[WIP] removed fetch_if_missing global
#606
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
harry-hov
force-pushed
the
fetch-if-missing
branch
from
e249ebd
to
8275da6
Compare
|
Just noticed this |
Indeed. I had pointed out this issue in git#662 (comment) already, and mistook the GitHub notification about this PR to refer to that other PR, sorry about that. I could have helped you much earlier if I had realized that this is a fresh start. |
|
for the past few days.. I'm learning about |
|
Do you want others to have a look? You could push your current state... |
I messed up a bit. but yes yes yes |
harry-hov
force-pushed
the
fetch-if-missing
branch
from
8275da6
to
b61ce0f
Compare
|
There is an issue in commit ac1cebd704fc919012b56073a1467560697e2367: |
|
There is an issue in commit de6772cd03f2a715db1ef56fd2d54d1d2585c663: |
|
There is an issue in commit fff701ca682956fa6ee472d7b746eaeb002561f1: |
|
There is an issue in commit 6fac52585dadf5235907711fc1ae15ce8a3edf95: |
|
There is an issue in commit b61ce0f99bcc15a5f46518aeaa00dacba9f3f831: |
harry-hov
changed the title
fetch_if_missing global into a field of struct repositoryfetch_if_missing global
harry-hov
changed the title
fetch_if_missing globalfetch_if_missing global
|
I'm also not sure about how to get rid of |
Looks like I am not really sure, though, what to do about this. My best suggestion: explain very clearly in the cover letter that |
Commit 6462d5e ("fetch: remove fetch_if_missing=0", 2019-11-08) strove to remove the need for fetch_if_missing=0 from the fetching mechanism, so it is plausible to attempt removing fetch_if_missing=0 from fetch-pack as well. Signed-off-by: Hariom Verma <hariom18599@gmail.com>
Commit 6462d5e ("fetch: remove fetch_if_missing=0", 2019-11-08) strove to remove the need for fetch_if_missing=0 from the fetching mechanism, so it is plausible to attempt removing fetch_if_missing=0 from index-pack as well. Signed-off-by: Hariom Verma <hariom18599@gmail.com>
harry-hov
force-pushed
the
fetch-if-missing
branch
from
b61ce0f
to
82f7473
Compare
|
@dscho I'm left with fetch_if_missing in Is it ok if I mention all these in cover letter? |
|
After removing any clue on this? |
Of course! Even better if you ask for guidance there... |
Not spontaneously. In any case, this would be better discussed on the Git mailing list. |
|
/submit |
|
Error: Refusing to submit a patch series that does not merge cleanly. |
|
/submit |
|
Submitted as pull.606.git.1587412477.gitgitgadget@gmail.com |
|
On the Git mailing list, Christian Couder wrote (reply to this): |
![gitgitgadget[bot]](https://avatars.githubusercontent.com/in/12836?s=60&v=4){kind=small}
| @@ -57,8 +57,6 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix) | |||
| struct packet_reader reader; | |||
There was a problem hiding this comment.
On the Git mailing list, Christian Couder wrote (reply to this):
On Mon, Apr 20, 2020 at 9:57 PM Hariom Verma via GitGitGadget
<gitgitgadget@gmail.com> wrote:
>
> From: Hariom Verma <hariom18599@gmail.com>
>
> Commit 6462d5e ("fetch: remove fetch_if_missing=0", 2019-11-08)
> strove to remove the need for fetch_if_missing=0 from the fetching
> mechanism, so it is plausible to attempt removing fetch_if_missing=0
> from fetch-pack as well.
It's ok to refer to a previous commit, but I think it would be better
if you could repeat a bit the reasons why removing the
fetch_if_missing global is a good idea, and not just rely on the
previous commit.
"it is plausible" also doesn't make it very clear that it's what the
patch is actually doing.
Thanks,
Christian.
There was a problem hiding this comment.
On the Git mailing list, Junio C Hamano wrote (reply to this):
Christian Couder <christian.couder@gmail.com> writes:
> On Mon, Apr 20, 2020 at 9:57 PM Hariom Verma via GitGitGadget
> <gitgitgadget@gmail.com> wrote:
>>
>> From: Hariom Verma <hariom18599@gmail.com>
>>
>> Commit 6462d5e ("fetch: remove fetch_if_missing=0", 2019-11-08)
>> strove to remove the need for fetch_if_missing=0 from the fetching
>> mechanism, so it is plausible to attempt removing fetch_if_missing=0
>> from fetch-pack as well.
>
> It's ok to refer to a previous commit, but I think it would be better
> if you could repeat a bit the reasons why removing the
> fetch_if_missing global is a good idea, and not just rely on the
> previous commit.
>
> "it is plausible" also doesn't make it very clear that it's what the
> patch is actually doing.
I had the same reaction. You could even write a random gibberish in
your patch and write "it's plausible this set of random changes made
without understanding what is going on in the current code might
have some chance to work" in your log message, and we would not even
want to touch such a patch with 10-foot pole.
The proposed log message above unfortunately makes this patch
indistinguishable from such a trash, unless we follow the codepaths
that are *not* touched by this patch and think about ramifications
of the removal *ourselves*. In other words, it does nothing to help
the readers to support the change.
There was a problem hiding this comment.
On the Git mailing list, Hariom verma wrote (reply to this):
On Thu, May 7, 2020 at 6:47 PM Christian Couder
<christian.couder@gmail.com> wrote:
>
> It's ok to refer to a previous commit, but I think it would be better
> if you could repeat a bit the reasons why removing the
> fetch_if_missing global is a good idea, and not just rely on the
> previous commit.
I agree with that.
> "it is plausible" also doesn't make it very clear that it's what the
> patch is actually doing.
Thanks for pointing it out. Will improve.
Regards,
Hariom
There was a problem hiding this comment.
On the Git mailing list, Hariom verma wrote (reply to this):
On Thu, May 7, 2020 at 8:32 PM Junio C Hamano <gitster@pobox.com> wrote:
>
> Christian Couder <christian.couder@gmail.com> writes:
>
> > It's ok to refer to a previous commit, but I think it would be better
> > if you could repeat a bit the reasons why removing the
> > fetch_if_missing global is a good idea, and not just rely on the
> > previous commit.
> >
> > "it is plausible" also doesn't make it very clear that it's what the
> > patch is actually doing.
>
> I had the same reaction. You could even write a random gibberish in
> your patch and write "it's plausible this set of random changes made
> without understanding what is going on in the current code might
> have some chance to work" in your log message, and we would not even
> want to touch such a patch with 10-foot pole.
>
> The proposed log message above unfortunately makes this patch
> indistinguishable from such a trash, unless we follow the codepaths
> that are *not* touched by this patch and think about ramifications
> of the removal *ourselves*. In other words, it does nothing to help
> the readers to support the change.
>
I understand it must be too hard for you to deal with such [trash]patches.
My apologies. Will improve in next revision
Thanks,
Hariom
| @@ -57,8 +57,6 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix) | |||
| struct packet_reader reader; | |||
There was a problem hiding this comment.
On the Git mailing list, Jonathan Tan wrote (reply to this):
> From: Hariom Verma <hariom18599@gmail.com>
>
> Commit 6462d5e ("fetch: remove fetch_if_missing=0", 2019-11-08)
> strove to remove the need for fetch_if_missing=0 from the fetching
> mechanism, so it is plausible to attempt removing fetch_if_missing=0
> from fetch-pack as well.
>
> Signed-off-by: Hariom Verma <hariom18599@gmail.com>
As Christian said [1], please include tests like in the commit you
mentioned. For a change like this, I think that the test is the most
important part.
Also include a justification for why it's safe to remove
fetch_if_missing=0. You can probably cite the aforementioned commit to
say that it covers the fetch_pack() method, and then go through the rest
of the code to see if any may inadvertently fetch an object.
Also, the fetch-pack and index-pack parts can be sent in separate patch
sets, so you might want to concentrate on one command first.
[1] https://lore.kernel.org/git/CAP8UFD2SNnpKWtYUztZ76OU7zBsrXyYhG_Zds1wi+NqBKCv+Qw@mail.gmail.com/
> diff --git a/fetch-pack.c b/fetch-pack.c
> index 1734a573b01..1ca643f6491 100644
> --- a/fetch-pack.c
> +++ b/fetch-pack.c
> @@ -1649,7 +1649,7 @@ static void update_shallow(struct fetch_pack_args *args,
> struct oid_array extra = OID_ARRAY_INIT;
> struct object_id *oid = si->shallow->oid;
> for (i = 0; i < si->shallow->nr; i++)
> - if (has_object_file(&oid[i]))
> + if (has_object_file_with_flags(&oid[i], OBJECT_INFO_SKIP_FETCH_OBJECT))
> oid_array_append(&extra, &oid[i]);
> if (extra.nr) {
> setup_alternate_shallow(&shallow_lock,
Hmm...this triggers when the user requests a clone that is both partial
and shallow, and the server reports a shallow object that it didn't send
back as a packfile; and it causes another fetch to be sent. This is a
separate issue, but Hariom, if you'd like to take a look at this, that
would work out too. You'll need to figure out how to make the server
send back shallow lines referencing objects that are not in the packfile
- one way to do it is to use one-time-perl. (Search the codebase to see
how it is used.) This is probably more complex, though.
There was a problem hiding this comment.
On the Git mailing list, Hariom verma wrote (reply to this):
On Fri, May 8, 2020 at 1:13 AM Jonathan Tan <jonathantanmy@google.com> wrote:
>
> As Christian said [1], please include tests like in the commit you
> mentioned. For a change like this, I think that the test is the most
> important part.
>
I will definitely add tests.
> Also include a justification for why it's safe to remove
> fetch_if_missing=0. You can probably cite the aforementioned commit to
> say that it covers the fetch_pack() method, and then go through the rest
> of the code to see if any may inadvertently fetch an object.
>
> Also, the fetch-pack and index-pack parts can be sent in separate patch
> sets, so you might want to concentrate on one command first.
>
Thanks, Will split and concentrate on one at a time.
>
> > diff --git a/fetch-pack.c b/fetch-pack.c
> > index 1734a573b01..1ca643f6491 100644
> > --- a/fetch-pack.c
> > +++ b/fetch-pack.c
> > @@ -1649,7 +1649,7 @@ static void update_shallow(struct fetch_pack_args *args,
> > struct oid_array extra = OID_ARRAY_INIT;
> > struct object_id *oid = si->shallow->oid;
> > for (i = 0; i < si->shallow->nr; i++)
> > - if (has_object_file(&oid[i]))
> > + if (has_object_file_with_flags(&oid[i], OBJECT_INFO_SKIP_FETCH_OBJECT))
> > oid_array_append(&extra, &oid[i]);
> > if (extra.nr) {
> > setup_alternate_shallow(&shallow_lock,
>
> Hmm...this triggers when the user requests a clone that is both partial
> and shallow, and the server reports a shallow object that it didn't send
> back as a packfile; and it causes another fetch to be sent. This is a
> separate issue, but Hariom, if you'd like to take a look at this, that
> would work out too. You'll need to figure out how to make the server
> send back shallow lines referencing objects that are not in the packfile
> - one way to do it is to use one-time-perl. (Search the codebase to see
> how it is used.) This is probably more complex, though.
I'm clueless about "one-time-perl" thing(till now!). Will surely going
to learn about that.
Thanks for the nice explanation.
Regards,
Hariom
There was a problem hiding this comment.
On the Git mailing list, Kousik Sanagavarapu wrote (reply to this):
So here, we are partial-cloning from a shallow remote and some
objects are not sent due to our clone filters. Let's say that
the shallow remote has a 5-commit history and we are cloning it
into another repository with a blob:none filter. The expected
behavior is cloning the 5 commits, with no blobs, except
for the HEAD.
When executing the above process, it leads to errors:
fatal: the remote end hung up unexpectedly
fatal: protocol error: bad pack header
warning: Clone succeeded, but checkout failed
You can inspect what was checked out with 'git status'
and retry with 'git restore --source=HEAD :/'
I looked into it a bit and it seems that packet_read() is not
successful. I'm not really sure how packet reading fits into
the big picture but it looks like the buffer is not read
completely.
It is a similar case with "bad pack header" too. The function
read_pack_header() fails because the pack header was not fully read.
Also, is the shallow object not sent when cloning due to the partial
clone filter and hence a subsequent fetching is done to ask for this
object? If so, then will such a fetch counted as an args->update_shallow?There was a problem hiding this comment.
On the Git mailing list, Kousik Sanagavarapu wrote (reply to this):
Sorry, it seems that I misunderstood the whole situation here. I didn't
realize that the problem was on my end and not something to do with the
code. Please ignore the above email except for the last question, which
I still don't seem to understand. So, I would be grateful if you could
clarify.
Thanks,
KousikThere was a problem hiding this comment.
On the Git mailing list, Jonathan Tan wrote (reply to this):
Kousik Sanagavarapu <five231003@gmail.com> writes:
> Also, is the shallow object not sent when cloning due to the partial
> clone filter and hence a subsequent fetching is done to ask for this
> object? If so, then will such a fetch counted as an args->update_shallow?
What do you mean by the shallow object? If you mean the last commit that
is sent (that is, without its parents), then that is a commit and is not
excluded by the filter.
As for args->update_shallow, that's a good question. Just glancing at
the code, even if it is set, I don't think there would be any difference
in operation since the lazy fetch does not fetch any refs (and in fact,
in protocol v2, we skip the ref advertisement in this case, as far as I
can remember).
|
On the Git mailing list, Hariom verma wrote (reply to this): |
|
On the Git mailing list, Kousik Sanagavarapu wrote (reply to this): Hi,
Are you still working on this? If not, then I would like to take this up
and write the tests, if it is worth doing. I think it would be a better
exposure of the codebase and would be helpful for GSoC. |
|
User |
|
On the Git mailing list, Christian Couder wrote (reply to this): Hi Kousik,
On Fri, Feb 17, 2023 at 7:22 PM Kousik Sanagavarapu
<five231003@gmail.com> wrote:
> Are you still working on this? If not, then I would like to take this up
> and write the tests, if it is worth doing. I think it would be a better
> exposure of the codebase and would be helpful for GSoC.
I don't know what's the state of this. I think only Hariom could answer.
I am not so sure it will be helpful for any of the GSoC project ideas
we propose, but feel free to work on it if you want.
Thanks,
Christian. |
|
User |
|
On the Git mailing list, Kousik Sanagavarapu wrote (reply to this): On Sat, 18 Feb 2023 at 22:30, Christian Couder
<christian.couder@gmail.com> wrote:
>
> I am not so sure it will be helpful for any of the GSoC project ideas
> we propose, but feel free to work on it if you want.
Well, I wanted to work on something before I started working on my application
and found this to be fun. So even if it would not really be
helpful for the project ideas proposed, I would still like to work on it as
something that could go into my application.
Thanks,
Kousik |
|
On the Git mailing list, Christian Couder wrote (reply to this): On Sun, Feb 19, 2023 at 4:57 AM Kousik Sanagavarapu
<five231003@gmail.com> wrote:
>
> On Sat, 18 Feb 2023 at 22:30, Christian Couder
> <christian.couder@gmail.com> wrote:
>
> > I am not so sure it will be helpful for any of the GSoC project ideas
> > we propose, but feel free to work on it if you want.
>
> Well, I wanted to work on something before I started working on my application
> and found this to be fun. So even if it would not really be
> helpful for the project ideas proposed, I would still like to work on it as
> something that could go into my application.
Sure, you can mention anything you did related to the project in your
application and we will take it into account.
Thanks,
Christian. |
|
On the Git mailing list, Hariom verma wrote (reply to this): HI,
On Fri, Feb 17, 2023 at 10:50 PM Kousik Sanagavarapu
<five231003@gmail.com> wrote:
>
> Are you still working on this? If not, then I would like to take this up
> and write the tests, if it is worth doing. I think it would be a better
> exposure of the codebase and would be helpful for GSoC.
I'm not working on it. Feel free to take it forward.
Thanks,
Hariom |
|
User |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We are not much happy with global variable
fetch_if_missing.So, in commit 6462d5e ("fetch: remove fetch_if_missing=0", 2019-11-08) Jonathan Tan jonathantanmy@google.com attempted to remove the need for fetch_if_missing=0 from the fetching mechanism. After that,
fetch_if_missingis removed fromcloneandpromisor-remotetoo.I imitated the same logic to remove
fetch_if_missingfromfetch-pack&index-pack.I'm looking forward to remove
fetch_if_missingfrom otherplaces too, but I not sure about how to handle it.
In fsck,
fetch_if_missingis set to 0 in the beginning ofcmd_fsck().In rev-list,
fetch_if_missingis set to 0 inparse_missing_action_value(), and incmd_rev_list()while parsing the command-line parameters.(almost similar case inpack-objects)fixes #251
cc: Kousik Sanagavarapu five231003@gmail.com
cc: Christian Couder christian.couder@gmail.com
cc: Jonathan Tan jonathantanmy@google.com