midx: disable replace objects #1711
Conversation
Welcome to GitGitGadgetHi @blanet, and welcome to GitGitGadget, the GitHub App to send patch series to the Git mailing list from GitHub Pull Requests. Please make sure that either:
You can CC potential reviewers by adding a footer to the PR description with the following syntax: Also, it is a good idea to review the commit messages one last time, as the Git project expects them in a quite specific form:
It is in general a good idea to await the automated test ("Checks") in this Pull Request before contributing the patches, e.g. to avoid trivial issues such as unportable code. Contributing the patchesBefore you can contribute the patches, your GitHub username needs to be added to the list of permitted users. Any already-permitted user can do that, by adding a comment to your PR of the form Both the person who commented An alternative is the channel Once on the list of permitted usernames, you can contribute the patches to the Git mailing list by adding a PR comment If you want to see what email(s) would be sent for a After you submit, GitGitGadget will respond with another comment that contains the link to the cover letter mail in the Git mailing list archive. Please make sure to monitor the discussion in that thread and to address comments and suggestions (while the comments and suggestions will be mirrored into the PR by GitGitGadget, you will still want to reply via mail). If you do not want to subscribe to the Git mailing list just to be able to respond to a mail, you can download the mbox from the Git mailing list archive (click the curl -g --user "<EMailAddress>:<Password>" \
--url "imaps://imap.gmail.com/INBOX" -T /path/to/raw.txtTo iterate on your change, i.e. send a revised patch or patch series, you will first want to (force-)push to the same branch. You probably also want to modify your Pull Request description (or title). It is a good idea to summarize the revision by adding something like this to the cover letter (read: by editing the first comment on the PR, i.e. the PR description): To send a new iteration, just add another PR comment with the contents: Need help?New contributors who want advice are encouraged to join git-mentoring@googlegroups.com, where volunteers who regularly contribute to Git are willing to answer newbie questions, give advice, or otherwise provide mentoring to interested contributors. You must join in order to post or view messages, but anyone can join. You may also be able to find help in real time in the developer IRC channel, |
blanet
force-pushed
the
xx/midx-ignore-replace-objects
branch
from
1def36a
to
060edb2
Compare
|
/preview |
|
Error: User blanet is not yet permitted to use GitGitGadget |
blanet
force-pushed
the
xx/midx-ignore-replace-objects
branch
2 times, most recently
from
76e9069
to
b1c8389
Compare
|
/allow |
|
/submit |
|
User blanet is now allowed to use GitGitGadget. |
|
Submitted as pull.1711.git.1712495507815.gitgitgadget@gmail.com To fetch this version into To fetch this version to local tag |
|
On the Git mailing list, Taylor Blau wrote (reply to this): On Sun, Apr 07, 2024 at 01:11:47PM +0000, blanet via GitGitGadget wrote:
> From: Xing Xin <xingxin.xx@bytedance.com>
>
> We observed a series of clone failures arose in a specific set of
> repositories after we fully enabled the MIDX bitmap feature within our
> Codebase service. These failures were accompanied with error messages
> such as:
>
> fatal: did not receive expected object ...
> fatal: fetch-pack: invalid index-pack output
>
> Temporarily disabling the MIDX feature eliminated the reported issues.
> After some investigation we found that all repositories experiencing
> failures contain replace references, which seem to be improperly
> acknowledged by the MIDX bitmap generation logic.
I was suspicious that this might be related to the MIDX or MIDX bitmap,
but noticed something curious upon digging in. Applying the following on
top of your patch:
--- 8< ---
diff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh
index 5e4cdef6a8..8543f8d097 100755
--- a/t/t5326-multi-pack-bitmaps.sh
+++ b/t/t5326-multi-pack-bitmaps.sh
@@ -451,9 +451,7 @@ test_expect_success 'do not follow replace objects for MIDX bitmap' '
git rev-list --objects --no-object-names $B |sort >expected &&
git replace $A $C &&
- git repack -ad &&
- git multi-pack-index write --bitmap &&
- git rev-list --objects --no-object-names --use-bitmap-index $B |sort >actual &&
+ git rev-list --objects --no-object-names $B |sort >actual &&
test_cmp expected actual
)
'
--- >8 ---
, I can still produce the failure that you are seeing here. So I suspect
that while it's entirely possible that there is a bug in the MIDX/bitmap
code, that this test is not exercising it.
I think the first step to demonstrate a bug in the MIDX/bitmap machinery
would be to provide a reproducer that fails only when using a MIDX
and/or bitmap.
> @@ -273,6 +274,8 @@ int cmd_multi_pack_index(int argc, const char **argv,
> };
> struct option *options = parse_options_concat(builtin_multi_pack_index_options, common_opts);
>
> + disable_replace_refs();
> +
Supposing for a moment that this issue is in the MIDX, we know that
regardless of what replace refs might be in place, the MIDX should only
be storing the objects that are in the packs being indexed, not the
objects which are their replacements.
Are we storing objects in the MIDX that are replacements? Looking
at midx.c::fill_pack_entry(), I think the answer is "no", since we're
looking up packed objects by calling nth_packed_object_id(), which is
just a table read into the .idx, all of which is beneath the level of
replace refs.
> @@ -434,6 +434,30 @@ test_expect_success 'tagged commits are selected for bitmapping' '
> )
> '
>
> +test_expect_success 'do not follow replace objects for MIDX bitmap' '
> + rm -fr repo &&
> + git init repo &&
> + test_when_finished "rm -fr repo" &&
> + (
> + cd repo &&
> +
> + test_commit A &&
> + A=$(git rev-parse HEAD) &&
It's possible that much of this will be moot if the current test gets
rewritten, but here are a couple of suggestions for writing tests in
Git's suite:
- test_commit will create a tag for you, so there is no need to store
"$A", "$B", and "$C".
> + test_commit B &&
> + B=$(git rev-parse HEAD) &&
> + git checkout --orphan=orphan $A &&
> + test_commit orphan &&
> + C=$(git rev-parse HEAD) &&
> + git rev-list --objects --no-object-names $B |sort >expected &&
- We do not allow Git invocations on the left-hand side of a pipe, since
doing so will squelch its exit code. Instead, try:
git rev-list --objects --no-object-names B >expect.raw &&
sort expect.raw >expect &&
Thanks,
Taylor |
|
User |
|
On the Git mailing list, Taylor Blau wrote (reply to this): On Sun, Apr 07, 2024 at 10:16:28AM -0400, Taylor Blau wrote:
> , I can still produce the failure that you are seeing here. So I suspect
> that while it's entirely possible that there is a bug in the MIDX/bitmap
> code, that this test is not exercising it.
>
> I think the first step to demonstrate a bug in the MIDX/bitmap machinery
> would be to provide a reproducer that fails only when using a MIDX
> and/or bitmap.
I had a some more time to look into this, and I think that your original
fix is correct.
The issue is, as you suggest, due to the following (from your original
patch):
> After some investigation we found that all repositories experiencing
> failures contain replace references, which seem to be improperly
> acknowledged by the MIDX bitmap generation logic.
Indeed, the pack-bitmap-write machinery does not itself call
disable_replace_refs(). So when it generates a reachability bitmap, it
is doing so with the replace refs in mind. You can see that this is
indeed the cause of the problem by looking at the output of an
instrumented version of Git that indicates what bits are being set
during the bitmap generation phase.
With replace refs (incorrectly) enabled, we get:
[2, 4, 6, 8, 13, 3, 6, 7, 3, 4, 6, 8]
and doing the same after calling disable_replace_refs(), we instead get:
[2, 5, 6, 13, 3, 6, 7, 3, 4, 6, 8]
Single pack bitmaps are unaffected by this issue because we generate
them from within pack-objects, which does call disable_replace_refs().
It is tempting to instead do something like:
--- 8< ---
diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c
index c6c8f94cc5..cbc543caad 100644
--- a/pack-bitmap-write.c
+++ b/pack-bitmap-write.c
@@ -17,6 +17,7 @@
#include "trace2.h"
#include "tree.h"
#include "tree-walk.h"
+#include "replace-object.h"
struct bitmapped_commit {
struct commit *commit;
@@ -223,6 +224,8 @@ static void bitmap_builder_init(struct bitmap_builder *bb,
memset(bb, 0, sizeof(*bb));
init_bb_data(&bb->data);
+ parsed_object_pool_clear(the_repository->parsed_objects);
+
reset_revision_walk();
repo_init_revisions(writer->to_pack->repo, &revs, NULL);
revs.topo_order = 1;
--- >8 ---
But by then it is too late, because the replace refs have already been
taken into account for parsed objects.
An alternative is to clear the parsed_object_pool before (or after)
calling disable_replace_refs(), but I think that approach that feels
sub-optimal for a couple of reasons:
- We're wasting time re-parsing objects that we've already seen
- We're banking on the fact that the MIDX generation does not lookup
objects with the OBJECT_INFO_LOOKUP_REPLACE flag set, which would
cause the MIDX to be broken in the same way.
So I think that disabling replace refs at the outset within the
multi-pack-index builtin is the right way to go. In addition to the test
fixes I suggested earlier, I would instead demonstrate the bug by
showing a clone (which fails with MIDXs, but doesn't without MIDXs) like
so:
--- 8< ---
diff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh
index 5e4cdef6a8..1fb3b0f9d7 100755
--- a/t/t5326-multi-pack-bitmaps.sh
+++ b/t/t5326-multi-pack-bitmaps.sh
@@ -442,19 +442,16 @@ test_expect_success 'do not follow replace objects for MIDX bitmap' '
cd repo &&
test_commit A &&
- A=$(git rev-parse HEAD) &&
test_commit B &&
- B=$(git rev-parse HEAD) &&
- git checkout --orphan=orphan $A &&
+ git checkout --orphan=orphan A &&
test_commit orphan &&
- C=$(git rev-parse HEAD) &&
- git rev-list --objects --no-object-names $B |sort >expected &&
- git replace $A $C &&
- git repack -ad &&
- git multi-pack-index write --bitmap &&
- git rev-list --objects --no-object-names --use-bitmap-index $B |sort >actual &&
- test_cmp expected actual
+ git replace A HEAD &&
+ git repack -ad --write-midx --write-bitmap-index &&
+
+ # generating reachability bitmaps with replace refs
+ # enabled will result in broken clones
+ git clone --no-local --bare . clone.git
)
'
--- >8 ---
With the change in your patch to call disable_replace_refs() in
builtin/multi-pack-index.c, this test passes as expected. With that
change compiled out, we instead get:
[...]
+ git clone --no-local --bare . clone.git
Cloning into bare repository 'clone.git'...
remote: Enumerating objects: 8, done.
remote: Total 8 (delta 0), reused 0 (delta 0), pack-reused 8 (from 1)
Receiving objects: 100% (8/8), done.
fatal: did not receive expected object da5497437fd67ca928333aab79c4b4b55036ea66
fatal: fetch-pack: invalid index-pack output
error: last command exited with $?=128
not ok 352 - do not follow replace objects for MIDX bitmap
as expected.
Thanks,
Taylor
|
|
On the Git mailing list, Taylor Blau wrote (reply to this): On Sun, Apr 07, 2024 at 02:02:17PM -0400, Taylor Blau wrote:
> It is tempting to instead do something like:
>
> --- 8< ---
> diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c
> index c6c8f94cc5..cbc543caad 100644
> --- a/pack-bitmap-write.c
> +++ b/pack-bitmap-write.c
> @@ -17,6 +17,7 @@
> #include "trace2.h"
> #include "tree.h"
> #include "tree-walk.h"
> +#include "replace-object.h"
>
> struct bitmapped_commit {
> struct commit *commit;
> @@ -223,6 +224,8 @@ static void bitmap_builder_init(struct bitmap_builder *bb,
> memset(bb, 0, sizeof(*bb));
> init_bb_data(&bb->data);
>
> + parsed_object_pool_clear(the_repository->parsed_objects);
> +
> reset_revision_walk();
> repo_init_revisions(writer->to_pack->repo, &revs, NULL);
> revs.topo_order = 1;
> --- >8 ---
Oops. I meant to attach a diff here that calls disable_replace_refs(),
not parsed_object_pool_clear().
> But by then it is too late, because the replace refs have already been
> taken into account for parsed objects.
Thanks,
Taylor |
We observed a series of clone failures arose in a specific set of
repositories after we fully enabled the MIDX bitmap feature within our
Codebase service. These failures were accompanied with error messages
such as:
Cloning into bare repository 'clone.git'...
remote: Enumerating objects: 8, done.
remote: Total 8 (delta 0), reused 0 (delta 0), pack-reused 8 (from 1)
Receiving objects: 100% (8/8), done.
fatal: did not receive expected object ...
fatal: fetch-pack: invalid index-pack output
Temporarily disabling the MIDX feature eliminated the reported issues.
After some investigation we found that all repositories experiencing
failures contain replace references, which seem to be improperly
acknowledged by the MIDX bitmap generation logic.
A more thorough explanation about the root cause from Taylor Blau says:
Indeed, the pack-bitmap-write machinery does not itself call
disable_replace_refs(). So when it generates a reachability bitmap, it
is doing so with the replace refs in mind. You can see that this is
indeed the cause of the problem by looking at the output of an
instrumented version of Git that indicates what bits are being set
during the bitmap generation phase.
With replace refs (incorrectly) enabled, we get:
[2, 4, 6, 8, 13, 3, 6, 7, 3, 4, 6, 8]
and doing the same after calling disable_replace_refs(), we instead get:
[2, 5, 6, 13, 3, 6, 7, 3, 4, 6, 8]
Single pack bitmaps are unaffected by this issue because we generate
them from within pack-objects, which does call disable_replace_refs().
This patch updates the MIDX logic to disable replace objects within the
multi-pack-index builtin, and a test showing a clone (which would fail
with MIDX bitmap) is added to demonstrate the bug.
Helped-by: Taylor Blau <me@ttaylorr.com>
Signed-off-by: Xing Xin <xingxin.xx@bytedance.com>
blanet
force-pushed
the
xx/midx-ignore-replace-objects
branch
from
b1c8389
to
1be25b5
Compare
|
/submit |
|
Submitted as pull.1711.v2.git.1712554017808.gitgitgadget@gmail.com To fetch this version into To fetch this version to local tag |
|
On the Git mailing list, 鑫邢 wrote (reply to this): > I had a some more time to look into this, and I think that your original
> fix is correct.
>
> The issue is, as you suggest, due to the following (from your original
> patch):
>
> > After some investigation we found that all repositories experiencing
> > failures contain replace references, which seem to be improperly
> > acknowledged by the MIDX bitmap generation logic.
>
> Indeed, the pack-bitmap-write machinery does not itself call
> disable_replace_refs(). So when it generates a reachability bitmap, it
> is doing so with the replace refs in mind. You can see that this is
> indeed the cause of the problem by looking at the output of an
> instrumented version of Git that indicates what bits are being set
> during the bitmap generation phase.
>
> With replace refs (incorrectly) enabled, we get:
>
> [2, 4, 6, 8, 13, 3, 6, 7, 3, 4, 6, 8]
>
> and doing the same after calling disable_replace_refs(), we instead get:
>
> [2, 5, 6, 13, 3, 6, 7, 3, 4, 6, 8]
>
> Single pack bitmaps are unaffected by this issue because we generate
> them from within pack-objects, which does call disable_replace_refs().
Thank you for the comprehensive investigation. I have quoted them in the
commit message to provide a clearer explanation of the patch.
> In addition to the test fixes I suggested earlier, I would instead demonstrate
> the bug by showing a clone (which fails with MIDXs, but doesn't without
> MIDXs) like so:
>
> --- 8< ---
> diff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh
> index 5e4cdef6a8..1fb3b0f9d7 100755
> --- a/t/t5326-multi-pack-bitmaps.sh
> +++ b/t/t5326-multi-pack-bitmaps.sh
> @@ -442,19 +442,16 @@ test_expect_success 'do not follow replace objects for MIDX bitmap' '
> cd repo &&
>
> test_commit A &&
> - A=$(git rev-parse HEAD) &&
> test_commit B &&
> - B=$(git rev-parse HEAD) &&
> - git checkout --orphan=orphan $A &&
> + git checkout --orphan=orphan A &&
> test_commit orphan &&
> - C=$(git rev-parse HEAD) &&
> - git rev-list --objects --no-object-names $B |sort >expected &&
>
> - git replace $A $C &&
> - git repack -ad &&
> - git multi-pack-index write --bitmap &&
> - git rev-list --objects --no-object-names --use-bitmap-index $B |sort >actual &&
> - test_cmp expected actual
> + git replace A HEAD &&
> + git repack -ad --write-midx --write-bitmap-index &&
> +
> + # generating reachability bitmaps with replace refs
> + # enabled will result in broken clones
> + git clone --no-local --bare . clone.git
> )
> '
> --- >8 ---
>
> With the change in your patch to call disable_replace_refs() in
> builtin/multi-pack-index.c, this test passes as expected. With that
> change compiled out, we instead get:
>
> [...]
> + git clone --no-local --bare . clone.git
> Cloning into bare repository 'clone.git'...
> remote: Enumerating objects: 8, done.
> remote: Total 8 (delta 0), reused 0 (delta 0), pack-reused 8 (from 1)
> Receiving objects: 100% (8/8), done.
> fatal: did not receive expected object da5497437fd67ca928333aab79c4b4b55036ea66
> fatal: fetch-pack: invalid index-pack output
> error: last command exited with $?=128
> not ok 352 - do not follow replace objects for MIDX bitmap
>
> as expected.
>
Applied! The test looks much clearer now, thanks!
Xing Xin |
|
User |
|
On the Git mailing list, Junio C Hamano wrote (reply to this): "blanet via GitGitGadget" <gitgitgadget@gmail.com> writes:
> From: Xing Xin <xingxin.xx@bytedance.com>
> ...
> Helped-by: Taylor Blau <me@ttaylorr.com>
> Signed-off-by: Xing Xin <xingxin.xx@bytedance.com>
> ---
I think this took the review in
https://lore.kernel.org/git/ZhLfqU9VNUW+2mmV@nand.local/
into account and is in good shape?
Thanks, both.
|
|
This branch is now known as |
|
This patch series was integrated into seen via git@28a4d87. |
|
There was a status update in the "New Topics" section about the branch source: <pull.1711.v2.git.1712554017808.gitgitgadget@gmail.com> |
|
On the Git mailing list, Taylor Blau wrote (reply to this): On Wed, Apr 17, 2024 at 12:34:27PM -0700, Junio C Hamano wrote:
> "blanet via GitGitGadget" <gitgitgadget@gmail.com> writes:
>
> > From: Xing Xin <xingxin.xx@bytedance.com>
> > ...
> > Helped-by: Taylor Blau <me@ttaylorr.com>
> > Signed-off-by: Xing Xin <xingxin.xx@bytedance.com>
> > ---
>
> I think this took the review in
>
> https://lore.kernel.org/git/ZhLfqU9VNUW+2mmV@nand.local/
>
> into account and is in good shape?
Yes, sorry for not explicitly ack-ing, this version looks good to me.
Thanks,
Taylor |
|
On the Git mailing list, Junio C Hamano wrote (reply to this): Taylor Blau <me@ttaylorr.com> writes:
> On Wed, Apr 17, 2024 at 12:34:27PM -0700, Junio C Hamano wrote:
>> "blanet via GitGitGadget" <gitgitgadget@gmail.com> writes:
>>
>> > From: Xing Xin <xingxin.xx@bytedance.com>
>> > ...
>> > Helped-by: Taylor Blau <me@ttaylorr.com>
>> > Signed-off-by: Xing Xin <xingxin.xx@bytedance.com>
>> > ---
>>
>> I think this took the review in
>>
>> https://lore.kernel.org/git/ZhLfqU9VNUW+2mmV@nand.local/
>>
>> into account and is in good shape?
>
> Yes, sorry for not explicitly ack-ing, this version looks good to me.
Thanks. |
|
This patch series was integrated into seen via git@a918243. |
|
This patch series was integrated into seen via git@1f65942. |
|
This patch series was integrated into seen via git@2c57ba4. |
|
There was a status update in the "Cooking" section about the branch The procedure to build multi-pack-index got confused by the replace-refs mechanism, which has been corrected by disabling the latter. Will merge to 'next'. source: <pull.1711.v2.git.1712554017808.gitgitgadget@gmail.com> |
|
This patch series was integrated into seen via git@f180c0a. |
|
This patch series was integrated into seen via git@5f6e4d7. |
|
There was a status update in the "Cooking" section about the branch The procedure to build multi-pack-index got confused by the replace-refs mechanism, which has been corrected by disabling the latter. Will merge to 'next'. source: <pull.1711.v2.git.1712554017808.gitgitgadget@gmail.com> |
|
This patch series was integrated into seen via git@81df3dc. |
|
This patch series was integrated into seen via git@ede2d5f. |
|
This patch series was integrated into next via git@6c8f417. |
|
There was a status update in the "Cooking" section about the branch The procedure to build multi-pack-index got confused by the replace-refs mechanism, which has been corrected by disabling the latter. Will cook in 'next'. source: <pull.1711.v2.git.1712554017808.gitgitgadget@gmail.com> |
|
This patch series was integrated into seen via git@3f0148e. |
|
This patch series was integrated into seen via git@f9cbe77. |
|
This patch series was integrated into seen via git@9b135aa. |
|
This patch series was integrated into seen via git@0d19c0b. |
|
There was a status update in the "Cooking" section about the branch The procedure to build multi-pack-index got confused by the replace-refs mechanism, which has been corrected by disabling the latter. Will cook in 'next'. source: <pull.1711.v2.git.1712554017808.gitgitgadget@gmail.com> |
|
This patch series was integrated into next via git@55e5548. |
|
This patch series was integrated into seen via git@55e5548. |
|
This patch series was integrated into master via git@55e5548. |
|
Closed via 55e5548. |
cc: Taylor Blau me@ttaylorr.com
cc: 鑫邢 xingxin.xx@bytedance.com