clean: demonstrate a bug with pathspecs

[derrickstolee]

While integrating v2.25.0 into the microsoft/git fork, one of our
VFS for Git functional tests started failing. Looking into it, the only
possible place could have been where one of our integration
points with the virtualfilesystem hook was moved by c5c4edd
(dir: break part of read_directory_recursive() out for reuse, 2019-12-10)
and then used in the following two commits.

By reverting these two commits, we stopped the failure, but it
took a while before figuring out that it was a regression in Git
and not a failure in our integration to the new logic. Thanks
to Kevin Willford for producing a test case.

b9670c1 (dir: fix checks on common prefix directory, 2019-12-19)
is the culprit, so this patch is based on that. If rebased to c5c4edd,
then the test passes.

As for actually fixing this regression, I don't know how. This code
is pretty dense and I don't have a firm grasp of what is happening
in both b9670c1 and the following 777b420 (dir: synchronize
tread_leading_path() and read_directory_recursive()). Elijah
is CC'd in case he still has context on this area.

V2 fixes a typo in the commit short-sha references.

Thanks,
-Stolee

Cc: Kevin.Willford@microsoft.com, newren@gmail.com

[derrickstolee]

/submit

[gitgitgadget]

Submitted as pull.526.git.1579119946211.gitgitgadget@gmail.com

[gitgitgadget]

On the Git mailing list, Kyle Meyer wrote (reply to this):

"Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:

> From: Derrick Stolee <dstolee@microsoft.com>
>
> b9660c1 (dir: fix checks on common prefix directory, 2019-12-19)
> modified the way pathspecs are handled when handling a directory
> during "git clean -f <path>".

I can't find b9660c1.  I think this and other references below should
point to b9670c1f5e (dir: fix checks on common prefix directory,
2019-12-19), which matches the base-commit value for this patch.

[gitgitgadget]

On the Git mailing list, Jonathan Nieder wrote (reply to this):

Hi,

Derrick Stolee wrote:

> b9660c1 (dir: fix checks on common prefix directory, 2019-12-19)
> modified the way pathspecs are handled when handling a directory
> during "git clean -f <path>". While this improved the behavior
> for known test breakages, it also regressed in how the clean
> command handles cleaning a specified file.
>
> Add a test case that demonstrates this behavior. This test passes
> before b9660c1 then fails after.

Can this commit message say a little more about the nature of the
bug?  For example, what kind of workflow does this come up in for
end users?

[...]
>     While integrating v2.25.0 into the microsoft/git fork, one of our VFS
>     for Git functional tests started failing.

This is also useful information to put in the commit message: e.g.
"Noticed via VFS for Git's functional test <test name>".  It provides
useful context when looking at such a patch later.

[...]
>                                      Elijah is CC'd in case he still has
>     context on this area.

Thanks and hope that helps,
Jonathan

[gitgitgadget]

On the Git mailing list, Elijah Newren wrote (reply to this):

On Wed, Jan 15, 2020 at 12:25 PM Derrick Stolee via GitGitGadget
<gitgitgadget@gmail.com> wrote:
>
> From: Derrick Stolee <dstolee@microsoft.com>
>
> b9660c1 (dir: fix checks on common prefix directory, 2019-12-19)
> modified the way pathspecs are handled when handling a directory
> during "git clean -f <path>". While this improved the behavior
> for known test breakages, it also regressed in how the clean
> command handles cleaning a specified file.
>
> Add a test case that demonstrates this behavior. This test passes
> before b9660c1 then fails after.
>
> Helped-by: Kevin Willford <Kevin.Willford@microsoft.com>
> Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
> ---
>     clean: demonstrate a bug with pathspecs
>
>     While integrating v2.25.0 into the microsoft/git fork, one of our VFS
>     for Git functional tests started failing. Looking into it, the only
>     possible place could have been where one of our integration points with
>     the virtualfilesystem hook was moved by c5c4edd (dir: break part of
>     read_directory_recursive() out for reuse, 2019-12-10) and then used in
>     the following two commits.
>
>     By reverting these two commits, we stopped the failure, but it took a
>     while before figuring out that it was a regression in Git and not a
>     failure in our integration to the new logic. Thanks to Kevin Willford
>     for producing a test case.
>
>     b9660c1 (dir: fix checks on common prefix directory, 2019-12-19) is the
>     culprit, so this patch is based on that. If rebased to c5c4edd, then the
>     test passes.
>
>     As for actually fixing this regression, I don't know how. This code is
>     pretty dense and I don't have a firm grasp of what is happening in both
>     b9660c1 and the following 777b420 (dir: synchronize tread_leading_path()
>     and read_directory_recursive()). Elijah is CC'd in case he still has
>     context on this area.
>
>     Thanks, -Stolee
>
> Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-526%2Fderrickstolee%2Fclean-bug-v1
> Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-526/derrickstolee/clean-bug-v1
> Pull-Request: https://github.com/gitgitgadget/git/pull/526
>
>  t/t7300-clean.sh | 9 +++++++++
>  1 file changed, 9 insertions(+)
>
> diff --git a/t/t7300-clean.sh b/t/t7300-clean.sh
> index 6e6d24c1c3..782e125c89 100755
> --- a/t/t7300-clean.sh
> +++ b/t/t7300-clean.sh
> @@ -737,4 +737,13 @@ test_expect_success MINGW 'handle clean & core.longpaths = false nicely' '
>         test_i18ngrep "too long" .git/err
>  '
>
> +test_expect_failure 'clean untracked paths by pathspec' '
> +       git init untracked &&
> +       mkdir untracked/dir &&
> +       echo >untracked/dir/file.txt &&
> +       git -C untracked clean -f dir/file.txt &&
> +       ls untracked/dir >actual &&
> +       test_must_be_empty actual
> +'
> +
>  test_done
>
> base-commit: b9670c1f5e6b98837c489a03ac0d343d30e08505
> --
> gitgitgadget

Is there an inverted phrase corresponding to "the gift that keeps on
giving", something like "the punishment that keeps on punishing"?  If
so, it would be a very appropriate description of dir.c.

Yeah, I still have context.  I even think I've got an idea about what
the fix might be, though with dir.c my ideas about fixes usually just
serve as starting points for debugging before I find the real fix.
I'll try to dig in.

[gitgitgadget]

On the Git mailing list, Derrick Stolee wrote (reply to this):

On 1/15/2020 7:38 PM, Elijah Newren wrote:
> Is there an inverted phrase corresponding to "the gift that keeps on
> giving", something like "the punishment that keeps on punishing"?  If
> so, it would be a very appropriate description of dir.c.

At least we will continue adding tests until we converge towards
correctness, and the behavior issues are even more contrived and
special case (like this one).

> Yeah, I still have context.  I even think I've got an idea about what
> the fix might be, though with dir.c my ideas about fixes usually just
> serve as starting points for debugging before I find the real fix.
> I'll try to dig in.

Thanks! I'll try to review it carefully when it arrives. Good luck.

-Stolee

[gitgitgadget]

On the Git mailing list, Derrick Stolee wrote (reply to this):

On 1/15/2020 6:30 PM, Kyle Meyer wrote:
> "Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:
> 
>> From: Derrick Stolee <dstolee@microsoft.com>
>>
>> b9660c1 (dir: fix checks on common prefix directory, 2019-12-19)
>> modified the way pathspecs are handled when handling a directory
>> during "git clean -f <path>".
> 
> I can't find b9660c1.  I think this and other references below should
> point to b9670c1f5e (dir: fix checks on common prefix directory,
> 2019-12-19), which matches the base-commit value for this patch.

Sorry for the digit swap. Thanks for pointing that out!

[gitgitgadget]

On the Git mailing list, Derrick Stolee wrote (reply to this):

On 1/15/2020 7:03 PM, Jonathan Nieder wrote:
> Hi,
> 
> Derrick Stolee wrote:
> 
>> b9660c1 (dir: fix checks on common prefix directory, 2019-12-19)
>> modified the way pathspecs are handled when handling a directory
>> during "git clean -f <path>". While this improved the behavior
>> for known test breakages, it also regressed in how the clean
>> command handles cleaning a specified file.
>>
>> Add a test case that demonstrates this behavior. This test passes
>> before b9660c1 then fails after.
> 
> Can this commit message say a little more about the nature of the
> bug?  For example, what kind of workflow does this come up in for
> end users?

I honestly don't know why anyone would call `git clean -f <path>` on a
file instead of using `rm <path>`. But, the behavior _did_ change, which
is why I'm bringing it up.

If the community instead said "this is not important functionality. We
should just expect the given pathspec to only match directories" then I
would accept that and just delete the file in another way. That seems
unlikely.

> [...]
>>     While integrating v2.25.0 into the microsoft/git fork, one of our VFS
>>     for Git functional tests started failing.
> 
> This is also useful information to put in the commit message: e.g.
> "Noticed via VFS for Git's functional test <test name>".  It provides
> useful context when looking at such a patch later.

I'm not sure the test [1] will shed much light on the issue. It sort of
accidentally reveals this bug because it happens to use "git clean -f <path>".

The test itself is holding a handle on <path> on a commit where <path>
is untracked, then tries to checkout a commit where <path> is tracked. On
Windows, this should fail. With the virtualization layer in VFS for Git,
Git doesn't actually try to write to <path> but instead VFS for Git tries
to update the virtualization at <path>, colliding with what Git is trying
to do. Hence, we need to make sure the Git command actually fails in this
attempt.

Perhaps that context isn't actually helpful. And you could understand why
I stared at this test for a long while before realizing that it was actually
a failure in "git clean -f" and then Kevin did the real work to find that
VFS for Git wasn't causing the issue.

-Stolee

[1] https://github.com/microsoft/VFSForGit/blob/1aec263033cc3c05d0389e1792b7958d9a2e70c6/GVFS/GVFS.FunctionalTests.Windows/Windows/Tests/WindowsUpdatePlaceholderTests.cs#L38-L72

[gitgitgadget]

On the Git mailing list, Elijah Newren wrote (reply to this):

On Wed, Jan 15, 2020 at 5:23 PM Derrick Stolee <stolee@gmail.com> wrote:
>
> On 1/15/2020 7:38 PM, Elijah Newren wrote:
> > Is there an inverted phrase corresponding to "the gift that keeps on
> > giving", something like "the punishment that keeps on punishing"?  If
> > so, it would be a very appropriate description of dir.c.
>
> At least we will continue adding tests until we converge towards
> correctness, and the behavior issues are even more contrived and
> special case (like this one).

This doesn't seem any more contrived or special case than most my
previous fixes for dir.c...

> > Yeah, I still have context.  I even think I've got an idea about what
> > the fix might be, though with dir.c my ideas about fixes usually just
> > serve as starting points for debugging before I find the real fix.
> > I'll try to dig in.
>
> Thanks! I'll try to review it carefully when it arrives. Good luck.

Man, I'm such a bozo.  It turns out, for once, that my idea for the
fix was correct but after digging a bit I realized that it was
essentially a bug I fixed not that long ago once already -- and that I
myself re-introduced it (for a slightly different case) in some
commits where I used some strongly worded disgust that "this bad code
structure is going to cause someone to mess up in <this way>" and then
I made that exact kind of mistake I was complaining about in the
commit message...as part of that EXACT commit, to boot.

At least it'll make for a fun new commit message explaining it all...


Anyway, I'm going to pull your commit into my series so I can put my
fix on top, and lump it in with Peff's two patches over at
https://lore.kernel.org/git/20200115202146.GA4091171@coredump.intra.peff.net/
since all these patches are basically "more fill_directory() fixes".
Let me know if you have any concerns with that.

Elijah

[gitgitgadget]

On the Git mailing list, Junio C Hamano wrote (reply to this):

Elijah Newren <newren@gmail.com> writes:

> Anyway, I'm going to pull your commit into my series so I can put my
> fix on top, and lump it in with Peff's two patches over at
> https://lore.kernel.org/git/20200115202146.GA4091171@coredump.intra.peff.net/
> since all these patches are basically "more fill_directory() fixes".

Thanks.  Then I'll refrain from applying those two patches we saw
earlier (including the one you have the URL in your message).

[gitgitgadget]

This patch series was integrated into pu via git@25f4715.

[derrickstolee]

This is incorporated into Elija's dir() fixes.