fsmonitor: only enable it in non-bare repositories

[dscho]

For quite a while now, I run with the built-in FSMonitor via my user config. Happily, the only issue I ran into was that FSMonitor tried to run in a bare repository the other day. But an FSMonitor makes only sense if we have a worktree. So let's disable it automatically in bare repositories.

This patch applies near the top of jh/rfc-builtin-fsmonitor. I would like to keep it as a separate topic because the built-in FSMonitor did not introduce this bug. This bug has been in Git's FSMonitor feature for a long, long time.

Changes since v1:

• Using NULL instead of 0 (d'oh!)

cc: Ævar Arnfjörð Bjarmason avarab@gmail.com
cc: Jeff Hostetler git@jeffhostetler.com

[derrickstolee]

Since I released Git for Windows v2.31.0, with brief interruption of two weeks, I enabled the built-in FSMonitor via my user config, and today was the first time I did anything in a bare repository. I was somewhat surprised that FSMonitor gave me trouble there, as the FSMonitor does not even make sense there...

This patch applies on top of jh/rfc-builtin-fsmonitor.

Maybe just add it to that branch, since it's only on v1 right now?

[dscho]

Maybe just add it to that branch, since it's only on v1 right now?

I'd rather not, for three reasons:

• This is on top of Jeff's patch series only because it would otherwise cause merge conflicts
• What it fixes is actually not the built-in FSMonitor. The bug existed for a long, long time before that patch series.
• Even if it was related, I think that Jeff's patch series really can stand on its own and I don't want to complicate reviews.

[dscho]

/submit

[gitgitgadget]

Submitted as pull.942.git.1619682362363.gitgitgadget@gmail.com

To fetch this version into FETCH_HEAD:

git fetch https://github.com/gitgitgadget/git pr-942/dscho/bare-repositories-need-no-fsmonitor-v1

To fetch this version to local tag pr-942/dscho/bare-repositories-need-no-fsmonitor-v1:

git fetch --no-tags https://github.com/gitgitgadget/git tag pr-942/dscho/bare-repositories-need-no-fsmonitor-v1

[j6t]

This should be = NULL, otherwise, sparse will complain about "Using plain integer as NULL pointer".

[gitgitgadget]

This branch is now known as jh/rfc-builtin-fsmonitor.

[gitgitgadget]

This patch series was integrated into seen via git@7607dc6.

[gitgitgadget]

This patch series was integrated into seen via git@5f1b369.

[gitgitgadget]

On the Git mailing list, Junio C Hamano wrote (reply to this):

"Johannes Schindelin via GitGitGadget" <gitgitgadget@gmail.com>
writes:

> From: Johannes Schindelin <johannes.schindelin@gmx.de>
>
> The entire point of the FSMonitor is to monitor the worktree changes in
> a more efficient manner than `lstat()`ing all worktree files every time
> we refresh the index.
>
> But if there is no worktree, FSMonitor has nothing to monitor.
>
> So let's ignore if an FSMonitor is configured (e.g. in `~/.gitconfig`)
> and we're running in a repository without worktree.
>
> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
> ---
>     fsmonitor: only enable it in non-bare repositories
>     
>     Since I released Git for Windows v2.31.0, with brief interruption of two
>     weeks, I enabled the built-in FSMonitor via my user config, and today
>     was the first time I did anything in a bare repository. I was somewhat
>     surprised that FSMonitor gave me trouble there, as the FSMonitor does
>     not even make sense there...
>     
>     This patch applies on top of jh/rfc-builtin-fsmonitor (not because it
>     fixes a problem in the built-in FSMonitor, the bug existed for a long,
>     long time before those patches, but because it would otherwise cause
>     merge conflicts with that patch series).
>
> Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-942%2Fdscho%2Fbare-repositories-need-no-fsmonitor-v1
> Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-942/dscho/bare-repositories-need-no-fsmonitor-v1
> Pull-Request: https://github.com/gitgitgadget/git/pull/942
>
>  config.c | 6 ++++++
>  1 file changed, 6 insertions(+)
>
> diff --git a/config.c b/config.c
> index 53e7dedc60de..fc5e744d81ca 100644
> --- a/config.c
> +++ b/config.c
> @@ -2515,6 +2515,12 @@ int git_config_get_max_percent_split_change(void)
>  
>  int repo_config_get_fsmonitor(struct repository *r)
>  {
> +	if (!r->worktree) {
> +		/* FSMonitor makes no sense in bare repositories */
> +		core_fsmonitor = 0;

Use NULL instead of integer 0 to mollify SP.


> +		return 1;
> +	}
> +
>  	if (r->settings.use_builtin_fsmonitor > 0) {
>  		core_fsmonitor = "(built-in daemon)";
>  		return 1;
>
> base-commit: 14d50074ff19e68e7a8d718b22d138882087bbc9

[gitgitgadget]

This patch series was integrated into seen via git@8826990.

[dscho]

/submit

[gitgitgadget]

Submitted as pull.942.v2.git.1620033105872.gitgitgadget@gmail.com

To fetch this version into FETCH_HEAD:

git fetch https://github.com/gitgitgadget/git pr-942/dscho/bare-repositories-need-no-fsmonitor-v2

To fetch this version to local tag pr-942/dscho/bare-repositories-need-no-fsmonitor-v2:

git fetch --no-tags https://github.com/gitgitgadget/git tag pr-942/dscho/bare-repositories-need-no-fsmonitor-v2

[gitgitgadget]

On the Git mailing list, Ævar Arnfjörð Bjarmason wrote (reply to this):

On Thu, Apr 29 2021, Johannes Schindelin via GitGitGadget wrote:

> From: Johannes Schindelin <johannes.schindelin@gmx.de>
>
> The entire point of the FSMonitor is to monitor the worktree changes in
> a more efficient manner than `lstat()`ing all worktree files every time
> we refresh the index.
>
> But if there is no worktree, FSMonitor has nothing to monitor.
>
> So let's ignore if an FSMonitor is configured (e.g. in `~/.gitconfig`)
> and we're running in a repository without worktree.
>
> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
> ---
>     fsmonitor: only enable it in non-bare repositories
>     
>     Since I released Git for Windows v2.31.0, with brief interruption of two
>     weeks, I enabled the built-in FSMonitor via my user config, and today
>     was the first time I did anything in a bare repository. I was somewhat
>     surprised that FSMonitor gave me trouble there, as the FSMonitor does
>     not even make sense there...
>     
>     This patch applies on top of jh/rfc-builtin-fsmonitor (not because it
>     fixes a problem in the built-in FSMonitor, the bug existed for a long,
>     long time before those patches, but because it would otherwise cause
>     merge conflicts with that patch series).
>
> Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-942%2Fdscho%2Fbare-repositories-need-no-fsmonitor-v1
> Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-942/dscho/bare-repositories-need-no-fsmonitor-v1
> Pull-Request: https://github.com/gitgitgadget/git/pull/942
>
>  config.c | 6 ++++++
>  1 file changed, 6 insertions(+)
>
> diff --git a/config.c b/config.c
> index 53e7dedc60de..fc5e744d81ca 100644
> --- a/config.c
> +++ b/config.c
> @@ -2515,6 +2515,12 @@ int git_config_get_max_percent_split_change(void)
>  
>  int repo_config_get_fsmonitor(struct repository *r)
>  {
> +	if (!r->worktree) {
> +		/* FSMonitor makes no sense in bare repositories */
> +		core_fsmonitor = 0;
> +		return 1;
> +	}
> +
>  	if (r->settings.use_builtin_fsmonitor > 0) {
>  		core_fsmonitor = "(built-in daemon)";
>  		return 1;
>
> base-commit: 14d50074ff19e68e7a8d718b22d138882087bbc9

This is surely a correct fix for now, but wouldn't it in the future also
be useful to run it in bare repositories e.g. to be able cache lookups
for non-existing loose objects?

[gitgitgadget]

User Ævar Arnfjörð Bjarmason <avarab@gmail.com> has been added to the cc: list.

[gitgitgadget]

On the Git mailing list, Jeff Hostetler wrote (reply to this):

On 5/3/21 9:58 AM, Ævar Arnfjörð Bjarmason wrote:
> 
> On Thu, Apr 29 2021, Johannes Schindelin via GitGitGadget wrote:
> 
>> From: Johannes Schindelin <johannes.schindelin@gmx.de>
>>
>> The entire point of the FSMonitor is to monitor the worktree changes in
>> a more efficient manner than `lstat()`ing all worktree files every time
>> we refresh the index.
>>
>> But if there is no worktree, FSMonitor has nothing to monitor.
>>
>> So let's ignore if an FSMonitor is configured (e.g. in `~/.gitconfig`)
>> and we're running in a repository without worktree.
>>
>> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
>> ---
...
>>
>> base-commit: 14d50074ff19e68e7a8d718b22d138882087bbc9
> 
> This is surely a correct fix for now, but wouldn't it in the future also
> be useful to run it in bare repositories e.g. to be able cache lookups
> for non-existing loose objects?
> 

No, the FSMonitor feature only expects data for paths within the
working directory.  (And is independent of whether the FS change
data is provided by my fsmonitor--daemon or provided by a hook-based
provider, such as Watchman.)  The FSMonitor feature uses that data to
shortcut scans of the working directory.

There is no interaction with the contents of the .git/objects
directory and I'm not sure how that would work.

Jeff

[gitgitgadget]

User Jeff Hostetler <git@jeffhostetler.com> has been added to the cc: list.

[gitgitgadget]

This patch series was integrated into seen via git@47c3902.

[gitgitgadget]

This patch series was integrated into seen via git@2e08481.

[gitgitgadget]

On the Git mailing list, Ævar Arnfjörð Bjarmason wrote (reply to this):

On Mon, May 03 2021, Jeff Hostetler wrote:

> On 5/3/21 9:58 AM, Ævar Arnfjörð Bjarmason wrote:
>> On Thu, Apr 29 2021, Johannes Schindelin via GitGitGadget wrote:
>> 
>>> From: Johannes Schindelin <johannes.schindelin@gmx.de>
>>>
>>> The entire point of the FSMonitor is to monitor the worktree changes in
>>> a more efficient manner than `lstat()`ing all worktree files every time
>>> we refresh the index.
>>>
>>> But if there is no worktree, FSMonitor has nothing to monitor.
>>>
>>> So let's ignore if an FSMonitor is configured (e.g. in `~/.gitconfig`)
>>> and we're running in a repository without worktree.
>>>
>>> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
>>> ---
> ...
>>>
>>> base-commit: 14d50074ff19e68e7a8d718b22d138882087bbc9
>> This is surely a correct fix for now, but wouldn't it in the future
>> also
>> be useful to run it in bare repositories e.g. to be able cache lookups
>> for non-existing loose objects?
>> 
>
> No, the FSMonitor feature only expects data for paths within the
> working directory.  (And is independent of whether the FS change
> data is provided by my fsmonitor--daemon or provided by a hook-based
> provider, such as Watchman.)  The FSMonitor feature uses that data to
> shortcut scans of the working directory.

Indeed, hence "in the future". I'm not suggesting that it'll do anything
useful by watching anything in the .git directory now, but that it might
be an interesting thing to explore.

> There is no interaction with the contents of the .git/objects
> directory and I'm not sure how that would work.

We'd watch .git/objects and .git/objects/{aa..ff}, then when about to
check for a loose object we'd avoid hitting the FS.

I don't know how useful that is post-61c7711cfea (sha1-file: use loose
object cache for quick existence check, 2018-11-12), but e.g. on NFS
this sort of thing still mattered. I had a "bigger hammer" approach with
[1] that ran (and still does, I believe) on a big corporate
installation.

More generally, if you strace .git access during repo operations you'll
find we're doing all sorts of existence checks etc. all the time. Loose
objects, refs, seeing what packs there are (better with the MIDX, but do
we still fall back?) etc. If we had up-to-date inotify/fsmonitor info we
could ask the daemon about it.

1. https://lore.kernel.org/git/20181028225023.26427-5-avarab@gmail.com/

[gitgitgadget]

This patch series was integrated into seen via git@3a1bc05.

[gitgitgadget]

This patch series was integrated into seen via git@a1ff0b6.

[gitgitgadget]

This patch series was integrated into seen via git@ab0bdee.

[gitgitgadget]

There was a status update in the "Stalled" section about the branch jh/rfc-builtin-fsmonitor on the Git mailing list:

An attempt to write and ship with a watchman equivalent tailored
for our use.

[gitgitgadget]

This patch series was integrated into seen via git@19e3c35.

[gitgitgadget]

This patch series was integrated into seen via git@5216d6f.

[gitgitgadget]

This patch series was integrated into seen via git@a51023d.

[gitgitgadget]

On the Git mailing list, Jeff Hostetler wrote (reply to this):

On 5/3/21 1:56 PM, Ævar Arnfjörð Bjarmason wrote:
> On Mon, May 03 2021, Jeff Hostetler wrote:
> 
>> On 5/3/21 9:58 AM, Ævar Arnfjörð Bjarmason wrote:
>>> On Thu, Apr 29 2021, Johannes Schindelin via GitGitGadget wrote:
>>>
>>>> From: Johannes Schindelin <johannes.schindelin@gmx.de>
>>>>
>>>> The entire point of the FSMonitor is to monitor the worktree changes in
>>>> a more efficient manner than `lstat()`ing all worktree files every time
>>>> we refresh the index.
>>>>
>>>> But if there is no worktree, FSMonitor has nothing to monitor.
>>>>
>>>> So let's ignore if an FSMonitor is configured (e.g. in `~/.gitconfig`)
>>>> and we're running in a repository without worktree.
>>>>
>>>> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
>>>> ---
>> ...
>>>>
>>>> base-commit: 14d50074ff19e68e7a8d718b22d138882087bbc9
>>> This is surely a correct fix for now, but wouldn't it in the future
>>> also
>>> be useful to run it in bare repositories e.g. to be able cache lookups
>>> for non-existing loose objects?
>>>
>>
>> No, the FSMonitor feature only expects data for paths within the
>> working directory.  (And is independent of whether the FS change
>> data is provided by my fsmonitor--daemon or provided by a hook-based
>> provider, such as Watchman.)  The FSMonitor feature uses that data to
>> shortcut scans of the working directory.
> 
> Indeed, hence "in the future". I'm not suggesting that it'll do anything
> useful by watching anything in the .git directory now, but that it might
> be an interesting thing to explore.
> 
>> There is no interaction with the contents of the .git/objects
>> directory and I'm not sure how that would work.
> 
> We'd watch .git/objects and .git/objects/{aa..ff}, then when about to
> check for a loose object we'd avoid hitting the FS.
> 
> I don't know how useful that is post-61c7711cfea (sha1-file: use loose
> object cache for quick existence check, 2018-11-12), but e.g. on NFS
> this sort of thing still mattered. I had a "bigger hammer" approach with
> [1] that ran (and still does, I believe) on a big corporate
> installation.
> 
> More generally, if you strace .git access during repo operations you'll
> find we're doing all sorts of existence checks etc. all the time. Loose
> objects, refs, seeing what packs there are (better with the MIDX, but do
> we still fall back?) etc. If we had up-to-date inotify/fsmonitor info we
> could ask the daemon about it.
> 
> 1. https://lore.kernel.org/git/20181028225023.26427-5-avarab@gmail.com/
> 

Interesting.

That's certainly something to look into later.  I know there is
code in the object lookup code to rescan/reload the packfiles or
loose objects (under the assumption that another process just created
a new packfile (and after our process loaded the packed-git list)).
Such a daemon might be helpful to improve something like that.

But I can't think about any of that right now.  I'd like to finish the
current fsmonitor--daemon patch series and let it settle down
before starting to think about an orthogonal use case such as this.

Thanks
Jeff

[gitgitgadget]

This patch series was integrated into seen via git@a9451c3.

[gitgitgadget]

There was a status update in the "Stalled" section about the branch jh/rfc-builtin-fsmonitor on the Git mailing list:

An attempt to write and ship with a watchman equivalent tailored
for our use.

[gitgitgadget]

This patch series was integrated into seen via git@24242a4.

[gitgitgadget]

This patch series was integrated into seen via git@79b0482.

[gitgitgadget]

This patch series was integrated into seen via git@a2867f7.

[gitgitgadget]

This patch series was integrated into seen via git@2eba377.

[gitgitgadget]

This patch series was integrated into seen via git@4024aca.

[gitgitgadget]

This patch series was integrated into seen via git@500f2ba.

[gitgitgadget]

This patch series was integrated into seen via git@e6e6b8d.

[gitgitgadget]

There was a status update in the "Stalled" section about the branch jh/rfc-builtin-fsmonitor on the Git mailing list:

An attempt to write and ship with a watchman equivalent tailored
for our use.

[gitgitgadget]

This patch series was integrated into seen via git@14b21e6.

[gitgitgadget]

This patch series was integrated into seen via git@bf43371.

[gitgitgadget]

There was a status update in the "Stalled" section about the branch jh/rfc-builtin-fsmonitor on the Git mailing list:

An attempt to write and ship with a watchman equivalent tailored
for our use.

[gitgitgadget]

This patch series was integrated into seen via git@bc58b98.

[gitgitgadget]

This patch series was integrated into seen via git@3fd7bc3.

[gitgitgadget]

There was a status update in the "Stalled" section about the branch jh/rfc-builtin-fsmonitor on the Git mailing list:

An attempt to write and ship with a watchman equivalent tailored
for our use.

[dscho]

This is no longer relevant.