Skip to content

Conversation

nipunn1313
Copy link

@nipunn1313 nipunn1313 commented Oct 20, 2020

UPDATE SINCE v1

  • Removed include of dir.h from fsmonitor.h as it's no longer needed

Credit to alexmv again - I'm rebasing these changes from a couple years ago for contribution.

Full comments are available here -
https://public-inbox.org/git/01ad47b4-aa5e-461a-270b-dd60032afbd1@gmail.com/
To summarize the relevant points

Re: Inlining mark_fsmonitor_[in]valid
peartben said

I'm fine with these not being inline.  I was attempting to minimize the 
performance impact of the fsmonitor code when it was not even turned on. 
  Inlineing these functions allowed it to be kept to a simple test but I 
suspect (especially with modern optimizing compilers) that the overhead 
of calling a function to do that test is negligible.

Re test-dump-fsmonitor
peartben suggested keeping the +- syntax as well as the summary counts
dscho suggested dumping the invalid entries

cc: Taylor Blau me@ttaylorr.com
cc: Nipunn Koorapati nipunn1313@gmail.com

@nipunn1313 nipunn1313 force-pushed the fsmonitor branch 3 times, most recently from e46d7e5 to 5985210 Compare October 21, 2020 13:55
@nipunn1313
Copy link
Author

/submit

@gitgitgadget
Copy link

gitgitgadget bot commented Oct 21, 2020

Submitted as pull.767.git.1603303474.gitgitgadget@gmail.com

To fetch this version into FETCH_HEAD:

git fetch https://github.com/gitgitgadget/git pr-767/nipunn1313/fsmonitor-v1

To fetch this version to local tag pr-767/nipunn1313/fsmonitor-v1:

git fetch --no-tags https://github.com/gitgitgadget/git tag pr-767/nipunn1313/fsmonitor-v1

@gitgitgadget
Copy link

gitgitgadget bot commented Oct 21, 2020

On the Git mailing list, Taylor Blau wrote (reply to this):

Hi Nipunn,

On Wed, Oct 21, 2020 at 06:04:32PM +0000, Nipunn Koorapati via GitGitGadget wrote:
> Credit to alexmv again - I'm rebasing these changes from a couple years ago
> for contribution.
>
> Full comments are available here -
> https://public-inbox.org/git/01ad47b4-aa5e-461a-270b-dd60032afbd1@gmail.com/
> To summarize the relevant points

I'm fine with both of these patches, but it may help to have a bit
more information about how they will be used. Presumably more patches
are coming that make use of the new public functions, but it'd be good
to know a little bit of why these changes are necessary.

Thanks,
Taylor

@gitgitgadget
Copy link

gitgitgadget bot commented Oct 21, 2020

User Taylor Blau <me@ttaylorr.com> has been added to the cc: list.


/* Now that we've updated istate, save the last_update_token */
FREE_AND_NULL(istate->fsmonitor_last_update);
istate->fsmonitor_last_update = strbuf_detach(&last_update_token, NULL);
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On the Git mailing list, Taylor Blau wrote (reply to this):

On Wed, Oct 21, 2020 at 06:04:33PM +0000, Alex Vandiver via GitGitGadget wrote:
> From: Alex Vandiver <alexmv@dropbox.com>
>
> These were inline'd when they were first introduced, presumably as an
> optimization for cases when they were called in tight loops.  This
> complicates using these functions, as untracked_cache_invalidate_path
> is defined in dir.h.
>
> Leave the inline'ing up to the compiler's decision, for ease of use.

Letting the compiler inline these is fine, but...

> diff --git a/fsmonitor.h b/fsmonitor.h
> index 739318ab6d..6249020692 100644
> --- a/fsmonitor.h
> +++ b/fsmonitor.h
> @@ -49,14 +49,7 @@ void refresh_fsmonitor(struct index_state *istate);
>   * called any time the cache entry has been updated to reflect the
>   * current state of the file on disk.
>   */
> -static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache_entry *ce)
> -{
> -	if (core_fsmonitor && !(ce->ce_flags & CE_FSMONITOR_VALID)) {
> -		istate->cache_changed = 1;
> -		ce->ce_flags |= CE_FSMONITOR_VALID;
> -		trace_printf_key(&trace_fsmonitor, "mark_fsmonitor_clean '%s'", ce->name);
> -	}
> -}
> +extern void mark_fsmonitor_valid(struct index_state *istate, struct cache_entry *ce);
>
>  /*
>   * Clear the given cache entry's CE_FSMONITOR_VALID bit and invalidate
> @@ -65,13 +58,6 @@ static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache
>   * trigger an lstat() or invalidate the untracked cache for the
>   * corresponding directory
>   */
> -static inline void mark_fsmonitor_invalid(struct index_state *istate, struct cache_entry *ce)
> -{
> -	if (core_fsmonitor) {
> -		ce->ce_flags &= ~CE_FSMONITOR_VALID;
> -		untracked_cache_invalidate_path(istate, ce->name, 1);
> -		trace_printf_key(&trace_fsmonitor, "mark_fsmonitor_invalid '%s'", ce->name);
> -	}
> -}
> +extern void mark_fsmonitor_invalid(struct index_state *istate, struct cache_entry *ce);
>
>  #endif

Any reason that these need to be externed explicitly? Note that these
functions are already externed by default since you haven't said
otherwise (and for no other reason than this'd be the only explicitly
externed function in fsmonitor.h).

Thanks,
Taylor

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On the Git mailing list, Junio C Hamano wrote (reply to this):

Taylor Blau <me@ttaylorr.com> writes:

>> +extern void mark_fsmonitor_invalid(struct index_state *istate, struct cache_entry *ce);
> ...
> Any reason that these need to be externed explicitly? Note that these
> functions are already externed by default since you haven't said
> otherwise (and for no other reason than this'd be the only explicitly
> externed function in fsmonitor.h).

Possibly due to the recent discussion?

https://lore.kernel.org/git/xmqqtuv3ryhr.fsf_-_@gitster.c.googlers.com/

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On the Git mailing list, Taylor Blau wrote (reply to this):

On Wed, Oct 21, 2020 at 02:24:22PM -0700, Junio C Hamano wrote:
> Taylor Blau <me@ttaylorr.com> writes:
> > Any reason that these need to be externed explicitly? Note that these
> > functions are already externed by default since you haven't said
> > otherwise (and for no other reason than this'd be the only explicitly
> > externed function in fsmonitor.h).
>
> Possibly due to the recent discussion?
>
> https://lore.kernel.org/git/xmqqtuv3ryhr.fsf_-_@gitster.c.googlers.com/

Ah, thanks. I remember the thread, but I wasn't sure where the
discussion ended up. After re-reading it, it sounds like new function
declarations in header files should be prefixed with 'extern' (making
this patch correct as it already is).

Tangential to this discussion: are you still expecting a tree-wide
change to start use extern everywhere?

Thanks,
Taylor

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On the Git mailing list, Junio C Hamano wrote (reply to this):

Taylor Blau <me@ttaylorr.com> writes:

> Tangential to this discussion: are you still expecting a tree-wide
> change to start use extern everywhere?

I think before we start opening the tree for new topics is the best
time to do so, if we were to follow through, but after we have dealt
with brown-paper-bag fixes to the release.  The Makefile tweak for
the skip-dashed thing is the only one for 2.29, I think, so ...

Thanks.


Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On the Git mailing list, Nipunn Koorapati wrote (reply to this):

> Letting the compiler inline these is fine, but...
>
> Any reason that these need to be externed explicitly? Note that these
> functions are already externed by default since you haven't said
> otherwise (and for no other reason than this'd be the only explicitly
> externed function in fsmonitor.h).

Did not have a reason or strong opinion here. It was this way, because this was
the way alexmv used it originally - but it does compile in either manner. The
thread Junio linked does seem to indicate preference for extern to avoid
confusion.

--Nipunn

@gitgitgadget
Copy link

gitgitgadget bot commented Oct 21, 2020

This branch is now known as av/fsmonitor-cleanup.

@gitgitgadget
Copy link

gitgitgadget bot commented Oct 21, 2020

This patch series was integrated into seen via git@0329cdd.

@gitgitgadget gitgitgadget bot added the seen label Oct 21, 2020
Alex Vandiver added 2 commits October 22, 2020 00:18
These were inline'd when they were first introduced, presumably as an
optimization for cases when they were called in tight loops.  This
complicates using these functions, as untracked_cache_invalidate_path
is defined in dir.h.

Leave the inline'ing up to the compiler's decision, for ease of use.

Signed-off-by: Alex Vandiver <alexmv@dropbox.com>
Signed-off-by: Nipunn Koorapati <nipunn@dropbox.com>
After displaying one very long line, summarize the contents of that
line.  The tests do not currently rely on any content except the first
line ("no fsmonitor" / "fsmonitor last update").

Signed-off-by: Alex Vandiver <alexmv@dropbox.com>
Signed-off-by: Nipunn Koorapati <nipunn@dropbox.com>
@gitgitgadget
Copy link

gitgitgadget bot commented Oct 21, 2020

On the Git mailing list, Nipunn Koorapati wrote (reply to this):

> I'm fine with both of these patches, but it may help to have a bit
> more information about how they will be used. Presumably more patches
> are coming that make use of the new public functions, but it'd be good
> to know a little bit of why these changes are necessary.

I believe the externs are just there to avoid pulling in `dir.h` from
the include file - since
it's only needed in the implementation. I believe at the time
(12/2017) `dir.h` was not imported from
`fsmonitor.h`, but it does appear imported now. I've eliminated the
import of `dir.h` as it
no longer appears necessary - which I will include in the next
iteration of this diff.

The test helper merely makes it easier to debug fsmonitor tests - will
be useful to any
developer working on fsmonitor related changes. I have an upcoming one
related to
fsmonitor in git checkout, which I've also revived, but I'm not 100%
sure I'll get it through,
but regardless, I believe this test-debugging change will be good.

--Nipunn

@gitgitgadget
Copy link

gitgitgadget bot commented Oct 21, 2020

User Nipunn Koorapati <nipunn1313@gmail.com> has been added to the cc: list.

@nipunn1313
Copy link
Author

/submit

@gitgitgadget
Copy link

gitgitgadget bot commented Oct 22, 2020

Submitted as pull.767.v2.git.1603326066.gitgitgadget@gmail.com

To fetch this version into FETCH_HEAD:

git fetch https://github.com/gitgitgadget/git pr-767/nipunn1313/fsmonitor-v2

To fetch this version to local tag pr-767/nipunn1313/fsmonitor-v2:

git fetch --no-tags https://github.com/gitgitgadget/git tag pr-767/nipunn1313/fsmonitor-v2

@gitgitgadget
Copy link

gitgitgadget bot commented Oct 22, 2020

This patch series was integrated into seen via git@0f00a96.

@gitgitgadget
Copy link

gitgitgadget bot commented Oct 22, 2020

On the Git mailing list, Taylor Blau wrote (reply to this):

Hi Nipunn,

On Thu, Oct 22, 2020 at 12:21:04AM +0000, Nipunn Koorapati via GitGitGadget wrote:
> UPDATE SINCE v1
>
>  * Removed include of dir.h from fsmonitor.h as it's no longer needed

This version all looks sensible to me.

I'm still iffy on whether or not this series makes sense to apply
without the rest of the code that depends on it, but I'll leave that up
to Junio whether he wants to take the series as it is now, or wait for
other patches to come in on top.

In either case, these two patches are:

  Reviewed-by: Taylor Blau <me@ttaylorr.com>

Thanks,
Taylor

@gitgitgadget
Copy link

gitgitgadget bot commented Oct 22, 2020

On the Git mailing list, Junio C Hamano wrote (reply to this):

Taylor Blau <me@ttaylorr.com> writes:

> Hi Nipunn,
>
> On Thu, Oct 22, 2020 at 12:21:04AM +0000, Nipunn Koorapati via GitGitGadget wrote:
>> UPDATE SINCE v1
>>
>>  * Removed include of dir.h from fsmonitor.h as it's no longer needed
>
> This version all looks sensible to me.
>
> I'm still iffy on whether or not this series makes sense to apply
> without the rest of the code that depends on it, but I'll leave that up
> to Junio whether he wants to take the series as it is now, or wait for
> other patches to come in on top.

Sorry but I am not sure what you mean by "the code that depends on
it".  Are these two functions unused anywhere in the code?  If so,
the right way to clean them up may not be to turn them from inline
to a proper definition, but to remove them ;-).  

If they have existing callers and it can be demonstrated that their
callers do not benefit from them being inline, that by itself is a
worthy clean-up, without adding any more callers, no?

Confused...

> In either case, these two patches are:
>
>   Reviewed-by: Taylor Blau <me@ttaylorr.com>
>
> Thanks,
> Taylor

@gitgitgadget
Copy link

gitgitgadget bot commented Oct 22, 2020

On the Git mailing list, Taylor Blau wrote (reply to this):

On Thu, Oct 22, 2020 at 11:32:51AM -0700, Junio C Hamano wrote:
> Taylor Blau <me@ttaylorr.com> writes:
> > I'm still iffy on whether or not this series makes sense to apply
> > without the rest of the code that depends on it, but I'll leave that up
> > to Junio whether he wants to take the series as it is now, or wait for
> > other patches to come in on top.
>
> Sorry but I am not sure what you mean by "the code that depends on
> it".  Are these two functions unused anywhere in the code?  If so,
> the right way to clean them up may not be to turn them from inline
> to a proper definition, but to remove them ;-).
>
> If they have existing callers and it can be demonstrated that their
> callers do not benefit from them being inline, that by itself is a
> worthy clean-up, without adding any more callers, no?
>
> Confused...

Sorry for the confusion. I mean the following:

  - These functions have existing callers that Nipunn claims do not need
    to be explicitly inlined.

  - These functions are being moved to be part of the fsmonitor public
    interface (presumably so that new callers can be added).

...And I was wondering whether you wanted to wait for new callers
before applying these to your tree.

Thanks,
Taylor

@gitgitgadget
Copy link

gitgitgadget bot commented Oct 22, 2020

On the Git mailing list, Junio C Hamano wrote (reply to this):

Taylor Blau <me@ttaylorr.com> writes:

> Sorry for the confusion. I mean the following:
>
>   - These functions have existing callers that Nipunn claims do not need
>     to be explicitly inlined.

I guess "claims" is the key phrase in your responsehere.  Do you
feel that the claim is not sufficiently substantiated?

Those without fsmonitor would pay the call/return cost for no good
reason if core_fsmonitor is not set, and checking that on the caller
side may make a big difference.  How big?  That needs measurement.

This is a tangent, but with or without inlining, I find it iffy to
see that untracked_cache_invalidate_path() is called only when
fsmonitor is in use.  Does untracked_cache depend on fsmonitor for
its correct operation?  Why is it OK not to invlidate when the
caller would tell fsmonitor that a path is invalid if fsmonitor were
in use?  The call is a statement of fact that the path is no longer
valid, and that bit of information would be useful to the parts of
the system outside fsmonitor, no?  Puzzled....

>   - These functions are being moved to be part of the fsmonitor public
>     interface (presumably so that new callers can be added).

They used to be implemented as static inline functions in the
fsmonitor.h header file, so they have been part of the public
interface anyway.  Anybody that includes fsmonitor.h can use it,
with or without the patch.  So I think this one would not be
a problem.

> ...And I was wondering whether you wanted to wait for new callers
> before applying these to your tree.

Thanks.

I still do not know about the "should the inline be kept" question.
The proposed log message for the commit does not explain (let alone
justify) why "optimization" is unneeded for the fuctions in the
first place, which does not help.



@gitgitgadget
Copy link

gitgitgadget bot commented Oct 22, 2020

This patch series was integrated into seen via git@7803745.

@gitgitgadget
Copy link

gitgitgadget bot commented Oct 22, 2020

This patch series was integrated into seen via git@152d49d.

@gitgitgadget
Copy link

gitgitgadget bot commented Oct 22, 2020

On the Git mailing list, Nipunn Koorapati wrote (reply to this):

> from Taylor
> I'm still iffy on whether or not this series makes sense to apply
> without the rest of the code that depends on it

Sorry for confusion. I don't think we should assume there is more code coming
related to this. I think this is intended to stand on its own.
It's not a required dependency either. Rather, it's motivated by
simplicity
- remove the dir.h dependency from fsmonitor.h.
- Keep implementation in fsmonitor.c and definitions in fsmonitor.h

> From Junio
> Those without fsmonitor would pay the call/return cost for no good
> reason if core_fsmonitor is not set, and checking that on the caller
> side may make a big difference.  How big?  That needs measurement.

Noted! This is not called out or measured - it is simply assumed based
on earlier
conversation. I should be able to run the fsmonitor perf suite before/after this
change and include the results in the commit message.

> This is a tangent, but with or without inlining, I find it iffy to
> see that untracked_cache_invalidate_path() is called only when
> fsmonitor is in use.  Does untracked_cache depend on fsmonitor for
> its correct operation?  Why is it OK not to invlidate when the
> caller would tell fsmonitor that a path is invalid if fsmonitor were
> in use?  The call is a statement of fact that the path is no longer
> valid, and that bit of information would be useful to the parts of
> the system outside fsmonitor, no?  Puzzled....

I did some source diving in an attempt to understand what's happening here.
I believe that untracked_cache_invalidate_path() is called in dir.c
whenever an entry is added or removed from
a directory.
This is an additional call when fsmonitor is enabled - because
fsmonitor's whole purpose
is to avoid the lstat on the other path. There is a nice explanation
in the original commit message

Commit 883e248b (fsmonitor: teach git to optionally utilize a file
system monitor to speed up detecting new or changed files.,
2017-09-22)

--Nipunn

@gitgitgadget
Copy link

gitgitgadget bot commented Oct 23, 2020

This patch series was integrated into seen via git@0eaf313.

@gitgitgadget
Copy link

gitgitgadget bot commented Oct 23, 2020

This patch series was integrated into seen via git@3830ded.

@gitgitgadget
Copy link

gitgitgadget bot commented Oct 23, 2020

This patch series was integrated into seen via git@7398e42.

@gitgitgadget
Copy link

gitgitgadget bot commented Oct 26, 2020

This patch series was integrated into seen via git@8377462.

@gitgitgadget
Copy link

gitgitgadget bot commented Oct 27, 2020

This patch series was integrated into seen via git@6689f5b.

@gitgitgadget
Copy link

gitgitgadget bot commented Oct 27, 2020

This patch series was integrated into seen via git@e82be5f.

@gitgitgadget
Copy link

gitgitgadget bot commented Oct 28, 2020

This patch series was integrated into seen via git@b48426e.

@gitgitgadget
Copy link

gitgitgadget bot commented Oct 29, 2020

This patch series was integrated into seen via git@765fca7.

@gitgitgadget
Copy link

gitgitgadget bot commented Oct 29, 2020

This patch series was integrated into seen via git@74718a3.

@gitgitgadget
Copy link

gitgitgadget bot commented Oct 30, 2020

This patch series was integrated into seen via git@38c3f2a.

@gitgitgadget
Copy link

gitgitgadget bot commented Nov 2, 2020

This patch series was integrated into seen via git@4f79d43.

@gitgitgadget
Copy link

gitgitgadget bot commented Nov 5, 2020

This patch series was integrated into seen via git@a1ea2a2.

@gitgitgadget
Copy link

gitgitgadget bot commented Nov 9, 2020

This patch series was integrated into seen via git@71084aa.

@gitgitgadget
Copy link

gitgitgadget bot commented Nov 10, 2020

This patch series was integrated into seen via git@5222df7.

@gitgitgadget
Copy link

gitgitgadget bot commented Nov 11, 2020

This patch series was integrated into seen via git@4bf2d47.

@gitgitgadget
Copy link

gitgitgadget bot commented Nov 11, 2020

This patch series was integrated into seen via git@87827c4.

@gitgitgadget
Copy link

gitgitgadget bot commented Nov 12, 2020

This patch series was integrated into seen via git@3db4720.

@gitgitgadget
Copy link

gitgitgadget bot commented Nov 16, 2020

This patch series was integrated into seen via git@e8be73c.

@gitgitgadget
Copy link

gitgitgadget bot commented Nov 17, 2020

This patch series was integrated into seen via git@41b8a5e.

@gitgitgadget
Copy link

gitgitgadget bot commented Nov 18, 2020

This patch series was integrated into seen via git@14e53e3.

@gitgitgadget
Copy link

gitgitgadget bot commented Nov 20, 2020

This patch series was integrated into seen via git@809e811.

@nipunn1313 nipunn1313 closed this Jan 14, 2021
@nipunn1313 nipunn1313 deleted the fsmonitor branch January 14, 2021 22:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant