Here is V3 of my patch series to add a builtin FSMonitor daemon to Git. I
rebased this series onto v2.32.0.

V3 addresses most of the previous review comments and things we've learned
from our experimental testing of V2. (A version of V2 was shipped as an
experimental feature in the v2.32.0-based releases of Git for Windows and
VFS for Git.)

There are still a few items that I need to address, but that list is getting
very short.

The following items from my V2 cover have not yet been addressed:

[ ] Revisit the how the client handles the IPC_STATE__NOT_LISTENING state
(where a daemon appears to be running, but is non-responsive)

[ ] Consider having daemon chdir() out of the working directory to avoid
directory handle issues on Windows.

[ ] On Windows, If the daemon is started as an elevated process, then client
commands might not have access to communicate with it.

[ ] Review if/how we decide to shutdown the FSMonitor daemon after and a
significant idle period.

Also, there are potential problems with the untracked-cache that we have
been looking at. Concurrently, Tao Klerks independently noticed similar
problems with the untracked-cache and has reported/discussed them here on
the mailing list. I would like to get to the bottom of them before going
further -- at this point I don't know they are related to FSMonitor or not.

In this version, the first commit updates the Simple IPC API to make it
easier to pass binary data using {char *, size_t} rather than assuming that
the message is a null-terminated string. FSMonitor does not use binary
messages and doesn't really need this API change, but I thought it best to
fix the API now before we have other callers of IPC.

This patch series contains 34 commits and is rather large. If it would help
with the review, I could try to divide it into a client-side part 1 and a
daemon-side part 2 -- if there is interest.

Jeff Hostetler (34):
  simple-ipc: preparations for supporting binary messages.
  fsmonitor--daemon: man page
  fsmonitor--daemon: update fsmonitor documentation
  fsmonitor-ipc: create client routines for git-fsmonitor--daemon
  help: include fsmonitor--daemon feature flag in version info
  fsmonitor: config settings are repository-specific
  fsmonitor: use IPC to query the builtin FSMonitor daemon
  fsmonitor--daemon: add a built-in fsmonitor daemon
  fsmonitor--daemon: implement 'stop' and 'status' commands
  t/helper/fsmonitor-client: create IPC client to talk to FSMonitor
    Daemon
  fsmonitor-fs-listen-win32: stub in backend for Windows
  fsmonitor-fs-listen-macos: stub in backend for MacOS
  fsmonitor--daemon: implement 'run' command
  fsmonitor--daemon: implement 'start' command
  fsmonitor: do not try to operate on bare repos
  fsmonitor--daemon: add pathname classification
  fsmonitor--daemon: define token-ids
  fsmonitor--daemon: create token-based changed path cache
  fsmonitor-fs-listen-win32: implement FSMonitor backend on Windows
  fsmonitor-fs-listen-macos: add macos header files for FSEvent
  fsmonitor-fs-listen-macos: implement FSEvent listener on MacOS
  fsmonitor--daemon: implement handle_client callback
  t/helper/test-touch: add helper to touch a series of files
  t/perf/p7519: speed up test using "test-tool touch"
  t/perf: avoid copying builtin fsmonitor files into test repo
  t/perf/p7519: add fsmonitor--daemon test cases
  t7527: create test for fsmonitor--daemon
  fsmonitor--daemon: periodically truncate list of modified files
  fsmonitor--daemon: use a cookie file to sync with file system
  fsmonitor: enhance existing comments
  fsmonitor: force update index after large responses
  t7527: test status with untracked-cache and fsmonitor--daemon
  fsmonitor: handle shortname for .git
  t7527: test FS event reporing on MacOS WRT case and Unicode

 .gitignore                                   |    1 +
 Documentation/config/core.txt                |   56 +-
 Documentation/git-fsmonitor--daemon.txt      |   75 +
 Documentation/git-update-index.txt           |   27 +-
 Documentation/githooks.txt                   |    3 +-
 Makefile                                     |   17 +
 builtin.h                                    |    1 +
 builtin/fsmonitor--daemon.c                  | 1567 ++++++++++++++++++
 builtin/update-index.c                       |   20 +-
 cache.h                                      |    1 -
 compat/fsmonitor/fsmonitor-fs-listen-macos.c |  497 ++++++
 compat/fsmonitor/fsmonitor-fs-listen-win32.c |  663 ++++++++
 compat/fsmonitor/fsmonitor-fs-listen.h       |   49 +
 compat/simple-ipc/ipc-unix-socket.c          |   14 +-
 compat/simple-ipc/ipc-win32.c                |   14 +-
 config.c                                     |   14 -
 config.h                                     |    1 -
 config.mak.uname                             |    4 +
 contrib/buildsystems/CMakeLists.txt          |    8 +
 environment.c                                |    1 -
 fsmonitor--daemon.h                          |  140 ++
 fsmonitor-ipc.c                              |  179 ++
 fsmonitor-ipc.h                              |   48 +
 fsmonitor.c                                  |  189 ++-
 fsmonitor.h                                  |   14 +-
 git.c                                        |    1 +
 help.c                                       |    4 +
 repo-settings.c                              |   48 +
 repository.h                                 |   11 +
 simple-ipc.h                                 |    7 +-
 t/README                                     |    4 +-
 t/helper/test-fsmonitor-client.c             |  121 ++
 t/helper/test-simple-ipc.c                   |   34 +-
 t/helper/test-tool.c                         |    2 +
 t/helper/test-tool.h                         |    2 +
 t/helper/test-touch.c                        |  126 ++
 t/perf/p7519-fsmonitor.sh                    |   51 +-
 t/perf/perf-lib.sh                           |    2 +-
 t/t7519-status-fsmonitor.sh                  |   38 +
 t/t7527-builtin-fsmonitor.sh                 |  679 ++++++++
 t/test-lib.sh                                |    6 +
 41 files changed, 4618 insertions(+), 121 deletions(-)
 create mode 100644 Documentation/git-fsmonitor--daemon.txt
 create mode 100644 builtin/fsmonitor--daemon.c
 create mode 100644 compat/fsmonitor/fsmonitor-fs-listen-macos.c
 create mode 100644 compat/fsmonitor/fsmonitor-fs-listen-win32.c
 create mode 100644 compat/fsmonitor/fsmonitor-fs-listen.h
 create mode 100644 fsmonitor--daemon.h
 create mode 100644 fsmonitor-ipc.c
 create mode 100644 fsmonitor-ipc.h
 create mode 100644 t/helper/test-fsmonitor-client.c
 create mode 100644 t/helper/test-touch.c
 create mode 100755 t/t7527-builtin-fsmonitor.sh

base-commit: ebf3c04b262aa27fbb97f8a0156c2347fecafafb

Submitted-As: https://lore.kernel.org/git/pull.923.v3.git.1625150864.gitgitgadget@gmail.com
In-Reply-To: https://lore.kernel.org/git/pull.923.git.1617291666.gitgitgadget@gmail.com
In-Reply-To: https://lore.kernel.org/git/pull.923.v2.git.1621691828.gitgitgadget@gmail.com