Language-aware diff tools (e.g., Difftastic) and format-specific analyzers
can produce better line matching than Git's builtin diff algorithm, but
diff.<driver>.command replaces Git's diff output with the program's own
output, so display features like word diff, function context, and color
cannot operate on it; and because the program is consulted only for that
patch output, blame, --stat, and git log -L fall back to Git's builtin line
matching and cannot benefit from the tool at all.

This series adds diff.<driver>.process, a long-running subprocess protocol
that lets an external tool control which lines Git considers changed while
Git handles all output formatting. The protocol follows
filter.<driver>.process: pkt-line over stdin/stdout, capability negotiation,
one process per Git invocation.

The tool receives both file versions and returns changed regions (line
ranges in the old and new file). Git validates and feeds them into the xdiff
pipeline in place of the builtin diff algorithm. When the tool returns no
hunks, Git treats the files as having no changes, which propagates through
patch output, the --stat summary, blame, and git log -L. The request also
carries the two blobs' object names (old-oid/new-oid) so a tool can cache
its analysis keyed on the pair.

 * Patch 1: document how an external diff driver (diff.<driver>.command)
   relates to the rest of Git's diff features, so the contrast with the new
   process driver is clear.
 * Patch 2: xdiff plumbing for externally supplied hunks.
 * Patch 3: diff.<driver>.process config key.
 * Patch 4: refactor subprocess API to separate process lifecycle from
   hashmap management, since the diff process stores its subprocess on the
   userdiff driver rather than in a hashmap.
 * Patch 5: the main feature, including the old-oid/new-oid request metadata
   for blob-pair caching.
 * Patch 6: bypass knobs (--no-ext-diff, format-patch).
 * Patch 7: blame integration so the tool can declare commits as having no
   changes; introduces the shared xdi_diff_process() consult-then-diff
   helper that blame and git log -L both use.
 * Patch 8: --stat/--numstat/--shortstat consult the tool, so the summary
   agrees with the patch output.
 * Patch 9: git log -L range tracking consults the tool, so a reformat-only
   commit is dropped from the log rather than shown with an empty diff.

A "Which features consult the diff process" section in gitattributes(5) lays
out, per feature, why each does or does not consult the process (patch
output, blame, summary formats, and the -L line-range view do; pickaxe -G,
patch-id, merge, range-diff, --check, and --raw do not, with reasons).
Combined diffs (--cc) remain on the builtin algorithm and are noted as
future work.

Changes since v4:

 * New preparatory doc patch (patch 1): document how an external diff driver
   (diff.<driver>.command) relates to the rest of Git's diff features, so
   the contrast with the process driver added later in the series is
   explicit.

 * New: --stat/--numstat/--shortstat now consult the process (patch 8). A
   file the tool reports as equivalent contributes no stat line, matching
   the empty patch git diff produces for it. Summary formats do not apply
   textconv, so the tool is fed raw content there, as the builtin --stat
   already counts raw lines.

 * New: git log -L range tracking now consults the process (patch 9).
   Previously the tracking pass used the builtin diff while the displayed
   diff used the tool, so a reformat-only commit could be selected by
   tracking and then rendered with an empty diff. Tracking now agrees with
   the display and the commit is dropped.

 * New: the request carries the old and new blob object names
   (old-oid/new-oid), so a tool can cache its line matching keyed on the
   blob pair. A side's oid is sent only when the tool receives that raw
   blob; it is omitted under textconv (which rewrites the bytes) and for a
   working-tree side with no stored object. This is where the process
   differs from diff.<name>.command, which never composes with textconv and
   so always feeds the raw blob its oid names.

 * Refactor: blame and git log -L now consult the process through a single
   xdi_diff_process() helper instead of open-coding the consult-then-diff
   dance; builtin_diff() keeps its own path so it can short-circuit
   equivalent files before its funcname/word-diff setup. The whitespace
   bypass keys off the effective diff parameters (xpp), which removes
   blame's separate -w guard, and blame's diff_hunks() sheds an intermediate
   variant it no longer needs.

 * Correctness: external-hunk validation now checks per-gap alignment, not
   just the total unchanged-line count. xdl_build_script() walks the two
   files in lockstep over unchanged lines, so a hunk set whose totals
   balance but whose per-gap line counts do not (e.g. hunk 1 1 3 1)
   previously passed validation and produced a corrupt diff, --stat, and
   blame attribution. Such responses are now rejected and Git falls back to
   the builtin diff.

 * Robustness: hunk accumulation is bounded by the two files' combined line
   count, so a misbehaving tool that floods hunk lines cannot grow memory
   without bound before validation.

 * Forward-compat: the hunk-line parser now ignores unknown trailing fields
   after the four counts, so a future protocol version can append a field
   without older clients rejecting it.

 * Feature interactions: the whitespace-ignoring options (-w,
   --ignore-space-change, --ignore-blank-lines, ...), -I, and --anchored
   bypass the process (the tool is never told about them and could not honor
   them), and git blame -w does the same. A change that only adds or removes
   the trailing newline cannot be expressed as line hunks, so it also falls
   back to the builtin diff (preserving the "\ No newline at end of file"
   marker).

 * Documentation: added the per-feature "Which features consult the diff
   process" section; documented that the process trusts the tool's notion of
   "unchanged" (it is not byte-validated, so like git diff -w such a patch
   may not apply against the old blob), that --exit-code/--quiet report
   success for tool- equivalent files, and that diff.<name>.command takes
   precedence when both it and .process are configured.

 * Tests: t4080 grew coverage for the per-gap check, the hunk- flood cap,
   the whitespace/-w bypasses, the trailing-newline and added/deleted-file
   fallbacks, multi-hunk output through patch and --stat, --stat
   --exit-code, a mode-only change, and a mixed equivalent/changed
   multi-file diffstat.

Michael Montalbo (9):
  gitattributes: document how external diff drivers relate to diff
    features
  xdiff: support external hunks via xpparam_t
  userdiff: add diff.<driver>.process config
  sub-process: separate process lifecycle from hashmap management
  diff: add long-running diff process via diff.<driver>.process
  diff: bypass diff process with --no-ext-diff and in format-patch
  blame: consult diff process for no-hunk detection
  diff: consult diff process for --stat counts
  line-log: consult diff process for range tracking

 Documentation/config/diff.adoc           |   5 +
 Documentation/diff-algorithm-option.adoc |   3 +
 Documentation/diff-options.adoc          |   4 +-
 Documentation/gitattributes.adoc         | 274 +++++++
 Makefile                                 |   2 +
 blame.c                                  |  24 +-
 builtin/log.c                            |   7 +
 diff-process.c                           | 529 +++++++++++++
 diff-process.h                           |  75 ++
 diff.c                                   |  57 +-
 diff.h                                   |   6 +
 line-log.c                               |  33 +-
 meson.build                              |   1 +
 sub-process.c                            |  28 +-
 sub-process.h                            |   9 +-
 t/helper/meson.build                     |   1 +
 t/helper/test-diff-process-backend.c     | 381 +++++++++
 t/helper/test-tool.c                     |   1 +
 t/helper/test-tool.h                     |   1 +
 t/meson.build                            |   1 +
 t/t4080-diff-process.sh                  | 950 +++++++++++++++++++++++
 userdiff.c                               |   7 +
 userdiff.h                               |   5 +
 xdiff-interface.c                        |   7 +-
 xdiff/xdiff.h                            |  16 +
 xdiff/xdiffi.c                           |  84 +-
 xdiff/xprepare.c                         |  10 +
 xdiff/xprepare.h                         |   1 +
 28 files changed, 2504 insertions(+), 18 deletions(-)
 create mode 100644 diff-process.c
 create mode 100644 diff-process.h
 create mode 100644 t/helper/test-diff-process-backend.c
 create mode 100755 t/t4080-diff-process.sh

base-commit: e9019fcafe0040228b8631c30f97ae1adb61bcdc

Submitted-As: https://lore.kernel.org/git/pull.2120.v5.git.1784149323.gitgitgadget@gmail.com
In-Reply-To: https://lore.kernel.org/git/pull.2120.git.1779415884.gitgitgadget@gmail.com
In-Reply-To: https://lore.kernel.org/git/pull.2120.v2.git.1779733799.gitgitgadget@gmail.com
In-Reply-To: https://lore.kernel.org/git/pull.2120.v3.git.1780087700.gitgitgadget@gmail.com
In-Reply-To: https://lore.kernel.org/git/pull.2120.v4.git.1781463564.gitgitgadget@gmail.com