r26.07.19

[christian-schilling]

core

New features

• New :select[F] filter as the complement of :exclude[F] (Add the Select filter as the complement of Exclude #2250)
  :select[F] keeps only the paths present in F's result. It's the exact set-complement of
  :exclude[F].

• The insertion filter is no longer experimental (Make the Insert filter non-experimental #2246)
  :$path=<content> no longer requires JOSH_EXPERIMENTAL_FEATURES=1. :$path=<oid> now
  also accepts a tree oid, inserting a whole subtree (Rename Op::Blob to Op::Insert and support tree oids #2231), and :$.=<oid> replaces the
  whole output tree instead of silently producing an empty one (Support inserting a tree at the root with :$.=<oid> #2251).

• history="no-splice" flag to suppress synthetic splice merges (Add history="no-splice" flag to disable synthetic splice merges #2209)
  The history meta value now accepts a comma-separated list of flags; no-splice drops
  the synthetic merge parents that splice in the prior history of paths made visible by a
  filter change.

Breaking changes

• :pin can no longer be combined with history splicing (Reject :pin combined with history splicing #2224)
  The combination is undefined, so josh now errors out instead of producing an ill-defined
  result. Filters using :pin must set history="no-splice".

Bug fixes

• Correct :rev / :hook reversal per commit (Reverse per-commit filters (:rev/:hook) per commit #2240)
  Reversing :rev(...) or :hook went through the generic invert path, breaking roundtrips.
  They are now reversed per commit like stored filters.

• Insert-filter oids are reachable from persisted filter trees (Reference the object of Op::Insert from its tree representation #2259)
  The oid of an :$path=<oid> insert was persisted as a hex-string blob, so the object
  could be missing when the filter tree is transferred. It is now a real tree entry, and
  distributed cache refs are named after the persisted id.

• Cache backend errors no longer crash the process (Propagate cache backend errors instead of panicking #2261)
  Transaction cache lookups turned any backend error into a panic via expect(); errors
  now propagate as Result through the already-fallible callers.

• Removals of whole subtrees show up in tree diffs again (Fix diff_paths dropping removals of whole subtrees #2297)
  A dead branch in diff_paths meant deleting an entire subdirectory reported no removed
  paths, affecting downstack dependency tracking and the GraphQL changedFiles diff.

• Composing insertion filters with shared directory prefixes no longer drops entries (Invert generative filters to empty to fix compose collapse #2232, Keep generative leaves out of common_pre hoisting #2233, Bump CACHE_VERSION to 29 for the generative filter invert change #2268)
  A filter like :[:$x/a="1",:$x/b="2",:$y/a="3",:$y/b="4"] dropped every group but the
  first.

Performance

• Pack filtered objects through an in-memory ODB backend (Pack filtered objects via an in-memory ODB backend #2212, Pack filtered objects on a background thread #2217, Route distributed cache writes through an in-memory object store #2295, Overlap distributed cache packing with filtering #2294)
  Loose-object writes dominated large rewrites; objects now buffer in a per-transaction
  in-memory libgit2 ODB backend and are flushed to packfiles, bounded by a memory limit
  (128 MiB by default).
  Packfile writing is handed to a process-global background worker so the filter hot path
  keeps running, with a blocking drain at transaction and subprocess boundaries.

• Leaner distributed cache storage format (Store distributed cache entries as gitlinks in a two-level fanout #2293, Do not embed the filter tree in distributed cache commits #2260)
  Cache entries are now gitlinks in a two-level fanout instead of hex-text blobs over three
  levels, and cache commits no longer embed the filter tree.

• Distributed cache eligibility checks no longer read the ODB (Decide distributed cache eligibility from the cached history hint #2284)
  The parent count is encoded in the cached history-graph hint instead of read from the ODB
  per query. Also fixes dropped entries for intermediate commits of chained filter specs.

• The distributed cache backend is read-only by default (Make the distributed cache backend read-only by default #2296)
  Plain clones and fetches no longer append flush commits that grow the shard chains;
  josh cache build remains the producer path.

• Cache 3-way merge results during downstacking (Cache 3-way merge_trees results per transaction #2168)
  The merged tree is a pure function of its three input trees, so results are memoized on
  the transaction, collapsing identical merges across a stack into hashmap hits.

• Faster downstacking via a dedicated filter op (Reimplement stack downstacking as a :_=<sha> filter op #2107, Compute downstack deps via tree::diff_paths #2169, Cache downstack_commit_deps per transaction #2167)
  Downstacking is now a filter op, written :_=<sha> (experimental), so results
  participate in the filter cache; dependencies are computed by a direct tree walk,
  memoized per transaction.

• Intern filter ops into hash-consed nodes (Intern filter Ops into hash-consed nodes #2199)
  The optimizer deep-cloned ops and re-serialized them to compute OIDs; ops are now
  interned into hash-consed nodes that cache their OID.

• Skip futile trailing-Compose distribution in the optimizer (Skip futile trailing-Compose distribution; flatten fully off hot paths #2223)
  Distributing a trailing compose out of a chain was immediately re-merged by a later step —
  an O(files × pins) blow-up on wide pin filters. Hot paths now skip it.

• Disable libgit2 strict object creation (Disable libgit2 strict object creation in josh #2191)
  Josh only writes objects whose referenced objects it has just produced or read, so
  libgit2's per-write existence verification is pure overhead and is now turned off.

• Keep git_odb__freshen off the filter write path (Keep git_odb__freshen off the filter write path #2202)
  libgit2 stats/touches loose objects on every write to refresh mtimes against gc, which
  josh never relies on during filtering; the in-memory backend now bypasses it.

• Build tree results with a single tree builder (Build tree results with a single tree builder #2254)
  subtract, intersect, pathstree and other tree operations re-seeded and wrote a
  TreeBuilder per entry — quadratic on wide trees. Each now collects its children into a
  single builder and writes the tree once.

• Skip reading the roots blob when only the sequence number is needed (Skip reading the roots blob when only the sequence number is needed #2257)
  The cached hint carries the sequence number directly, skipping a reachable-roots blob
  read per input commit.

• Build pin overlay with a single tree intersect (Build the pin overlay with a single tree intersect #2249)
  The :pin step that re-contents pinned paths from the parent's filtered tree is now a
  single path-intersection pass using tree::intersect.

josh-cli

New features

• Wrap a pushed commit in a merge with josh push --merge (Wrap pushed commit in a merge on top of --base via --merge #2206)
  The CLI equivalent of git push -o merge: the reverse-filtered commit is wrapped in a
  two-parent merge on top of the base, recording an explicit integration commit. Requires
  --base=<ref> or an existing destination ref.

• Reparent orphan commits onto --base when pushing (Reparent orphan commits onto --base when pushing #2207)
  Unrelated (orphan) history is overlaid onto the base instead of pushed as a disconnected
  root, matching what josh-proxy does for git push -o base=<ref>.

• Global --no-distributed-cache flag (Add a global --no-distributed-cache flag to the josh CLI #2278)
  Disables the distributed cache for a command: filtering neither reads nor writes
  refs/josh/cache/*.

Bug fixes

• Semantic :~() meta args of remote filters take effect
  A remote filter like :~(history="keep-trivial-merges")[:/sub1] had its meta args ignored
  when filtering.

josh-proxy

Improvements

• Better diagnostics in the GraphQL proxy's ref fetching (Rework fetch request mechanism in graphql #2228)
  The confusing "ref query not allowed" error shown when a remote did not return a
  requested OID is replaced with a contextualized message.

documentation

• Simplified Chinese README and documentation (Simplified Chinese version of README. #2165, Use human translation for zh_CN FAQ #2163)
  The README and the documentation are now available in Simplified Chinese.

---

This discussion was created from the release r26.07.19.