Skip to content

Source control command list: git, hg, jj

Jump to bottom
Rain edited this page Jan 21, 2025 · 30 revisions

The most common use of this crate is with version control systems. The examples below cover three such systems: Git, Mercurial (including Sapling), and Jujutsu (v0.25.0).

Considerations

This section covers some of the facts you should keep in mind when working with paths in a version control system.

Glossary

Working directory: The root directory on your local machine containing the current version of your files that you can directly view and edit. Also known as "worktree" or "working copy".

Current directory: The directory a command is being run from, as returned by APIs like std::env::current_dir. For source control commands, this is often—but not always—the same as the working directory.

Merge base: The most recent common ancestor of two commits in version control history. Generally unique, but see the discussion on criss-cross merges below.

DAG: Short for directed acyclic graph: a graph where the edges have arrows, and you can't traverse the graph from any node and get back to itself. Version control histories, and hash-based graphs (i.e. Merkle graphs) in general, are always DAGs.

Null-separated paths

Paths as produced by tools like git diff --name-only are typically separated by newline characters (\n). However, on Unix platforms filenames can themselves have newlines embedded in them, so source control systems often end up quoting newlines and other "unusual" characters.

A robust, lossless way to retrieve a list of paths is by separating them with null characters (i.e. NUL or \0: a byte with a value of zero). Both Unix and Windows guarantee that a path can never have embedded null characters. This crate is geared towards supporting these kinds of outputs. This means that this doesn't handle any kind of escaping or quoting, instead assuming that the paths are already correctly formatted.

Both Git and Mercurial support null-separated paths. Jujutsu as of version 0.25.0 does not, but will hopefully support them in the future. (With current versions, commands like jj file list will return paths separated by newlines and will not perform any kind of quoting or escaping on them, so this crate is compatible with most real-world paths.)

Windows path separators

Unix platforms always use forward slashes (/) as the path separator. Windows platforms use \ as the main separator, but also supports / as an alternative in most situations.

On Windows, Git always returns file paths with forward slashes. (This is related to Git's history as a Unix-first tool.) Mercurial and Jujutsu, both being native Windows utilities, always return paths with backslashes.

Independently of the system in use, based on your requirements, you might prefer:

  • Paths with forward slashes (Unix-style or Git-style) on all platforms.
  • Paths using the native separator on each platform.

This crate provides a way to convert any of these representations to one of your choosing.

Untracked files in Git

Several of the Git recipes say that you need to obtain the list of untracked files, and combine it with the list obtained from a command. Note that the lists can have duplicate entries in some situations. (For example, a file was marked removed, then re-created as untracked.)

So be sure to use a set union operation on the files, not just a list concatenation.

Current directory

In all of the recipes below, paths are returned with respect to the current directory. In most cases, you will want to run the below commands from the root of the repository.

Recipes

Here's a set of recipes for how to make such paths in Git, Mercurial, and Jujutsu.

First, you need to decide:

  1. What query do you want to perform? In general, you may either want to list all files in the working directory or in a revision, or to list changed files by performing a diff of some kind.

  2. If you're listing changed files, what are you diffing from and to? There are three possible answers here:

    1. Between two revisions.
    2. From the merge base of two revisions, to a particular revision.
    3. Between a revision and the working directory.

    In general, if you're concerned with what changed in a branch, you'll want to use the merge base. If you're concerned with what changed in a commit, you'll want to diff a commit against its parent(s).

  3. If the working directory is part of what you're doing, do you only want to consider tracked files, or do you want to include untracked files as well? (In general, most users will exclude ignored files.)

    Untracked files matter most for local workflows. Including untracked files makes sense if you think of them as files that will soon be committed. But many users leave untracked files in their repositories for long periods of time, making this a weaker heuristic. There is unfortunately not a clear answer here, and the right answer will probably depend on the specifics of the problem and users' habits.

    In CI and automation, you're unlikely to see untracked files. Also, Jujutsu does not have a notion of untracked files.

  4. Do you need to filter by directory? Most commands here let you filter to a directory by passing in -- path/to/directory to the command.

    However, doing so might cause paths to be returned relative to the directory rather than relative to the root of the repo. You're encouraged to test commands out against your own repositories to see how this works.

(This is all a bit complicated! Feel free to take a break at this point if this is too much.)

Commands to run

Here's how this breaks down:

List all files in the working directory, excluding untracked files:

  • Git: git ls-files -z produces null-separated paths with forward slashes.
  • Mercurial: hg files --print0 produces null-separated paths with the native separator.
  • Jujutsu: jj file list produces newline-separated paths with the native separator.

List all files in the working directory, including untracked files:

  • Git: git ls-files -z --cached --others --exclude-standard produces null-separated paths with forward slashes.
  • Mercurial: hg status --print0 -mardu --no-status produces null-separated paths with the native separator.
  • Jujutsu: Jujutsu does not have a notion of untracked files.

List all files present in revision $REV:

  • Git: git ls-tree -z -r --name-only $REV produces null-separated paths with forward slashes.
  • Mercurial: hg status --print0 --no-status --rev null --rev $REV produces null-separated paths with the native separator.
  • Jujutsu: jj file list --revision $REV produces newline-separated paths with the native separator.

List changed files from a revision $REV to the working directory, excluding untracked files:

  • Git: git diff-index -m -z --name-only $REV produces null-separated paths with forward slashes.
  • Mercurial: hg status --no-status --print0 --rev $REV produces null-separated paths with the native separator.
  • Jujutsu: jj diff --from $REV --name-only produces newline-separated paths with the native separator.

List changed files from a revision $REV to the working directory, including untracked files:

  • Git: There isn't a single command to do this. Instead, first obtain the diff excluding untracked files, then combine with the list of untracked files obtained from git ls-files -z --others --exclude-standard.
  • Mercurial: hg status --no-status --print0 -mardu --rev $REV produces null-separated paths with the native separator.
  • Jujutsu: Jujutsu does not have a notion of untracked files.

List changed files from the merge base of a revision $REV to the working directory, excluding untracked files:

  • Git: git diff-index -m -z --merge-base --name-only $REV produces null-separated paths with forward slashes.
  • Mercurial: hg status --no-status --print0 --rev 'ancestor($REV,.)' produces null-separated paths with the native separator.
  • Jujutsu: jj diff --from 'fork_point(@ | $REV)' --name-only produces newline-separated paths with the native separator.

List changed files from the merge base of a revision $REV to the working directory, including untracked files:

  • Git: There isn't a single command to do this. Instead, first obtain the diff excluding untracked files, then combine with the list of untracked files obtained from git ls-files -z --others --exclude-standard.
  • Mercurial: hg status --no-status --print0 -mardu --rev 'ancestor($REV,.)' produces null-separated paths with the native separator.
  • Jujutsu: Jujutsu does not have a notion of untracked files.

List changed files in a revision compared to its parent(s):

  • Git: git diff-tree -z --name-only $REV produces null-separated paths with forward slashes.
  • Mercurial: hg status --no-status --print0 --change $REV produces null-separated paths with the native separator.
  • Jujutsu: jj diff --revision $REV --name-only produces newline-separated paths with the native separator.

List changed files from $OLD_REV to $NEW_REV:

  • Git: git diff-tree -z --name-only $OLD_REV $NEW_REV produces null-separated paths with forward slashes.
  • Mercurial: hg status --no-status --print0 --rev $OLD_REV --rev $NEW_REV produces null-separated paths with the native separator.
  • Jujutsu: jj diff --from $OLD_REV --to $NEW_REV --name-only produces newline-separated paths with the native separator.

List changed files from the merge base of $OLD_REV and $NEW_REV, to $NEW_REV:

  • Git: git diff-tree -z --name-only --merge-base $OLD_REV $NEW_REV produces null-separated paths with forward slashes.
  • Mercurial: hg status --no-status --print0 --rev 'ancestor($OLD_REV,$NEW_REV)' --rev $NEW_REV produces null-separated paths with the native separator.
  • Jujutsu: jj diff --from 'fork_point($OLD_REV | $NEW_REV)' --to $NEW_REV --name-only produces newline-separated paths with the native separator.

Advanced details

This section covers other details that some users need to be aware of.

In all of the below DAGs:

  • time flows to the right
  • solid arrows go from child commits to parent ones
  • dashed arrows go from branch pointers to commits
  • A, B, C etc represent published commits on the main branch
  • Pn represents in-progress commits

The discussion below applies to arbitrary DAGs, and does not depend on source control specifics like branch pointers, published work, and/or in-progress commits. The discussion uses these terms purely to help build an intuition.

Criss-cross merges

In the above commands, we've generally assumed that two commits have a unique merge base. This is always true in linear histories (repositories without merges).

graph RL;
    subgraph published
    main -.-> C
    C --> B
    B --> A
    end
    subgraph in-progress
    wip -.-> P1
    P1 --> A
    end
Loading

In this case, the merge base of C and P1 is A.

Merge bases are also unique if merges happen in one direction. This is commonly seen if the history of main is linear, but developers use merges locally.

graph RL;
    subgraph published
    main -.-> C
    C --> B
    B --> A
    end
    subgraph in-progress
    wip -.-> P3
    P3 --> P2
    P2 --> B
    P2 --> P1
    P1 --> A
    end
Loading

In this case, the merge base of P3 and C is B.

But if merges go in both directions, merge bases might not be unique. This situation is called a criss-cross merge.

graph RL;
    subgraph published
    main
    C
    B
    A
    end
    subgraph in-progress
    wip
    P3
    P2
    P1
    end

    main -.-> C
    C --> B
    B --> A
    C --> P1

    wip -.-> P3
    P3 --> P2
    P2 --> B
    P2 --> P1
    P1 --> A
Loading

In this case, there is a merge from published to in-progress work, as well as one from in-progress work to published commits. Here, C and P3 have two merge bases: B and P1.

This is a very real problem! Statements like "list all files changed in the branch", which are meaningful with a unique merge base, become much harder to interpret.

There are a number of ways to resolve criss-cross merges, but all of them have upsides and downsides.

The behavior of the above commands with criss-cross merges varies.

  • With Git and Jujutsu, the commands above fail.
  • With Mercurial, one of the merge bases is arbitrarily picked.

In general, you will have an easier time if your repositories don't have criss-cross merges. The easiest way to systematically prevent the existence of criss-cross merges is to maintain a linear history on main: if merges are one-directional, criss-cross merges will never happen. Users might still create criss-cross merges locally, but in the writer's extensive experience deploying source control to tens of thousands of users over many years, users almost never create criss-cross merges within their own work.

(This is also the strongest technical reason why for published branches, linear histories are better than merges. The lack of criss-cross merges makes a number of source control algorithms function better.)

Submodules

The above commands don't pay much attention to Git submodules. The author believes that submodules are a feature of last resort that should be avoided as much as possible. If you need to support submodules, you're encouraged to do your own research.

References

Git:

Mercurial:

Jujutsu:

Clone this wiki locally