On jj run

[arxanas]

I've been using git-branchless's git test for some time now and thought I'd provide notes on the workflow in practice.

Terminology note: when rebasing B on top of A, we reapply the logical patch induced by B on top of A. There's another relevant operation when it comes to fixing commits where we reapply B on top of A by simply reusing B's tree, without any attempt to reapply the patch. I'll call this A "adopting" its old (orphaned) descendant B, rather than rebasing its descendants.

There are a few common modes for git test run:

• Run test(s) on some number of commits. The commits shouldn't be modified.
  • In some cases, we use git test run simply to "sandbox" a command run:
    • It should be run in the background on an alternate worktree, because you don't want to touch the working copy you're developing with.
    • Resulting changes to the working directory shouldn't be reflected or applied in the parent. (This can happen for some cases of snapshot tests, or simply unfortunate/buggy test implementation 😬.)
• Fix some number of commits with a command that writes to the working copy. There are two main implementation ideas:
  • Use a command that runs on the entire working copy (generally only feasible on smaller repositories).
    • This lets us run the command in parallel on each commit and collect the resulting working copy trees, then replace the original commit trees and adopt descendants. For the user, it's reasonably simple to implement and reason about (i.e. don't need to configure a formatter to run only on a subset of files).
  • Use a command that runs on only changes in the current commit.
    • The natural implementation doesn't work with adoption (because descendant commits will discard the changes of their adoptive ancestors), and therefore isn't well parallelizable.
    • As discussed on Discord, to solve the adoption problem, hg fix will run fixes on the set of all files changes since the roots of the commit set being processed. This means that each commit being processed performs duplicate work for files which were changed in an ancestor, but, in exchange, the command processing can be parallelized. This approach is sensible and almost certainly efficient in practice, but requires additional complexity or concessions if caching is desired.
    • User commands have to be able to run on a subset of files. hg fix actually requires that user commands run on the textual contents of files; as I understand it, hg fix won't materialize the commit working copies at all. Obviously, this requires significant integration with the tool in question — for example, many formatters are not equipped to process files without a working copy — but you can get significant speedups if implemented correctly.

And then some less-common modes, but still worth considering:

• Run a command on a commit for side-effects only, such as to push a branch or create a pull request. (git test has a --no-cache option to decline both reading and writing to the cache.)
• Find (and optionally fix) the first commit. Sometimes, a check is expensive, and the user wants to minimize the number of executions of their command.
  • Example: an expensive integration test.
  • Example: on with Github non-stacked PRs, I sometimes have lint failures introduced somewhere in the stack, but don't know specifically which commit in the PR introduced them.
  • To minimize the number of checks, binary search on the commit set is often better than linear search.
  • If the command has a fix, then we usually don't want to adopt descendants, because we don't want to run the expensive command on all descendants. Instead, we want to apply the fix and rebase descendants. git test actually doesn't (currently) have a mode to fix a commit without adopting, which can cause mistakes if you're not careful.
• Modify commit messages. Occasionally, metadata is stored in the commit message and we want to consume it and possibly update the commit message.
  • Phabricator's arc tool famously does this to embed a change ID in the commit message immediately prior to uploading.
  • At my work, you can put a list of test commands in the commit message, and then run a command which will run the tests and check off any successful tests in the commit message.
• The same command setup and organization could be useful for "merge drivers" (tools which can automatically resolve conflicts).

Other notes:

• Checking and fixing oftentimes come in pairs. There are both cases where you want to check without (yet) fixing and where you want to fix.
  • Sometimes you know that the fixes may not be safe to automatically apply and you want to manually inspect any fixes.
  • Some fixes aren't intended to be sound, i.e. they guess on how to fix the problem but the result may not actually work. It's oftentimes nice to use the fix as a "starting point" for your own fix on top of that. (For example, swapping a deprecated method with its new version might not compile due to some signature change, but you'd still rather have automation apply that than do it yourself.)
  • Sometimes fixing takes longer than checking, and you'd rather manually fix some commits because it would be faster (especially if fixing requires re-running on descendant commits and adopting them).
  • Sometimes you don't want to touch the tree contents because it'll invalidate some caches or trigger a rebuild in CI at an inopportune time.
  • Nonetheless, logically speaking, it would be nice for the user to consider the two separate checking and fixing modes as one command. That is, I don't like having to use separate -c arguments for git test run -c lint vs git test fix -c lint-fix.
• Caching is really useful.
  • The workflow changes psychologically so that you feel free to constantly run git test without worrying about wasting time.
  • I might have had a false positive at some point (where the result of a cached test run is no longer valid due to some external change), but I don't remember any, and it's fairly obvious when caching occurred because the output says (cached) and completes instantly.
• git test has separate run and fix subcommands because it wasn't clear how much the use-cases would overlap. In my experience so far, there are definitely cases where you don't want to fix using the results, but I think using the same command (with an optional --fix argument) makes sense. The main impedance mismatch is that you often want to specify different underlying commands for the "check" and "fix" modes of a logical command.
  • Currently, the only difference in supported options is that git test fix doesn't support --search <strategy>. I think there's even a reasonable interpretation of that argument, in which you try to find the first failing commit and then fix it and rebase (not adopt) descendants, as discussed above.
• git-branchless also has revset support for tests, so that you could say something like git submit 'stack() - tests.failed(). The most useful thing in practice is switching to the earliest/latest commit that failed tests (something like git sw 'roots(descendants(stack() & tests.failed()))... definitely worth aliasing if you're going to use that). That being said, I don't think revset support is a priority, particularly if we can directly enable some workflows like finding and fixing the first failing commit.
• It might be sensible to develop jj run and other more experimental commands (jj sync?) as a separate command which uses jututsu-lib. This would let us test Jujutsu's API and mimic how some organizations will probably create their own commands. (Conceivably, we could even change git-branchless to support jj as a backend to execute its utility commands like git test and git sync...)

[martinvonz]

For reference, the design doc for jj run is here (thanks, @PhilipMetzger, for writing that).

[PhilipMetzger]

First sorry for the late response, as I've tried multiple times to write a reply but always failed to express myself. So here it goes.

Run test(s) on some number of commits. The commits shouldn't be modified.

• In some cases, we use git test run simply to "sandbox" a command run:
  • It should be run in the background on an alternate worktree, because you don't want to touch the working copy you're developing with.
  • Resulting changes to the working directory shouldn't be reflected or applied in the parent. (This can happen for some cases of snapshot tests, or simply unfortunate/buggy test implementation 😬.)
    Fix some number of commits with a command that writes to the working copy. There are two main implementation ideas:
• Use a command that runs on the entire working copy (generally only feasible on smaller repositories).
  • This lets us run the command in parallel on each commit and collect the resulting working copy trees, then replace the original commit trees and adopt descendants. For the user, it's reasonably simple to implement and reason about (i.e. don't need to configure a formatter to run only on a subset of files).
    Use a command that runs on only changes in the current commit.
  • The natural implementation doesn't work with adoption (because descendant commits will discard the changes of their adoptive ancestors), and therefore isn't well parallelizable.
  • As discussed on Discord, to solve the adoption problem, hg fix will run fixes on the set of all files changes since the roots of the commit set being processed. This means that each commit being processed performs duplicate work for files which were changed in an ancestor, but, in exchange, the command processing can be parallelized. This approach is sensible and almost certainly efficient in practice, but requires additional complexity or concessions if caching is desired.
  • User commands have to be able to run on a subset of files. hg fix actually requires that user commands run on the textual contents of files; as I understand it, hg fix won't materialize the commit working copies at all. Obviously, this requires significant integration with the tool in question — for example, many formatters are not equipped to process files without a working copy — but you can get significant speedups if implemented correctly.

The current approach which we'll use is documented in the design doc. I also recently thought of adding an option like --files to the command, if the user wants to have a subset of the repository before executing run, but this can wait until we have users with this requirement.

Run a command on a commit for side-effects only, such as to push a branch or create a pull request. (git test has a --no-cache option to decline both reading and writing to the cache.)
Modify commit messages. Occasionally, metadata is stored in the commit message and we want to consume it and possibly update the commit message.

• Phabricator's arc tool famously does this to embed a change ID in the commit message immediately prior to uploading.
• At my work, you can put a list of test commands in the commit message, and then run a command which will run the tests and check
  off any successful tests in the commit message.

The same command setup and organization could be useful for "merge drivers" (tools which can automatically resolve conflicts).

This is planned and will be supported as soon as run understands the Workspace of a repository.

And supporting special footers, e.g TEST=//my/target and PLATFORMS=x86,ARM64 should probably be a backend/server responsibility. But run could be invoked in some backends when these appear.

Find (and optionally fix) the first commit. Sometimes, a check is expensive, and the user wants to minimize the number of executions of their command.

• Example: an expensive integration test.
• Example: on with Github non-stacked PRs, I sometimes have lint failures introduced somewhere in the stack, but don't know specifically which commit in the PR introduced them.
• To minimize the number of checks, binary search on the commit set is often better than linear search.
• If the command has a fix, then we usually don't want to adopt descendants, because we don't want to run the expensive command on all descendants. Instead, we want to apply the fix and rebase descendants. git test actually doesn't (currently) have a mode to fix a commit without adopting, which can cause mistakes if you're not careful.

This is a really interesting case to support, so I think this could be added to run in the future.

git test has separate run and fix subcommands because it wasn't clear how much the use-cases would overlap. In my experience so far, there are definitely cases where you don't want to fix using the results, but I think using the same command (with an optional --fix argument) makes sense. The main impedance mismatch is that you often want to specify different underlying commands for the "check" and "fix" modes of a logical command.

As noted in the design doc, I'm not really a fan of adding --fix to run as it may stomp on the design of other commands, e.g fix and format which build on top of run. And I think jj run 'cargo clippy --fix' --fix is a bit repetitive. But as you previously brought up, checking and fixing comes in pairs and I'd expect modern tooling (linters and co) to have a fix subcommand builtin. I'm open to change my opinion on it though.

So what does jj run $command --fix bring, which wasn't mentioned in the doc?

Caching is really useful.
The workflow changes psychologically so that you feel free to constantly run git test without worrying about wasting time.
I might have had a false positive at some point (where the result of a cached test run is no longer valid due to some external change), but I don't remember any, and it's fairly obvious when caching occurred because the output says (cached) and completes instantly.

Caching is planned and hopefully makes run as cheap as possible.

git-branchless also has revset support for tests, so that you could say something like git submit 'stack() - tests.failed(). The most useful thing in practice is switching to the earliest/latest commit that failed tests (something like git sw 'roots(descendants(stack() & tests.failed()))... definitely worth aliasing if you're going to use that). That being said, I don't think revset support is a priority, particularly if we can directly enable some workflows like finding and fixing the first failing commit.

I think this is another feature request for Jujutsu, as no current VCS has a builtin understanding of tests. But I like the idea.

It might be sensible to develop jj run and other more experimental commands (jj sync?) as a separate command which uses jututsu-lib. This would let us test Jujutsu's API and mimic how some organizations will probably create their own commands. (Conceivably, we could even change git-branchless to support jj as a backend to execute its utility commands like git test and git sync...)

I agree, as every backend benefits from run but we'll need some new traits in jujutsu-lib to abstract over the different virtual filesystems. Something similar to LLVM's VFS will be useful.

[arxanas]

So what does jj run $command --fix bring, which wasn't mentioned in the doc?

My main concern is that there's no way to run jj run without applying changes (as far as I can tell?), and that perhaps the default of applying changes is dangerous.

As noted in the design doc, I'm not really a fan of adding --fix to run as it may stomp on the design of other commands

I also thought this, but, surprisingly, every option in git-branchless-test works when combined with --fix so far. Now I'm wondering what options wouldn't compose well with --fix. I also don't think git test run needs to be a separate command from git test fix (but jj run already is only one command, not two, unless we plan a separate jj fix?).

[PhilipMetzger]

My main concern is that there's no way to run jj run without applying changes (as far as I can tell?), and that perhaps the default of applying changes is dangerous.

That's a great point. Although we did not specify it in the design doc, it should be possible to run a command without applying changes.

I also thought this, but, surprisingly, every option in git-branchless-test works when combined with --fix so far. Now I'm wondering what options wouldn't compose well with --fix. I also don't think git test run needs to be a separate command from git test fix (but jj run already is only one command, not two, unless we plan a separate jj fix?).

I think we should add something like fix and format as they only operate on a patch's content and not the whole repository. But nothing is planned yet.

[arxanas]

The git-branchless progress indicator shows a linear list of commits being processed, but it would be nice if it modeled the graph structure instead. It's often difficult for me to tell if a specific branch/stack has been fully processed yet.