copy-info: initial design doc

docs/design/copy-tracking.md

@@ -0,0 +1,294 @@
+# Copy Tracking and Tracing Design
+
+Authors: [Daniel Ploch](mailto:dploch@google.com)
+
+**Summary:** This Document documents an approach to tracking and detecting copy
+information in jj repos, in a way that is compatible with both Git's detection
+model and with custom backends that have more complicated tracking of copy
+information. This design affects the output of diff commands as well as the
+results of rebasing across remote copies.
+
+## Objective
+
+Implement extensible APIs for recording and retrieving copy info for the
+purposes of diffing and rebasing across renames and copies more accurately.
+This should be performant both for Git, which synthesizes copy info on the fly
+between arbitrary trees, and for custom extensions which may explicitly record
+and re-serve copy info over arbitrarily large commit ranges.
+
+The APIs should be defined in a way that makes it easy for custom backends to
+ignore copy info entirely until they are ready to implement it.
+
+## Interface Design
+
+### Read API
+
+Copy information will be served both by a new Backend trait method described
+below, as well as a new field on Commit objects for backends that support copy
+tracking:
+
+```rust
+/// An individual copy source.
+pub struct CopySource {
+    /// The source path a target was copied from.
+    ///
+    /// It is not required that the source path is different than the target
+    /// path. A custom backend may choose to represent 'rollbacks' as copies
+    /// from a file unto itself, from a specific prior commit.
+    path: RepoPathBuf,
+    file: FileId,
+    /// The source commit the target was copied from. If not specified, then the
+    /// parent of the target commit is the source commit. Backends may use this
+    /// field to implement 'integration' logic, where a source may be
+    /// periodically merged into a target, similar to a branch, but the
+    /// branching occurs at the file level rather than the repository level. It
+    /// also follows naturally that any copy source targeted to a specific
+    /// commit should avoid copy propagation on rebasing, which is desirable
+    /// for 'fork' style copies.
+    ///
+    /// If specified, it is required that the commit id is an ancestor of the
+    /// commit with which this copy source is associated.
+    commit: Option<CommitId>,
+}
+
+pub enum CopySources {
+    Resolved(CopySource),
+    Conflict(HashSet<CopySource>),
+}
+
+/// An individual copy event, from file A -> B.
+pub struct CopyRecord {
+    /// The destination of the copy, B.
+    target: RepoPathBuf,
+    /// The CommitId where the copy took place.
+    id: CommitId,
+    /// The source of the copy, A.
+    sources: CopySources,
+}
+
+/// Backend options for fetching copy records.
+pub struct CopyRecordOpts {
+    // TODO: Probably something for git similarity detection
+}
+
+pub type CopyRecordStream = Pin<Box<dyn Stream<Item = BackendResult<CopyRecord>>>>;
+
+pub trait Backend {
+    /// Get all copy records for `paths` in the dag range `roots..heads`.
+    /// 
+    /// The exact order these are returned is unspecified, but it is guaranteed
+    /// to be reverse-topological. That is, for any two copy records with
+    /// different commit ids A and B, if A is an ancestor of B, A is streamed
+    /// after B.
+    /// 
+    /// Streaming by design to better support large backends which may have very
+    /// large single-file histories. This also allows more iterative algorithms
+    /// like blame/annotate to short-circuit after a point without wasting
+    /// unnecessary resources.
+    async fn get_copy_records(&self, paths: &[RepoPathBuf], roots: &[CommitId], heads: &[CommitId]) -> CopyRecordStream;
+}
+```
+
+Obtaining copy records for a single commit requires first computing the files
+list for that commit, then calling get_copy_records with `heads = [id]` and
+`roots = parents()`. This enables commands like `jj diff` to produce better
+diffs that take copy sources into account.
+
+### Write API
+
+Backends that support tracking copy records at the commit level will do so
+through a new field on `backend::Commit` objects:
+
+```rust
+pub struct Commit {
+    ...
+    copies: Option<HashMap<RepoPathBuf, CopySources>>,
+}
+
+pub trait Backend {
+    /// Whether this backend supports storing explicit copy records on write.
+    fn supports_copy_tracking(&self) -> bool;
+}
+```
+
+This field will be ignored by backends that do not support copy tracking, and
+always set to `None` when read from such backends. Backends that do support copy
+tracking are required to preserve the field value always.
+
+This API will enable the creation of new `jj` commands for recording copies:
+
+```shell
+jj cp $SRC $DEST [OPTIONS]
+jj mv $SRC $DEST [OPTIONS]
+```
+
+These commands will rewrite the target commit to reflect the given move/copy
+instructions in its tree, as well as recording the rewrites on the Commit
+object itself for backends that support it (for backends that do not,
+these copy records will be silently discarded).
+
+Flags for the first two commands will include:
+
+```
+-r/--revision
+    perform the copy or move at the specified revision
+    defaults to the working copy commit if unspecified
+-f
+    force overwrite the destination path
+--after
+    record the copy retroactively, without modifying the targeted commit tree
+--resolve
+    overwrite all previous copy intents for this $DEST
+--allow-ignore-copy
+    don't error if the backend doesn't support copy tracking
+--from REV
+    specify a commit id for the copy source that isn't the parent commit
+```
+
+For backends which do not support copy tracking, it will be an error to use
+`--after`, since this has no effect on anything and the user should know that.
+The `supports_copy_tracking()` trait method is used to determine this.
+
+An additional command is provided to deliberately discard copy info for a
+destination path, possibly as a means of resolving a conflict.
+
+```shell
+jj forget-cp $DEST [-r REV]
+```
+
+## Behavioral Changes
+
+### Rebase Changes
+
+In general, we want to support the following use cases:
+
+-   A rebase of an edited file A across a rename of A->B should transparently move the edits to B.
+-   A rebase of an edited file A across a copy from A->B should _optionally_ copy the edits to B. A configuration option should be defined to enable/disable this behavior.
+-   TODO: Others?
+
+Using the aforementioned copy tracing API, both of these should be feasible. A
+detailed approach to a specific edge case is detailed in the next section.
+
+#### Rename of an added file
+
+A well known and thorny problem in Mercurial occurs in the following scenario:
+
+1.  Create a new file A
+1.  Create new commits on top that make changes to file A
+1.  Whoops, I should rename file A to B. Do so, amend the first commit.
+1.  Because the first commit created file A, there is no rename to record; it's changing to a commit that instead creates file B.
+1.  All child commits get sad on evolve
+
+In jj, we have an opportunity to fix this because all rebasing occurs atomically
+and transactionally within memory. The exact implementation of this is yet to be
+determined, but conceptually the following should produce desirable results:
+
+1.  Rebase commit A from parents [B] to parents [C]
+1.  Get copy records from [D]->[B] and [D]->[C], where [D] are the common ancestors of [B] and [C]
+1.  DescendantRebaser maintains an in-memory map of commits to extra copy info, which it may inject into (2). When squashing a rename of a newly created file into the commit that creates that file, DescendentRebase will return this rename for all rebases of descendants of the newly modified commit. The rename lives ephemerally in memory and has no persistence after the rebase completes.
+1.  A to-be-determined algorithm diffs the copy records between [D]->[B] and [D]->[C] in order to make changes to the rebased commit. This results in edits to renamed files being propagated to those renamed files, and avoiding conflicts on the deletion of their sources. A copy/move may also be undone in this way; abandoning a commit which renames A->B should move all descendant edits of B back into A.
+
+### Conflicts
+
+With copy-tracking, a whole new class of conflicts become possible. These need
+to be well-defined and have well documented resolution paths. Because copy info
+in a commit is keyed by _destination_, conflicts can only occur at the
+_destination_ of a copy, not at a source (that's called forking).
+
+#### Split conflicts
+
+Suppose we create commit A by renaming file F1 -> F2, then we split A. What
+happens to the copy info? I argue that this is straightforward:
+
+-   If F2 is preserved at all in the parent commit, the copy info stays on the parent commit.
+-   Otherwise, the copy info goes onto the child commit.
+
+Things get a little messier if A _also_ modifies F1, and this modification is
+separated from the copy, but I think this is messy only in an academic sense and
+the user gets a sane result either way. If they want to separate the
+modification from the copy while still putting it in an earlier commit, they can
+express this intent after with `jj cp --after --from`.
+
+#### Merge commit conflicts
+
+Suppose we create commit A by renaming file F1 -> F, then we create a sibling
+commit B by renaming file F2 -> F. What happens when we create a merge commit
+with parents A and B?
+
+In terms of _copy info_ there is no conflict here, because C does not have copy
+info and needs none, but resolving the contents of F becomes more complicated.
+We need to (1) identify the greatest common ancestor of A and B (D)
+(which we do anyway), and (2) invoke `get_copy_records()` on F for each of
+`D::A` and `D::B` to identify the 'real' source file id for each parent. If
+these are the same, then we can use that as the base for a better 3-way merge.
+Otherwise, we must treat it as an add+add conflict where the base is the empty
+file id.
+
+It is possible that F1 and F2 both came from a common source file G, but that
+these copies precede D. In such case, we will not produce as good of a merge
+resolution as we theoretically could, but (1) this seems extremely niche and
+unlikely, and (2) we cannot reasonably achieve this without implementing some
+analogue of Mercurial's linknodes concept, and it would be nice to avoid that
+additional complexity.
+
+#### Squash conflicts
+
+Suppose we create commit A by renaming file F1 -> F, then we create child
+commit B in which we replace F by renaming F2 -> F. This touches on two issues.
+
+Firstly, if these two commits are squashed together, then we have a destination
+F with two copy sources, F1 and F2. In this case, we can store a
+`CopySources::Conflict([F1, F2])` as the copy source for F, and treat this
+commit as 'conflicted' in `jj log`. `jj status` will need modification to show
+this conflicted state, and `jj resolve` will need some way of handling the
+conflicted copy sources (possibly printing them in some structured text form,
+and using the user's merge tool to resolve them). Alternatively, the user can
+'resolve directly' by running `jj cp --after --resolve` with the desired copy
+info.
+
+Secondly, who is to say that commit B is 'replacing' F at all? In some version
+control systems, it is possible to 'integrate' a file X into an existing file Y,
+by e.g. propagating changes in X since its previous 'integrate' into Y, without
+erasing Y's prior history in that moment for the purpose of archaeology. With
+the commit metadata currently defined, it is not possible to distinguish
+between a 'replacement' operation and an 'integrate' operation.
+
+##### Track replacements explicitly
+
+One solution is to add a `destructive: bool` field or similar to the
+`CopySource` struct, to explicitly distinguish between these two types of copy
+records. It then becomes possible to record a non-destructive copy using
+`--after` to recognize that a file F was 'merged into' its destination, which
+can be useful in handling parallel edits of F that later sync this information.
+
+##### Always assume replacement
+
+Alternatively, we can keep copy-tracking simple in jj by taking a stronger
+stance here and treating all copies-onto-existing-files as 'replacement'
+operations. This makes integrations with more complex VCSs that do support
+'integrate'-style operations trickier, but it is possible that a more generic
+commit extension system is better suited to such backends.
+
+### Future Changes
+
+An implementation of `jj blame` or `jj annotate` does not currently exist, but
+when it does we'll definitely want it to be copy-tracing aware to provide
+better annotations for users doing archaeology. The Read APIs provided are
+expected to be sufficient for these use cases.
+
+## Non-goals
+
+### Tracking copies in Git
+
+Git uses rename detection rather than copy tracking, generating copy info on
+the fly between two arbitrary trees. It does not have any place for explicit
+copy info that _exchanges_ with other users of the same git repo, so any
+enhancements jj adds here would be local only and could potentially introduce
+confusion when collaborating with other users.
+
+### Directory copies/moves
+
+All copy/move information will be read and written at the file level. While
+`jj cp|mv` may accept directory paths as a convenience and perform the
+appropriate tree modification operations, the renames will be recorded at the
+file level, one for each copied/moved file.