automatic conflict resolution

[problame]

zrepl should be able to resolve replication conflicts automatically.

Background / Use Cases

Replication conflicts are situations where there is something on the receiving side that prevents replication of one or more file systems from the sender to the receiver.
Examples and use cases:

• receiver-only snapshots
  • zfs snapshot -r receivingpool@pre_upgrade_snapshot
  • forgotten exception in auto-snaphotting tool on receive side
  • ...
• unsnapshotted modifications on the receiver => recv fails with characteristic error; cc STEP-ERROR: destination has been modified since most recent snapshot #405
• renamed and/or re-created file systems on the sender => looks like complete divergence to replication algorithm; cc Handle filesystem destroy / rename #70

UX

• zrepl should continue to opt for non-destructive-by-default behavior
• Renames should be detected automatically, e.g. by tracking GUIDs that are invariant to replication cc Handle filesystem destroy / rename #70.
  Renames are thus not actually part of the conflict resolution UX.
• With the daemon, no user interaction is required for conflict resolution.
• Instead, a conflict resolution policy defined in the config is applied automatically to all filesystems matched by a job.
• If the conflict resolution gives up, the user is informed via logs, metrics and zrepl status.
• Optional The zrepl signal (or similar) command allows the user to override the conflict resolution policy on a per-filesystem basis for the next replication attempt.

Config Syntax:

type: push # active side
...
conflict_resolution:

  # what to do if the receiver diverged and we can't replicate:
  receiver_diverged:

    # if there are (only) local modifications but no snapshots:
    unsnapshotted_modifications:
      # Policy Option #1: fail filesystems affected by this
      policy:
        type: fail
      # Policy Option #2: rollback filesystem to latest snapshot
      policy:
        type: rollback_to_latest_snapshot
        max_written: 10MiB # max value for the `written` zfs property of the dataset (fail otherwise)

    # if there are snapshots on the receiver that prevent replication
    receiver_only_snapshots:
      # Policy Option #1: fail filesystems affected by this
      policy:
        type: fail

      # Policy Option #2: destroy the receiver-only snapshots that prevent replication
      policy:
        type: destroy
        max_time: 1d # the max time window that is allowed to be rolled
        max_count: 10 # the max number of snapshots that are allowed to be rolled back. -1 for inifinity, 0 is invalid
        snapshots: # regex that match the snapshot names (NOTE: syntax should be the same as for snapshot filtering)
          - regex: "^zrepl_.*"
            action: rollback
          - regex: ".*"
            action: prohibit

      # Policy Option #3: rename the conflicting dataset and start over using initial replication
      policy:
        type: conflict_copy
        clone: # whether to try cloning instead of initial replication if there is still a common snapshot between sender and receiver

          # Policy Option #1: don't clone
          type: no

          # Policy Option #2: try it within some limits, otherwise
          type: try
          max_time: 1d # the max time window for which we should consider cloning
          max_count: 10 # the max number of snapshots that we should go back before giving up on trying to clone
          max_written: 1GiB # max value for the written@ property between the clone origin candidate
          otherwise: fail | copy # copy does full initial replication

Security Considerations

• Future use cases such as append-only sink cc @mdtancsa
  • What could an attacker accomplish if they take over the sender but not the receiver? Our goal should be that the conflict resolution can be configured such that the attacker cannot provoke deletion of snapshots through the conflict resolution policy.

Implementation

• Open question, let's discuss this after we have settled on features.

[problame]

Additional case

type: push # active side
...
conflict_resolution:
  receiver_diverged:
    # if there are no common snapshots between sender and receiver
    no_common_snapshots:
      # Option 1: fial
      policy:
        type: fail
      # Option 2
      policy:
        type: conflict_copy
        ...
      # Option 3
      policy:
        type: destroy #perhaps 'replace' would be a better name?

[problame]

Additional use case

type: push
...
conflict_resultion:
  # what should happen if there is no corresponding receive-side file system
  initial_replication:
    # Option 1: fail
    # Option 2: send most recent snapshot (current behavior)
    # Option 3: send all snapshots on the sender