Add target-diff

[JSCU-CNI]

This PR adds the command target-diff, which can be used to compare two or more targets against one another:

$ target-diff --help

target-diff

positional arguments:
  {shell,fs,query}      Mode for differentiating targets
    shell               Open an interactive shell to compare two or more targets.
    fs                  Yield records about differences between target filesystems.
    query               Differentiate plugin outputs between two or more targets.

options:
  -d, --deep            Compare file contents even if metadata suggests they have been left unchanged (default: False)
  -l LIMIT, --limit LIMIT
                        How many bytes to compare before assuming a file is left unchanged (0 for no limit) (default:
                        32768)

fs mode outputs records denoting filesystem changes from one target to the other:

$ target-diff --deep fs src.tar dst.tar

<differential/file/created hostname=None domain=None src_target='src.tar' dst_target='dst.tar' path='/changes/only_on_dst'>
<differential/file/deleted hostname=None domain=None src_target='src.tar' dst_target='dst.tar' path='/changes/only_on_src'>
<differential/file/modified hostname=None domain=None src_target='src.tar' dst_target='dst.tar' path='/changes/changed' diff=[b'--- \n', b'+++ \n', b'@@ -1 +1 @@\n', b'-SRC', b'+DST']>

Using query mode, you can compare plugin outputs from one target to the other:

$ target-diff query -f users src.tar dst.tar

<differential/record/unchanged hostname=None domain=None src_target='src.tar' dst_target='dst.tar' record=<unix/user hostname='dst_target' domain=None name='root' passwd='x' uid=0 gid=0 gecos='root' home='/root' shell='/bin/bash' source='/etc/passwd'>>
<differential/record/unchanged hostname=None domain=None src_target='src.tar' dst_target='dst.tar' record=<unix/user hostname='dst_target' domain=None name='user' passwd='x' uid=1000 gid=1000 gecos='user' home='/home/user' shell='/bin/bash' source='/etc/passwd'>>
<differential/record/created hostname=None domain=None src_target='src.tar' dst_target='dst.tar' record=<unix/user hostname='dst_target' domain=None name='dst_user' passwd='x' uid=1001 gid=1001 gecos='dst_user' home='/home/dst_user' shell='/bin/bash' source='/etc/passwd'>>
<differential/record/deleted hostname=None domain=None src_target='src.tar' dst_target='dst.tar' record=<unix/user hostname='src_target' domain=None name='src_user' passwd='x' uid=1001 gid=1001 gecos='src_user' home='/home/src_user' shell='/bin/bash' source='/etc/passwd'>>

In shell mode, you can browse the target filesystems like in target-shell, where directory listings will show which files / directories have been changed, added or deleted. Using the plugin command, plugin outputs can be compared from within the shell context.

$ target-diff shell src.tar dst.tar

(dst_target/src_target)/diff />help

Target Diff
==========


Documented commands (type help <topic>):
=================================================================
cat  clear  diff   exit  help  ls    plugin  previous  set
cd   cyber  enter  find  list  next  prev    python  

(dst_target/src_target)/diff />cd changes
(dst_target/src_target)/diff /changes>ls
changed
only_on_dst (deleted)
only_on_src (created)
subdirectory_both
subdirectory_dst (deleted)
subdirectory_src (created)
unchanged

target-diff depends on fox-it/flow.record#107. To allow tests to run for this PR we've temporarily bumped flow.record to 3.15.dev10 in pyproject.toml

When three or more targets are provided, you can choose between treating every target as a 'delta' or compare every target against one 'absolute' target. Treating targets as 'deltas' is useful if you have multiple snapshots of the same target from different points in time. Treating targets as 'absolutes' can be useful in situations where you have a 'golden image' that you want to compare different targets against.

To keep code duplication low between tools/diff.py and tools/shell.py, this PR adds a superclass ExtendedCmd to shell.py that contains most of the functionality that is shared between the two. Both TargetCmd and DifferentialCli inherit from this class.

[yunzheng]

I did not review the whole PR, just added some small flow.record quality of life suggestions since the merge of fox-it/flow.record#115

Available in flow.record==3.15.

[yunzheng]

Suggested change

	"flow.record~=3.15.dev10 ",
	"flow.record~=3.15.0",

[yunzheng]

We can now use the ignore_fields_for_comparison context manager.

Suggested change

	from flow.record import (
	IGNORE_FIELDS_FOR_COMPARISON,
	Record,
	RecordOutput,
	set_ignored_fields_for_comparison,
	)
	from flow.record import (
	ignore_fields_for_comparison,
	Record,
	RecordOutput,
	)

[yunzheng]

This can now be wrapped in with ignore_fields_for_comparison([..]): block