v0.9.2

v0.9.2

For anyone running 0.9.1. This release checks every action before the first snapshot is taken, makes a
saved plan script as careful as applying with Y, stops a scan within about a second, keeps hashing
fast in a large scan, refuses command lines it would half-ignore, and makes mouse clicks and keys act
on what the screen shows. The database stays at schema v6, so nothing is migrated.

Applying actions

• Before the first snapshot, every action is checked: a read-only file system; chattr +i or +a on
  the file, its directory or the quarantine directory; no space or an exhausted quota; a file reached
  through a second mount (a bind mount) or lying outside its dataset's mount point; for a hardlink,
  the keeper as well. Such an action is refused with its reason and without reading any file; the
  others run. A snapshot is taken only of the datasets where there is work.
• If nothing in a batch can run, nothing changes and no snapshot is taken:
  nothing done — N actions cannot run: <reason>; no snapshot taken, marks kept; first: <path>, and
  the plan goes back to the window where it was confirmed.
• Reflink on a host without block cloning (OpenZFS older than 2.2.1, or zfs_bclone_enabled=0), or on
  a pool whose feature@block_cloning is disabled, is refused before the confirmation window, with
  the number of marks and what to do instead.
• A hardlink or reflink whose keeper is on another dataset is refused by the plan itself, before any
  snapshot or read: cannot hardlink or reflink across datasets (N marks) — …. Reflink across the
  datasets of one pool never worked; the manual no longer promises it.
• Hardlink and reflink of a file whose name is up to the 255-byte limit no longer fail with
  File name too long, and a CSV export can be written to a file with such a name.
• A failed replacement names the real cause: the prepared replacement vanished, no space, the quota,
  a read-only file system, an I/O error. If another file took the original's place, so that the
  original cannot go back, the message gives its exact path in the quarantine. A failed database
  write during a scan names the operating system's error, and a problem with the database file or
  the lock says what was found under that name.

A saved plan script

The script that F11 → S saves now takes the same steps as applying with Y:

• Before each action it compares the file with its keeper byte for byte (cmp) and leaves alone a
  file that has changed or become a symbolic link (dedcom: skip <file>: …).
• The replacement, a link or a clone, is made beside the file first. Only then does the file go to
  the quarantine, and it comes back if the replacement cannot be put in its place.
• A reflink copy gets the owner, mode, times, extended attributes and ACL of the original.
• An mv -n that moved nothing no longer counts as a deletion, and a file whose quarantine is on
  another mount (a bind mount, or a link to another file system) is refused before the move.
• One failed action no longer stops the rest. The script ends with
  dedcom: N of M actions not carried out — each is named above and exit code 1.
• The Commands tab shows one function call per action; the functions are defined at the top of the
  script.

Scanning

• A stop during hashing (Esc on the scan screen; Ctrl+C, SIGTERM or SIGHUP for --scan) takes
  effect within about a second, even in the middle of a large file. Before, the scan first finished
  reading every file of the current batch of up to 64: tens of minutes for a 500 GB disk image. The
  files being read at that moment get no hash and are read again from the start on resume.
• Hashing no longer slows down as a large scan goes on: 100,000 small files take 11 s instead of
  6 minutes on this project's test machine, and a scan of 2 million files on a home server hashes
  about 300 files per second instead of 14.
• --scan of a root that does not exist or cannot be read is refused with exit code 1 before the
  lock and the database are touched, instead of ending as an empty "successful" scan that trimmed the
  history. A resumed scan prints the settings it keeps and refuses a flag it would ignore;
  --no-resume starts over.
• A scan that finds no file under a root where earlier scans of the same roots found files, usually
  the empty mount point of a dataset that did not mount, no longer sends those scans to the trash and
  says History kept: ….
• A directory that cannot be opened (removed during the scan, a path longer than 4,095 bytes, no
  search permission, a disk error) is one walk error instead of the end of the scan: Omissions: … walk errors, ⚠ gaps in the interface, and a walk error: <path>: <reason> line in dedcom.log.
• --include-ext tar.gz finds photos.tar.gz, '*.JPG' means jpg on the command line and in
  presets.json, and a value with / is refused.

The command line and config.json

• An argument that is not valid UTF-8 no longer stops dedcom with a panic: the state directory and
  the export file accept such names, and a scan root gets a clear refusal.
• Two modes at once, a repeated --export-csv, or a flag the run does not read are refused with exit
  code 2 and an explanation, instead of half of the command being dropped. So are --verify,
  --merkle-dirs, --strict-verify and --no-resume beside --read-only. A window started with
  --read-only answers an action with (started with --read-only).
• Refusals show invisible direction-control characters escaped.
• A config.json that cannot be read (not JSON, not an object, not a regular file) is no longer
  rewritten and decides nothing. Each writing run says so in one line; automatic VACUUM and history
  trimming are skipped until the file is fixed, and the lock policy falls back to ask. A field of
  the wrong kind turns off only its own decision. config.json is now written by replacing the file,
  with mode 0600.

Clicks and keys act on what is drawn

• A click on a function-key number in the commander's bottom bar runs that key at any terminal
  width. Before, at widths such as 80, 100, 160 and 200 columns, eight of the twelve numbers ran the
  key to their left: "9 Menu" set a Delete mark, "11 Exec" quit.
• A click or a double-click in a list of groups or of a group's files, in the wizard and in the
  commander, scrolled or not, takes the row drawn under the pointer. Before, the wizard took the row
  below it, and a double-click saved the wrong keeper; in the commander such a click dropped the
  cursor. PgUp and PgDn move by the visible part of the list.
• In the wizard's group view, 1 opens Folders and 2 opens Files, as labelled.
• While the help, the start-up notice, the role prompt, a function-key window or a yes/no question
  is open, a click does nothing: close the window first. Before, a click under the help could set a
  Delete mark. On the Triage Board, a click no longer moves the focus while a receiver waits for its
  digit.
• The "Please wait" box is no longer drawn over a question or a window that is waiting for a key.
  Before, it could hide "Move to trash?" or "Purge from trash?", where Enter means yes.

A late answer does not take over an open window

• When F2, F3 or F11 answers after you opened the F9 menu, another window, the help or the Triage
  Board, its window no longer opens over them, and no scan starts from under them. The status line
  says what arrived and which key to press again ("Close this window, press F2 again: …").
• A scan result that finishes opening after you opened a menu, a window or the Triage Board, or moved
  to another screen, opens without switching the screen; the status line names the scan and says how
  to show it. The help does not count: the groups open under it. Before, the group view took over,
  and an Enter meant for the menu saved a keeper. A plan finished after you left the group view is
  dropped; r builds it again.

Names that are not valid UTF-8

• For a file whose name shows with a ?, F3 no longer shows another file's checksum and duplicates,
  "duplicates of the cursor" no longer shows another file's group, and a directory no longer shows
  another directory's total size. Both screens say instead that the scan has no such name, and why.
• The manual (§11.4) no longer says such a name is exported. It says where these files are counted
  (Omissions: in the --scan output, ⚠ gaps in the TUI) and how to ask about one file (F3 in
  the commander). The sample --scan output in §11.1 matches what the program prints.

The cursor in a large scan

In "duplicates of the cursor" a cursor step costs a fraction of a millisecond and no longer grows
with the scan; in 0.9.1 it was about 70 ms on a scan of a million files, on this project's test
machine. Building the F11 plan no longer walks the scan for every mark and group.

The manual

• A new §8.9 on backups and the stores of other programs (Time Machine, restic, borg, kopia, Arq,
  Duplicacy, Proxmox Backup Server, iPhone backups, virtual machine disks): do not scan a store, stop
  the program before applying, apply with Y rather than a saved script, verify with the program's
  own tools before cleaning anything. Wherever the manual cleans the quarantine, the command is
  dedcom --purge-quarantine --yes.
• §11 gives a ready cron line that runs at low priority and checks that the pool is mounted.
  Chapter 13 quotes every reason an action can be refused before the snapshots, and every line a
  saved script prints; a test checks the quotes against the program.

What it does not establish

• Unicode formatting characters in names, such as bidirectional overrides and zero-width characters,
  are not escaped on screen; a terminal with bidi support may reorder a row that shows one.
  Command-line refusals now show them escaped.
• A plan saved as .sh holds the real bytes of the names, as it must to run, so cat of that file
  still passes an escape sequence in a name to the terminal. The Commands tab shows them escaped.
• The confirmation window does not yet show which actions cannot run; their refusal comes after you
  confirm. Planned for 0.9.3.
• A stop is not seen during the --verify byte comparison, nor while Y re-reads both files before
  an action: those finish first. Planned for 0.9.3.
• nohup does not keep --scan running after an SSH session drops: the hang-up stops the scan. Run a
  long scan over SSH in tmux or screen. Planned for 0.9.3.
• A stopped --scan exits with code 0, as a finished one does. Planned for 0.9.3.
• The memory warning before grouping advises Esc, but a stop is noticed only after grouping has built
  its structures. Planned for 0.9.3.
• A dataset below a scan root that did not mount is not noticed: the scan finds files in the rest of
  the root and trims the history as usual.
• A path within about 60 bytes of the 4,095-byte limit can still fail a hardlink or reflink after
  the snapshot. A directory the walk entered but could not read the metadata of still stops the scan.
• The first visit to a very large directory still costs one pass over its rows, seconds at the top
  of a pool of millions of files.
• A files panel shows only the marks set while it was open. Marks saved earlier show in "group
  files" and on the F11 confirmation.
• While a scan opens, the "Please wait" box still covers the middle of the screen, and keys act on
  what is under it: in the commander that can be the file under the cursor.
• If the list of scans is read again while you are on it, the cursor can go back to the first row,
  and the next Enter or Del acts on that row; "Move to trash?" and "Purge from trash?" do not name
  the scan. Planned for 0.9.3.
• If a scan finishes but its result fails to open, the scanning screen stays, and only q leaves
  it. Planned for 0.9.3.

On upgrade

• The 0.9.1 database opens as it is.
• A command line that combines modes, repeats --export-csv, or passes a flag the run does not read
  now fails with exit code 2. Check cron lines and aliases: alias dedcom='dedcom --strict-verify'
  now breaks dedcom --stats and dedcom --read-only.
• An unfinished scan started by 0.9.1 with --include-ext '*.jpg' does not resume from the same
  command line (it kept *.jpg, the new run asks for jpg): start it over with --no-resume.
• A saved plan script needs cmp (diffutils) and findmnt (util-linux) besides coreutils, all three
  required packages on Debian 12 and 13. It reads every file it works on and its keeper in full: a
  pair of 100 GiB images on a pool of hard disks at 150–200 MB/s takes 17–23 minutes.
• Snapshots are taken only of the datasets where at least one action can run. The pools'
  feature@block_cloning is read once at start: restart dedcom after changing it.
• dedcom --read-only no longer starts when dedcom.lock is a directory.
• After --stats, --export-csv or a --read-only window, dedcom.db-wal (0 bytes) and
  dedcom.db-shm (32 KiB) can stay in the state directory. They are harmless and the next writing
  run removes them; do not delete them by hand.
• From 0.9.0-beta.4 or earlier, what the v0.9.1 notes say on upgrade applies as well.