pipeline_v4

---

title: Pipeline v4 Notes description: published: true date: 2023-03-16T23:03:56.226Z tags: editor: markdown dateCreated: 2022-05-03T15:40:45.851Z

Checking for differences

Need a script, or tool, part of the pipeline scripts themselves ideally, to calculate hashes for input files and input directories, as well as hashes for output files and output directories.

To speed up the calculation, use an SQLite database to cache the hashes for files by name, size, and modification date.

(In general, if either the input or output hashes mismatch, the process needs to be run, otherwise it can be skipped. All processes will need to report on their input and output directories and files.)

(Since we'll have all input files, and output files, we'll also be able to add a final cleanup pass that removes stray files.)