add: performance and reliability issues

[skshetry]

• Repeated dvc add is not skipped.

  $ dvc add data
  $ dvc add data

  In 1.X, it'd have been skipped. And, dvc still deletes the file and tries to restore it from the cache making it slower.

• DVC uses move-then-checkout logic. It moves the file from the workspace to the cache and then checks it out again, rather than just using copy.

  This is slow and might result in data loss if it happens to fail in between the operations.

• DVC deletes the stage file, before even adding those files. This means that if the dvc add operation fails, the existing pointer file is lost, which is the only way to get access to the data.

• DVC resets the stages multiple times (only if multiple targets are provided) and forces the stage recollection which is slow.

• To the same effect, it resets the internal state of the repo after creating each stage, which also happens to reset dulwich's ignore manager, making it horribly slow if using too many targets (or, -R).

https://github.com/iterative/dvc/blob/4e792ae61c5927ab2e5f6a6914d985d43aa705b4/dvc/repo/add.py#L266

[pared]

DVC uses move-then-checkout logic. It moves the file from the workspace to the cache and then checks it out again, rather than just using copy.

Wasn't this intended to enforce cache link type? I guess in case of copy it would make sense but what about others?

[skshetry]

For other links, the one I suggested was to change copy behaviour to be move + link that works atomically.
@efiop also suggested using hardlinks instead.

[dberenbaum]

@skshetry Do you think we should include this as part of the data epic?

[skshetry]

Closed by

• state/cache: implement get_many/set_many dvc-data#522
• diff: do not recurse when trying to get stats dvc-data#543
• checkout: remove unnecessary exists/remove call dvc-data#544
• hashfile/checkout: use save_many to save state dvc-data#545
• checkout: figure out whether to link or not dvc-data#548
• speedup localfs.info  dvc-data#552
• pass checksum_jobs dvc-data#554
• relink: try to determine iscopy() from metadata if possible dvc-data#562
• checkout: reduce makedirs call by caching dvc-data#561
• deps: bump dvc-data to >=3.16.0 #10513
• bump dvc-data to >=3.16.2 #10519

and, released in https://github.com/iterative/dvc/releases/tag/3.54.0.