Feature: ipfs add/get --reflink

[Artoria2e5]

Although filestores are nice, they carry the issue of someone carelessly changing the file content and the cost of checks related to this problem. With a bit of filesystem magic, we can actually do a basically no-cost copy of data from a file to the blockstore or vice versa. The specific API is FICLONERANGE (Linux) and FSCTL_DUPLICATE_EXTENTS_TO_FILE (Windows).

A new boolean / string option --reflink should be added to ipfs add and ipfs get. It accepts three values: false (disable reflink), auto (enable reflink with fallback), and true (enable reflink no fallback).

• For ipfs add, --reflink only has an effect when the block store is a flatfs. --nocopy will disable --reflink since the block store isn't even used.
• For ipfs get, --reflink has an effect in two cases: when the block store is flatfs and when a block is being retrived from a filestore. The proposed --nocopy option (ipfs get --nocopy #6687) will work with this switch.

A big issue here is the datastore interface, as many bunches of file:offset:length information will need to be passed around for reflinking to happen in flatfs. (It could happen with other stores but no serious concurrent backend would expose these sort of internal data structure.) It might be easier for filestore since it already does that.

[aschmahmann]

issue of someone carelessly changing the file content

Seems like this could be handled by making files read-only, no?

---

@Artoria2e5 using Copy-on-Write behavior to improve the filestore seems like a pretty reasonable idea, since as you noted the main issue with the filestore is how to let people continue to use their files even though the data has been added immutably to IPFS.

A few notes:

• I wouldn't try and optimize for FlatFS. We're hoping to switch to Badger as the default datastore Make badger-ds the default datastore #4279 anyway so making design decisions informed by FlatFS seems like a generally idea
• While FICLONERANGE is Copy-on-Write for Linux, FSCTL_DUPLICATE_EXTENTS_TO_FILE seems to just be a faster copy and isn't supported on most Windows installations and Filesystems (i.e. Windows Server 2016 with ReFS). This makes this essentially a Linux-only feature
• It might be more reasonable to toss your proposed ipfs add modification and replace your ipfs get proposal with "ipfs get" using the Filestore #3981