Advanced Copy Options - Symlinks and more!

[Ryex]

Lets let users copy instances with some more advanced settings, potentially preserving disk-space.

Symbolic links and Hard links are the standard, but I was even able to do CoW on ZFS, BTRFS, XFS, APFS on MacOS, and even BTRFS (via winbrtfs driver) and ReFS on windows!

fix: #761

related: #422, #38

• Windows
  • Filesystem::create_link (hard links) (no perms needed!)
  • Filesystem::create_link (symbolic) (need elevated process)
  • Filesystem::clone (winbtrfs and ReFS only, clone function uses FSCTL_DUPLICATE_EXTENTS_TO_FILE and DeviceIoControl )
• Linux
  • Filesystem::create_link (hard links)
  • Filesystem::create_link (symbolic)
  • Filesystem::clone (COW via ioctl_ficlone)
• Macos
  • Filesystem::create_link (hard links)
  • Filesystem::create_link (symbolic)
  • Filesystem::clone (COW via clonefile)
• Export process, knows to follow symlinks.
• safety feature so users know when they are modifying linked content.

[Ryex]

OK. basic functionality is in place. Now there just needs to be a discussion of the safety feature that need to be in play.

please review!

[Ryex]

I updated the logic on links
Now there is no danger of an instance overwriting the original config when "copied" via symbolic or hard links
if "link recursively" is turned off instance-folder/* is linked to new-instance-folder/* (top level items are linked except for instance.cfg)
if "link recursively" is turned on instance-folder/**/* is linked to new-instance-folder/**/* (all items are linked except for instance.cfg)

UI logic is cleaned up:
soft and hard links are mutually exclusive check-boxes now. Both are exclusive to clone.

[Ryex]

windows btrfs reflink support by calling ReflinkCopyW from shellbtrfs.dll provided by winbtrfs!

NOTE: This is slight shenanigans because ReflinkCopyW is a bit broken atm
maharmstone/btrfs#556

but the code should be transparently backwards compatible if it get fixed.

[Ryex]

I'v now implimented windows Reflink on both winbtrfs and ReFS via DeviceIoControl calls.

The old code remains in place for now until we can be confident there are no issues.

[Ryex]

Latest change adds the UAC shield to the symlink checkbox on windows. selecting it adds the shield icon to the ok button too.

[Ryex]

and... done.

now there are warnings when resources are symbolically linked or hard linked.

[getchoo]

works as expected on ntfs, but i think i came across a bit of an issue: currently, if you try to delete the instance that a copy is symbolically linked to, the copied instance is now completely unusable due to the links being unresolved. imo, a good solution here would be to delete the symlinks and copy over the files from the original instance being deleted - or at least add a warning to the deletion dialog ;)

this is awesome though, i love it!

edit: forgor to specify symbolic links

[Ryex]

works as expected on ntfs, but i think i came across a bit of an issue: currently, if you try to delete the instance that a copy is symbolically linked to, the copied instance is now completely unusable due to the links being unresolved. imo, a good solution here would be to delete the symlinks and copy over the files from the original instance being deleted - or at least add a warning to the deletion dialog ;)

this is awesome though, i love it!

edit: forgor to specify symbolic links

I'm not really sure what to do in this situation. It's nigh impossible to detect when a symlink exists without knowing exactly where to look for it. The only way to warn users when they are deleting an instance another instance links files form would be to keep track of all the symlinks an instance has and do a search through all other instances when doing a delete... I'm not sure that would be smart or quick.

[getchoo]

works as expected on ntfs, but i think i came across a bit of an issue: currently, if you try to delete the instance that a copy is symbolically linked to, the copied instance is now completely unusable due to the links being unresolved. imo, a good solution here would be to delete the symlinks and copy over the files from the original instance being deleted - or at least add a warning to the deletion dialog ;)

this is awesome though, i love it!

edit: forgor to specify symbolic links

I'm not really sure what to do in this situation. It's nigh impossible to detect when a symlink exists without knowing exactly where to look for it. The only way to warn users when they are deleting an instance another instance links files form would be to keep track of all the symlinks an instance has and do a search through all other instances when doing a delete... I'm not sure that would be smart or quick.

i can understand speed when recursing through all instances, but keeping track of links not being smart? i would have to disagree with that
at bare minimum, warning users that other instances could be effected by deleting another would be an infinitely better solution than having an instance break without warning - and though you could say symbolic links make that self explanatory, a majority of people may not understand how they work or the consequences of deleting the original file.

i think that a great solution here would be to add a key to instance.cfg (if possible) that contains the source of an instance using symbolic links. this would allow the ability to easily recurse through all instances in a performance friendly way, while limiting the more time consuming task of finding symlinks. afterwards, a prompt could be shown explaining something along the lines of "the files of this instance are used by , would you like to delete it as well or copy the files over?"

just an idea though, i'm not sure how hard that would be to implement - but again, at bare minimum, i think there needs to be a warning here or else there's a good chance people using this feature are going to come across this and be very confused about what comes up in the error log

[TayouVR]

maybe give the original a list in the instance.cfg with all symlinked instances (their IDs), and then when deleting there could be a warning popup with
"The Instance(s) %1 might reference files in this instance. Deleting it could break the other instance(s), are you sure?"

[Ryex]

re-based because clean git history is cool.

[flowln]

i may be able to finish it off in the weekend

i lied

sorry, this PR gives me a lot of brain fatigue (your work is great, but os-specific stuff gives me nightmares) 😵

[Ryex]

i may be able to finish it off in the weekend

i lied

sorry, this PR gives me a lot of brain fatigue (your work is great, but os-specific stuff gives me nightmares) dizzy_face

no worries. don't feel too overwhelmed by it though. when it comes down to it it's pretty simple stuff despite appearances. I documented the parts that need external research to really understand.

[Continous]

@getchoo @TayouVR I've added a somewhat naive link tracking system. each instance when it is made via copy saves a linkedInstancesList setting with is a json list containing the ID of the original instance. When an instance is asked to be deleted there is now an additional second message box that lists the instances that say they are linked to the deletion candidate, warns that these instances may break, and asks for confirmation.

@DioEgizio I made sure the _filelink.exe was included in the setup exe, I know you mentioned that somewhere.

Is there any possibility we could have an additional option to allow a deletion of the original instance to prompt the user to have the original files migrated to the linked instance? If it's not too difficult, we could even allow linked instances to become "real" instances on command from the user in such case they may want or need it. A good example I can think of is if someone plans on migrating from one version to the next, and has finally decided to remove the previous instance.

[Ryex]

@getchoo @TayouVR I've added a somewhat naive link tracking system. each instance when it is made via copy saves a linkedInstancesList setting with is a json list containing the ID of the original instance. When an instance is asked to be deleted there is now an additional second message box that lists the instances that say they are linked to the deletion candidate, warns that these instances may break, and asks for confirmation.
@DioEgizio I made sure the _filelink.exe was included in the setup exe, I know you mentioned that somewhere.

Is there any possibility we could have an additional option to allow a deletion of the original instance to prompt the user to have the original files migrated to the linked instance? If it's not too difficult, we could even allow linked instances to become "real" instances on command from the user in such case they may want or need it. A good example I can think of is if someone plans on migrating from one version to the next, and has finally decided to remove the previous instance.

perhaps in another PR. that's a fair bit of work and this one is big enough already.

[flowln]

i swear we will finish this soon 🥺

[flowln]

good job 🙂 /gen

[DioEgizio]

2 approvals 👀

[Scrumplex]

Let's merge, I guess.

I probably don't have much to add after @flowln's thorough analysis :D

And if the feature has issues, we can hopefully catch them before an eventual release.

[Un1q32]

1.20 prerelease 7 gives a warning when opening worlds that use symlinks, and requires allowed symlinks to be added to a text file to silence the warning. Will this cause problems?

[DioEgizio]

Make an issue for it please

[Un1q32]

Make an issue for it please

I haven't actually tested anything so I don't know if its a problem, I just randomly remembered it so figured I would mention it.

[RNanB]

1.20 prerelease 7 gives a warning when opening worlds that use symlinks, and requires allowed symlinks to be added to a text file to silence the warning. Will this cause problems?

No, this change only affects linking of individual worlds. Linking the saves folder itself will still work the same

[Ryex]

1.20 prerelease 7 gives a warning when opening worlds that use symlinks, and requires allowed symlinks to be added to a text file to silence the warning. Will this cause problems?

No, this change only affects linking of individual worlds. Linking the saves folder itself will still work the same

This could still have an effect if the user tries to recursively link during a copy. That will need the file created