Large file support

[ganguin]

Do you have already plans for large file support?

IMO large file support should allow:

• decide to get or skip the files: maybe something like partial clone in Microsoft scalar?
• ability to delete a local copy: should it also allow to delete the last copy of a file?
• it would be nice to have releases/builds part of the large files. Maybe by keeping track if the file is "required" or can be built.
• ability to make/tag as "optional" object any file in the repo history. maybe like a shallow clone? but with the ability to get the history "on-demand".
• "preview"/statistics: in case of a "clone everything", it would be nice to have something telling how big the clone will be. Or a warning? are you sure you want to get 5TB of releases/builds XD.

These are some thoughts, there is probably much more to it ;-)

Example:

• One idea would be to make it possible to version control a binary package repository.
• Keep a clone as small as needed. Be able to clone "source code" only or with "builds".
• Drop history to get on par with a shallow clone

git lfs is nice because it is simple to use. IMO the biggest drawbacks are:

• Selectively manage which file to get, which to skip is a pain. Either you get everything, or you get nothing and selectively pull what you need, but there is no way to "remove" a local copy (beside delete everything, re-clone and start over selective pull).
• It requires a separate channel to pull/push lfs objects.

git annex is quite more powerful than git lfs in how you manage your big files and to keep track in which repo they are, but:

• It can get quickly overcomplicated
• You can too easily make a mess with git annex sync

[martinvonz]

Sorry that it took me so long to reply (I was on vacation and then just forgot).

I haven't thought much at all about large file support and I haven't used either Git LFS or git-annex. I hope partial clones will be sufficient as that seems most user-friendly. I think Git's partial clones support filtering by file size, so you can ask to get only files smaller than 1MB, for example.

However, since jj uses libgit2, which doesn't have support for partial clones yet, it's probably going to take a long time before we get there. We'll see if gitoxide gets support for it before libgit2. Then we can consider switching to that.

[joyously]

I was reminded of SnowTrack today and wondered about how jj commands (and tests) handle binary files.
https://www.snowtrack.io/

[martinvonz]

I was reminded of SnowTrack today and wondered about how jj commands (and tests) handle binary files. https://www.snowtrack.io/

There's no special handling of binary files. We have #80 about support for Git LFS. Custom backends can do better by not loading files until their needed. The Git backend uses libgit2, which doesn't support partial clone, so it downloads all files in the initial clone.