Archive/Download do not include LFS files or Submodules

[tredeske]

• Gitea version (or commit ref): 1.5.0
• Git version: 2.16.4
• Operating system: centos 7
• Database (use [x]):
  • PostgreSQL
  • MySQL
  • MSSQL
  • SQLite
• Can you reproduce the bug at https://try.gitea.io:
  • Yes (provide example URL)
  • No
  • Not relevant
• Log gist: Downloads of LFS files

Description

With git-lfs installed and enabled on both the gitea server host and the client host, LFS controlled files do not get added properly to the .zip or .tar.gz files when:

• Using the Download Repository button
• Downloading a release

Instead of the expected file in the .zip or .tar.gz, a text file of the same name is placed in the file.

The rest endpoint also functions in the same way.

GET /repos/{owner}/{repo}/raw/{filepath}

In other respects, git-lfs works as expected when using git command line to interact with the repo.

Screenshots

Text files look like this:

version https://git-lfs.github.com/spec/v1
oid sha256:a7da80fc96bc0dd73ea0416fda5dfe1321910517634d4b142903a9fbab24f196
size 1465634

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs during the next 2 weeks. Thank you for your contributions.

[stale]

This issue has been automatically closed because of inactivity. You can re-open it if needed.

[jjstoo]

Are there any updates, fixes or thoughts on how to approach this issue?

We would love to use Gitea and its API to download releases directly onto deployment servers and end users, but Gitea not including any LFS objects to the downloads is a huge problem. Using git to clone the repository is not an option as we cannot mandate our customers to install any extra software.

[zeripath]

So there is at least now a GET /repos/{owner}/{repo}/media/{filepath} endpoint which means that you can get the actual lfs'd data.

Could you give me some information as to how you create the zips - I don't immediately know where to look to find the code that creates them.

[schmittlauch]

#7209 might be related

[zeripath]

@schmittlauch I'm not certain. I would have to dive to see how these zips are created.

My suspicion is that these zips do not even attempt to dereference the LFS pointers whereas on #7209 your problem is different.

[zeripath]

OK so yeah #7209 is not relevant to this.

The issue is that we use git archive to create these archives. That doesn't include submodules either - so I think this needs a complete rethink.

[lunny]

And how github archive did that?

[zeripath]

I suspect they rewrote the command. Back in November 2018 git-lfs/git-lfs#1322 (comment) states that they didn't include lfs files (and likely submodules) in their zips.

I think that's what we're going to have to do unfortunately.

[zeripath]

This again leads to the slightly annoying issue whereby we don't know what files are LFS files except by reading them and checking if they're a pointer or not.

Similarly we need to do this zipping in the context of the current user and repository. In the case of submodules - it's conceivable that the zip that one user downloads may not be the same as the zip another user gets - I guess that's ok but it means caching these might be difficult unless we cache them with the associated permission state.

Finally we must be very careful indeed about which submodules we're happy to include, if any - perhaps just allow those that are local to the gitea instance?

[RogueRaider]

Just wanted to add my two cents to this conversation.
Without this feature there is not much point in using git lfs at all. All the development work happens using lfs and when a production version is produced, it contains lots of nasty surprises in blank pointer files.
My current approach is to not use lfs at all and handle big binaries separately to git. This creates a lot of extra mess that would be much easier if I could just download the archive.

[drewcassidy]

I know it wouldn't be as fast as git archive, but could gitea just checkout the repository to a temp directory and archive that? that way any smudge filters and submodules would be handled without directly having to handle them. It could even cache results for the head of the main branch to prevent it from having to run multiple times on subsequent downloads

[zeripath]

Gitea checking out the repository is not a good idea.

[simchanu29]

There is a wrapper around git archive that would allows to at least handles submodules.
https://github.com/fabacab/git-archive-all.sh/blob/master/git-archive-all.sh

Maybe gitea could use it ?

[hramrach]

AFAIK github does not solve this. The archive you get from github may include LFS but not submodules.

Release archives published on github are created and uploaded by the developer to include the submodules.

If no release archive is uploaded by the developer submodules are not included.

It is probably possible to automate with github actions.