-
Notifications
You must be signed in to change notification settings - Fork 17.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
proxy.golang.org: support git-lfs #47241
Comments
As a matter of policy, However, as a practical matter, changing the way Git behaves while downloading modules is a dangerous thing to do, since |
As suggested in the Workarounds section above, using the If the proxy is just using † The go-git project is close to supporting git-lfs by way of filters.
I suspect the OP in #39720 used an old git version, in which
and is easily confirmed by some console messages about downloading the git-lfs files and the listed file sizes.
I understand your concern. One idea is to:
This approach means that all existing checksums previously computed by The hash computed above is over the "raw repository": you might call it a "repo" checksum. You might then consider adding a second "content" hash to the |
The module services are closed source, but the part you're interested in just calls
This distinction doesn't exist in the This is a question for the cmd/go team, @bcmills @jayconrod @matloob. But my guess is that we want to actively disable LFS support unless a user manually opts in. The module ecosystem really isn't set up to have multiple valid views of the same repository. I understand this is going to make life hard for LFS users and I don't know of a good answer to that. |
We certainly cannot have the |
I would argue that LFS support should be on by default, with an opt out for cases like #29987. As it stands, GOPROXY can break a build that otherwise works without: it seems odd to opt-in for non-breaking behavior. An alternative which I would support is a new VCS called "git-lfs". In this case the "opt in" is by the package author (in their go-import meta tag; or VCS qualifier and import comment), and not the package user. In this way, checksums will be consistent because all repository accesses would require LFS and therefore a single-view of it.
These issues were cited in the top-comment -- all of them are about a year old now and I've not seen any discussion or proposals on how LFS might be properly supported. What is the process for seeing it through?
I am all for this: the git-lfs project is implemented in go, licensed under MIT, and can be incorporated into Implementing a new "git-lfs" VCS would satisfy all the needs brought up in this thread so far:
|
Maybe! Treating (Unfortunately, the |
An explicit If |
It also seems like a reasonable option to me. But anyone who isn't using a vanity server is stuck -- we can't switch the protocol for all of github.com (etc) without causing trouble. I don't know if that's good enough. |
It would presumably get its own VCS path extension too. So you could do something like:
and then Actually, that's a really nice property to have, because it gives different module paths (with different checksums) for |
(Then we would just need some mechanism for forwarding references from the old path to the one with the |
@jpap, I think that idea (an explicit |
I've added a proposal here. Feedback in that issue is welcome.
If a module owner opts in "new versions" to Git LFS, old (tagged) commits remain available via the original import path (lacking the If the module user requests "old versions" but now using the new VCS qualifier, old (tagged) commits will now be fetched with Git LFS. Whether or not those old versions hosted Git LFS content, each checksum is still unique because it associates with a module path that now includes the The only downside to the last scenario is that a GOPROXY might end up storing the same content twice, if there was no LFS content in the old version: once for the original module path, then again for the new module path with (That duplication depends on the implementation: if the proxy stores all of its downloaded module data keyed on checksum, it could effectively deduplicate them. Pity about the ZIP contents filename The checksum mismatch between the current GOPROXY and using There is one more scenario whereby the module user requests one of the "new versions" of the module that opts into Git LFS, but does so without specifying the new VCS qualifier (explicitly or via go-import). This scenario might be a module user updating their dependency. In this last case, we want the user to use Git LFS because that's what the module owner wants for that version. This is handled by the "upgrade" scheme in Part 5 of the new proposal: if that proves too complex or magic, I am happy to replace it with "all references to a module (package) that has opted into Git LFS must use the git-lfs qualifier". This makes importing a GOPATH package using Git LFS less clean, but I could live with that given that use case is being deprecated. This last part may relate to the "forwarding references" you mentioned. Your feedback is welcome. EDIT: I've removed the "upgrade" scheme in the proposal and replaced it with the error out clause... |
The Google-hosted
proxy.golang.org
does not smudge git-lfs pointer files in public repos. This results in broken builds when modules include binary syso files that are hosted under git-lfs. (The linker sees a text file instead of the object file, ignores it, then explodes with a lot of missing symbol errors which isn't very intuitive.)While it might be unreasonable to expect every git extension be supported on
proxy.golang.org
, git-lfs is quite common: it's supported by many of the major git hosts (e.g. GitHub, GitLab, Bitbucket, Azure Repos) with many others having a pending feature-request for it.Workarounds
Use
GOPRIVATE
for all packages that need git-lfs. This can result in a complex configuration environment for the developer who now needs to track this across possibly many dependencies, some of which may be transitory.The nuclear option: turn off the proxy with
GOPROXY=direct
.Change all modules to avoid git-lfs and commit all binary syso files directly into each repo.
None of these options are great...
Similar issues
Issue #41708 talks about a mod checksum mismatch between proxy and direct. One solution is to ignore the smudge and get matching checksums, which can work in that case because the git-lfs file(s) are inconsequential to the build. That approach doesn't work in the scenario described above.
The following issues are related to git-lfs, but aren't as relevant:
proxy.golang.org
) and where skipping the smudge worked for them.git archive
didn't smudge git-lfs files usinggo get
. Worked when I tried thegit archive
it on a GitHub repo with git-lfs files... perhaps they have used an old git version?The text was updated successfully, but these errors were encountered: