New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: cmd/go: go get should support source archives #19614

Open
iand opened this Issue Mar 20, 2017 · 11 comments

Comments

Projects
None yet
4 participants
@iand
Contributor

iand commented Mar 20, 2017

This is a tester to see if there is any support for this idea. If there is then I will write up a more detailed proposal.

Currently the go get command expects to interact with a source control system to check out source code. There is no way to package up a source archive (e.g. tar or zip) and use go get to retrieve this.

This proposal would modify the go tool to look for a source archive according to a to-be-defined scheme when the user types go get package. The go tool would download and unpack the source archive in the correct location. Alternatively the go tool could be modified to keep the archive file and support reading source from the archive directly during a build.

One possible strawman scheme would be to extend the .vcs package path syntax to allow the specification of an archive format, e.g. example.com/code/package.zip. This would be the direct URL of the source archive of the package.

There are a number of advantages to using source archives:

  • simpler publishing of packages: just place the archive file on a web server
  • removes need for installing support utilities such as git or mercurial
  • convenient versioning of packages by adopting a semver naming convention for the archive
  • verifiability through pre-existing content hash and signing schemes
  • better caching of source downloads
  • opportunities for mirroring
  • potential for cleaner inclusion with OS package systems (e.g. apt, yum etc)
  • smaller download sizes since no need to replicate entire vcs history
@bradfitz

This comment has been minimized.

Member

bradfitz commented Mar 20, 2017

Dup of #16545

@iand

This comment has been minimized.

Contributor

iand commented Mar 20, 2017

As @bradfitz noted, this is suggesting similar functionality to #16545. That proposal got stuck on the behaviour of go get -u, especially in the situation where the downloaded source had been edited.

I think this complexity can be avoided by stating that source archives are immutable. go get -u example.com/code/package-1.0.zip would have no effect if the directory $GOPATH/src/example.com/code/package-1.0 already exists.

To upgrade to a new version of a package, a new package path needs to be used, e.g. go get -u example.com/code/package-1.1.zip would download and extract source files to $GOPATH/src/example.com/code/package-1.1

@bradfitz

This comment has been minimized.

Member

bradfitz commented Mar 20, 2017

What if package-1.0.zip changed? Or more realistically, if a user put up mycode.zip and tried to use go get -u on it? Would -u be an error if it already existed on disk? Would we save the server's Content-Length & ETag? Where?

@iand

This comment has been minimized.

Contributor

iand commented Mar 20, 2017

Changing package-1.0.zip breaks the contract that source archives are immutable. The user would need to intervene and remove the old version manually. This is the same situation that occurs when getting from a vcs where the user has modified the local source and not committed it or the remote source has rebased the user's current commit out of existence. go get should not try and fix these problems.

-u would be a no-op if the directory existed. The source is assumed to be immutable so it cannot be updated if it already exists. This is similar (I think) to the behaviour of go get in a detached head situation.

I think saving the etag/sha256sum is out of scope of this change. I would expect the go get process to check the sha256sum of the file during download if a convention is defined for the naming of a file containing the sum, e.g. example.com/code/package-1.0.zip.sha256sum

@bradfitz

This comment has been minimized.

Member

bradfitz commented Mar 20, 2017

Changing package-1.0.zip breaks the contract that source archives are immutable.

I don't think that's a sufficient answer. Users are going to violate it anyway, either on purpose or by accident, and we need an answer to why two users have different results when one ran "go get" at time 1 and the other ran "go get" at time 2, then both ran "go get -u" and things were still different. We deal with enough of those support complaints already. This would make it worse. It's easy enough to say "just go rm -rf $GOPATH/src/foo", but in this case "foo" might be a far away dependency of the thing the user actually cared about.

In other words, we already support the "-u" flag. I don't think it's acceptable to just say that sometimes "-u" doesn't work for certain packages, seemingly randomly to the user.

@iand

This comment has been minimized.

Contributor

iand commented Mar 20, 2017

As you pointed out on #16545 we don't want to implement a vcs system. If users want to modify their code in place and reuse the same package path then they should use the pre-existing go get support for common vcs systems. For source archives, versioning is achieved by naming the package paths and assuming the archive is immutable. -u would not download a replacement for a source archive but it would traverse the dependency tree and recursively update imports as normal. -u -v would indicate that the package source is assumed to be up to date.

If there were a convention for locating a remote sha256sum then potentially we could add a way of verifying that the local source has the same hash as the remote but this is only informative.

I don't think it's acceptable to just say that sometimes "-u" doesn't work for certain packages, seemingly randomly to the user.

I don't think it would appear to be at random. The user is typing in a package path that ends with .zip or .tar. This is a signal that something is different and using -v would explain that difference.

Immutability of source archives could be seen as a goal in of itself. The current mechanism for version pinning has confusing behaviour for end users. For example github.com/aktau/github-release uses semver to tag releases (and coincidentally is a tool for doing that). A user can pin a release by checking out a tag, e.g. cd $GOPATH/src/github.com/aktau/github-release; git checkout v0.5

Subsequent usages of go get -u result in a potentially confusing error:

$ go get -u -v github.com/aktau/github-release
github.com/aktau/github-release (download)
# cd /home/iand/src/github.com/aktau/github-release; git pull --ff-only
You are not currently on a branch.
Please specify which branch you want to merge with.
See git-pull(1) for details.

    git pull <remote> <branch>

package github.com/aktau/github-release: exit status 1

This is made more confusing because the user believes they have just switched to the v0.5 "branch". Also -u stops here and doesn't update dependencies of the package.

The source archive approach would retain the semantics of "pinning", guaranteeing that the source is unchanged, but continue to check and update dependencies (which may also be pinned).

@bradfitz

This comment has been minimized.

Member

bradfitz commented Mar 20, 2017

I don't think it would appear to be at random. The user is typing in a package path that ends with .zip or .tar.

No, I was referring to cases where the .zip is a dependency of something else.

Consider:

User 1

# Time 1:
$ go get github.com/foo/bar
# Time 3:
$ go get -u github.com/foo/bar

User 2

# Time 2:
$ go get github.com/foo/bar
# Time 3:
$ go get -u github.com/foo/bar

Question: do User 1 and User 2 have the same code on disk? Will the github.com/foo/bar package be the same?

With your proposal, you no longer know. Because foo/bar might transitively depend on what you're assuming is an "immutable" package, but only because you're wishfully declaring that. A user could actually put my-foo-bar-dep.zip on an HTTP server and update it, but then Users 1 and 2 above could have different code on disk, even after a go get -u.

@bradfitz

This comment has been minimized.

Member

bradfitz commented Mar 20, 2017

If you want to proceed with this, please write a design doc. See https://github.com/golang/proposal

@iand

This comment has been minimized.

Contributor

iand commented Mar 20, 2017

Thanks, will do.

@gopherbot gopherbot added this to the Proposal milestone Mar 20, 2017

@gopherbot gopherbot added the Proposal label Mar 20, 2017

@ghost

This comment has been minimized.

ghost commented Mar 21, 2017

this will also solve --depth=1 problem (on github) #18423.
e.g go get https://github.com/USER/REPO/zipball/master and go get https://github.com/USER/REPO/tarball/master

@rsc

This comment has been minimized.

Contributor

rsc commented Mar 27, 2017

On hold for package management, which I expect will have an answer for this problem.

@rsc rsc added the Proposal-Hold label Mar 27, 2017

@rsc rsc changed the title from proposal: go get should support source archives to proposal: cmd/go: go get should support source archives Mar 27, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment