Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: Add a command to zip the current project and push it to a hosted repository #33312

Open
jlstephens89 opened this issue Jul 27, 2019 · 17 comments

Comments

@jlstephens89
Copy link

commented Jul 27, 2019

Problem

With go.mod and GOPROXY the go tooling now has a good story for proxying open source packages but doesn't have a clear story for how best to manage and consume private packages. This is especially problematic for Repository Managers (such as Nexus Repository Manager) and makes it difficult for Go developers to share their private packages within their organizations.

Goal

Add a new command the Go CLI that creates a zip in a format that can be consumed via GOPROXY and push/POST that zip to a HTTP(s) endpoint.

This would allow Repository Managers to implement this endpoint and consume Go packages, making those packages available to Go developers via the download APIs. Go developers could then use this for sharing their private packages within their organizations.

@triztian

This comment has been minimized.

Copy link

commented Jul 27, 2019

I think this can be achieved this by this zipping up $GOPATH/pkg/mod/cache/download and then extracting it elsewhere, after extracting the zip archive you can use the GOPROXY env var with an absolute path pointing to the extracted location:

Something like:

zip -r gomods.zip $GOPATH/pkg/cache/download
curl -X POST -H "Content-Type: application/zip" --data @gomods.zip
https://example.org/gomods/download

And then elswhere:

GOPROXY=file:///path/to/extracted/contents go mod verify

Unless I'm missing something?

@jlstephens89

This comment has been minimized.

Copy link
Author

commented Jul 27, 2019

@triztian I’m referring to zipping your current project rather than its dependencies and then publishing that to a remote server such as Nexus Repository Manager. This would make for a better user experience for companies that are developing their own private libraries for use by other developers or teams.

@bcmills

This comment has been minimized.

Copy link
Member

commented Jul 28, 2019

This is closely related to #28835.

@bcmills

This comment has been minimized.

Copy link
Member

commented Jul 28, 2019

@triztian, there are some constraints on the file paths within the zip files: they must be slash-separated and have a “<module>@<version>/” prefix.

They also need to omit any directory subtrees containing a go.mod file.

@bcmills

This comment has been minimized.

Copy link
Member

commented Jul 28, 2019

@bcmills bcmills added this to the Go1.14 milestone Jul 28, 2019

@bcmills

This comment has been minimized.

Copy link
Member

commented Jul 28, 2019

Perhaps we should start with a function in golang.org/x/mod or standalone binary in golang.org/x/tools (CC @ianthehat).

@triztian

This comment has been minimized.

Copy link

commented Jul 28, 2019

@jlstephens89 Got it that makes sense, then yes, it's definitively missing such feature specially with the limitations that @bcmills mentioned. After looking at the linked issue #28835 it seems that the same could be achieved with a combination of go mod edit and downloading the remote source from the repository manager (i.e. Nexus Repository Manager).

For example one could still zip up the private source and then download it as a zip file and extract it somewhere on disk.

After extracting one could use go mod edit to replace the imports to point to the version on disk, for example, suppose that the package is provider.com/some/library and that we are able to download and extract the source of such package to /path/to/extracted/sources in our dev machine, then one could do something like this (assuming that the sources have been zipped as-is and pushed to repsmanager.org):

wget https://repsmanager.org/provider.com/some/library.zip # downloads library.zip
mkdir -p /path/to/extracted/sources
unzip library.zip -d /path/to/extracted/sources

cd myproject # the root of the project that depends on provider.com/some/library package
go mod edit -replace=provider.com/some/library=/path/to/extracted/sources
go build -mod=readonly .

Doing this avoids having to pre-populate the $GOPATH/pkg/mod/cache/download with the deps of the project, on the publishers side all that she/he has to do is zip the sources and make them available for download. The Repository Managers would also have access to the sources because they arere plain zip files so as long as the zip file is pushed to the RM no special command is needed. Also by using replace statements one no longer has to deal with the setup of GOPROXY:

I just did a sample test of the go mod edit, before the edit:

λ ui ~> cat go.mod
module gioui.org/ui

go 1.12

require (
	golang.org/x/image v0.0.0-20190703141733-d6a02ce849c9
	golang.org/x/sys v0.0.0-20190626221950-04f50cda93cb
)

Performing the edit:

 λ ui ~> go mod edit -replace=golang.org/x/image=/opt/thirdparty/go/sources/x/image

After the edit:

 λ ui ~> cat go.mod
module gioui.org/ui

go 1.12

require (
	golang.org/x/image v0.0.0-20190703141733-d6a02ce849c9
	golang.org/x/sys v0.0.0-20190626221950-04f50cda93cb
)

replace golang.org/x/image => /opt/thirdparty/go/sources/x/image

Based on the replace docs I think one could script to replace all deps to point to the Repository Manager.

@jlstephens89

This comment has been minimized.

Copy link
Author

commented Jul 28, 2019

@triztian. GOPROXY and the download API make for a very natural user experience when working with a repository manager.

If we went the route of adding a command that created a zip with the correct file paths then any developer that has set GOPROXY to point to a “Group” repository, a repository type that aggregates public and private sources (or alternatively list the URLs comma separated), would have access to both internal and external dependencies with no additional effort. This would be a one time configuration for all developers in an organisation and from then on would allow centralised tooling teams to manage remotes/sources in a single place.

To publish a private dependency developers could do something like “go push http://repourl” and that dependency would then be available to an entire organisation.

@jayconrod

This comment has been minimized.

Copy link
Contributor

commented Jul 29, 2019

At the moment, it would be hard to implement this in an external tool or in x/mod: most of the logic for this is in cmd/go/internal/modfetch and cmd/go/internal/modfetch/codehost. Those packages should eventually be moved to x/mod, but at the moment, they are tightly integrated with many other packages in the go command.

@bcmills

This comment has been minimized.

Copy link
Member

commented Jul 29, 2019

Actually, I think all of the logic for actually constructing the zip file is in one location, at the end of (*modfetch.codeRepo).Zip.

I think it would be pretty straightforward to extract that body out to a function.

@triztian

This comment has been minimized.

Copy link

commented Jul 29, 2019

So basically it'd be a tool that performs the following HTTP (or similar) requests for the given module path?:

POST $GOPROXY/<module>/@v/list 
<version>

Append the version of this module to the ist of all known versions of the given module, one per line.

PUT $GOPROXY/<module>/@v/<version>.info

Set JSON-formatted metadata about that version of the given module.

PUT $GOPROXY/<module>/@v/<version>.mod 

Set the go.mod file for that version of the given module.

PUT $GOPROXY/<module>/@v/<version>.zip

Set the zip archive for that version of the given module.

@jlstephens89

This comment has been minimized.

Copy link
Author

commented Jul 29, 2019

@triztian that would work and some ecosystems work that way. Alternatively it could just POST the zip and leave the rest of the work to the repository manager. The repo manager could extract the mod from the zip and could update .info and list based of the version and module name provided in the upload.

@bcmills

This comment has been minimized.

Copy link
Member

commented Aug 1, 2019

@thepudds

This comment has been minimized.

Copy link

commented Aug 7, 2019

I think the k8s team might have already put together something in terms of a utility that can create valid Go module .zip files. I have not looked at it carefully myself, but could be something to examine for anyone here that is interested.

kubernetes/publishing-bot#185

From https://github.com/kubernetes/publishing-bot/tree/master/cmd/gomod-zip/zip.go:

Creates a zip file at $GOPATH/pkg/mod/cache/download/<package-name>/@v/<pseudo-version>.zip.
The zip file has the same hash as if it were created by go mod download.
This tool can be used to package modules which haven't been uploaded anywhere
yet and are only available locally.
This tool assumes that the package is already checked out at the commit
pointed by the pseudo-version.

There is also https://pagure.io/modist/, which I think might include the ability to create a proper Go modules zip starting with files on disk, but I haven’t looked at that recently, so not 100% sure.

I don't believe either of those include pushing the result somewhere else, but one or both of them might solve the piece of the problem discussed here about how to create a canonical zip file with the same resulting hash as a zip file created by the go command itself.

CC @nikhita

@nikhita

This comment has been minimized.

Copy link

commented Aug 8, 2019

The k8s author here 👋 Thanks for surfacing that, @thepudds!

https://github.com/kubernetes/publishing-bot/blob/master/cmd/gomod-zip/zip.go is indeed the tool used to create .zip files with the same hash as that created by go itself. As a side note, we had some trouble using the zip command directly since that wasn't creating the correct zip files, more details here: https://github.com/kubernetes/publishing-bot/blob/b05cc3ce103eb1c4a70e3aa84ec7ac11a461c634/cmd/gomod-zip/zip.go#L115-L122.

Actually, I think all of the logic for actually constructing the zip file is in one location, at the end of (*modfetch.codeRepo).Zip.

I think it would be pretty straightforward to extract that body out to a function.

The tool essentially duplicates the internal go code to achieve this. It would be really helpful if this logic were extracted out though.

I don't believe either of those include pushing the result somewhere else

We "fake" publish the repo to the local go mod cache. The code for it can be found here:

While having a zip command can be helpful, there would still be some more steps needed to actually publish the repo (as mentioned in #33312 (comment)), so IMHO a command like #28835 would be much more helpful than a simple zip command.

@jlstephens89

This comment has been minimized.

Copy link
Author

commented Aug 8, 2019

While having a zip command can be helpful, there would still be some more steps needed to actually publish the repo (as mentioned in #33312 (comment)), so IMHO a command like #28835 would be much more helpful than a simple zip command.

@nikhita My purpose with this issue is for publishing to a Repository Manager such as Nexus Repository Manager rather than the module cache. The RM can then take care of generating/extracting the additional metadata files provided the information is available within the zip. For those purposes a simple zip and POST command would be the easiest route.

@edganiukov

This comment has been minimized.

Copy link

commented Aug 9, 2019

We are using jfrog artifactory as a GOPROXY and jfrog-cli to publish go modules into the jfrog repository. For example, we use it for generated swagger API clients - we don't store generated code in git, but in Go repository. Also, we are publishing releases of libraries. I found this very convenient and I think it makes sense to have a RW Go proxy (or Go repository) and be able to publish/download modules from it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
7 participants
You can’t perform that action at this time.