Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: add GOPROXY support for blob storage #32433

Closed
carnott-snap opened this issue Jun 4, 2019 · 7 comments
Closed

cmd/go: add GOPROXY support for blob storage #32433

carnott-snap opened this issue Jun 4, 2019 · 7 comments

Comments

@carnott-snap
Copy link

@carnott-snap carnott-snap commented Jun 4, 2019

What did you do?

GOPROXY=gs://goproxy go get golang.org/x/xerrors

What did you see instead?

go get golang.org/x/xerrors: invalid $GOPROXY setting: malformed URL or invalid scheme (must be http, https, file)

What would you like to see changed?

Currently GOPROXY supports http, https, file protocols. It would be nice if support could be added for blob storage mechanisms like GCS, S3, etc.

@hyangah

This comment has been minimized.

Copy link
Contributor

@hyangah hyangah commented Jun 4, 2019

Why not running a static webserver in front of blob storage (e.g. https://cloud.google.com/storage/docs/hosting-static-website) or accessing the buckets or blob storages using the default web server the service providers provide?

@carnott-snap

This comment has been minimized.

Copy link
Author

@carnott-snap carnott-snap commented Jun 4, 2019

That would certainly be a stopgap, however it adds extra infrastructure and would not allow you to use default credentials for invisible authentication. Other workarounds could involve FUSE, or some GCS proxy, but it would be nice to have native support. Some of this desire comes out of the lack of GOAUTH support, see #26232, however it seems somewhat orthogonal.

This is something that other build systems have added support for, e.g. gradle. While it would add provider specific authentication logic and pull in external dependencies, this could be mitigated by shelling out to a binary, a la git helpers, and allow for more protocol flexibility.

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Jun 6, 2019

Native gs:// support in the go command would require the go command to link in the various blob-storage clients for everybody, not just the folks who need it.

I agree with @hyangah that a blob-to-HTTP proxy seems like the right solution. It should be fairly trivial to write one that you can run locally on your machine.

If you really don't want to write your own, you could also consider a feature request with one of the existing private-proxy implementations.

CC @jayconrod @marwan-at-work.

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Jun 6, 2019

Shelling out to a binary doesn't seem particularly simpler than having a long-running local proxy, and an ephemeral binary couldn't easily cache the connections to the blob-storage backend. It seems better to stick to the mechanism we already have (GOPROXY).

@bcmills bcmills closed this Jun 6, 2019
@carnott-snap

This comment has been minimized.

Copy link
Author

@carnott-snap carnott-snap commented Jan 13, 2020

@bcmills, considering that work on #26232 has stalled, and custom transport providers could workaround the auth issue, is there interest in pursuing a go-proxy-<transport> helper implementation? (where transport:// is provided to GOPROXY)

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Jan 22, 2020

Is there interest? I have no idea; that's part of why #26232 has stalled. (When we fixed the .netrc bug, there seemed to be a lot less interest in alternative auth mechanisms.)

If you want to take a stab at an alternative-transport proxy, I'd be happy to review the code.

@marwan-at-work

This comment has been minimized.

Copy link
Contributor

@marwan-at-work marwan-at-work commented Jan 22, 2020

@bcmills my 2 cents is that I still believe #26232 would be valuable to a number of use cases. Even if it's just a slimmed down version where you can tell the Go command to pass static headers. Happy to continue the conversation there and try to contribute that feature 😄

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.