Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: mod download doesn't allow insecure download #27332

Closed
lizihuai opened this issue Aug 29, 2018 · 56 comments
Closed

cmd/go: mod download doesn't allow insecure download #27332

lizihuai opened this issue Aug 29, 2018 · 56 comments

Comments

@lizihuai
Copy link

@lizihuai lizihuai commented Aug 29, 2018

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.11 linux/amd64

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

GO111MODULE=on go mod download

If possible, provide a recipe for reproducing the error.

error:

go: k8s.io/client-go@v8.0.0+incompatible: unrecognized import path "k8s.io/client-go" (https fetch: Get https://k8s.io/client-go?go-get=1: x509: certificate signed by unknown authority)
go: k8s.io/api@v0.0.0-20180824172530-dd5c735cbff9: unrecognized import path "k8s.io/api" (https fetch: Get https://k8s.io/api?go-get=1: x509: certificate signed by unknown authority)

A complete runnable program is good.
A link on play.golang.org is best.

What did you expect to see?

What did you see instead?

@oiooj
Copy link
Member

@oiooj oiooj commented Aug 29, 2018

@lizihuai Could you run this command and log it.

echo | openssl s_client -showcerts -servername k8s.io -connect k8s.io:443 2>/dev/null | openssl x509 -inform pem -noout -text

It seems that you have no letsencrypt's CA.

@lizihuai
Copy link
Author

@lizihuai lizihuai commented Aug 29, 2018

@oiooj
I run the command is failure,error message
unable to load certificate
140308543207312:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:707:Expecting: TRUSTED CERTIFICATE

Explain that I have no CA?

Why do I need a CA when I use go mod?

@lizihuai
Copy link
Author

@lizihuai lizihuai commented Aug 29, 2018

@oiooj but I use curl -kv https://k8s.io:443 is success

@fraenkel
Copy link
Contributor

@fraenkel fraenkel commented Aug 29, 2018

@lizihuai Drop the -k and the failure should occur.

@lizihuai
Copy link
Author

@lizihuai lizihuai commented Aug 29, 2018

@fraenkel Drop the -k is failure . how can I do ? Why can I access other https URLs? I can't access this.

@fraenkel
Copy link
Contributor

@fraenkel fraenkel commented Aug 29, 2018

Most OSes have added the LetsEncrypt CA as a trusted CA.
But you might be able to get away with a go get -insecure instead. I am not sure if go mod download is missing the -insecure flag.

@agnivade
Copy link
Contributor

@agnivade agnivade commented Aug 29, 2018

That is the exact issue - go mod download doesn't have an insecure flag.

go help mod download
usage: go mod download [-dir] [-json] [modules]

/cc @rsc @bcmills

@agnivade agnivade changed the title [go mod download] use go mod download pkg cmd/go:mod download doesn't allow insecure download Aug 29, 2018
@agnivade agnivade added this to the Go1.12 milestone Aug 29, 2018
@lizihuai
Copy link
Author

@lizihuai lizihuai commented Aug 30, 2018

@fraenkel
use go get insecure can download k8s.io.but the other problem can't download
https://gopkg.in/yaml.v2?go-get=1
@agnivade GO 1.11 can fix this problem ?
Why is there such a problem, what is the cause of this problem? thanks

@FiloSottile FiloSottile changed the title cmd/go:mod download doesn't allow insecure download cmd/go: mod download doesn't allow insecure download Aug 30, 2018
@bcmills bcmills added the modules label Sep 13, 2018
@gcstang
Copy link

@gcstang gcstang commented Sep 13, 2018

I have the same issue for a self-generated cert on my private repository when using
go mod tidy
go mod download
etc...

@agnivade
Copy link
Contributor

@agnivade agnivade commented Sep 14, 2018

Why is there such a problem, what is the cause of this problem? thanks

It is how https works. The client has to verify the cert returned by the server. We can explicitly choose to ignore validating the cert. go get has an option to allow that, go mod download doesn't. Hence the issue.

@gopherbot
Copy link

@gopherbot gopherbot commented Sep 17, 2018

Change https://golang.org/cl/135735 mentions this issue: cmd/go/internal/modcmd: mod download allow insecure download

@gcstang
Copy link

@gcstang gcstang commented Sep 17, 2018

I see in the recent change the flag has been added will this also allow it to pick up the setting in GO_FLAGS like other commands can?

i.e. GOFLAGS=-insecure

@bcmills
Copy link
Member

@bcmills bcmills commented Sep 17, 2018

@bcmills
Copy link
Member

@bcmills bcmills commented Sep 17, 2018

If the problem is that go get is missing some needed root certificate, I would rather we add a mechanism to tell go get about the missing certificate rather than ignore all of the ones it already has.

@bcmills
Copy link
Member

@bcmills bcmills commented Sep 17, 2018

(Or, to put it another way: I would prefer that we deprecate the -insecure flag rather than expanding its reach.)

@gcstang
Copy link

@gcstang gcstang commented Sep 17, 2018

I'm good either way, as long as I can add a self generated certificate for it to work.

@fearful-symmetry
Copy link

@fearful-symmetry fearful-symmetry commented Oct 3, 2018

Seconding this. My work has a number of internal gitlab instances with self-signed certs. An -insecure flag would be very helpful.

@sgoodrow
Copy link

@sgoodrow sgoodrow commented Nov 2, 2018

We are dealing with this problem as well. An -insecure flag would be very helpful.

@FiloSottile
Copy link
Member

@FiloSottile FiloSottile commented Nov 2, 2018

How many people would need the full -insecure shotgun, and why, as opposed to an environment variable to point to a private CA (cough) or self-signed certificate to trust?

@myitcv
Copy link
Member

@myitcv myitcv commented Nov 13, 2018

I guess an alternative here would be to have a GOPROXY implementation that handles this.

@bcmills bcmills modified the milestones: Go1.12, Go1.13 Nov 15, 2018
@daqingshu
Copy link

@daqingshu daqingshu commented Dec 3, 2018

How many people would need the full -insecure shotgun, and why, as opposed to an environment variable to point to a private CA (cough) or self-signed certificate to trust?

example1: in some company, use private Gitlab env, just use http and ssh, no https

@cvigo
Copy link

@cvigo cvigo commented Dec 3, 2018

I vote for this.

I filed a related issue this weekend #29059

@daqingshu
Copy link

@daqingshu daqingshu commented Dec 14, 2018

but they would have to deal with the certificates pain

Setting up a private CA is practically a breeze with mkcert or various other tools out there. The entire world is moving to https. There are various web features (like SWs) which are not available on http at all.

What @bcmills and @FiloSottile are trying to understand is why folks are unable to move to https. And so far the answer is that "certificate pain".

I may have minimal experience on this, so @FiloSottile can chime in, but installing a private CA takes no time at all. And for external servers, you have letsencrypt. Although I agree that one has to still ramp up on these tools and in some corporate environments, there are levels of red tape. So it is not always a one command install.

A tough call.

I can't convince IT (a private gitlab cluster with hundreds of servers deployed in different cities, I don't know what the actual risk on the production environment), just as I can't convince you.
glide can help me
go mod can't
Sorry, I shut up.

@cvigo
Copy link

@cvigo cvigo commented Dec 14, 2018

When you have hundreds of certificates (=> a few expiring every single day) and thousands of developers, then the breeze becomes a hurricane.

But if these certificates automatically update as required (which is the case with many LetsEncrypt integrations), then the cost is ~ zero. And with a bit of trivial scripting, it would be easy to automate the updating of non-private server certificates (via DNS challenge mode). And that's all before we consider wildcard certificates which might even make much of this discussion moot.

Good in theory. Never seen it in practice. Just as an example, I have 7 different private CA's installed and trusted in my laptop. Not sure who manages each. Security dept. sometimes answer "that one is not mine, try with XXX dept.". Not a definition of breeze

@bitfield
Copy link

@bitfield bitfield commented Dec 22, 2018

Seems to me this boils down to "Our Gitlab server has an invalid TLS certificate, so please disable security for every Go user in the world so we don't have to fix our certificate".

@cvigo
Copy link

@cvigo cvigo commented Dec 25, 2018

Who has said that?

@samart
Copy link

@samart samart commented Dec 26, 2018

We have mcafee web gateway in the middle which I think is causing this error and making go mod unusable. To get certs added to that infra (if that is a fix) would mean many discussion with clueless security teams that would then want to know how golang works and then likely create more roadblocks for us. Is an insecure option possible with go mod?

@bitfield
Copy link

@bitfield bitfield commented Dec 27, 2018

I vote to close this as "works as designed".

@samart
Copy link

@samart samart commented Dec 27, 2018

How about allowing an optional, non default —insecure flag? For those in environments where go mod won’t work. Seems like it would help some and hurt no one.

@gcstang
Copy link

@gcstang gcstang commented Dec 27, 2018

I have an alternative solution but I can understand the dilemma some of these developers find themselves in.

Maybe I don't understand why this is such an issue to add a flag to give support to the community that needs it in order to continue down the path that was given. Make this an optional flag or at least supply a fix that will allow those that need it as a way to progress that is realistic.

I've never seen such push back on something that would be so simple to address.
Is this push back just due to not wanting to do something or just to get developers of the Go project riled up? At this point it just seems antagonistic which is just the opposite of what I've seen and experienced in the Go community, I hope this simple request can be resolved in a friendlier manner.

@dawnhammond
Copy link

@dawnhammond dawnhammond commented Dec 27, 2018

All I wanted for Christmas was the --insecure flag.

@bitfield
Copy link

@bitfield bitfield commented Dec 28, 2018

From the discussion on the CL:

it sounds like most of the cases where this would be useful are cases where go get is missing a root certificate for the requested domain.

I'd prefer that we address that case directly rather than adding a bypass flag: we don't really want folks to put -insecure in their GOFLAGS and end up downloading and running untrusted code.

Nothing to do with antagonism or being unfriendly; it's a security issue. Using HTTPS links without actually checking the certificates gives a false sense of security. If you can't set up a verifiable TLS cert chain, you should be using a non-HTTPS URL, to make it clear that this is not a secure connection.

I've never seen such push back on something that would be so simple to address.

Then perhaps there's a good reason.

@samart
Copy link

@samart samart commented Dec 28, 2018

When not on the corporate network it works fine. When on the corporate network, we have mcafee web gateway in the middle which .. i think ... needs some cert installed or adds something to the chain.. What cert I don’t know. That’s my guess.

But this is huge issue for go development in my team and at my company. We are completely stuck whilst this back and forth happens.

@bitfield I don’t understand why an option would be so bad. You don’t have to use it and neither does anyone else for whom it’s working fine. At least we can code. Otherwise we have to change programming languages? That’s extreme.

note, the errors we are getting are on golang.org, etc, downloading crypto packages, etc (not k8s.io as in the original post).

@FiloSottile
Copy link
Member

@FiloSottile FiloSottile commented Dec 28, 2018

A global flag defeating all security is uncomfortable because users will naturally use it and go on with their day instead of reporting the specific reason they need it, which we might be able to address safely. Remember that modules support is still in experimental state, so now is the time to learn. For example:

  • if we are failing to recognize an installed root CA, like in #24652, we should fix that
  • if users need to use external root CAs more easily, a proposal to add an environment variable to add them to the pool without installing them might make sense
  • if internal insecure modules (http:// or git://) are widespread, we might consider a bypass list in a config file the same way we will have an exclude list for internal modules that should not be fetched from the mirror; this critically only defeats security for some modules, maybe the ones that are fetched over a VPN anyway; or maybe this need will be addressed by a GOPROXY implementation that will have much better flexibility in authentication methods than we could ever provide in the go tool.

If any of the above seem to match your use case (or not), please open a specific issue with as many details as possible of what you are trying to do and in what environment (not just the solution you want) and tag me and @bcmills. I am closing this issue to encourage per-scenario discussion, not because there are no valid use cases here.

(If instead you do indeed need to defeat all security because of some security appliance, sounds like you should bring that up as an issue with your vendor as soon as possible.)

@berniedurfee-ge
Copy link

@berniedurfee-ge berniedurfee-ge commented Mar 7, 2019

Disappointed to see this request being closed. Many large enterprises don't manage certificates ideally for a variety of reasons. Very often developers in those environments don't have enough authority or influence to change how those enterprises manage certificates.

I understand the reluctance to introduce a flag allowing users to disable part of TLS is not ideal, but reality is often not ideal. This is precisely why most tools include a -insecure flag of some sort to allow those in less than ideal environments to use the tool. Not including this flag in an effort to make the world safer is a noble ideal, but ultimately just makes life more difficult for those developers who don't have the ability to change their environments.

Hopefully this decision is reconsidered and the -insecure flag is added to go mod vendor. Tools like this should be focused on delivering functionality in any environment. Again, it's noble to take advantage of the leverage you have as tool maintainers to make the world a better place. However, it's also okay to allow institutions and users to decide for themselves how they want to manage risk.

@samart
Copy link

@samart samart commented Mar 7, 2019

@berniedurfee-ge We should write to the head of the golang project at google. Low level engineers dont see the big picture.

@chinglinwen
Copy link

@chinglinwen chinglinwen commented Jun 11, 2019

I don't know go mod download before, I just using go get most of the time

Is go mod download serve for different purpose than go get

If that so, maybe there's different constraint or assumption?

@chiefy
Copy link

@chiefy chiefy commented Aug 3, 2019

We have our own CA and must go through a corporate proxy. Unless I want to install our CA on every docker image I pull, I usually use an -insecure flag. The fact that go mod download doesn't support this is not cool. Now in order to run go mod download I have to disconnect from our VPN, run the command and then get back onto our VPN which is a waste of at least 2-3 minutes each time.

@andesli
Copy link

@andesli andesli commented Aug 7, 2019

need -insecure

@bcmills
Copy link
Member

@bcmills bcmills commented Aug 7, 2019

@andesli, see http://golang.org/wiki/NoPlusOne.

(Also note that this issue is closed, and we have no process in place to follow up on closed issues.)

@golang golang deleted a comment from ttaranov Sep 23, 2019
@FiloSottile
Copy link
Member

@FiloSottile FiloSottile commented Sep 23, 2019

This issue is closed, but as I said we do want to address the use case, even if not with a blunt -insecure flag. The third case in my comment is being expanded on in #32966. If that solution doesn't work for anyone, please do open a new issue.

@Haiyung
Copy link

@Haiyung Haiyung commented Nov 22, 2019

I vote for this.

I filed a related issue this weekend #29059

vote +1

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Nov 22, 2019

@Haiyung This issue is closed. Also, please see https://golang.org/wiki/NoPlusOne. Thanks.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet