go1.18: [Tracker] go1.18 breaks CSR validation (fix pending in go1.18.1)

[MadhavJivrajani]

Which jobs are failing?

https://testgrid.k8s.io/sig-release-master-blocking#integration-master

Which tests are failing?

https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/108905/pull-kubernetes-integration/1506459579091783680
an instance of a failure, the tests here are the ones failing

Since when has it been failing?

My guess is since #108870 was merged

Testgrid link

https://testgrid.k8s.io/sig-release-master-blocking#integration-master

Reason for failure (if possible)

https://tip.golang.org/doc/go1.18#sha1

Anything else we need to know?

No response

Relevant SIG(s)

/sig testing

[MadhavJivrajani]

/priority critical-urgent
/cc @dims @aojea

[nikhita]

https://tip.golang.org/doc/go1.18#sha1

The failures for https://testgrid.k8s.io/sig-release-master-blocking#integration-master look like they are related to this.

Example below:

x509: cannot verify signature: insecure algorithm SHA1-RSA (temporarily override with GODEBUG=x509sha1=1)

[MadhavJivrajani]

/assign

[aojea]

So, presubmits jobs didn't catch these things?

[MadhavJivrajani]

A temporary fix, quick fix would be setting the env variable as suggested:

GODEBUG=x509sha1=1

Continuing to look
/sig auth
(for visibility)

[MadhavJivrajani]

So, presubmits jobs didn't catch these things?

yeah, I'm wondering this too

[aojea]

it seems golang has to do a minor release golang/go#41682 (comment), they are being too strict and sha1 in signatures seems to be allowed

https://go-review.googlesource.com/c/go/+/394294/3/src/crypto/x509/x509.go

[aojea]

/cc @liggitt

[dims]

presubmits were using image based on golang 1.17, so we hit it AFTER the 1.18 in k/k merged and then the master image was updated from 1.17 to 1.18

[MadhavJivrajani]

The CI failures should now be fixed (temporarily) on account of: #108901
Thanks Dims!

Not closing this out as we still need to fix things (regenerate hard-coded certs) in order to be fully back in business.

[MadhavJivrajani]

/remove-priority critical-urgent
/priority important-soon

[liggitt]

we still need to fix things (regenerate hard-coded certs) in order to be fully back in business

Actually, I think the CSR failures are a go bug we need to wait for go1.18.1 for

[dims]

it looks like all the integration test failures are for CSR objects, which will be resolved in go1.18.1

marking this as a blocker for 1.24

@liggitt do we have commitment from the golang team to ship go1.18.1 with the fix we need in time for 1.24? (what are our chances?)

[liggitt]

I don't know, but given it is a recovery release fixing a regression, I would expect it sooner than the normal ~month cadence

[dims]

Thanks @liggitt ! Looks like the backport issue is here: golang/go#51852

[dims]

@MadhavJivrajani do you wanna open a CL in golang repo? golang/go#51852 (comment)

[liggitt]

don't open a new CL, the one against master is under review and will be picked once ready

[dims]

@liggitt thanks. the linked comment caused me to wonder!

[dims]

So for folks following along:

• https://go-review.googlesource.com/c/go/+/394294/ needs to land first in golang master
• Then a backport CL has to be opened (by a bot?) for 1.18 branch which will get linked to crypto/x509: reject SHA-1 signatures in Verify [1.18 backport] golang/go#51852
• then a release has to be made by golang (looks like 1.18.1 which is expected in 3-4 days may not have the fix above)
• then we test the fix by dropping the temporary changes we made ( https://cs.k8s.io/?q=x509sha1%3D1&i=nope&files=&excludeFiles=&repos= )
• then we cut v1.24.0 final

sounds right @liggitt ?

[liggitt]

golang master CL merged, pick open to release-1.18 https://go-review.googlesource.com/c/go/+/398074

[dims]

for this one ... i want to be proven wrong :) (i said, it may not go into 1.18.1 on record!)

[liggitt]

https://go-review.googlesource.com/c/go/+/398074 merged to go 1.18 branch, appears it will be in go1.18.1

[dims]

/meow
/woof

[k8s-ci-robot]

@dims:

In response to this:

/meow
/woof

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-ci-robot]

@dims:

In response to this:

/meow
/woof

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[MadhavJivrajani]

Folks, Go 1.18.1 is postponed to 12th April.
Please see: https://groups.google.com/g/golang-announce/c/vtbMjE04kPk/m/TY7AIrwSCgAJ?utm_medium=email&utm_source=footer
cc @palnabarun @dims

[pacoxu]

https://github.com/golang/go/releases/tag/go1.18.1 is out.
https://go.googlesource.com/go/+/refs/tags/go1.18.1

[pacoxu]

kubernetes/release#2499 will update the dependency

[dims]

fixed k/k last remaining problem with 1.18.1 in #109312

/close

[k8s-ci-robot]

@dims: Closing this issue.

In response to this:

fixed k/k last remaining problem with 1.18.1 in #109312

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.