Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

E2E tests fail on Prow with newer ko versions #2479

Closed
adrcunha opened this issue Nov 13, 2018 · 19 comments
Closed

E2E tests fail on Prow with newer ko versions #2479

adrcunha opened this issue Nov 13, 2018 · 19 comments
Assignees
Labels
area/test-and-release It flags unit/e2e/conformance/perf test issues for product features kind/bug Categorizes issue or PR as related to a bug.

Comments

@adrcunha
Copy link
Contributor

When making Prow use go 1.11.1 and Cloud SDK 225, E2E tests started failing with different errrors:

https://gubernator.knative.dev/build/knative-prow/pr-logs/pull/knative_serving/2468/pull-knative-serving-integration-tests/1062426675620352001/

W1113 19:37:16.237] 2018/11/13 19:37:16 error processing import paths in "config/webhook.yaml": BLOB_UNKNOWN: "Manifest \"sha256:e392d2a37aeba7425e33c18b57a1bf4a4663e35f5d67445727cafe2527edca6f\" blob \"sha256:f39c3f3ec4ec5beb7fbe9400562f4b186b5e784501d5d79f537d1ad376057b61\" unknown to registry."
...
I1113 19:37:40.400] >> Patching Istio
W1113 19:37:40.511] Error from server (NotFound): horizontalpodautoscalers.autoscaling "knative-ingressgateway" not found
I1113 19:37:40.612] ERROR: Knative Serving installation failed

https://gubernator.knative.dev/build/knative-prow/pr-logs/pull/knative_serving/2437/pull-knative-serving-integration-tests/1062427808908709890/

W1113 19:41:33.420] 2018/11/13 19:41:33 Unexpected error running "go build": exit status 2
W1113 19:41:33.421] # github.com/knative/serving/pkg/apis/serving/v1alpha1
W1113 19:41:33.422] pkg/apis/serving/v1alpha1/revision_validation.go:66:11: undefined: apis.ErrOutOfBoundsValue
W1113 19:41:33.422] 2018/11/13 19:41:33 error processing import paths in "config/activator.yaml": exit status 2
...
I1113 19:41:57.178] >> Patching Istio
W1113 19:41:57.282] Error from server (NotFound): horizontalpodautoscalers.autoscaling "knative-ingressgateway" not found
I1113 19:41:57.383] ERROR: Knative Serving installation failed

https://gubernator.knative.dev/build/knative-prow/logs/ci-knative-serving-continuous/1062420259731935234

W1113 19:13:15.335] 2018/11/13 19:13:15 error processing import paths in "config/999-cache.yaml": BLOB_UNKNOWN: "Manifest \"sha256:496427954201b3d3bb709160cc9bb44d87f0946ab6526e52cdd4745321c89e0f\" blob \"sha256:2992b18231ba9f736eadad4889e6cdcb1cc5e80b1646fc4d61a5d39afe09d391\" unknown to registry."
...
I1113 19:13:39.642] >> Patching Istio
W1113 19:13:39.758] Error from server (NotFound): horizontalpodautoscalers.autoscaling "knative-ingressgateway" not found
I1113 19:13:39.859] ERROR: Knative Serving installation failed

To reproduce it, you must start a Prow test job using prow-tests:v20181113-07b942c-dirty.

@knative-prow-robot knative-prow-robot added area/test-and-release It flags unit/e2e/conformance/perf test issues for product features kind/bug Categorizes issue or PR as related to a bug. labels Nov 13, 2018
@adrcunha
Copy link
Contributor Author

Actually, the last link points to a Prow job using go 1.10 and cloud SDK 221. The root cause for these errors might be unrelated to Prow or the go/cloud version.

@adrcunha
Copy link
Contributor Author

The "error processing import paths" issue still happpens with go 1.11 and Cloud SDK 228:

W1214 23:25:15.835] 2018/12/14 23:25:15 error processing import paths in "config/config-controller.yaml": BLOB_UNKNOWN: "Manifest \"sha256:49f2e2fc674322ac1aa1385f19bdacd26f079a4b7036a5ffd34a503d83ea9e06\" blob \"sha256:85cffe608e625a1a4f494ad319d83f9b6117e6f57dd838c25fcc52f10c3d03e2\" unknown to registry."

https://storage.googleapis.com/knative-prow/pr-logs/pull/knative_serving/2725/pull-knative-serving-integration-tests/1073718706254647296/build-log.txt

@srinivashegde86
Copy link
Contributor

@adrcunha
Copy link
Contributor Author

This is actually unrelated to Cloud SDK and go version. It failed for the same reason with Cloud SDK 221 and go 1.10:

https://gubernator.knative.dev/build/knative-prow/pr-logs/pull/knative_serving/2725/pull-knative-serving-integration-tests/1074773657357651968/

I'm now suspicious of ko.

@adrcunha
Copy link
Contributor Author

adrcunha commented Dec 17, 2018

prow-tests image v20181025-0fcaba1 works fine. It uses Cloud SDK 221 and go 1.10:

https://prow.knative.dev/log?job=pull-knative-serving-integration-tests&id=1074778816099913728

@jonjohnsonjr
Copy link
Contributor

It looks like this is happening with github.com/knative/serving/cmd/queue a lot.

Two things I've noticed:

  1. That package has symlinks in kodata. We saw another failure in build-pipeline with the same for the webhook package.
  2. That package is being built and published twice for serving: here and here.

I think there might be several issues, but one of them is probably symlink related + a race.

Also, it seems that invoking go build concurrently isn't safe, which is exactly what ko does, but I would be surprised if we were downloading modules because we vendor everything... right?

@mattmoor any thoughts?

@adrcunha adrcunha changed the title E2E tests fail on Prow with go 1.11 / cloud SDK 225 E2E tests fail on Prow with newer ko versions Dec 18, 2018
@adrcunha
Copy link
Contributor Author

Nailed down the culprit to ko. Details and logs in #2745. The culprit seems to be google/go-containerregistry#297

Jon, you authored that PR and has way more context than me, so I'm assigning this issue to you.

/assign @jonjohnsonjr

@jonjohnsonjr
Copy link
Contributor

@adrcunha thank you for narrowing it down to that commit, that's really helpful. This is interesting because I've suspected something about the test environment might be causing an issue (which is why I can't reproduce it locally). Is it possible to get a dump of the environment variables?

I suspect there's a bug/race somewhere that involves all of this + the symlinks stuff in my comment:

And I suspect there's an environment difference between {prow, me, that commit} that triggers the bug/race in our test environment but not when run locally...

@adrcunha
Copy link
Contributor Author

@jonjohnsonjr environment variables are dumped here:

https://prow.knative.dev/log?job=pull-knative-serving-integration-tests&id=1075118053097738241

Search for @@@, they're right after the first occurrence.

@jonjohnsonjr
Copy link
Contributor

I have a theory that google/go-containerregistry#297 has a hard requirement on a later version of kubectl than we're using, perhaps?

I1218 19:58:36.888] >> kubectl version
I1218 19:58:36.985] Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.7", GitCommit:"0c38c362511b20a098d7cd855f1314dad92c2780", GitTreeState:"clean", BuildDate:"2018-08-20T10:09:03Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
I1218 19:58:36.985] Server Version: version.Info{Major:"1", Minor:"9+", GitVersion:"v1.9.7-gke.11", GitCommit:"dc4f6dda6a08aae2108d7a7fdc2a44fa23900f4c", GitTreeState:"clean", BuildDate:"2018-11-10T20:22:02Z", GoVersion:"go1.9.3b4", Compiler:"gc", Platform:"linux/amd64"}

Might be related to google/go-containerregistry#328 as well.

@jonjohnsonjr
Copy link
Contributor

I believe this was fixed by google/go-containerregistry#330

@jonjohnsonjr
Copy link
Contributor

/assign @adrcunha

@adrcunha
Copy link
Contributor Author

@jonjohnsonjr the newest ko doesn't throw "error processing import paths" anymore, but the E2E tests are failing consistently with

W1222 02:12:32.821] unable to recognize "/tmp/tmp.HqUwlDylgM/release-no-mon.yaml": no matches for kind "Image" in version "caching.internal.knative.dev/v1alpha1"

e.g. https://gubernator.knative.dev/build/knative-prow/pr-logs/pull/knative_serving/2745/pull-knative-serving-integration-tests/1076297323824287744/

Any clues?

@tcnghia
Copy link
Contributor

tcnghia commented Dec 22, 2018

@adrcunha this can be fixed by retrying. It's a race condition of kubectl.

@srinivashegde86
Copy link
Contributor

We should add the retry even to our current prow job setup. I sometimes see this error during the current e2e setup as well.

@adrcunha
Copy link
Contributor Author

I'm not convinced that this is a race condition of kubectl. The E2E tests failed 3 times in a row for the same reason. It doesn't happen with the older version of ko, as you can see in the CI flow.

@adrcunha
Copy link
Contributor Author

Actually, this is the real culprit:

I1218 06:48:00.380] >> Creating test resources (test/config/)
I1218 06:48:00.391] ERROR: Knative Serving installation failed

@adrcunha
Copy link
Contributor Author

#2745 confirmed that the issues was fixed by google/go-containerregistry#330.

@adrcunha
Copy link
Contributor Author

Sample job using the latest ko:

https://prow.knative.dev/log?job=pull-knative-serving-integration-tests&id=1079514393449861120

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/test-and-release It flags unit/e2e/conformance/perf test issues for product features kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Development

No branches or pull requests

6 participants