Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CI test builds failing with "invalid action cache entry" #20599

Closed
mkbehr opened this issue Mar 30, 2022 · 6 comments · Fixed by #20741
Closed

CI test builds failing with "invalid action cache entry" #20599

mkbehr opened this issue Mar 30, 2022 · 6 comments · Fixed by #20741
Labels
area/build bug

Comments

@mkbehr
Copy link
Contributor

mkbehr commented Mar 30, 2022 •
edited
Loading

Title: CI test builds failing with "invalid action cache entry"

Description:
I'm seeing CI runs fail to build changed files with messages like:

ERROR: /source/test/extensions/filters/http/cache/BUILD:71:24: Compiling test/extensions/filters/http/cache/cache_filter_test.cc failed: (Exit 34): Invalid action cache entry 0da235ebddfcb79aa14ea81f1aa904e9fdf4b016e4fb535094a666317649c5b5: expected output test/extensions/filters/http/cache/_objs/cache_filter_test/cache_filter_test.pic.o does not exist.

I've seen a couple different PRs with this problem: my results here for #20593 and another failure here for #20581.

There's a recent Bazel bug with similar error messages, so we can probably fix this by pulling in the fix there.

Repro steps:
Run the linux_x64 gcc presubmit on #20593.
@mkbehr mkbehr added bug triage Issue requires triage labels Mar 30, 2022
@mkbehr mkbehr mentioned this issue Mar 30, 2022
Closed
@keith
Copy link
Member

keith commented Mar 30, 2022

According to that bazel issue it is fixed in bazel 5.1, which I updated main too yesterday fd92486 🤔

@keith
Copy link
Member

keith commented Mar 30, 2022

In case the CI log rotates here's more of the stack:

ERROR: /source/test/extensions/filters/http/cache/BUILD:71:24: Compiling test/extensions/filters/http/cache/cache_filter_test.cc failed: (Exit 34): Invalid action cache entry 0da235ebddfcb79aa14ea81f1aa904e9fdf4b016e4fb535094a666317649c5b5: expected output test/extensions/filters/http/cache/_objs/cache_filter_test/cache_filter_test.pic.o does not exist.
java.io.IOException: Invalid action cache entry 0da235ebddfcb79aa14ea81f1aa904e9fdf4b016e4fb535094a666317649c5b5: expected output test/extensions/filters/http/cache/_objs/cache_filter_test/cache_filter_test.pic.o does not exist.
	at com.google.devtools.build.lib.remote.RemoteExecutionService.downloadOutputs(RemoteExecutionService.java:1031)
	at com.google.devtools.build.lib.remote.RemoteSpawnRunner.downloadAndFinalizeSpawnResult(RemoteSpawnRunner.java:367)
	at com.google.devtools.build.lib.remote.RemoteSpawnRunner.lambda$exec$2(RemoteSpawnRunner.java:284)
	at com.google.devtools.build.lib.remote.Retrier.execute(Retrier.java:244)
	at com.google.devtools.build.lib.remote.RemoteRetrier.execute(RemoteRetrier.java:125)
	at com.google.devtools.build.lib.remote.RemoteRetrier.execute(RemoteRetrier.java:114)
	at com.google.devtools.build.lib.remote.RemoteSpawnRunner.exec(RemoteSpawnRunner.java:239)

@keith keith mentioned this issue Mar 30, 2022
Closed
@keith
Copy link
Member

keith commented Mar 31, 2022

I'm wondering now if this degraded in Bazel 5.1 and we should downgrade back to 5.0 🤔

@daixiang0 daixiang0 added area/build and removed triage Issue requires triage labels Mar 31, 2022
@daixiang0
Copy link
Member

I think we should pin a version rather than HEAD to avoid this case.

keith added a commit to keith/envoy that referenced this issue Mar 31, 2022
@keith
Revert "build: Update to bazel 5.1.0"
ab70087 
This avoids this issue envoyproxy#20599
and I do not believe we were hit by the original issue that introduced
this regression.

There is a fix in bazelbuild/bazel#15151 that
should get us past this in bazel 5.2+. It seems this warning was just a
red herring and actually meant your actions failed.

This reverts commit fd92486.

Signed-off-by: Keith Smiley <keithbsmiley@gmail.com>
@keith keith mentioned this issue Mar 31, 2022
Merged
@keith
Copy link
Member

keith commented Mar 31, 2022

#20615

keith added a commit that referenced this issue Mar 31, 2022
@keith
Revert "build: Update to bazel 5.1.0"
96c3178 
This avoids this issue #20599
and I do not believe we were hit by the original issue that introduced
this regression.

There is a fix in bazelbuild/bazel#15151 that
should get us past this in bazel 5.2+. It seems this warning was just a
red herring and actually meant your actions failed.

This reverts commit fd92486.

Signed-off-by: Keith Smiley <keithbsmiley@gmail.com>
@keith keith mentioned this issue Apr 8, 2022
Merged
@keith
Copy link
Member

keith commented Apr 8, 2022

#20741

keith added a commit to keith/envoy that referenced this issue Apr 9, 2022
@keith
bazel: Update to 5.1.1
7a2621a 
We tried updating to 5.1.0 which fixed some remote cache issues, but
that had a regression where all action failures would appear to be
remote cache flakes instead, this was fixed in 5.1.1

https://github.com/bazelbuild/bazel/releases/tag/5.1.1

Fixes: envoyproxy#20599

Signed-off-by: Keith Smiley <keithbsmiley@gmail.com>
keith added a commit that referenced this issue Apr 11, 2022
@keith
bazel: Update to 5.1.1
0cb4ec8 
We tried updating to 5.1.0 which fixed some remote cache issues, but
that had a regression where all action failures would appear to be
remote cache flakes instead, this was fixed in 5.1.1

https://github.com/bazelbuild/bazel/releases/tag/5.1.1

Fixes: #20599

Signed-off-by: Keith Smiley <keithbsmiley@gmail.com>
vehre-x41 pushed a commit to vehre-x41/envoy that referenced this issue Apr 19, 2022
@keith @vehre-x41
bazel: Update to 5.1.1
80635c9 
We tried updating to 5.1.0 which fixed some remote cache issues, but
that had a regression where all action failures would appear to be
remote cache flakes instead, this was fixed in 5.1.1

https://github.com/bazelbuild/bazel/releases/tag/5.1.1

Fixes: envoyproxy#20599

Signed-off-by: Keith Smiley <keithbsmiley@gmail.com>

Signed-off-by: Andre Vehreschild <vehre@x41-dsec.de>
ravenblackx pushed a commit to ravenblackx/envoy that referenced this issue Jun 8, 2022
@keith @ravenblackx
Revert "build: Update to bazel 5.1.0"
56ba6be 
This avoids this issue envoyproxy#20599
and I do not believe we were hit by the original issue that introduced
this regression.

There is a fix in bazelbuild/bazel#15151 that
should get us past this in bazel 5.2+. It seems this warning was just a
red herring and actually meant your actions failed.

This reverts commit fd92486.

Signed-off-by: Keith Smiley <keithbsmiley@gmail.com>
ravenblackx pushed a commit to ravenblackx/envoy that referenced this issue Jun 8, 2022
@keith @ravenblackx
bazel: Update to 5.1.1
da5ef0b 
We tried updating to 5.1.0 which fixed some remote cache issues, but
that had a regression where all action failures would appear to be
remote cache flakes instead, this was fixed in 5.1.1

https://github.com/bazelbuild/bazel/releases/tag/5.1.1

Fixes: envoyproxy#20599

Signed-off-by: Keith Smiley <keithbsmiley@gmail.com>
phlax pushed a commit to phlax/envoy that referenced this issue Nov 17, 2022
@keith @phlax
bazel: Update to 5.1.1
8b5a9f5 
We tried updating to 5.1.0 which fixed some remote cache issues, but
that had a regression where all action failures would appear to be
remote cache flakes instead, this was fixed in 5.1.1

https://github.com/bazelbuild/bazel/releases/tag/5.1.1

Fixes: envoyproxy#20599

Signed-off-by: Keith Smiley <keithbsmiley@gmail.com>

Signed-off-by: Ryan Northey <ryan@synca.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/build bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

bazel: Update to 5.1.1 keith/envoy
3 participants
@mkbehr @keith @daixiang0