New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Corrupt remote cache entry shouldn't fail the whole build #14693
Comments
Builds should be resilient against remote cache connectivity issues. This is a problem with unpacking the (successfully) downloaded cache entry. Can you please post a build scan or a full stacktrace? |
There is no build scan or full stacktrace available. I've enabled full stacktrace and will update the issue once it happens again. Regardless I think even a corrupted artifact shouldn't fail a build. 😅 |
Here is the stacktrace: Execution failed for task ':services:notifications:api:compileJava'.
> Failed to load cache entry for task ':services:notifications:api:compileJava'
* Try:
Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Exception is:
org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':services:notifications:api:compileJava'.
at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:38)
at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.executeTask(EventFiringTaskExecuter.java:77)
at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:55)
at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:52)
at org.gradle.internal.operations.DefaultBuildOperationExecutor$CallableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:409)
at org.gradle.internal.operations.DefaultBuildOperationExecutor$CallableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:399)
at org.gradle.internal.operations.DefaultBuildOperationExecutor$1.execute(DefaultBuildOperationExecutor.java:157)
at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:242)
at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:150)
at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:94)
at org.gradle.internal.operations.DelegatingBuildOperationExecutor.call(DelegatingBuildOperationExecutor.java:36)
at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:52)
at org.gradle.execution.plan.LocalTaskNodeExecutor.execute(LocalTaskNodeExecutor.java:41)
at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:372)
at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:359)
at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:352)
at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:338)
at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.lambda$run$0(DefaultPlanExecutor.java:127)
at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.execute(DefaultPlanExecutor.java:191)
at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.executeNextNode(DefaultPlanExecutor.java:182)
at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.run(DefaultPlanExecutor.java:124)
at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56)
Caused by: java.lang.RuntimeException: Failed to load cache entry for task ':services:notifications:api:compileJava'
at org.gradle.internal.execution.steps.CacheStep.lambda$executeWithCache$4(CacheStep.java:116)
at org.gradle.internal.Try$Failure.getOrMapFailure(Try.java:229)
at org.gradle.internal.execution.steps.CacheStep.executeWithCache(CacheStep.java:114)
at org.gradle.internal.execution.steps.CacheStep.execute(CacheStep.java:71)
at org.gradle.internal.execution.steps.CacheStep.execute(CacheStep.java:43)
at org.gradle.internal.execution.steps.StoreExecutionStateStep.execute(StoreExecutionStateStep.java:44)
at org.gradle.internal.execution.steps.StoreExecutionStateStep.execute(StoreExecutionStateStep.java:33)
at org.gradle.internal.execution.steps.RecordOutputsStep.execute(RecordOutputsStep.java:38)
at org.gradle.internal.execution.steps.RecordOutputsStep.execute(RecordOutputsStep.java:24)
at org.gradle.internal.execution.steps.SkipUpToDateStep.executeBecause(SkipUpToDateStep.java:92)
at org.gradle.internal.execution.steps.SkipUpToDateStep.lambda$execute$0(SkipUpToDateStep.java:85)
at org.gradle.internal.execution.steps.SkipUpToDateStep.execute(SkipUpToDateStep.java:55)
at org.gradle.internal.execution.steps.SkipUpToDateStep.execute(SkipUpToDateStep.java:39)
at org.gradle.internal.execution.steps.ResolveChangesStep.execute(ResolveChangesStep.java:76)
at org.gradle.internal.execution.steps.ResolveChangesStep.execute(ResolveChangesStep.java:37)
at org.gradle.internal.execution.steps.legacy.MarkSnapshottingInputsFinishedStep.execute(MarkSnapshottingInputsFinishedStep.java:36)
at org.gradle.internal.execution.steps.legacy.MarkSnapshottingInputsFinishedStep.execute(MarkSnapshottingInputsFinishedStep.java:26)
at org.gradle.internal.execution.steps.ResolveCachingStateStep.execute(ResolveCachingStateStep.java:94)
at org.gradle.internal.execution.steps.ResolveCachingStateStep.execute(ResolveCachingStateStep.java:49)
at org.gradle.internal.execution.steps.CaptureStateBeforeExecutionStep.execute(CaptureStateBeforeExecutionStep.java:79)
at org.gradle.internal.execution.steps.CaptureStateBeforeExecutionStep.execute(CaptureStateBeforeExecutionStep.java:53)
at org.gradle.internal.execution.steps.ValidateStep.execute(ValidateStep.java:74)
at org.gradle.internal.execution.steps.SkipEmptyWorkStep.lambda$execute$2(SkipEmptyWorkStep.java:78)
at org.gradle.internal.execution.steps.SkipEmptyWorkStep.execute(SkipEmptyWorkStep.java:78)
at org.gradle.internal.execution.steps.SkipEmptyWorkStep.execute(SkipEmptyWorkStep.java:34)
at org.gradle.internal.execution.steps.legacy.MarkSnapshottingInputsStartedStep.execute(MarkSnapshottingInputsStartedStep.java:39)
at org.gradle.internal.execution.steps.LoadExecutionStateStep.execute(LoadExecutionStateStep.java:40)
at org.gradle.internal.execution.steps.LoadExecutionStateStep.execute(LoadExecutionStateStep.java:28)
at org.gradle.internal.execution.impl.DefaultWorkExecutor.execute(DefaultWorkExecutor.java:33)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeIfValid(ExecuteActionsTaskExecuter.java:195)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:187)
at org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:114)
at org.gradle.api.internal.tasks.execution.FinalizePropertiesTaskExecuter.execute(FinalizePropertiesTaskExecuter.java:46)
at org.gradle.api.internal.tasks.execution.ResolveTaskExecutionModeExecuter.execute(ResolveTaskExecutionModeExecuter.java:62)
at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:57)
at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:56)
at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:36)
at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.executeTask(EventFiringTaskExecuter.java:77)
at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:55)
at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:52)
at org.gradle.internal.operations.DefaultBuildOperationExecutor$CallableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:409)
at org.gradle.internal.operations.DefaultBuildOperationExecutor$CallableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:399)
at org.gradle.internal.operations.DefaultBuildOperationExecutor$1.execute(DefaultBuildOperationExecutor.java:157)
at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:242)
at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:150)
at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:94)
at org.gradle.internal.operations.DelegatingBuildOperationExecutor.call(DelegatingBuildOperationExecutor.java:36)
at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:52)
at org.gradle.execution.plan.LocalTaskNodeExecutor.execute(LocalTaskNodeExecutor.java:41)
at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:372)
at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:359)
at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:352)
at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:338)
at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.lambda$run$0(DefaultPlanExecutor.java:127)
at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.execute(DefaultPlanExecutor.java:191)
at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.executeNextNode(DefaultPlanExecutor.java:182)
at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.run(DefaultPlanExecutor.java:124)
at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56)
Caused by: org.gradle.api.GradleException: Build cache entry 72548751717cc3d72d4101ddf794a3ef from remote build cache is invalid
at org.gradle.caching.internal.controller.DefaultBuildCacheController.lambda$load$0(DefaultBuildCacheController.java:120)
at org.gradle.caching.local.internal.DefaultBuildCacheTempFileStore.withTempFile(DefaultBuildCacheTempFileStore.java:46)
at org.gradle.caching.internal.controller.DefaultBuildCacheController.load(DefaultBuildCacheController.java:112)
at org.gradle.internal.execution.steps.CacheStep.lambda$executeWithCache$0(CacheStep.java:78)
at org.gradle.internal.Try.ofFailable(Try.java:39)
at org.gradle.internal.execution.steps.CacheStep.executeWithCache(CacheStep.java:77)
... 57 more
Caused by: java.io.UncheckedIOException: java.io.EOFException: Unexpected end of ZLIB input stream
at org.gradle.caching.internal.controller.DefaultBuildCacheController$Unpack$1.run(DefaultBuildCacheController.java:152)
at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:395)
at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:387)
at org.gradle.internal.operations.DefaultBuildOperationExecutor$1.execute(DefaultBuildOperationExecutor.java:157)
at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:242)
at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:150)
at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:84)
at org.gradle.internal.operations.DelegatingBuildOperationExecutor.run(DelegatingBuildOperationExecutor.java:31)
at org.gradle.caching.internal.controller.DefaultBuildCacheController$Unpack.execute(DefaultBuildCacheController.java:146)
at org.gradle.caching.internal.controller.DefaultBuildCacheController.lambda$load$0(DefaultBuildCacheController.java:118)
... 62 more
Caused by: java.io.EOFException: Unexpected end of ZLIB input stream
at org.apache.commons.compress.archivers.tar.TarArchiveInputStream.read(TarArchiveInputStream.java:639)
at org.gradle.internal.hash.DefaultStreamHasher.doHash(DefaultStreamHasher.java:52)
at org.gradle.internal.hash.DefaultStreamHasher.hashCopy(DefaultStreamHasher.java:43)
at org.gradle.caching.internal.packaging.impl.TarBuildCacheEntryPacker.unpackFile(TarBuildCacheEntryPacker.java:260)
at org.gradle.caching.internal.packaging.impl.TarBuildCacheEntryPacker.unpackDirectoryTree(TarBuildCacheEntryPacker.java:299)
at org.gradle.caching.internal.packaging.impl.TarBuildCacheEntryPacker.unpackTree(TarBuildCacheEntryPacker.java:255)
at org.gradle.caching.internal.packaging.impl.TarBuildCacheEntryPacker.unpack(TarBuildCacheEntryPacker.java:199)
at org.gradle.caching.internal.packaging.impl.TarBuildCacheEntryPacker.unpack(TarBuildCacheEntryPacker.java:161)
at org.gradle.caching.internal.packaging.impl.GZipBuildCacheEntryPacker.unpack(GZipBuildCacheEntryPacker.java:49)
at org.gradle.caching.internal.controller.impl.DefaultBuildCacheCommandFactory$LoadCommand.load(DefaultBuildCacheCommandFactory.java:93)
at org.gradle.caching.internal.controller.DefaultBuildCacheController$Unpack$1.run(DefaultBuildCacheController.java:150)
... 71 more |
Multiple Gradle builds with Kotlin compiler in CI server fail with this error frequently. Is there any possible workaround? |
@socar-jian what are you using for remote caching? Gradle Enterprise? |
I think my case is more like #15050, not remote-cache-related. I'm sorry for confusing. |
I have the same problem. In my case it's caused when my login session of my corporate network where the cache is hosted has expired. The expired single sign on cookie is rejected, followed by almost the same stacktrace as @fkorotkov posted (differs only in the compression method: |
Beam is hitting this issue as well. |
Also seeing these errors in my build, not much details:
|
@tvalentyn can you give some more context please? What Gradle version, what is the exact message? One of the reasons we are typically unable to load stuff from cache is that we cannot delete the output directory. The error message is arguably obscure, and we are going to improve that in 7.5, see #17208. |
Currently we use Gradle 7.3.2, looking forward to 7.5. |
Looks like multiple builds colliding: https://scans.gradle.com/s/tetxtcns3nmge/failure#1
|
We are going to improve the rendering of build cache problems like this in Gradle 7.5: #19672 |
What is the expected behavior here? I guess I would hope that (1) multiple builds share the cache, since that's sort of the whole point (2) true concurrent conflicts would resolve either by blocking or by treating it as a cache miss. |
That's a good question. We haven't been able to reproduce this problem yet. Can you please follow up on #8375? It would be good to see what the other PID is doing. I know that at least on Windows we've seen virus scanners keeping ahold of the lock, but I don't think this is the case on the Linux build here. |
We run these tests on Jenkins workers. Each worker has 4 slots, so 2 suites can run concurrently, triggerring 2 |
would that be consistent with the failure mode? |
Just want to chime in here -- we have started seeing this occasionally in our CI builds. Twice in the last week, users have reported a similar issue with task output from this plugin https://github.com/autonomousapps/dependency-analysis-android-gradle-plugin Both times it was the
and
|
Seeing similar issues when running parallel stages in Jenkins. It would be great if Gradle could handle this errors in a way that doesn't break the CI builds since it produces clutter and hides true errors. Similarly to what has been stated above, we have a jenkins node with multiple executors to allow for parallel execution, we attempted to remediate the issues by giving each task a dedicated cache directory based on the executor and the stage its running (i.e. Example failures as follows:
|
Right now when a local Gradle build fails to download a cache entry or there is some other issue the whole build fails with the following message:
Expected Behavior
This should be treated like a cache miss and build should just recompile it.
Current Behavior
The whole build fails.
The text was updated successfully, but these errors were encountered: