Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build cache miss when jars have been hashed with different normalization configuration #16144

Closed
ghale opened this issue Feb 13, 2021 · 3 comments · Fixed by #16145
Closed

Build cache miss when jars have been hashed with different normalization configuration #16144

ghale opened this issue Feb 13, 2021 · 3 comments · Fixed by #16145
Assignees
Labels
a:bug affects-version:6.8 @execution in:core
Milestone

Comments

@ghale
Copy link
Member

@ghale ghale commented Feb 13, 2021

We've received multiple reports of cache misses on runtime normalization input properties for jar files that should be identical. At least one reporter confirmed that using a different gradle user home resolved the issue. It turns out some of the changes to runtime classpath normalization in 6.8 don't accurately capture changes to the normalization configuration, which can lead to a sequence of events like the following:

  1. CI server runs a build that hashes a 3rd party library in a project without properties file normalization rules. The hash of this file is cached in the resources cache.
  2. Another build runs that does have properties file normalization rules, but when the hasher gets to the library, it does not properly capture the rules when calculating the key for the resources cache and calculates the same key as the build without normalization rules. Rather than hashing the file again, it gets a hit on the resources cache and incorporates this hash into the task's build cache key.
  3. A build on another system that does not have the library's hash in the resources cache runs the build with the properties file normalization rules, gets a miss for that file in the resources cache, and does the fully normalized hash of the library. The resulting hash is different which results in a different cache key for the task, causing a build cache miss.

Expected Behavior

We should get.a build cache hit on the task.

Current Behavior

The build cache key can be incorrectly calculated on one side or the other, causing a miss.

Context

Poorer build perfomance due to cache misses.

Steps to Reproduce

See above.

@ghale ghale added this to the 6.8.3 milestone Feb 13, 2021
@ghale ghale self-assigned this Feb 13, 2021
ghale added a commit that referenced this issue Feb 13, 2021
Normalization rules were not being properly captured when calculating the
cache key for file hashes during runtime classpath normalization.  This was
causing incorrect build cache misses under the right sequence of events.

Fixes #16144
ghale added a commit that referenced this issue Feb 13, 2021
Normalization rules were not being properly captured when calculating the
cache key for file hashes during runtime classpath normalization.  This was
causing incorrect build cache misses under the right sequence of events.

Fixes #16144
ghale added a commit that referenced this issue Feb 13, 2021
Normalization rules were not being properly captured when calculating the
cache key for file hashes during runtime classpath normalization.  This was
causing incorrect build cache misses under the right sequence of events.

Fixes #16144
ghale added a commit that referenced this issue Feb 13, 2021
Normalization rules were not being properly captured when calculating the
cache key for file hashes during runtime classpath normalization.  This was
causing incorrect build cache misses under the right sequence of events.

Fixes #16144
ghale added a commit that referenced this issue Feb 13, 2021
Normalization rules were not being properly captured when calculating the
cache key for file hashes during runtime classpath normalization.  This was
causing incorrect build cache misses under the right sequence of events.

Fixes #16144
ghale added a commit that referenced this issue Feb 13, 2021
Normalization rules were not being properly captured when calculating the
cache key for file hashes during runtime classpath normalization.  This was
causing incorrect build cache misses under the right sequence of events.

Fixes #16144
ghale added a commit that referenced this issue Feb 13, 2021
Normalization rules were not being properly captured when calculating the
cache key for file hashes during runtime classpath normalization.  This was
causing incorrect build cache misses under the right sequence of events.

Fixes #16144
ghale added a commit that referenced this issue Feb 16, 2021
Normalization rules were not being properly captured when calculating the
cache key for file hashes during runtime classpath normalization.  This was
causing incorrect build cache misses under the right sequence of events.

Fixes #16144
@wolfs wolfs removed this from the 6.8.3 milestone Feb 17, 2021
@wolfs wolfs added this to the 7.0 RC1 milestone Feb 17, 2021
@wolfs wolfs added the @execution label Feb 17, 2021
@wolfs
Copy link
Member

@wolfs wolfs commented Feb 17, 2021

I created #16192 to backport the fix to 6.8.3.

@wolfs wolfs removed this from the 7.0 RC1 milestone Feb 19, 2021
@wolfs wolfs added this to the 6.8.3 milestone Feb 19, 2021
github-actions bot pushed a commit to w4bo/github-actions-helloworld that referenced this issue Feb 23, 2021
This is a patch release for Gradle 6.8.

This fixes [a critical bug](gradle/gradle#16144) present in Gradle 6.8, 6.8.1 and 6.8.2.

[All issues fixed in this patch release](https://github.com/gradle/gradle/milestone/171?closed=1)

Please don’t use the original 6.8 release or previous patch releases, and instead upgrade to 6.8.3.

## Upgrade Instructions

Switch your build to use Gradle 6.8.3 by updating your wrapper:

`./gradlew wrapper --gradle-version=6.8.3`

See the [Gradle 6.x upgrade guide](https://docs.gradle.org/6.8.3/userguide/upgrading_version_6.html#changes_6.8) to learn about deprecations, breaking changes and other considerations when upgrading to Gradle 6.8.3.

## Reporting Problems

If you find a problem with this release, please file a bug on [GitHub Issues](https://github.com/gradle/gradle/issues) adhering to our issue guidelines.
If you're not sure you're encountering a bug, please use the [forum](https://discuss.gradle.org/c/help-discuss).
tobiasschaefer added a commit to camunda-community-hub/micronaut-camunda-platform-7 that referenced this issue Feb 23, 2021
martisaw pushed a commit to camunda-community-hub/micronaut-camunda-platform-7 that referenced this issue Feb 23, 2021
@carstenhag
Copy link

@carstenhag carstenhag commented Feb 23, 2021

Is it really a "critical bug" as per the 6.8.3 release notes? Seems like it's only performance relevant? :)

@wolfs
Copy link
Member

@wolfs wolfs commented Feb 23, 2021

This can lead to incorrect builds, which makes it a critical bug.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
a:bug affects-version:6.8 @execution in:core
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants