New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

First configuration to resolve determines effective dynamic version cache policy #3019

Closed
DanielThomas opened this Issue Sep 25, 2017 · 1 comment

Comments

Projects
None yet
3 participants
@DanielThomas
Contributor

DanielThomas commented Sep 25, 2017

When a dynamic version is first resolved, that version list cached for the duration of the build. However, if the first configuration to resolve doesn't carry the desired cacheDynamicVersionsFor setting (as is the case for detached configurations), that will lead to stale dependencies being selected.

Expected Behavior

The individual cache TTL is honored.

Current Behavior

The cache TTL of the first configuration to resolve determines the staleness of dynamic dependencies.

Context

When we update dependency locks, we set changing/dynamic caching to 0, however plugins like Spring Dependency Management use detached configurations as part of dependency resolution, causing that configuration to be ignored.

Steps to Reproduce (for bugs)

Build

Run gw resolve --debug | grep DynamicVersion -A5 against the following build:

plugins {
	id 'java'
}

repositories {
	mavenCentral()
}

dependencies {
	compile 'com.google.guava:guava:latest.release'
}

configurations.all {
	resolutionStrategy {
	 	cacheDynamicVersionsFor 0, 'seconds'
	}
}

task resolve() {
	doFirst {
		// Resolve a configuration that doesn't have the desired dynamic TTL
		configurations.detachedConfiguration(dependencies.create('com.google.guava:guava:latest.release')).files

		// Resolve the desired configuration
		configurations.compile.files
	}
}

Observation

Note that the first dynamic version lookup indicates Using cached module metadata and the following requests perform no lookup:

14:48:42.749 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.ivyresolve.DynamicVersionResolver] Attempting to resolve version for com.google.guava:guava:latest.release using repositories [MavenRepo]
14:48:42.749 [INFO] [org.gradle.cache.internal.DefaultCacheAccess] Creating new cache for metadata-2.23/module-versions, path /Users/dannyt/.gradle/caches/modules-2/metadata-2.23/module-versions.bin, access org.gradle.cache.internal.DefaultCacheAccess@1d5f484
14:48:42.750 [DEBUG] [org.gradle.cache.internal.btree.BTreePersistentIndexedCache] Opening cache module-versions.bin (/Users/dannyt/.gradle/caches/modules-2/metadata-2.23/module-versions.bin)
14:48:42.750 [INFO] [org.gradle.cache.internal.DefaultCacheAccess] Creating new cache for metadata-2.23/module-metadata, path /Users/dannyt/.gradle/caches/modules-2/metadata-2.23/module-metadata.bin, access org.gradle.cache.internal.DefaultCacheAccess@1d5f484
14:48:42.750 [DEBUG] [org.gradle.cache.internal.btree.BTreePersistentIndexedCache] Opening cache module-metadata.bin (/Users/dannyt/.gradle/caches/modules-2/metadata-2.23/module-metadata.bin)
14:48:42.751 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.ivyresolve.CachingModuleComponentRepository] Using cached module metadata for module 'com.google.guava:guava:23.0' in 'MavenRepo'
--
14:48:42.751 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.ivyresolve.DynamicVersionResolver] Using com.google.guava:guava:23.0 from Maven repository 'MavenRepo'
14:48:42.751 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.resolveengine.graph.DependencyGraphBuilder] Selecting new module version com.google.guava:guava:23.0
14:48:42.751 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.resolveengine.graph.DependencyGraphBuilder] Visiting configuration com.google.guava:guava:23.0(default).
14:48:42.751 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.resolveengine.graph.DependencyGraphBuilder] Visiting dependency com.google.guava:guava:23.0(default) -> dependency: com.google.code.findbugs:jsr305:1.3.9, scope: Compile, optional: false
14:48:42.751 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.resolveengine.graph.DependencyGraphBuilder] Selecting new module version com.google.code.findbugs:jsr305:1.3.9
14:48:42.751 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.ivyresolve.RepositoryChainComponentMetaDataResolver] Attempting to resolve component for com.google.code.findbugs:jsr305:1.3.9 using repositories [MavenRepo]
--
14:48:42.757 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.ivyresolve.DynamicVersionResolver] Attempting to resolve version for com.google.guava:guava:latest.release using repositories [MavenRepo]
14:48:42.758 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.ivyresolve.DynamicVersionResolver] Using com.google.guava:guava:23.0 from Maven repository 'MavenRepo'
14:48:42.758 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.resolveengine.graph.DependencyGraphBuilder] Selecting new module version com.google.guava:guava:23.0
14:48:42.758 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.resolveengine.graph.DependencyGraphBuilder] Visiting configuration com.google.guava:guava:23.0(default).
14:48:42.758 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.resolveengine.graph.DependencyGraphBuilder] Visiting dependency com.google.guava:guava:23.0(default) -> dependency: com.google.code.findbugs:jsr305:1.3.9, scope: Compile, optional: false
14:48:42.758 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.resolveengine.graph.DependencyGraphBuilder] Selecting new module version com.google.code.findbugs:jsr305:1.3.9
14:48:42.758 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.ivyresolve.RepositoryChainComponentMetaDataResolver] Attempting to resolve component for com.google.code.findbugs:jsr305:1.3.9 using repositories [MavenRepo]

Your Environment

Gradle 3.5.1

@bigdaz

This comment has been minimized.

Member

bigdaz commented Dec 24, 2017

Thanks @DanielThomas for tracking this down.

This is a pretty nasty issue, caused by our in-memory resolution cache not respecting the different cache expiry settings for different configurations.
A similar problem exists where Maven SNAPSHOT modules will not be updated when they are first resolved by a detached configuration (that misses any cache expiry settings).

I hope to have a fix ready for 4.5.

@bigdaz bigdaz self-assigned this Dec 31, 2017

@bigdaz bigdaz added this to the 4.6 RC1 milestone Jan 5, 2018

@wolfs wolfs referenced this issue Feb 5, 2018

Closed

Fix for #3109 #4263

bigdaz added a commit that referenced this issue Feb 5, 2018

Add tests demonstrating #3019
These tests demonstrate that Gradle only considers the cache expiry
policy for the first configuration that resolves a particular
dynamic version or snapshot module. This is particularly problematic
since detached configurations never get any cache expiry configured
via `configurations.all`.

As such, if a detached configuration is the first to resolve a dynamic
version or snapshot during a build invocation, it may cause the user-
configured cache expiry to be ignored. Several popular plugins,
including the Spring dependency management plugin, add detached
configurations that will be resolved early in a build invocation.

bigdaz added a commit that referenced this issue Feb 5, 2018

Add failing test for #4261
In-memory caching of module metadata is done on a per-repository
basis: the same repository definition in different projects will
allow sharing of in-memory cached module metadata.

Component metadata rules are defined on a per-project basis. This
means that in the scenario where different component metadata rules
have been defined for different projects, only the rules from the
first project to resolve a particular module from a particular
repository will apply.

This fix for this issue will require further changes beyond the fix
for #3019.

bigdaz added a commit that referenced this issue Feb 5, 2018

Simple fix for cache-expiry by removing in-memory cache
This fix demonstrates that the incorrect dynamic version behaviour
of #3019 is due to the in-memory cache of module version lists.
A more complete fix will involve moving the in-memory caching to live
with the other filesystem caching, allowing the cache expiry policy
to be correctly honoured.

bigdaz added a commit that referenced this issue Feb 5, 2018

Preserve Maven snapshot ID through dependency resolution
When resolving a Maven snapshot that has a unique timestamp, Gradle internally
generates a unique component identifier for this snapshot. Previously this
unique snapshot id was discarded during resolution, meaning that artifacts from
different unique snapshots could have the same identifier.

With this change, the `MavenUniqueSnapshotIdentifier` is preserved in
resolution, and is made available via the resolution result.

This change is required to fix #3019 for maven snapshots, to avoid the
in-memory cache of resolved artifacts providing a stale result for
unique snapshot artifacts.

bigdaz added a commit that referenced this issue Feb 5, 2018

Cache processed metadata in-memory
Processing component metadata rules for resolved module metadata
is expensive, and the fix for #3019 introduced a serious performance
regression. This change adds in-memory caching for the post-processed
metadata on a per-repository basis.

Unfortunately, this re-introduces the bug #4261, where component
metadata rule outputs can leak between projects for a repository.
Since defining different metadata rules in different projects is
unlikely to be commonplace, this is considered an acceptable
interim position. This issue should be addressed with the
introduction of general-purpose caching of component metadata
rules.

bigdaz added a commit that referenced this issue Feb 5, 2018

Add tests demonstrating #3019
These tests demonstrate that Gradle only considers the cache expiry
policy for the first configuration that resolves a particular
dynamic version or snapshot module. This is particularly problematic
since detached configurations never get any cache expiry configured
via `configurations.all`.

As such, if a detached configuration is the first to resolve a dynamic
version or snapshot during a build invocation, it may cause the user-
configured cache expiry to be ignored. Several popular plugins,
including the Spring dependency management plugin, add detached
configurations that will be resolved early in a build invocation.

bigdaz added a commit that referenced this issue Feb 5, 2018

Add failing test for #4261
In-memory caching of module metadata is done on a per-repository
basis: the same repository definition in different projects will
allow sharing of in-memory cached module metadata.

Component metadata rules are defined on a per-project basis. This
means that in the scenario where different component metadata rules
have been defined for different projects, only the rules from the
first project to resolve a particular module from a particular
repository will apply.

This fix for this issue will require further changes beyond the fix
for #3019.

bigdaz added a commit that referenced this issue Feb 5, 2018

Simple fix for cache-expiry by removing in-memory cache
This fix demonstrates that the incorrect dynamic version behaviour
of #3019 is due to the in-memory cache of module version lists.
A more complete fix will involve moving the in-memory caching to live
with the other filesystem caching, allowing the cache expiry policy
to be correctly honoured.

bigdaz added a commit that referenced this issue Feb 5, 2018

Cache processed metadata in-memory
Processing component metadata rules for resolved module metadata
is expensive, and the fix for #3019 introduced a serious performance
regression. This change adds in-memory caching for the post-processed
metadata on a per-repository basis.

Unfortunately, this re-introduces the bug #4261, where component
metadata rule outputs can leak between projects for a repository.
Since defining different metadata rules in different projects is
unlikely to be commonplace, this is considered an acceptable
interim position. This issue should be addressed with the
introduction of general-purpose caching of component metadata
rules.

@bigdaz bigdaz closed this in b761585 Feb 6, 2018

bigdaz added a commit that referenced this issue Feb 6, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment