[MBUILDCACHE-22] added flag to force rebuild per project or globally #24

eltsovalex · 2022-08-26T15:28:54Z

added possibility to skip generated sources restore on per-project level
added some logging

Following this checklist to help us incorporate your
contribution quickly and easily:

Make sure there is a JIRA issue filed
for the change (usually before you start working on it). Trivial changes like typos do not
require a JIRA issue. Your pull request should address just this issue, without
pulling in other changes.
Each commit in the pull request should have a meaningful subject line and body.
Format the pull request title like [MNG-XXX] - Fixes bug in ApproximateQuantiles,
where you replace MNG-XXX with the appropriate JIRA issue. Best practice
is to use the JIRA issue title in the pull request title and in the first line of the
commit message.
Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
Run mvn clean verify to make sure basic checks pass. A more thorough check will
be performed on your pull request automatically.
You have run the Core IT successfully.

If your pull request is about ~20 lines of code you don't need to sign an
Individual Contributor License Agreement if you are unsure
please ask on the developers list.

To make clear that you license your contribution under
the Apache License Version 2.0, January 2004
you have to acknowledge this by using the following check-box.

I hereby declare this contribution to be licenced under the Apache License Version 2.0, January 2004
In any other case, please file an Apache Individual Contributor License Agreement.

added possibility to skip generated sources restore on per-project level added some logging

eltsovalex · 2022-08-26T15:34:40Z

(https://issues.apache.org/jira/browse/MBUILDCACHE-22)
(https://issues.apache.org/jira/browse/MBUILDCACHE-23)

Basically this PR is about adding 2 new features which are required to use this extension in yet another complex multi-module project in Deutsche Bank

ability to skip cache lookup for a particular module or whole project (required to get some modules always built e.g. via a profile - e.g. I have some additional artifacts published by CI). Alternatively it allows an easy "force" rebuild of a whole project
The difference with not using cache is that results are put into cache, just current matching versions are not taken from caches
ability to disable generated source restoration on a per-module level. This is required to overcome issues with some weird Maven plugins which go nuts when cached generated sources are present and produce incorrect build result

maximilian-novikov-db · 2022-08-29T07:46:50Z

@eltsovalex https://issues.apache.org/jira/projects/MBUILDCACHE for this project

maximilian-novikov-db · 2022-08-29T17:48:17Z

src/main/java/org/apache/maven/buildcache/xml/CacheConfigImpl.java

@@ -87,6 +87,16 @@ public class CacheConfigImpl implements org.apache.maven.buildcache.xml.CacheCon
    public static final String RESTORE_GENERATED_SOURCES_PROPERTY_NAME = "maven.build.cache.restoreGeneratedSources";
    public static final String ALWAYS_RUN_PLUGINS = "maven.build.cache.alwaysRunPlugins";

+    /**
+     * Flag to control if we should skip lookup for cached artifacts globally r for a particular project even if


hboutemy · 2022-09-05T06:41:59Z

is it possible to add a note in documentation about this feature and how to use it, please?
https://github.com/apache/maven-build-cache-extension/tree/master/src/site/markdown

gnodet

Looks good, please fix the typo in the javadoc in CacheConfigImpl.java and document its usage in the documentation.

eltsovalex · 2022-09-06T09:03:45Z

Typos fixed in javadoc + new parameters described in documentation

AlexanderAshitkin · 2022-09-14T13:22:30Z

src/main/java/org/apache/maven/buildcache/BuildCacheMojosExecutionStrategy.java

            {
-                result = cacheController.findCachedBuild( session, project, mojoExecutions );
+                result = cacheController.findCachedBuild( session, project, mojoExecutions, skipCacheLookup );


right now cache could be accessed when no initialized (it seems). not sure if it could cause any issues but in terms of naming and contract clarity it is not so clear . skipCache should work better

@AlexanderAshitkin renamed to skipCache

AlexanderAshitkin · 2022-09-14T13:34:06Z

src/main/java/org/apache/maven/buildcache/BuildCacheMojosExecutionStrategy.java

@@ -96,6 +97,7 @@ public void execute( List<MojoExecution> mojoExecutions,
        // execute clean bound goals before restoring to not interfere/slowdown clean
        CacheState cacheState = DISABLED;
        CacheResult result = CacheResult.empty();
+        boolean skipCacheLookup = cacheConfig.isSkipLookup() || MavenProjectInput.shouldSkipCacheLookup( project );


regarding the naming - lets call it skipCache. Gradle has similar parameter to force rebuild named --no-build-cache.

@AlexanderAshitkin renamed

AlexanderAshitkin · 2022-09-14T13:39:59Z

src/main/java/org/apache/maven/buildcache/CacheControllerImpl.java

-                result = localBuild;
-            }
-            else
+            // remote build first


could you please change this comment to something like Central repository build first
remote and local notation is confusing here

This is not my comment - just part of code by @gnodet that I've moved inside a block

AlexanderAshitkin · 2022-09-14T13:41:38Z

src/main/java/org/apache/maven/buildcache/DefaultNormalizedModelProvider.java

@@ -123,10 +127,16 @@ private List<Plugin> normalizePlugins( List<Plugin> plugins )
                {
                    Plugin copy = plugin.clone();
                    List<String> excludeProperties = cacheConfig.getEffectivePomExcludeProperties( copy );
-                    removeBlacklistedAttributes( copy.getConfiguration(), excludeProperties );


where is it now?

@AlexanderAshitkin the code is still there - just after debug output

AlexanderAshitkin · 2022-09-14T13:43:10Z

src/main/java/org/apache/maven/buildcache/checksum/MavenProjectInput.java

                filteredOutPaths.add( Paths.get( properties.getProperty( propertyName ) ) );
            }
        }
+        CacheUtils.debugPrintCollection( LOGGER, filteredOutPaths,
+                "List of excluded paths (checked either by fileName or by startsWith prefix)",
+                "Pah entry" );


typo. Pah entry is a medical term likely ¯_(ツ)_/¯

@AlexanderAshitkin fixed

AlexanderAshitkin · 2022-09-14T13:45:04Z

src/site/markdown/parameters.md

@@ -32,6 +32,7 @@ This documents contains various configuration parameters supported by cache engi
 | `-Dmaven.build.cache.lazyRestore=(true/false)`             | Restore artifacts from remote cache lazily                                                                                               | Performance optimization |
 | `-Dmaven.build.cache.restoreGeneratedSources=(true/false)` | Do not restore generated sources and directly attached files                                                                             | Performance optimization |
 | `-Dmaven.build.cache.alwaysRunPlugins=<list of plugins>`   | Comma seprated list of plugins to always run regardless of cache state. An example: `maven-deploy-plugin:*,maven-install-plugin:install` | Remote cache setup/tuning/troubleshooting |
+| `-Dmaven.build.cache.skipLookup=(true/false)`              | Skip looking up artifacts in caches. Does not affect writing artifacts to caches, disables only reading when set to `true`               | May be used to trigger a forced rebuild when maching artifatcs do exist in caches|


From command line and pom perspective -Dorg.maven.build.cache.rerun is more expressive because it assumes that cache will be invalidated and rebuilt. Another option is which is also look better is -Dorg.maven.build.cache.skipCache just seems more expressive to me

@AlexanderAshitkin renamed to -Dorg.maven.build.cache.skipCache

AlexanderAshitkin · 2022-09-14T13:47:38Z

src/main/java/org/apache/maven/buildcache/checksum/MavenProjectInput.java

+                    LOGGER.debug( "Adding an excludePath from property '{}', values is '{}', path is '{}' ",
+                            propertyName, propertyValue, path );
+                }
+
                filteredOutPaths.add( Paths.get( properties.getProperty( propertyName ) ) );


no longer need this line, to be removed?

@AlexanderAshitkin thanks, fixed

AlexanderAshitkin · 2022-09-14T13:52:56Z

src/main/java/org/apache/maven/buildcache/checksum/MavenProjectInput.java

@@ -819,4 +833,22 @@ public int compare( Path f1, Path f2 )
        }
    }

+    public static boolean shouldSkipCacheLookup( MavenProject project )


skipCache or isSkipCache will be better

@AlexanderAshitkin fixed

AlexanderAshitkin · 2022-09-14T14:02:59Z

I would rename this pr into "flag to force rebuild" which better describes new functionality

AlexanderAshitkin · 2022-09-15T15:07:44Z

src/site/markdown/parameters.md

@@ -32,7 +32,7 @@ This documents contains various configuration parameters supported by cache engi
 | `-Dmaven.build.cache.lazyRestore=(true/false)`             | Restore artifacts from remote cache lazily                                                                                               | Performance optimization |
 | `-Dmaven.build.cache.restoreGeneratedSources=(true/false)` | Do not restore generated sources and directly attached files                                                                             | Performance optimization |
 | `-Dmaven.build.cache.alwaysRunPlugins=<list of plugins>`   | Comma seprated list of plugins to always run regardless of cache state. An example: `maven-deploy-plugin:*,maven-install-plugin:install` | Remote cache setup/tuning/troubleshooting |
-| `-Dmaven.build.cache.skipLookup=(true/false)`              | Skip looking up artifacts in caches. Does not affect writing artifacts to caches, disables only reading when set to `true`               | May be used to trigger a forced rebuild when maching artifatcs do exist in caches|
+| `-Dmaven.build.cache.skip=(true/false)`                    | Skip looking up artifacts in caches. Does not affect writing artifacts to caches, disables only reading when set to `true`               | May be used to trigger a forced rebuild when maching artifatcs do exist in caches|


small typo. maching -> matching

@AlexanderAshitkin - thx, fixed

maximilian-novikov-db · 2022-10-14T09:20:39Z

@gnodet please merge

added possibility to skip cache lookup per proejct or globally

fed81b0

added possibility to skip generated sources restore on per-project level added some logging

maximilian-novikov-db approved these changes Aug 30, 2022

View reviewed changes

eltsovalex changed the title ~~added possibility to skip cache lookup per project or globally~~ [MBUILDCACHE-22] added possibility to skip cache lookup per project or globally Aug 30, 2022

hboutemy requested a review from gnodet September 5, 2022 06:42

gnodet requested changes Sep 5, 2022

View reviewed changes

[MBUILDCACHE-22] fixed typos in javadocs + added missing documentation

dd45b4f

maximilian-novikov-db approved these changes Sep 8, 2022

View reviewed changes

AlexanderAshitkin reviewed Sep 14, 2022

View reviewed changes

eltsovalex added 2 commits September 15, 2022 16:39

[MBUILDCACHE-22] renamed parameter to skipCache as per review comments

50d6330

[MBUILDCACHE-22] removed extra line to as per review comment

731604f

AlexanderAshitkin reviewed Sep 15, 2022

View reviewed changes

AlexanderAshitkin approved these changes Sep 15, 2022

View reviewed changes

[MBUILDCACHE-22] more review fixes

79c1392

eltsovalex changed the title ~~[MBUILDCACHE-22] added possibility to skip cache lookup per project or globally~~ [MBUILDCACHE-22] added flag to force rebuild per project or globally Sep 15, 2022

[MBUILDCACHE-22] fixed typo in logger

ddb9898

gnodet merged commit dd09baa into apache:master Oct 16, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[MBUILDCACHE-22] added flag to force rebuild per project or globally #24

[MBUILDCACHE-22] added flag to force rebuild per project or globally #24

eltsovalex commented Aug 26, 2022 •

edited

eltsovalex commented Aug 26, 2022 •

edited

maximilian-novikov-db commented Aug 29, 2022

maximilian-novikov-db Aug 29, 2022

eltsovalex Sep 6, 2022

hboutemy commented Sep 5, 2022

gnodet left a comment •

edited

eltsovalex commented Sep 6, 2022

AlexanderAshitkin Sep 14, 2022 •

edited

eltsovalex Sep 15, 2022

AlexanderAshitkin Sep 14, 2022 •

edited

eltsovalex Sep 15, 2022

AlexanderAshitkin Sep 14, 2022 •

edited

eltsovalex Sep 15, 2022 •

edited

AlexanderAshitkin Sep 14, 2022

eltsovalex Sep 15, 2022

AlexanderAshitkin Sep 14, 2022 •

edited

eltsovalex Sep 15, 2022

AlexanderAshitkin Sep 14, 2022 •

edited

eltsovalex Sep 15, 2022

AlexanderAshitkin Sep 14, 2022 •

edited

eltsovalex Sep 15, 2022

AlexanderAshitkin Sep 14, 2022 •

edited

eltsovalex Sep 15, 2022

AlexanderAshitkin commented Sep 14, 2022

AlexanderAshitkin Sep 15, 2022

eltsovalex Sep 15, 2022

maximilian-novikov-db commented Oct 14, 2022

[MBUILDCACHE-22] added flag to force rebuild per project or globally #24

[MBUILDCACHE-22] added flag to force rebuild per project or globally #24

Conversation

eltsovalex commented Aug 26, 2022 • edited

eltsovalex commented Aug 26, 2022 • edited

maximilian-novikov-db commented Aug 29, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hboutemy commented Sep 5, 2022

gnodet left a comment • edited

Choose a reason for hiding this comment

eltsovalex commented Sep 6, 2022

AlexanderAshitkin Sep 14, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AlexanderAshitkin Sep 14, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AlexanderAshitkin Sep 14, 2022 • edited

Choose a reason for hiding this comment

eltsovalex Sep 15, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AlexanderAshitkin Sep 14, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AlexanderAshitkin Sep 14, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AlexanderAshitkin Sep 14, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AlexanderAshitkin Sep 14, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AlexanderAshitkin commented Sep 14, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

maximilian-novikov-db commented Oct 14, 2022

eltsovalex commented Aug 26, 2022 •

edited

eltsovalex commented Aug 26, 2022 •

edited

gnodet left a comment •

edited

AlexanderAshitkin Sep 14, 2022 •

edited

AlexanderAshitkin Sep 14, 2022 •

edited

AlexanderAshitkin Sep 14, 2022 •

edited

eltsovalex Sep 15, 2022 •

edited

AlexanderAshitkin Sep 14, 2022 •

edited

AlexanderAshitkin Sep 14, 2022 •

edited

AlexanderAshitkin Sep 14, 2022 •

edited

AlexanderAshitkin Sep 14, 2022 •

edited