[JENKINS-38992] Allow caching library versions #50

julienduchesne · 2018-08-20T14:38:09Z

We have a jenkins instance that need to pull a lot of libraries. When many jobs are running simultaneously it can sometimes take up to 30 seconds to load a single library.

Probably fixes https://issues.jenkins-ci.org/browse/JENKINS-38992 and https://issues.jenkins-ci.org/browse/JENKINS-48210 in some measure

abayer

Definitely needs tests for the actual caching behavior.

julienduchesne · 2018-08-22T15:30:48Z

Alright, and I still have to fix failing tests. FYI, that's how it looks in the UI

julienduchesne · 2018-08-23T11:43:55Z

Added tests, the build failure seems unrelated:
java.lang.RuntimeException: Error grabbing Grapes -- [download failed: org.apache.commons#commons-math3;3.4.1!commons-math3.jar]

julienduchesne · 2018-09-04T10:33:13Z

Any update on this?

dwnusbaum

Idea looks good to me. It would be nice to refactor LibraryCacheConfiguration to implement Describable so that we have CasC support, and it seem like the caching logic might be better suited to be part of of SCMSourceRetriver so that the cache can use take the SCM into account.

The only high-level thing I am concerned about is some way of automatically clearing out long-unused caches. Maybe we could the time each cache was used most recently and have an AsyncPeriodicWork instance that deletes caches that haven't been used in some amount of time (could be configurable, but a week also seems like a decent default).

src/main/java/org/jenkinsci/plugins/workflow/libs/LibraryConfiguration.java

src/main/java/org/jenkinsci/plugins/workflow/libs/LibraryAdder.java

src/main/java/org/jenkinsci/plugins/workflow/libs/LibraryCachingConfiguration.java

dwnusbaum · 2018-10-08T16:56:16Z

src/main/java/org/jenkinsci/plugins/workflow/libs/LibraryAdder.java

+                }
+            }
+
+            if(versionCacheDir.exists()) {


If we change the SCM configuration (i.e. switch to a new repository), but not the library name, then the cache is still valid. It seems like it might be better to move the caching logic inside of SCMSourceRetriver (and SCMRetriever if possible) so that we can use SCM#getKey as part of the cache name so that SCM changes automatically invalidate the cache.

julienduchesne · 2018-10-10T17:10:35Z

Hey @dwnusbaum. I've fixed every issue except putting the config in SCMSourceRetriever. I put the config in LibraryConfiguration on purpose, that way it is independant from the SCM. The same way that other configs such as the default version and default version override are. I also don't think that it's a bug if I change the repository but the version stays the same. Let me know what you think.

Looks obsolete.

src/main/java/org/jenkinsci/plugins/workflow/libs/LibraryConfiguration.java

jglick · 2018-10-13T14:45:30Z

...rg/jenkinsci/plugins/workflow/libs/LibraryCachingConfiguration/help-excludedVersionsStr.html

@@ -0,0 +1,3 @@
+<div>
+    Space separated list of versions excluded from caching. Ex: "master release10 release9"


But there is no possibility here to specify a regular expression.

FWIW my proposal in JENKINS-38992 was to make this automatic—zero configuration. The SCM should know whether or not a given version was deterministic. The current SCMSource.retrieve(String, TaskListener) does not suffice since at least in the case of AbstractGitSCMSource the resulting SCMRevision.isDeterministic will always be true—the method resolves the actual current head of a branch. Thus we would need to extend the SCMHead API in scm-api to let the SCM source indicate whether the head is deterministic (like a hash, or arguably a tag) vs. nondeterministic (like a branch). This would be easy to implement in git at least.

I agree that it should support regex and maybe add an include field as well (only include specific versions or regexes) but is it really user friendly to cache some versions but not others. In our use case, it would be annoying to cache tags by default because we use semantic versioning and publish minor versions as well as patch versions (ex: release v4.2.1 also releases v4.2 and v4).

All I'm saying is I'm not a fan of hidden behavior and the current implementation is very simple. Is it really necessary to complicate it?

d3rp3tt3 · 2018-10-16T17:41:08Z

Based on input here, this might not be the best implementation.

julienduchesne · 2018-10-16T18:20:41Z

It works very well though. We are using it on a jenkins instance running about 20 very short jobs (15 sec to 5 minutes) concurrently at all times during the day. Each of those jobs pull two shared libraries. We used to only be able to do a full cycle of our workflow in over an hour before this. It now takes under 20 minutes.

criley-eagle · 2018-11-29T14:08:43Z

"Based on input here, this might not be the best implementation."
@jennbriden Is this pr still viable, or is there a better implementation in the works, as we are definitely hitting up against the same issue this pr was attempting to resolve?

julienduchesne · 2020-07-09T10:30:03Z

👋 I haven't received any "pings" until now. What needs to be done to have this merged?

msrb · 2020-08-28T05:10:52Z

@oleg-nenashev LGTM? :)

oleg-nenashev · 2020-09-28T13:29:53Z

👋 I haven't received any "pings" until now. What needs to be done to have this merged?

Looks like somebody needs to take ownership of the plugin or to get the current maintainers to review/merge the pull request. Unfortunately I cannot do much more to push it forward at the moment.

atmcarmo · 2020-10-30T23:49:38Z

Thanks for this PR! Hope this gets merged. It's really an issue for us the time it gets to clone and load a library when many pipelines are starting to execute at the same time.

Includes pipeline caching pull request jenkinsci/workflow-cps-global-lib-plugin#50

robodenitro · 2021-01-07T10:50:40Z

any updates here?

MarkEWaite · 2021-01-07T12:44:29Z

@robodenitro I've been running this pull request since early November 2020 without detecting any issue. You could do the same by installing the file from MarkEWaite/docker-lfs@408684e . Once you do that, your specific need should be resolved and you can report your test results

It is more persuasive to report successful use of a pull request than to ask for a status update.

nnadams · 2021-02-19T18:03:22Z

Following Mark's example, I have been using this pull request since June 2020 and seen no issues. We run thousands of jobs weekly that all make use of this caching. Our CI has been faster and much more robust. I hope this can be merged. Either way thank you @julienduchesne!

MarkEWaite

Remove trailing white space in diffs

MarkEWaite · 2021-02-19T21:05:34Z

src/test/java/org/jenkinsci/plugins/workflow/libs/ResourceStepTest.java

-        sampleRepo.git("add", "src", "resources");
-        sampleRepo.git("commit", "--message=init");
+        initFixedContentLibrary();
+


Remove trailing white space

Suggested change

MarkEWaite · 2021-02-19T21:05:52Z

src/test/java/org/jenkinsci/plugins/workflow/libs/ResourceStepTest.java

+    @Test public void cachingExcludedLibrary() throws Exception {
+        clearCache("stuff");
+        initFixedContentLibrary();
+


Remove trailing white space

Suggested change

MarkEWaite · 2021-02-19T21:06:28Z

src/test/java/org/jenkinsci/plugins/workflow/libs/ResourceStepTest.java

+    @Test public void caching() throws Exception {
+        clearCache("stuff");
+        initFixedContentLibrary();
+


Remove trailing white space

Suggested change

MarkEWaite · 2021-02-19T21:06:58Z

src/test/java/org/jenkinsci/plugins/workflow/libs/ResourceStepTest.java

+    @Test public void cachingRefresh() throws Exception {
+        clearCache("stuff");
+        initFixedContentLibrary();
+


Remove trailing white space

Suggested change

…aite) Includes caching as implemented in jenkinsci/workflow-cps-global-lib-plugin#50

ababushk · 2021-05-12T20:01:33Z

We use version of workflow-cps-global-lib with changes from this PR on production instance with huge amount of jobs and builds per day. No issues so far!
Moreover, this allowed us to run multiple concurrent builds (>1000 simultaneously) without waiting for a lock on git repo cache directory for our global library on Jenkins server, which has speeded up them drastically. Many thanks to @julienduchesne !

rfvmonteiro · 2021-05-18T13:27:22Z

Thanks for this contribution. We are testing it too and no issues found so far.
I believe this is a solid solution and should be merged soon 🙏

atmcarmo · 2021-05-18T13:42:25Z

This solution seems to be rock solid, really improving performance even with a very big amount of builds happening at the same time.

What needs to be done to merge to be included in a new release?

Thank you very much for your work 👏

michelgasser · 2021-06-21T07:23:50Z

Why is the PR not being merged? Why is it taking so long?

timja · 2021-06-21T07:26:30Z

@dwnusbaum @car-roll @bitwiseman Would it be possible for someone to take a look at this?

oleg-nenashev · 2021-06-21T11:25:19Z

As a CloudBees employee I spent so much time pinging the current maintainers. Doing the last attempt

…

On Mon, Jun 21, 2021, 09:26 Tim Jacomb ***@***.***> wrote: @dwnusbaum <https://github.com/dwnusbaum> @car-roll <https://github.com/car-roll> @bitwiseman <https://github.com/bitwiseman> Would it be possible for someone to take a look at this? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#50 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAW4RIFLX2BTVOC3FTGHACLTT3STLANCNFSM4FQQQ36A> .

car-roll

So, I'm just going over the comments here and I think this summarizes what went on here:

It does look like most issues were addressed, but there was a preference by both @dwnusbaum and @jglick to implement things slightly different, which is why this PR stalled. But because there was no significant effort to address the alternatives proposed by them, and since this PR was the ONLY solution available, it became widely adopted. Thus the wide adoption of this solution is now causing pressure for this PR to be merged.

Given that this PR has been actively used in the wild now for quite some time, I'm going ahead to merge it.

edit: The UI seems to be fine after resolving the latest merge

michelgasser · 2021-06-23T08:59:08Z

Cool, thanks a lot for the merge! :-)

MaximeAnsquer · 2021-10-07T19:46:44Z

How can I disable caching please?

MarkEWaite · 2021-10-07T22:34:47Z

How can I disable caching please?

Caching is disabled by default. If you have enabled caching in the "Manage Jenkins" -> "Configure System" page, you would disable it by unchecking the "Cache fetched versions on master for quick retrieval"

MaximeAnsquer · 2021-10-08T06:59:58Z

Thank you @MarkEWaite!

aaronr93 · 2022-02-13T12:41:57Z

Is there a way to programmatically clear the cache from a pipeline?

Here are my ideal use cases:

Cache all versions for a long time, then clear the cache when new revisions are available (for example, called directly from a job called by an SCM webhook or some other trigger). This would follow an industry-standard concept of "cache busting".
Clean the caches of old/deleted branches via a regularly-scheduled maintenance job, rather than remembering to manually click "Clear cache" every now and then.
Programmatically configure certain pipelines to not use caching (perhaps a parameter passed to library).

Regardless, this feature should be added to the "Extending with Shared Libraries" documentation.

Let me know if I should start a new issue for this.

MarkEWaite · 2022-02-13T12:56:04Z

Regardless, this feature should be added to the "Extending with Shared Libraries" documentation.

Let me know if I should start a new issue for this.

Best would be to submit the pull request proposing the changes to the existing documentation. See the "Improve this page" link at the bottom of that page.

If not ready to submit the documentation pull request, then submit the issue using the "Report a problem" at the bottom of that page.

aaronr93 · 2022-05-02T21:52:29Z

Hi @MarkEWaite, just following up on my first question above. Is there a way to programmatically clear the cache from a pipeline?

MarkEWaite · 2022-05-02T22:27:01Z

Hi @MarkEWaite, just following up on my first question above. Is there a way to programmatically clear the cache from a pipeline?

No method that I know to clear cache from a Pipeline.

Julien Duchesne added 3 commits August 20, 2018 10:33

Allow caching library versions

289da46

Fix LibraryStepTest failure

2de345f

Add refresh time and branch exclusion options

894e704

abayer requested review from jglick, svanoort, dwnusbaum and abayer August 22, 2018 15:15

abayer previously requested changes Aug 22, 2018

View reviewed changes

Julien Duchesne added 3 commits August 22, 2018 14:29

Fix failing tests

105b11a

Added tests to cover new library caching features

b66300f

Make LibraryCachingConfiguration properties final

54dd01c

julienduchesne force-pushed the cache-versions branch from 066dda8 to 54dd01c Compare August 23, 2018 11:29

dwnusbaum reviewed Oct 8, 2018

View reviewed changes

Julien Duchesne added 3 commits October 9, 2018 21:31

Merge remote-tracking branch 'origin/master' into cache-versions

293735e

Fix comments by @dwnusbaum

7cf56b0

Use touch to write lock file

ce38c82

Make versionDir if it doesn't exist

a643a60

jglick changed the title ~~Allow caching library versions~~ [JENKINS-38992] Allow caching library versions Oct 13, 2018

jglick reviewed Oct 13, 2018

View reviewed changes

Remove format changes

1688731

julienduchesne force-pushed the cache-versions branch from fb9ee83 to 1688731 Compare October 14, 2018 20:12

jglick mentioned this pull request Jan 8, 2019

[JENKINS-55341] Loading library only once and storing it in common place from where all pipeline builds which use that version can access it #58

Open

Julien Duchesne added 2 commits September 28, 2020 09:22

Merge tag 'workflow-cps-global-lib-2.17' into cache-versions

2b59338

Merge branch 'master' into cache-versions

09653a6

MarkEWaite added a commit to MarkEWaite/docker-lfs that referenced this pull request Nov 2, 2020

Use workflow cps global lib 2.18-rc536.f8c07562afbb

408684e

Includes pipeline caching pull request jenkinsci/workflow-cps-global-lib-plugin#50

MarkEWaite approved these changes Feb 19, 2021

View reviewed changes

MarkEWaite added a commit to MarkEWaite/docker-lfs that referenced this pull request Feb 19, 2021

Use workflow cps global lib plugin 2.19-SNAPSHOT (private-ab276be9-mw…

98d5c16

…aite) Includes caching as implemented in jenkinsci/workflow-cps-global-lib-plugin#50

Merge branch 'master' into cache-versions

a84db04

car-roll added the enhancement label Jun 21, 2021

car-roll approved these changes Jun 21, 2021

View reviewed changes

car-roll merged commit b367de6 into jenkinsci:master Jun 21, 2021

		@@ -0,0 +1,3 @@
		<div>
		Space separated list of versions excluded from caching. Ex: "master release10 release9"

[JENKINS-38992] Allow caching library versions #50

[JENKINS-38992] Allow caching library versions #50

Conversation

julienduchesne commented Aug 20, 2018

abayer left a comment

Choose a reason for hiding this comment

julienduchesne commented Aug 22, 2018

julienduchesne commented Aug 23, 2018

julienduchesne commented Sep 4, 2018

dwnusbaum left a comment

Choose a reason for hiding this comment

dwnusbaum Oct 8, 2018

Choose a reason for hiding this comment

julienduchesne commented Oct 10, 2018

jglick Oct 13, 2018

Choose a reason for hiding this comment

julienduchesne Oct 14, 2018 • edited

Choose a reason for hiding this comment

d3rp3tt3 commented Oct 16, 2018 • edited

julienduchesne commented Oct 16, 2018 • edited

criley-eagle commented Nov 29, 2018

julienduchesne commented Jul 9, 2020

msrb commented Aug 28, 2020

oleg-nenashev commented Sep 28, 2020

atmcarmo commented Oct 30, 2020

robodenitro commented Jan 7, 2021

MarkEWaite commented Jan 7, 2021

nnadams commented Feb 19, 2021

MarkEWaite left a comment

Choose a reason for hiding this comment

MarkEWaite Feb 19, 2021

Choose a reason for hiding this comment

MarkEWaite Feb 19, 2021

Choose a reason for hiding this comment

MarkEWaite Feb 19, 2021

Choose a reason for hiding this comment

MarkEWaite Feb 19, 2021

Choose a reason for hiding this comment

ababushk commented May 12, 2021

rfvmonteiro commented May 18, 2021

atmcarmo commented May 18, 2021

michelgasser commented Jun 21, 2021

timja commented Jun 21, 2021

oleg-nenashev commented Jun 21, 2021 via email

car-roll left a comment • edited

Choose a reason for hiding this comment

michelgasser commented Jun 23, 2021

MaximeAnsquer commented Oct 7, 2021

MarkEWaite commented Oct 7, 2021 • edited

MaximeAnsquer commented Oct 8, 2021

aaronr93 commented Feb 13, 2022 • edited

MarkEWaite commented Feb 13, 2022 • edited

aaronr93 commented May 2, 2022

MarkEWaite commented May 2, 2022

julienduchesne Oct 14, 2018 •

edited

d3rp3tt3 commented Oct 16, 2018 •

edited

julienduchesne commented Oct 16, 2018 •

edited

car-roll left a comment •

edited

MarkEWaite commented Oct 7, 2021 •

edited

aaronr93 commented Feb 13, 2022 •

edited

MarkEWaite commented Feb 13, 2022 •

edited