[JENKINS-62755] Introduce Fingerprint Cleanup in External Storage API #4817

stellargo · 2020-06-22T19:48:58Z

See JENKINS-62755.
See JEP-226.

Proposed changelog entries

Developer: Introduce new FingerprintStorage methods: iterateAndCleanupFingerprints, cleanFingerprint to allow external fingerprint storages to configure fingerprint cleanup.
User: Creates a Fingerprint cleanup configuration page which allows enabling/disabling fingerprint cleanup.

Proposed upgrade guidelines

N/A

Submitter checklist

(If applicable) Jira issue is well described
Changelog entries and upgrade guidelines are appropriate for the audience affected by the change (users or developer, depending on the change). Examples
- Fill-in the Proposed changelog entries section only if there are breaking changes or other changes which may require extra steps from users during the upgrade
Appropriate autotests or explanation to why this change has no tests
For dependency updates: links to external changelogs and, if possible, full diffs

Desired reviewers

@mention

Maintainer checklist

Before the changes are marked as ready-for-merge:

There are at least 2 approvals for the pull request and no outstanding requests for change
Conversations in the pull request are over OR it is explicit that a reviewer does not block the change
Changelog entries in the PR title and/or Proposed changelog entries are correct
Proper changelog labels are set so that the changelog can be generated automatically
If the change needs additional upgrade steps from users, upgrade-guide-needed label is set and there is a Proposed upgrade guidelines section in the PR title. (example)
If it would make sense to backport the change to LTS, a Jira issue must exist, be a Bug or Improvement, and be labeled as lts-candidate to be considered (see query).

stellargo · 2020-06-22T19:49:31Z

This PR is WIP currently.

stellargo · 2020-06-24T16:25:04Z

Requesting reviews @oleg-nenashev @afalko @mikecirioli

stellargo · 2020-06-24T16:35:39Z

In this PR, we expose the fingerprint cleanup API for external plugins to implement in the case fingerprint storage plugins want to implement their own fingerprint cleanup. Downstream PR: jenkinsci/redis-fingerprint-storage-plugin#23. We refactor the current fingerprint cleanup into FileFingerprintCleanupThread which can perform the fingerprint cleanup for file system based storage.

oleg-nenashev

With the current implementation I am not sure why it has to be a separate extension point. Could we just integrate the logic into the FingerprintStorage and invoke it from FingerprintCleanupThread which is universal for all extension points?

cleanFingerprint and execute implementations could be moved to a storage.

core/src/main/java/jenkins/fingerprints/FileFingerprintCleanupThread.java

core/src/main/java/jenkins/fingerprints/FingerprintCleanupThread.java

oleg-nenashev · 2020-07-12T16:12:46Z

@stellargo is it still in progress?

stellargo · 2020-07-12T16:16:05Z

@oleg-nenashev So it isn't in progress, the work is complete. But it will create merge conflicts with #4834. I think it is better to merge that before this.

oleg-nenashev · 2020-07-17T14:52:13Z

Unblocking, because #4834 is merged. Merge conflict needs to be resolved tho

stellargo · 2020-07-17T14:53:16Z

Yep currently working on resolving them :)

oleg-nenashev

The Fingerprint cleanup will not work as written, I suggest inverting it to "Disable Fingerprint cleanup"

The API iteself looks OK for a beta implementation. I would rather create an API method for iterator with handler parameter, but it can be done later. API is Beta anyway

core/src/main/java/jenkins/fingerprints/FileFingerprintStorage.java

core/src/main/java/jenkins/fingerprints/FingerprintStorage.java

core/src/main/java/jenkins/fingerprints/GlobalFingerprintConfiguration.java

oleg-nenashev · 2020-07-17T15:22:30Z

core/src/main/java/jenkins/fingerprints/FileFingerprintStorage.java

+        taskListener.getLogger().println("Cleaned up "+numFiles+" records");
+    }
+
+    private boolean cleanFingerprint(File fingerprintFile, TaskListener listener) {


If it is private, it does not override the implementation in upper class. Looks like a 🐛 for API users

So this isn't supposed to override the method of the upper class(signature is different). The method in the upper class isn't called by itself, it is -used- by plugin implementations.

stellargo · 2020-07-17T15:44:25Z

@oleg-nenashev

I would rather create an API method for iterator with handler parameter

So basically asking the fingerprint implementations to provide an iterator, and do the cleanup ourself, is that what you meant?

oleg-nenashev · 2020-07-17T15:50:58Z

So basically asking the fingerprint implementations to provide an iterator, and do the cleanup ourself, is that what you meant?

Might be. Anyway, it can be refactored later. I would not block the pull request by it

core/src/main/java/jenkins/fingerprints/GlobalFingerprintConfiguration.java

oleg-nenashev

I think this is ready to go. Thanks @stellargo !

timja · 2020-07-18T18:42:31Z

core/src/main/resources/jenkins/fingerprints/GlobalFingerprintConfiguration/config.jelly

-        <f:section title="${%Fingerprints}">
+    <f:section title="${%Fingerprints}">
+        <f:entry title="Disable Fingerprint Cleanup">
+            <f:checkbox field="fingerprintCleanupDisabled" default="${it.fingerprintCleanupDisabled}" />


Is there any reason to disable this if there isn’t an external plugin for fingerprints?

It might be a good feature to have in case the disk storage is cheaper for a user than having a periodically running process. Or say Jenkins is configured on a distributed disk storage. I don't have any strong opinions though :)

oleg-nenashev · 2020-07-20T05:07:06Z

We may merge it in 24 hours if there is no negative feedback. Please see the merge process documentation for more information about the merge process

Create new FingerprintCleanupThread

287f8df

stellargo changed the title ~~[JENKINS-62755] Introduce Fingerprint cleanup API in Jenkins core~~ [JENKINS-62755] Introduce External Fingerprint cleanup API in Jenkins core Jun 22, 2020

stellargo changed the title ~~[JENKINS-62755] Introduce External Fingerprint cleanup API in Jenkins core~~ [JENKINS-62755] Introduce External Fingerprint cleanup API Jun 22, 2020

stellargo added 3 commits June 23, 2020 09:21

Moved test to jenkins

59bdf8d

Add license

f954d10

Fix license

7950fab

daniel-beck added the work-in-progress The PR is under active development, not ready to the final review label Jun 23, 2020

stellargo added 8 commits June 23, 2020 21:19

Fix some tests

c54aeb7

Fix all tests

82d65b9

Remove redundant imports

c274871

Add documentation; add test

3f6d00e

Fix errors in fingerprintertest

a5bb69d

Merge remote-tracking branch 'upstream/master' into fp_cleanup

b198246

Remove default fingerprint cleanup implementation

5daa43c

Remove override from execute

da9664f

stellargo added 2 commits June 24, 2020 21:55

Undo author deletion

383c471

Remove redundant import

a20c73e

oleg-nenashev reviewed Jun 24, 2020

View reviewed changes

stellargo added 7 commits June 24, 2020 23:15

Move fingerprint cleanup to storage

83aa71a

Fix spacing

c858939

Add new line

6f237c7

Minor fixes

ec5ff5a

Fix tests

ac6dfd7

Merge remote-tracking branch 'upstream/master' into fp_cleanup

7bc7fea

Add documentation

383cbe8

stellargo requested a review from oleg-nenashev June 24, 2020 22:09

stellargo changed the title ~~[JENKINS-62755] Introduce External Fingerprint cleanup API~~ [JENKINS-62755] Introduce Fingerprint Cleanup in External Storage API Jun 24, 2020

Introduce shouldNotCleanFingerprintsWhenDisabled test

e8552cc

stellargo added 2 commits July 17, 2020 20:36

Resolve merge conflicts

cb95e92

Fix default cleanup value

342bea6

oleg-nenashev requested a review from afalko July 17, 2020 15:16

oleg-nenashev requested changes Jul 17, 2020

View reviewed changes

Minor fixes

a9e5a3b

Enabled->Disabled

2a3896d

stellargo marked this pull request as draft July 17, 2020 15:54

Fix tests

512f086

stellargo marked this pull request as ready for review July 17, 2020 17:23

stellargo requested a review from oleg-nenashev July 17, 2020 17:26

oleg-nenashev reviewed Jul 17, 2020

View reviewed changes

core/src/main/java/jenkins/fingerprints/GlobalFingerprintConfiguration.java Outdated Show resolved Hide resolved

get -> is

578b46f

stellargo requested a review from oleg-nenashev July 17, 2020 18:00

oleg-nenashev approved these changes Jul 18, 2020

View reviewed changes

timja approved these changes Jul 18, 2020

View reviewed changes

oleg-nenashev added the ready-for-merge The PR is ready to go, and it will be merged soon if there is no negative feedback label Jul 20, 2020

oleg-nenashev merged commit be28d58 into jenkinsci:master Jul 21, 2020

daniel-beck mentioned this pull request Jul 27, 2020

Update at-since Javadoc #4877

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[JENKINS-62755] Introduce Fingerprint Cleanup in External Storage API #4817

[JENKINS-62755] Introduce Fingerprint Cleanup in External Storage API #4817

stellargo commented Jun 22, 2020 •

edited

Loading

stellargo commented Jun 22, 2020

stellargo commented Jun 24, 2020

stellargo commented Jun 24, 2020

oleg-nenashev left a comment

oleg-nenashev commented Jul 12, 2020

stellargo commented Jul 12, 2020 •

edited

Loading

oleg-nenashev commented Jul 17, 2020

stellargo commented Jul 17, 2020

oleg-nenashev left a comment

oleg-nenashev Jul 17, 2020

stellargo Jul 17, 2020

stellargo commented Jul 17, 2020

oleg-nenashev commented Jul 17, 2020

oleg-nenashev left a comment

timja Jul 18, 2020

stellargo Jul 18, 2020

oleg-nenashev commented Jul 20, 2020

[JENKINS-62755] Introduce Fingerprint Cleanup in External Storage API #4817

[JENKINS-62755] Introduce Fingerprint Cleanup in External Storage API #4817

Conversation

stellargo commented Jun 22, 2020 • edited Loading

Proposed changelog entries

Proposed upgrade guidelines

Submitter checklist

Desired reviewers

Maintainer checklist

stellargo commented Jun 22, 2020

stellargo commented Jun 24, 2020

stellargo commented Jun 24, 2020

oleg-nenashev left a comment

Choose a reason for hiding this comment

oleg-nenashev commented Jul 12, 2020

stellargo commented Jul 12, 2020 • edited Loading

oleg-nenashev commented Jul 17, 2020

stellargo commented Jul 17, 2020

oleg-nenashev left a comment

Choose a reason for hiding this comment

oleg-nenashev Jul 17, 2020

Choose a reason for hiding this comment

stellargo Jul 17, 2020

Choose a reason for hiding this comment

stellargo commented Jul 17, 2020

oleg-nenashev commented Jul 17, 2020

oleg-nenashev left a comment

Choose a reason for hiding this comment

timja Jul 18, 2020

Choose a reason for hiding this comment

stellargo Jul 18, 2020

Choose a reason for hiding this comment

oleg-nenashev commented Jul 20, 2020

stellargo commented Jun 22, 2020 •

edited

Loading

stellargo commented Jul 12, 2020 •

edited

Loading