[FEATURE] Customize MaxRecurringJobRetain

[c3y1huang]

Is your feature request related to a problem? Please describe (👍 if you like this request)

Hardcoding the MaxRecurringJobRetain may be inadequate in some cases.

Describe the solution you'd like

Introduce a setting or field to allow customizing the max retaining number.

Describe alternatives you've considered

None

Additional context

longhorn/longhorn-manager#1827

[innobead]

longhorn/longhorn-manager#1827 (review)

[mantissahz]

Updates:

1. When the number of snapshots for a volume is up to 250, the recurring job pod will still try to create a new snapshot but it would be blocked about 30 minutes and then fail, and after retrying 3 times, the job status would show it failed.
2. Keeping snapshots of backups might need to be concerned.
3. Snapshot space management [FEATURE] Longhorn snapshot space management #6563

[longhorn-io-github-bot]

Pre Ready-For-Testing Checklist

• Where is the reproduce steps/test steps documented?
  The reproduce steps/test steps are at:

1. Setup a Longhorn system
2. Set the global setting recurring-job-max-retain to 10
3. Create a recurring job with the retain number 15 and it will fail.
4. Create a recurring job with the retain number 8 and it will succeed.
5. Set the global setting recurring-job-max-retain to 251 and it will fail.

• Is there a workaround for the issue? If so, where is it documented?
  The workaround is at:

• Does the PR include the explanation for the fix or the feature?

• Does the PR include deployment change (YAML/Chart)? If so, where are the PRs for both YAML file and Chart?
  The PR for the YAML change is at:
  The PR for the chart change is at:

• Have the backend code been merged (Manager, Engine, Instance Manager, BackupStore etc) (including backport-needed/*)?
  The PR is at
  feat: customize MaxRecurringJobRetain longhorn-manager#2379

• Which areas/issues this PR might have potential impacts on?
  Area
  Issues

• If labeled: require/LEP Has the Longhorn Enhancement Proposal PR submitted?
  The LEP PR is at
  feat(lep): customize maximum recurring job retain #6556

• If labeled: area/ui Has the UI issue filed or ready to be merged (including backport-needed/*)?
  The UI issue/PR is at

• If labeled: require/doc Has the necessary document PR submitted or merged (including backport-needed/*)?
  The documentation issue/PR is at

• If labeled: require/automation-e2e Has the end-to-end test plan been merged? Have QAs agreed on the automation test case? If only test case skeleton w/o implementation, have you created an implementation issue (including backport-needed/*)
  The automation skeleton PR is at
  The automation test case PR is at
  The issue of automation test case implementation is at (please create by the template)

• If labeled: require/automation-engine Has the engine integration test been merged (including backport-needed/*)?
  The engine automation PR is at

• If labeled: require/manual-test-plan Has the manual test plan been documented?
  The updated manual test plan is at

• If the fix introduces the code for backward compatibility Has a separate issue been filed with the label release/obsolete-compatibility?
  The compatibility issue is filed at

[chriscchien]

Verified pass on longhorn master(longhorn-manager 88d986) with test steps

1. set setting recurring-job-max-retention to 10 and create recurring job with the retain number 15 failed

failed to create recurring job c-ru27cr: admission webhook "validator.longhorn.io" denied the request: retain value should be less than or equal to 10

2. set setting recurring-job-max-retention to 251 failed

failed to set the setting recurring-job-max-retention with invalid value 251: value 251 of settings recurring-job-max-retention is invalid: the value 251 should be between 1 and 250

3. Create recurring job with the retain number 8, the recurring job worked well.
4. test_recurring_job.py worked well