Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement batch expiration for permanent deletes of expired objects #17792

Closed
wants to merge 1 commit into from

Conversation

harshavardhana
Copy link
Member

@harshavardhana harshavardhana commented Aug 3, 2023

Description

Implement batch expiration for permanent deletes of expired objects

Motivation and Context

expire:
  apiVersion: v1
  bucket: BUCKET
  prefix: PREFIX
  # optional flags-based filtering criteria
  # for all objects
  flags:
    filter:
      newerThan: "7d" # match objects newer than this value (e.g. 7d10h31s)
      olderThan: "7d" # match objects older than this value (e.g. 7d10h31s)
      createdAfter: "date" # match objects created after "date"
      createdBefore: "date" # match objects created before "date"
      tags:
        - key: "name"
          value: "pick*" # match objects with tag 'name', with all values starting with 'pick'
      metadata:
        - key: "content-type"
          value: "image/*" # match objects with 'content-type', with all values starting with 'image/'
    notify:
      endpoint: "https://notify.endpoint" # notification endpoint to receive job status events
      token: "Bearer xxxxx" # optional authentication token for the notification endpoint
    retry:
      attempts: 10 # number of retries for the job before giving up
      delay: "500ms" # least amount of delay between each retry

How to test this PR?

You would need two PRs

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Optimization (provides speedup with no functional changes)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • Fixes a regression (If yes, please add commit-id or PR # here)
  • Unit tests added/updated
  • Internal documentation updated
  • Create a documentation update request here

cmd/batch-expire.go Show resolved Hide resolved
cmd/batch-expire.go Show resolved Hide resolved
cmd/batch-expire.go Show resolved Hide resolved
@harshavardhana harshavardhana force-pushed the fix-batch branch 2 times, most recently from a40b91d to bd0ee65 Compare August 4, 2023 22:52
@harshavardhana harshavardhana changed the title Implement batch expiration for targeted ILM Implement batch expiration for permanent deletes of expired objects Aug 4, 2023
@harshavardhana harshavardhana force-pushed the fix-batch branch 2 times, most recently from b8cc6c7 to d64f740 Compare August 5, 2023 04:04
}
rnd := rand.New(rand.NewSource(time.Now().UnixNano()))

skip := func(info FileInfo) (ok bool) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
skip := func(info FileInfo) (ok bool) {
filter := func(info FileInfo) (ok bool) {

opts.WalkFitler takes a function which returns true if the entry should be returned as part of the Walk calls results. skip gives the opposite impression/idea. This is a minor change but will help future readers understand the intent better.

krisis pushed a commit to krisis/minio that referenced this pull request Aug 29, 2023
Based on an initial PR from Harshavardhana -
minio#17792
krisis pushed a commit to krisis/minio that referenced this pull request Aug 29, 2023
Based on an initial PR from Harshavardhana -
minio#17792
krisis pushed a commit to krisis/minio that referenced this pull request Aug 30, 2023
Based on an initial PR from Harshavardhana -
minio#17792
krisis pushed a commit to krisis/minio that referenced this pull request Sep 1, 2023
Based on an initial PR from Harshavardhana -
minio#17792
krisis pushed a commit to krisis/minio that referenced this pull request Sep 6, 2023
Based on an initial PR from Harshavardhana -
minio#17792
krisis pushed a commit to krisis/minio that referenced this pull request Sep 6, 2023
Based on an initial PR from Harshavardhana -
minio#17792
krisis pushed a commit to krisis/minio that referenced this pull request Oct 13, 2023
Based on an initial PR from Harshavardhana -
minio#17792
krisis pushed a commit to krisis/minio that referenced this pull request Oct 16, 2023
Based on an initial PR from Harshavardhana -
minio#17792
krisis pushed a commit to krisis/minio that referenced this pull request Oct 20, 2023
Based on an initial PR from Harshavardhana -
minio#17792
krisis pushed a commit to krisis/minio that referenced this pull request Oct 23, 2023
Based on an initial PR from Harshavardhana -
minio#17792
krisis pushed a commit to krisis/minio that referenced this pull request Nov 1, 2023
Based on an initial PR from Harshavardhana -
minio#17792
krisis pushed a commit to krisis/minio that referenced this pull request Nov 2, 2023
Based on an initial PR from Harshavardhana -
minio#17792
krisis pushed a commit to krisis/minio that referenced this pull request Nov 7, 2023
Based on an initial PR from Harshavardhana -
minio#17792
krisis pushed a commit to krisis/minio that referenced this pull request Nov 9, 2023
Based on an initial PR from Harshavardhana -
minio#17792
krisis pushed a commit to krisis/minio that referenced this pull request Nov 14, 2023
Based on an initial PR from Harshavardhana -
minio#17792
krisis pushed a commit to krisis/minio that referenced this pull request Nov 15, 2023
Based on an initial PR from Harshavardhana -
minio#17792
krisis pushed a commit to krisis/minio that referenced this pull request Nov 16, 2023
Based on an initial PR from Harshavardhana -
minio#17792
krisis pushed a commit to krisis/minio that referenced this pull request Nov 17, 2023
Based on an initial PR from Harshavardhana -
minio#17792
krisis pushed a commit to krisis/minio that referenced this pull request Nov 20, 2023
Based on an initial PR from Harshavardhana -
minio#17792
krisis pushed a commit to krisis/minio that referenced this pull request Nov 23, 2023
Based on an initial PR from Harshavardhana -
minio#17792
krisis pushed a commit to krisis/minio that referenced this pull request Nov 28, 2023
Based on an initial PR from Harshavardhana -
minio#17792
krisis pushed a commit to krisis/minio that referenced this pull request Nov 30, 2023
Based on an initial PR from Harshavardhana -
minio#17792
krisis pushed a commit to krisis/minio that referenced this pull request Dec 2, 2023
Based on an initial PR from Harshavardhana -
minio#17792
harshavardhana pushed a commit that referenced this pull request Dec 2, 2023
Based on an initial PR from -
#17792

But fully completes it with newer finalized YAML spec.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants