NC | lifecycle | expire objects with batching

[nadavMiz]

Describe the Problem

we populate the delete candidates in memory which might cause issues when the delete candidates list very large. also we want to promote fairness between the lifecycle rules, so one process / bucket will not take most of the processing time. this PR adds populating expiry candidates with batching so we process NC_LIFECYCLE_LIST_BATCH_SIZE of objects each time for every rule in a loop. there is also a loop between bucket so a bucket will process only NC_LIFECYCLE_BUCKET_BATCH_SIZE objects each time

Explain the Changes

1. add two config variables:
   • NC_LIFECYCLE_LIST_BATCH_SIZE - determines how much objects we process on each loop for each rule. (it is the limit used in list_objects function)
   • NC_LIFECYCLE_BUCKET_BATCH_SIZE - determines the amount of objects that each bucket process every time in the loop before moving to the next bucket
2. added two loops, one outer loop for going over the buckets in batches. one inner loop to go over the rules for every bucket
3. added stat property to the lifecycle status both to every rule, and to every bucket. this property saves the key marker where the rule stopped on the last batch, so we can resume in the next batch. and it saves the is_finished property that determines if we finished processing the rule/ bucket and we don't need to resume
4. once all buckets are finished, exit the loop

Issues: Gaps

1. add unit tests
2. change bucket state inside update_status function
3. restructure object counting per bucket. currently its done in the bucket state
4. need better way to decide if we should not have another batch in case expire statement doesn't exists. currently check if rule state is empty

Testing Instructions:

manual testing:

1. configure the config variables to be small values for testing. NC_LIFECYCLE_BUCKET_BATCH_SIZE should be bigger than NC_LIFECYCLE_LIST_BATCH_SIZE
2. create a bucket and add lifecycle rule
3. add nuber of objects larger than NC_LIFECYCLE_BUCKET_BATCH_SIZE
4. rules should apply to all buckets in a cyclic order with batching
5. lifecycle run (sudo cat /var/log/noobaa/lifecycle/lifecycle_run_<timetamp>.json) should show state of all buckets and rules with is_finished = true. in case run finished before all objects were processed, state should include key_marker or where the rule last stopped. you can make the process end prematurely by setting NC_LIFECYCLE_TIMEOUT_MS in the timeout to be a small enough value so the run will not finish, but high enough value so at least on cycle will finish (one list-object operation)

• Doc added/updated
• Tests added