Tiering with storage class support - TTL based worker + TMFS blocks migration

[tangledbytes]

Explain the changes

Context: NooBaa in its write flows marks an object for migration so that TMFS can migrate to another tier2 and NooBaa's read flow can recall the block from the other tier using TMFS calls but this would leave a bunch of objects on the disk (cache).

This PR adds a BG Worker which iterates over the chunks associated with a bucket (the bucket needs to have at least 2 tiers and for the case of TMFS, the second tier HAS to share the same pool) and select the chunks which haven't been read for longer than 30 mins (configurable) and move them to second tier and evict the associated the blocks.

Tested In the following way

1. Upload 2 objects, testobject1 and testobject2 of sizes 2G and 4G respectively.
2. Let the BG worker do its job.
3. Run find storage/backingstores -name '*.data' -type f | xargs -n1 -I{} sh -c "echo 'file: {} => '; xattr {}; echo" which correctly shows user._trigger.migrate being set on the files.
4. Run psql -d nbcore -U postgres -c "SELECT data FROM datachunks;" shows the second tier as the tier selected for datachunks.
5. TTL Worker does not iterates over the blocks which are in second tier.

• Doc added/updated
• Tests added

[guymguym]

@Utkarsh-pro The structure looks good.
Notice the comment on adding a time condition to the evictor.

[guymguym]

Hey @Utkarsh-pro
This PR is superb, but I raised several changes which are important (+ some cosmetic).
I'd love to discuss it further.
!!

[guymguym]

Ultimately we would want that tier 2 creation will be a system configuration and not require every bucket to be configured specifically. I would leave it like this for this PR, but this will follow soon.

[tangledbytes]

@guymguym lemme try to summarize the proposed changes (specifically storage class) here:

• The idea is to have the construct of storage classes in our tiers. Each tier can have <= 1 storage class specified.
• Storage class will just communicate some additional details to the underlying block store. This detail will be used by functions like move_blocks_to_storage_class.
• move_blocks_to_storage_class must consume block ids and storage class and should try to move it to that storage class doing something (here something would be different for each block store which plans to support this).
• TTL worker should invoke map_client.run() which will do all of the heavy lifting.
• Read flow already moves the chunks across the tiers. We will need to modify flow (maybe map_client.read_object_mapping?) such that it also starts invoking move_blocks_to_storage_class.

Clarifications

1. Are storage classes transparent to S3? By "transparent" I mean that S3 responses will always mention StorageClass: Standard?

[guymguym]

@Utkarsh-pro I think you got it, just two notes:

• TTL worker should invoke map_client.run() which will do all of the heavy lifting.
• Read flow already moves the chunks across the tiers. We will need to modify flow (maybe map_client.read_object_mapping?) such that it also starts invoking move_blocks_to_storage_class.

Both TTL worker and TTF worker already call build_chunks and therefore invoke map_client, so there is no need to add such a call in the workers, only to make this call also handle the storage class move as I suggested in this comment - #7313 (comment)

1. Are storage classes transparent to S3? By "transparent" I mean that S3 responses will always mention StorageClass: Standard?

Yes. For now we only handle the internal mappings. Exposing tiers to the clients is a whole new challenge.

[guymguym]

Just to avoid confusion, I would change to not call these block_store_fs methods with *glacier in their name, because while the storage class is called glacier, we are translating it to TMFS. So I think I would so something like this here, and if TMFS is not enabled we should probably continue and throw the error (unsupported storage class ) because we don't have a glacier tier to move to. WDYT?

Suggested change

	if (storage_class === 'GLACIER') {
	return this._move_blocks_to_glacier(block_ids);
	}
	if (storage_class === 'GLACIER') {
	if (config.BLOCK_STORE_FS_TMFS_ENABLED) {
	return this._move_blocks_to_tmfs(block_ids);
	}
	}

[guymguym]

how about using _.xor() instead of creating a special function for this esoteric case?

[tangledbytes]

I .. did not know about xor

[guymguym]

@Utkarsh-pro All in all this is absolutely fantastic! I added couple of small comments, but it can also be addressed in next PRs. Godspeed.