block_store_fs enhanced to use xattrs and migrate blocks to tier2

[guymguym]

Explain the changes

1. Enhance backing_store_fs to use xattr to store block_md json instead of in separate .meta files.
2. This allows reducing the number of files needed on the file system by half.
3. Add new config option (false by default) - config.BLOCK_STORE_FS_TIER2_ENABLED - enable to set custom xattr trigger to signal the filesystem to migrate the data to archive tier (for filesystems that support that).
4. Added new config option (false by default) - config.BLOCK_STORE_FS_MAPPING_INFO_ENABLED - enable to pass on extra recovery information (which bucket, object, offset, etc) to be added in block_md xattr, for cases when recovery without the db is needed.

Issues: Fixed #xxx / Gap #xxx

1. NA

Testing Instructions:

1. Regression of backing_store_fs (PV pools from the operator).
2. Enable the new configs to test the new options and check the xattr of the blocks (e.g xattr -l <filename> on mac).

[tangledbytes]

I am just trying to understand here. What happens if the unlink succeeds but this fails and _test_root_path_exists throws as well?

[guymguym]

The usage is not going to update in that case, which is indeed undesired, but since usage calculation is not atomic anyhow, any failure can cause this.

I explained to @romayalon's question in the comment above that the overwrite block case is not an actual flow we invoke on purpose.

The reason we decided that write_block should handle overwrites was that we didn't want to fail user uploads on one hand, and we didn't want the usage to start getting skewed because of retries so we had to check if this is an overwrite first and "reduce" that replaced size.

I guess that simply failing the request if the same block is written with different data could just be better.

[guymguym]

For now I will keep the existing behavior and we can revise as needed in the future.