[ENH]: Add ability to set different block sizes for different blockfiles

[sanketkedia]

Description of changes

Summarize the changes made by this PR.

• Improvements & Bug fixes
  • We persist the max_block_size_bytes in the arrow metadata now and also add a new blockfile version for handling backward compatibility with old data
  • This is need so that spann posting list can have a block size different from other blockfiles
  • This PR also sets the spann PL block size to 5MiB
• New functionality
  • ...

Test plan

How are these changes tested?

• Tests pass locally with pytest for python, yarn test for js, cargo test for rust

Documentation Changes

None

[sanketkedia]

• [ENH]: Add ability to set different block sizes for different blockfiles #4948 👈 (View in Graphite)
• [ENH]: granular lock for posting list #4937
• [ENH]: More concurrent blockfilewriter #4889
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[github-actions]

Reviewer Checklist

Please leverage this checklist to ensure your code review is thorough before approving

Testing, Bugs, Errors, Logs, Documentation

• Can you think of any use case in which the code does not behave as intended? Have they been tested?
• Can you think of any inputs or external events that could break the code? Is user input validated and safe? Have they been tested?
• If appropriate, are there adequate property based tests?
• If appropriate, are there adequate unit tests?
• Should any logging, debugging, tracing information be added or removed?
• Are error messages user-friendly?
• Have all documentation changes needed been made?
• Have all non-obvious changes been commented?

System Compatibility

• Are there any potential impacts on other parts of the system or backward compatibility?
• Does this change intersect with any items on our roadmap, and if so, is there a plan for fitting them together?

Quality

• Is this code of a unexpectedly high quality (Readability, Modularity, Intuitiveness)

[propel-code-bot]

Add Per-Blockfile Block Size Setting with Versioning and Backward Compatibility

This PR introduces the ability to specify and persist different block sizes for distinct blockfiles in the Arrow blockstore. It updates the Arrow metadata schema to persist the max_block_size_bytes per-blockfile, adds a new V1.2 blockfile version for backward compatibility, and sets Spann posting list block size to 5MiB. The necessary migration logic is added to upgrade older blockfiles, and the blockfile provider APIs, writer options, and internal handling are updated to support and test this functionality. Additional temporary debug logs regarding IO task limits and scheduling are also present.

Key Changes:
• Blockfile writers and readers can now set and use a per-blockfile max_block_size_bytes, persisted in Arrow schema metadata starting with version V1.2.
• Arrow root and blockfile structs, creation logic, and migrations updated to support the new block size metadata and ensure backward compatibility for V1 and V1.1.
• Blockfile provider, writer options, and underlying APIs extended to accept per-writer block size overrides.
• Spann index sets its posting list block size to 5MiB via new config path.
• Extensive test updates/additions for V1.2 support, metadata round-trip, forking, and compatibility with legacy files.
• Temporary println! and debug logs added for tracing scheduler and IO task behavior.

Affected Areas:
• rust/blockstore/src/arrow/root.rs
• rust/blockstore/src/arrow/blockfile.rs
• rust/blockstore/src/arrow/ordered_blockfile_writer.rs
• rust/blockstore/src/arrow/provider.rs
• rust/blockstore/src/arrow/migrations.rs
• rust/blockstore/src/types/writer_options.rs
• rust/index/src/spann/types.rs
• tests in blockstore and spann

This summary was automatically generated by @propel-code-bot

[HammadB]

Should add a justification for why this

[codetheweb]

should update this method name