Skip to content

Conversation

@mag1c-h
Copy link
Contributor

@mag1c-h mag1c-h commented Oct 21, 2025

Purpose

In the default design, a Block file has two states:

  • activated – while data is being written
  • archived – after the write is finished

The state is encoded in the file name and switched by an in-directory rename.
On some storage systems (e.g. PCFS) this rename is expensive.
This PR introduces an optional layout that keeps the two states in separate directories so the switch becomes a fast cross-directory move.

Modifications

  • Turn SpaceLayout into an interface
  • Keep the existing behaviour in SpaceShardLayout (state encoded in file name, activated & archived live in the same directory)
  • Add SpaceShardTempLayout; when tempDumpDirEnable=true activated files are written to a temporary directory and moved to the final directory on completion

Test

  • CI passed with new added/existing test.
  • E2E tests in NFSStore passed.

@mag1c-h mag1c-h self-assigned this Oct 21, 2025
@mag1c-h mag1c-h requested a review from ygwpz as a code owner October 21, 2025 08:24
@ygwpz ygwpz changed the title feat: pluggable SpaceLayout to avoid in-place rename for better performance on PCFS [Feature]: pluggable SpaceLayout to avoid in-place rename for better performance on PCFS Oct 22, 2025
@mag1c-h mag1c-h merged commit b53b23a into ModelEngine-Group:develop Oct 22, 2025
3 checks passed
@mag1c-h mag1c-h deleted the dev_space_layout_interface branch October 22, 2025 09:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants