Skip to content

HDDS-7428. Change AllocateBlock API for precise allocation and better abstraction#3898

Closed
kaijchen wants to merge 18 commits intoapache:masterfrom
kaijchen:HDDS-7428
Closed

HDDS-7428. Change AllocateBlock API for precise allocation and better abstraction#3898
kaijchen wants to merge 18 commits intoapache:masterfrom
kaijchen:HDDS-7428

Conversation

@kaijchen
Copy link
Member

@kaijchen kaijchen commented Oct 27, 2022

What changes were proposed in this pull request?

Currently, AllocateBlock takes two major arguments: size and num. There are some problems in the current API:

  1. For EC replication type, the allocated size for clients to write is size * ec.numData * num, which is not a good abstraction from the HDDS client's view.
  2. It assumes all blocks are in same size. However, sometimes it's useful to let SCM allocate arbitrary size the client wants (e.g. when client know the key size in advance).

In this Jira, I wish to add a requestedSize field in the AllocateBlock RPC.
SCM will calculate how many blocks to allocate based on requestedSize,
and return size of each block allocated to client.

The old size field is used for the client specified block size, if set to 0, SCM will decide the block size.
The old numBlock field is not used anymore.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-7428

How was this patch tested?

UT: TestSCMBlockProtocolServer#testAllocateBlock
CI: https://github.com/kaijchen/ozone/actions/runs/3345373585

@kaijchen
Copy link
Member Author

Hi @JacksonYao287, this PR implements the idea of better AllocateBlock abstraction #3882 (review).

It's still work in progress though (need to add tests and handle compatibility),
can you help review the API design first?

@kaijchen
Copy link
Member Author

In Ozone weekly meeting, @ChenSammi mentioned the client might need to specify custom block size for compatibility with HDFS. So I have reused the size field in proto to denote the client specified block size.

@kaijchen kaijchen marked this pull request as ready for review October 28, 2022 13:40
@kaijchen kaijchen marked this pull request as draft November 4, 2022 06:48
@kaijchen
Copy link
Member Author

kaijchen commented Nov 4, 2022

Converted to draft to address compatibility issues.

@kaijchen kaijchen changed the title HDDS-7428. Calculate number of blocks to allocate on SCM HDDS-7428. Change AllocateBlock for precise allocation and better abstraction Nov 4, 2022
@kaijchen kaijchen changed the title HDDS-7428. Change AllocateBlock for precise allocation and better abstraction HDDS-7428. Change AllocateBlock API for precise allocation and better abstraction Nov 4, 2022
@kaijchen kaijchen closed this Jun 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant