HDDS-13660. Ozone client support readVectored. #9402
Conversation
|
Thanks @ashishkumar50 for the patch. I have ported the corresponding contract test from Hadoop, which shows that some input validation is missing (negative length and offset, overlapping ranges, same ranges). Please feel free to pick it from adoroszlai@ee11cf3 and include in this PR. https://github.com/adoroszlai/ozone/actions/runs/19905480757/job/57061761264#step:13:5599 |
hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/MultipartInputStream.java
Outdated
Show resolved
Hide resolved
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/OzoneInputStream.java
Outdated
Show resolved
Hide resolved
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/OzoneFSInputStream.java
Outdated
Show resolved
Hide resolved
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/OzoneFSInputStream.java
Outdated
Show resolved
Hide resolved
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/OzoneFSInputStream.java
Outdated
Show resolved
Hide resolved
(cherry picked from commit ee11cf3)
|
@adoroszlai Thanks for the review, handled comments. |
| (offset, buffer) -> readRangeData(offset, buffer, initialPosition)); | ||
| } finally { | ||
| // Restore position | ||
| synchronized (this) { |
There was a problem hiding this comment.
what would happen in case of concurrent readVectored() calls?
There was a problem hiding this comment.
Synchronized now to avoid race condition in same concurrent stream reads.
hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/MultipartInputStream.java
Outdated
Show resolved
Hide resolved
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/VectoredReadUtils.java
Show resolved
Hide resolved
| }); | ||
| } finally { | ||
| // Restore position | ||
| synchronized (this) { |
There was a problem hiding this comment.
what would happen in case of concurrent readVectored() calls?
There was a problem hiding this comment.
Synchronized now to avoid race condition in same concurrent stream reads.
|
@chungen0126 @jojochuang @yandrey321 please take a look |
chungen0126
left a comment
chungen0126
left a comment
There was a problem hiding this comment.
Thanks for @ashishkumar50 the patch. Are there plans to optimize read vector further? The current implementation doesn't seem to fully support fully parallel reading for different ranges yet. I also added some comments.
| ); | ||
|
|
||
| // Restore position | ||
| seek(initialPosition); |
There was a problem hiding this comment.
I'm wondering if the seek to initialPosition is necessary here? Since the offset changes asynchronously, restoring it at this point might not work correctly. It seems like readRangeData already handles the position restoration correctly.
| ); | ||
|
|
||
| // Restore position before returning from method | ||
| seek(initialPosition); |
There was a problem hiding this comment.
Same comment as above.
6 participants
What changes were proposed in this pull request?
Ozone client support readVectored which allows reading multiple file ranges in parallel.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-13660
How was this patch tested?
Tests are added