Add Async Read and Poll APIs in FileSystem #9564

akankshamahajan15 · 2022-02-14T22:04:45Z

Summary: This PR adds support for new APIs Async Read that reads the data
asynchronously and Poll API that checks if requested read request has
completed or not.

Usage: In RocksDB, we are currently planning to prefetch data
asynchronously during sequential scanning and RocksDB will call these
APIs to prefetch more data in advanced.

Design:

ReadAsync API submits the read request to underlying FileSystem in
order to read data asynchronously. When read request is completed,
callback function will be called. cb_arg is used by RocksDB to track the
original request submitted and IOHandle is used by FileSystem to keep track
of IO requests at their level.
The Poll API is added in FileSystem because the call could end up handling
completions for multiple different files which is not specific to a
FSRandomAccessFile instance. There could be multiple outstanding file reads
from different files in future and they can complete in any order.

Test Plan: Test will be added in separate PR.

facebook-github-bot · 2022-02-14T23:06:44Z

@akankshamahajan15 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2022-02-15T00:07:19Z

@akankshamahajan15 has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2022-02-15T00:08:30Z

@akankshamahajan15 has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2022-02-15T00:09:05Z

@akankshamahajan15 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

include/rocksdb/file_system.h

mrambacher

Without an example, I am having trouble following exactly how this API is supposed to function. I was assuming that the caller would invoke ReadAsync, get back something that represented that operation (an IOHandle?) and the callback was invoked when the operation was complete. But who creates and manages the IOHandle is unclear to me (is it the caller or the FileSystem?).

The lifetime of the returned Slice is also unclear (who owns/manages the "scratch" data backing the Slice?).

env/file_system_tracer.cc

mrambacher · 2022-02-16T01:14:23Z

include/rocksdb/file_system.h

+  // IOStatus::NotSupported() indicates that polling is not supported, and
+  // RocksDB will be informed of IO completions via the callback on another
+  // thread.
+  virtual IOStatus Poll(IOHandle* /*io_handle*/) {


Why is the "Poll" off of the FileSystem while the ReadAsync is off the RandomAccessFile? Shouldn't we poll the file?

What is the Status returned if the read has completed?

This also should not take an IOHandle* -- perhaps by reference or const reference?

The Poll call could end up handling completions for multiple different files, so its not specific to a FSRandomAccessFile instance. There could be multiple outstanding file reads and they can complete in any order.

include/rocksdb/file_system.h

mrambacher · 2022-02-16T01:23:35Z

include/rocksdb/file_system.h

@@ -733,6 +747,19 @@ struct FSReadRequest {
  IOStatus status;
 };

+// A read IO response structure for use by asynchronously Read APIs.
+struct FSReadResponse {


I am not sure I understand why you need both the IOHandle and the FSReadResponse. Shouldn't the IOHandle contain everything to identify the request (offset, length, status) and the callback to invoke when the read is complete?

anand1976 · 2022-02-16T03:57:51Z

Without an example, I am having trouble following exactly how this API is supposed to function. I was assuming that the caller would invoke ReadAsync, get back something that represented that operation (an IOHandle?) and the callback was invoked when the operation was complete. But who creates and manages the IOHandle is unclear to me (is it the caller or the FileSystem?).

I agree. An example, or a test with a mock FileSystem implementation, would make it clearer.

The IOHandle would be created by the FileSystem to store whatever context it needs to store. Its lifecycle is managed by RocksDB, hence its defined as a type erased unique_ptr. IOHandle could have been a void*, but managing the lifecycle would have been complicated. Would the FileSystem delete it after calling the callback, or when RocksDB calls Poll?

The lifetime of the returned Slice is also unclear (who owns/manages the "scratch" data backing the Slice?).

The scratch data and returned Slice lifetime works exactly like Read, i.e RocksDB reader owns scratch and Slice refers to either scratch or an mmap'd file.

mrambacher · 2022-02-16T15:53:16Z

Without an example, I am having trouble following exactly how this API is supposed to function. I was assuming that the caller would invoke ReadAsync, get back something that represented that operation (an IOHandle?) and the callback was invoked when the operation was complete. But who creates and manages the IOHandle is unclear to me (is it the caller or the FileSystem?).

I agree. An example, or a test with a mock FileSystem implementation, would make it clearer.

The IOHandle would be created by the FileSystem to store whatever context it needs to store. Its lifecycle is managed by RocksDB, hence its defined as a type erased unique_ptr. IOHandle could have been a void*, but managing the lifecycle would have been complicated. Would the FileSystem delete it after calling the callback, or when RocksDB calls Poll?

I am not sure why an IOHandle is different than any other unique_ptr we return (like a FSRandomAccessFile). It feels very similar to me and should have functions off of it (for Poll, Complete, etc)

The lifetime of the returned Slice is also unclear (who owns/manages the "scratch" data backing the Slice?).

The scratch data and returned Slice lifetime works exactly like Read, i.e RocksDB reader owns scratch and Slice refers to either scratch or an mmap'd file.

As the current API is proposed, I do not think this is clean. The "scratch" is part of the Request but not part of the Response. So the Async is responsible for maintaining the scratch pointer. It is unclear what its lifetime will be. If there are multiple concurrent Async requests, then how is the caller supposed to manage their scratch buffers? There is no correlation between the request, the callback and the scratch buffer.

akankshamahajan15 · 2022-02-16T17:09:00Z

Let me add more details about what's the purpose of this PR, future plan where we will be using these APIs and test units.

anand1976 · 2022-02-16T17:41:35Z

I am not sure why an IOHandle is different than any other unique_ptr we return (like a FSRandomAccessFile). It feels very similar to me and should have functions off of it (for Poll, Complete, etc)

I actually think Poll should be a FileSystem method and should allow the caller to wait for multiple IO completions - IOStatus Poll(std::vector<IOHandle*>& handles, size_t min_completions). It would be inefficient to wait for one IO at a time.

facebook-github-bot · 2022-02-16T21:33:58Z

@akankshamahajan15 has updated the pull request. You must reimport the pull request before landing.

anand1976

Mostly LGTM. Would be good to have a test that gives an example of how the FS should implement ReadAsync.

env/file_system_tracer.cc

include/rocksdb/file_system.h

facebook-github-bot · 2022-02-17T21:23:25Z

@akankshamahajan15 has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2022-02-17T21:29:35Z

@akankshamahajan15 has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2022-02-17T21:33:37Z

@akankshamahajan15 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2022-02-17T22:23:00Z

@akankshamahajan15 has updated the pull request. You must reimport the pull request before landing.

anand1976

LGTM. The test needs some more clarity. I've left some comments inline.

include/rocksdb/file_system.h

anand1976 · 2022-02-17T23:10:57Z

env/env_test.cc

+  // 2. Read file.
+  {
+    std::unique_ptr<FSRandomAccessFile> file;
+    ASSERT_OK(Env::Default()->GetFileSystem()->NewRandomAccessFile(


I was hoping we could have a mock file system (derived from FileSystemWrapper) and a mock random access file (derived from FSRandomAccessFileWrapper). It can be very simple, just override NewRandomAccessFile and ReadAsync respectively.

anand1976 · 2022-02-17T23:13:32Z

env/env_test.cc

+      req.len = kSectorSize;
+      req.scratch = NewAligned(kSectorSize * 8, 0).get();
+      ASSERT_OK(file->ReadAsync(&req, opts, callback, cb_arg,
+                                nullptr /*io_handle*/, nullptr /*dbg*/));


I think io_handle should always be non-null. Since we don't know, we should assume that the FS implementation would want to allocate something to store in io_handle.

I am also confused here. I thought the ReadAsync operation was responsible for returning an IOHandle that could later be checked. If not, how would one ever Poll for operations -- how is an IOHandle associated with an outstanding Async request?

I am updating the test to implement MockFS. That would clear up few things.

IOHandle will be used by FileSystem to distinguish between different read requests submit at their level. I haven't implemented PosixFileSystem support yet that will for example store some context when it has different read requests in any order.

env/env_test.cc

mrambacher · 2022-02-18T01:37:39Z

env/env_test.cc

+      req.len = kSectorSize;
+      req.scratch = NewAligned(kSectorSize * 8, 0).get();
+      ASSERT_OK(file->ReadAsync(&req, opts, callback, cb_arg,
+                                nullptr /*io_handle*/, nullptr /*dbg*/));


I am also confused here. I thought the ReadAsync operation was responsible for returning an IOHandle that could later be checked. If not, how would one ever Poll for operations -- how is an IOHandle associated with an outstanding Async request?

include/rocksdb/file_system.h

facebook-github-bot · 2022-02-18T17:00:48Z

@akankshamahajan15 has updated the pull request. You must reimport the pull request before landing.

akankshamahajan15 · 2022-02-18T17:02:01Z

In Progress: Update test to have a mock file system and submit requests to each thread.

anand1976 · 2022-02-18T18:00:59Z

env/env_test.cc

+TEST_F(TestAsyncRead, ReadAsync) {
+  std::shared_ptr<AsyncReadFS> fs =
+      std::make_shared<AsyncReadFS>(env_->GetFileSystem());
+  std::unique_ptr<Env> env(new CompositeEnvWrapper(env_, fs));


The env is not necessary. You can call fs directly.

env/env_test.cc

facebook-github-bot · 2022-02-18T18:48:04Z

@akankshamahajan15 has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2022-02-18T18:51:53Z

@akankshamahajan15 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2022-02-18T19:08:14Z

@akankshamahajan15 has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2022-02-18T19:26:56Z

@akankshamahajan15 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2022-02-18T20:10:24Z

@akankshamahajan15 has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2022-02-18T20:12:02Z

@akankshamahajan15 has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2022-02-18T20:12:41Z

@akankshamahajan15 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2022-02-18T20:31:40Z

@akankshamahajan15 has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2022-02-18T20:33:06Z

@akankshamahajan15 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

Summary: This PR adds support for new APIs Async Read that reads the data asynchronously and Poll API that checks if requested read request has completed or not. Usage: In RocksDB, we are currently plannig to prefetch data asynchronously during sequential scanning and RocksDB will call these APIs to prefetch more data in advanced. Design: - ReadAsync API submits the read request to underlying FileSystem in order to read data asynchronously. When read request is completed, callback function will be called. cb_arg is used by RocksDB to track the original request submit and IOHandle is used by FileSystem to keep track of IO requests at their level. - The Poll API is added in FileSystem because the call could end up handling completions for multiple different files which is not specific to a FSRandomAccessFile instance. There could be multiple outstanding file reads from different files in future and they can complete in any order. Test Plan: make check -j64

Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:

facebook-github-bot · 2022-02-18T22:38:29Z

@akankshamahajan15 has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2022-02-18T22:38:48Z

@akankshamahajan15 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

Signed-off-by: pingyu <yuping@pingcap.com>

facebook-github-bot added the CLA Signed label Feb 14, 2022

akankshamahajan15 changed the title ~~Add Async Read and Poll APIs in FileSystem~~ [WIP] Add Async Read and Poll APIs in FileSystem Feb 14, 2022

akankshamahajan15 marked this pull request as draft February 14, 2022 22:05

akankshamahajan15 force-pushed the fs_apis branch 2 times, most recently from 6706bec to 53d0377 Compare February 14, 2022 22:16

akankshamahajan15 marked this pull request as ready for review February 14, 2022 23:03

akankshamahajan15 changed the title ~~[WIP] Add Async Read and Poll APIs in FileSystem~~ Add Async Read and Poll APIs in FileSystem Feb 14, 2022

akankshamahajan15 requested a review from anand1976 February 14, 2022 23:06

akankshamahajan15 force-pushed the fs_apis branch from 53d0377 to f01c48a Compare February 15, 2022 00:07

anand1976 reviewed Feb 15, 2022

View reviewed changes

include/rocksdb/file_system.h Outdated Show resolved Hide resolved

include/rocksdb/file_system.h Outdated Show resolved Hide resolved

include/rocksdb/file_system.h Outdated Show resolved Hide resolved

include/rocksdb/file_system.h Outdated Show resolved Hide resolved

mrambacher reviewed Feb 16, 2022

View reviewed changes

akankshamahajan15 force-pushed the fs_apis branch from 400cf35 to 12f6c06 Compare February 16, 2022 21:33

anand1976 reviewed Feb 17, 2022

View reviewed changes

env/file_system_tracer.cc Outdated Show resolved Hide resolved

include/rocksdb/file_system.h Show resolved Hide resolved

akankshamahajan15 force-pushed the fs_apis branch from 12f6c06 to 66fe848 Compare February 17, 2022 21:23

akankshamahajan15 force-pushed the fs_apis branch from 66fe848 to 7a187e1 Compare February 17, 2022 21:29

akankshamahajan15 force-pushed the fs_apis branch from 7a187e1 to 7d44cf5 Compare February 17, 2022 22:22

anand1976 approved these changes Feb 17, 2022

View reviewed changes

mrambacher reviewed Feb 18, 2022

View reviewed changes

akankshamahajan15 force-pushed the fs_apis branch from 7d44cf5 to a552372 Compare February 18, 2022 17:00

anand1976 reviewed Feb 18, 2022

View reviewed changes

akankshamahajan15 force-pushed the fs_apis branch from 4b0f018 to be94c46 Compare February 18, 2022 19:08

akankshamahajan15 requested a review from anand1976 February 18, 2022 19:26

anand1976 approved these changes Feb 18, 2022

View reviewed changes

akankshamahajan15 force-pushed the fs_apis branch from 928d982 to a8e42a6 Compare February 18, 2022 20:12

akankshamahajan15 added 5 commits February 18, 2022 14:36

Addressed comments

b7adb4a

Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:

Add unit test

668efd3

Addressed comments

04f9a05

Addressed comments

08e0c84

akankshamahajan15 force-pushed the fs_apis branch from e660581 to 08e0c84 Compare February 18, 2022 22:38

facebook-github-bot closed this in 559525d Feb 19, 2022

pingyu pushed a commit to pingyu/rocksdb that referenced this pull request Oct 22, 2022

pick 559525d (facebook#9564)

9144457

Signed-off-by: pingyu <yuping@pingcap.com>

Add Async Read and Poll APIs in FileSystem #9564

Add Async Read and Poll APIs in FileSystem #9564

Conversation

akankshamahajan15 commented Feb 14, 2022 • edited Loading

facebook-github-bot commented Feb 14, 2022

facebook-github-bot commented Feb 15, 2022

facebook-github-bot commented Feb 15, 2022

facebook-github-bot commented Feb 15, 2022

mrambacher left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anand1976 commented Feb 16, 2022

mrambacher commented Feb 16, 2022

akankshamahajan15 commented Feb 16, 2022

anand1976 commented Feb 16, 2022

facebook-github-bot commented Feb 16, 2022

anand1976 left a comment

Choose a reason for hiding this comment

facebook-github-bot commented Feb 17, 2022

facebook-github-bot commented Feb 17, 2022

facebook-github-bot commented Feb 17, 2022

facebook-github-bot commented Feb 17, 2022

anand1976 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

facebook-github-bot commented Feb 18, 2022

akankshamahajan15 commented Feb 18, 2022

Choose a reason for hiding this comment

facebook-github-bot commented Feb 18, 2022

facebook-github-bot commented Feb 18, 2022

facebook-github-bot commented Feb 18, 2022

facebook-github-bot commented Feb 18, 2022

facebook-github-bot commented Feb 18, 2022

facebook-github-bot commented Feb 18, 2022

facebook-github-bot commented Feb 18, 2022

facebook-github-bot commented Feb 18, 2022

facebook-github-bot commented Feb 18, 2022

facebook-github-bot commented Feb 18, 2022

facebook-github-bot commented Feb 18, 2022

akankshamahajan15 commented Feb 14, 2022 •

edited

Loading