Document Async decoder usage (#4043) (#78) #4046
Conversation
arrow-csv/src/reader/mod.rs
Outdated
| //! | ||
| //! # Async Usage | ||
| //! | ||
| //! The lower-level [`Decoder`] can be integrated with various forms of async data streams. |
There was a problem hiding this comment.
The "various forms" here is key, one of the major challenges in accommodating the async ecosystem is there is no one-size fits all async IO primitive, the intention behind the Decoder interface is to not be opinionated about where the bytes are coming from. This is similar to the AsyncFileReader trait we have for parquet, which similarly is not opinionated about what the underlying IO primitive actually is.
There was a problem hiding this comment.
This information should be part of the doc-string, not hidden in some PR comment.
tustvold
force-pushed
the
document-async-decoder
branch
from
250d95b
to
caa43fd
Compare
arrow-csv/src/reader/mod.rs
Outdated
| //! | ||
| //! # Async Usage | ||
| //! | ||
| //! The lower-level [`Decoder`] can be integrated with various forms of async data streams. |
There was a problem hiding this comment.
This information should be part of the doc-string, not hidden in some PR comment.
arrow-csv/src/reader/mod.rs
Outdated
| //! // Note: the decoder needs to be called with an empty | ||
| //! // array to delimit the final record |
There was a problem hiding this comment.
Where is this done in this example? I would have expected that there's some decoder.finish() method instead of some sentinel value for this purpose (I've seen a similar API design w/ some compressor crates before).
There was a problem hiding this comment.
The fallthrough of the branch above will end up here, I'll move it up to the match on poll_next_unpin to make it more clear
Which issue does this PR close?
Closes #4043
Closes #78
Rationale for this change
It isn't immediately obvious how to use these decoders in an async setting, so lets document it.
What changes are included in this PR?
Are there any user-facing changes?