ARROW-9869: [R] Implement full S3FileSystem/S3Options constructor #8197
Conversation
nealrichardson
force-pushed
the
r-s3-options
branch
2 times, most recently
from
5864fbf
to
425923d
Compare
|
@github-actions crossbow submit homebrew-r-autobrew |
|
Revision: 425923d Submitted crossbow builds: ursa-labs/crossbow @ actions-545
|
|
@github-actions crossbow submit homebrew-r-autobrew |
|
Revision: 2b89af1 Submitted crossbow builds: ursa-labs/crossbow @ actions-546
|
|
@github-actions crossbow submit homebrew-r-autobrew |
|
Revision: 51b93df Submitted crossbow builds: ursa-labs/crossbow @ actions-547
|
r/src/filesystem.cpp
Outdated
| /// If non-empty, override region with a connect string such as "localhost:9000" | ||
| SEXP endpoint_override = options["endpoint_override"]; | ||
| if (!Rf_isNull(endpoint_override)) { | ||
| s3_opts.endpoint_override = cpp11::as_cpp<std::string>(endpoint_override); |
There was a problem hiding this comment.
Why not reuse get_optional_string here?
There was a problem hiding this comment.
Because I don't want to set it if the user doesn't provide it (i.e. if it is NULL). Or is empty string "" the same thing? s3fs.h doesn't give these fields default values.
There was a problem hiding this comment.
The empty string is implicitly the default value.
nealrichardson
force-pushed
the
r-s3-options
branch
from
5180a51
to
1933a40
Compare
nealrichardson
force-pushed
the
r-s3-options
branch
from
cfbd63d
to
1e14dae
Compare
| #' @param ... extra parameters passed to `read_feather()`. | ||
| #' | ||
| #' @return A `data.frame` if `as_data_frame` is `TRUE` (the default), or an | ||
| #' Arrow [Table] otherwise | ||
| #' @seealso [read_feather()] for writing IPC files. [RecordBatchReader] for a | ||
| #' lower-level interface. | ||
| #' @export | ||
| read_ipc_stream <- function(file, as_data_frame = TRUE, ...) { | ||
| read_ipc_stream <- function(file, as_data_frame = TRUE, filesystem = NULL, ...) { | ||
| if (!inherits(file, "InputStream")) { |
There was a problem hiding this comment.
There are a lot of this happening. Does this really need file= and filesystem. Can this be something like:
read_ipc_stream(filesystem$open(file))?
There was a problem hiding this comment.
Unfortunately, it's spelled filesystem$OpenInputStream(file) if it's for reading, or filesystem$OpenOutputStream(file) for writing. I resisted adding the extra argument, but I thought it was less bad than requiring users to learn those.
But we could think of other solutions. One possibility is having some sort of FileLocator object that is filesystem + path; we've started exploring that in places in the C++ library, and I've been looking into similar things for ARROW-9870. Maybe we could revisit this question in that PR (next week-ish)?
|
I'm going to merge this and we can follow up with UX questions in the next PR. |
Also adds tests that run on minio, which I've tested locally, added documentation for how to run them, and added them to the macOS ("autobrew") nightly test run (because the Linux builds still don't have aws-sdk-cpp support).
Closes apache#8197 from nealrichardson/r-s3-options
Authored-by: Neal Richardson <neal.p.richardson@gmail.com>
Signed-off-by: Neal Richardson <neal.p.richardson@gmail.com>
Also adds tests that run on minio, which I've tested locally, added documentation for how to run them, and added them to the macOS ("autobrew") nightly test run (because the Linux builds still don't have aws-sdk-cpp support).
Closes apache#8197 from nealrichardson/r-s3-options
Authored-by: Neal Richardson <neal.p.richardson@gmail.com>
Signed-off-by: Neal Richardson <neal.p.richardson@gmail.com>
Also adds tests that run on minio, which I've tested locally, added documentation for how to run them, and added them to the macOS ("autobrew") nightly test run (because the Linux builds still don't have aws-sdk-cpp support).