-
Notifications
You must be signed in to change notification settings - Fork 3.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ARROW-16887: [R][Docs] Update Filesystem Vignette for GCS #13601
Conversation
|
r/vignettes/fs.Rmd
Outdated
> In Windows and macOS binary packages, S3 and GCS support are included. On Linux when | ||
installing from source, S3 and GCS support is not enabled by default, and it has | ||
additional system requirements. See `vignette("install", package = "arrow")` | ||
for details. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
IIRC we recently changed how the Linux installation works. Do these lines still apply?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@nealrichardson do you know the answer to this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oof, the answer is "it depends", it's not on by default in the pure source build, but if your system meets the requirements (which are too fussy to expand on here), you may get a prebuilt libarrow binary, which (may) have support for S3 and GCS.
@@ -355,7 +355,7 @@ specifying a S3 path: | |||
|
|||
.. code-block:: python | |||
|
|||
dataset = ds.dataset("s3://ursa-labs-taxi-data/", partitioning=["year", "month"]) | |||
dataset = ds.dataset("s3://voltrondata-labs-datasets/nyc-taxi/") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there also a reason for removing the partitioning
option?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a newer version of the dataset that is Hive partitioned, so doesn't require explicitly passing partitioning.
r/tests/testthat/test-filesystem.R
Outdated
capture.output(print(bucket)), | ||
"SubTreeFileSystem: gs://voltrondata-labs-datasets/" | ||
) | ||
skip_on_os("windows") # FIXME |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What happens on Windows exactly? These bare FIXMEs are not explanatory.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This was copied over from the S3 tests. I'm not sure what happens, but I can try and see.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I deleted the skips and seems to run fine locally and in CI.
@@ -18,6 +18,7 @@ vignettes/nyc-taxi/ | |||
arrow_*.tar.gz | |||
arrow_*.tgz | |||
extra-tests/files | |||
.deps |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I added this because it got a lot of files in it while building the R doc site; lmk if there's a good reason to not add it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No objection; what creates this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When I run pkgdown::build_site()
within RStudio, during the "Installing package into temporary library" it creates this directory.
r/man/array.Rd
Outdated
\if{html}{\out{<div class="sourceCode">}}\preformatted{a <- Array$create(x) | ||
\preformatted{a <- Array$create(x) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This isn't intentional. I just ran make doc
locally. Is something about my R out of date, or does that command need an update?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Upgrading to roxygen2 7.2 fixed this.
96f7897
to
176a7f5
Compare
cc @thisisnic would you be wiling to review? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I haven't read in detail, but this looks fine to me overall.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A few suggestions here and there but overall LGTM, thanks for taking this on.
@@ -18,6 +18,7 @@ vignettes/nyc-taxi/ | |||
arrow_*.tar.gz | |||
arrow_*.tgz | |||
extra-tests/files | |||
.deps |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No objection; what creates this?
$ tip_amount : num [1:10918] 3 0 4 1 1 6 0 1.35 0 5.9 ... | ||
$ total_amount : num [1:10918] 18.8 13.3 20.3 15.8 13.3 ... | ||
$ passenger_count: int [1:10918] 3 2 1 1 1 1 1 1 1 3 ... | ||
$ tip_pct : num [1:10918] 0.1596 0 0.197 0.0633 0.0752 ... |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not in scope here (and probably not relevant for this exact line, this just made me think of it): we should show an example of glimpse()
in this vignette somewhere.
r/vignettes/fs.Rmd
Outdated
> In Windows and macOS binary packages, S3 and GCS support are included. On Linux when | ||
installing from source, S3 and GCS support is not enabled by default, and it has | ||
additional system requirements. See `vignette("install", package = "arrow")` | ||
for details. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oof, the answer is "it depends", it's not on by default in the pure source build, but if your system meets the requirements (which are too fussy to expand on here), you may get a prebuilt libarrow binary, which (may) have support for S3 and GCS.
r/vignettes/fs.Rmd
Outdated
``` | ||
|
||
tells the `S3FileSystem` that it should allow the creation of new buckets and to | ||
talk to Google Storage instead of S3. The latter works because GCS implements an |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
! Why didn't we do this before then?!
r/vignettes/fs.Rmd
Outdated
``` | ||
gs://anonymous@voltrondata-labs-datasets/nyc-taxi/?retry_limit_seconds=10 | ||
``` | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It might be worth showing how this is equivalent to GcsFileSystem$create(anonymous = TRUE, retry_limit_seconds = 10)$path("voltrondata-labs-datasets/nyc-taxi/")
or whatever it is. As in, URI query params are useful but not the only way to provide these options.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh, since the FS objects are introduced below, maybe this note about equivalence should be introduced down there.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah I wasn't sure I wanted to do this refactor, but it did feel like it would almost be easier to introduce GcsFileSystem$create()
and then explain options in URIs.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've reordered them so we introduce the filesystems and that made it more natural to make the statement you suggested.
With combined vacation + rstudio::conf I won't have time to look at this til 2nd August, so probably best not wait for me! |
@jonkeane @nealrichardson can we go ahead and merge it? |
I already approved and don't feel the need to re-review, I'm sure @wjones127 took care of everything, please merge. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Merging, thanks @wjones127!
Benchmark runs are scheduled for baseline = ab8c92c and contender = 42647dc. 42647dc is a master commit associated with this PR. Results will be available as each benchmark for each run completes. |
['Python', 'R'] benchmarks have high level of regressions. |
This PR: * Moves minio integration tests into a generic suite that is now run on minio (S3 emulator) and GCS testbench (GCS emulator). This is run in CI. * Move Minio and GCS test server initialization to within the tests. This makes it easier to setup the background processes in a cross-platform way. * MinIO and GCS tests are now run on R Ubuntu CI. MinIO is now run on Windows CI. I couldn't get GCS to run on Windows CI yet, due to some issue where the tests hang (I believe this is an issue with the test setup and not the functionality). See follow up at: [ARROW-17149: [R] Enable GCS tests for Windows](https://issues.apache.org/jira/browse/ARROW-17149) * Sets the default retry timeout to 15 seconds to mitigate issue described by ARROW-17020. This affects explicitly-created fs with `GcsFileSystem$create()` (and `gs_bucket()` introducted in #13601), but not URIs. Authored-by: Will Jones <willjones127@gmail.com> Signed-off-by: Neal Richardson <neal.p.richardson@gmail.com>
This PR: * Moves minio integration tests into a generic suite that is now run on minio (S3 emulator) and GCS testbench (GCS emulator). This is run in CI. * Move Minio and GCS test server initialization to within the tests. This makes it easier to setup the background processes in a cross-platform way. * MinIO and GCS tests are now run on R Ubuntu CI. MinIO is now run on Windows CI. I couldn't get GCS to run on Windows CI yet, due to some issue where the tests hang (I believe this is an issue with the test setup and not the functionality). See follow up at: [ARROW-17149: [R] Enable GCS tests for Windows](https://issues.apache.org/jira/browse/ARROW-17149) * Sets the default retry timeout to 15 seconds to mitigate issue described by ARROW-17020. This affects explicitly-created fs with `GcsFileSystem$create()` (and `gs_bucket()` introducted in apache#13601), but not URIs. Authored-by: Will Jones <willjones127@gmail.com> Signed-off-by: Neal Richardson <neal.p.richardson@gmail.com>
…Hub issue numbers (#34260) Rewrite the Jira issue numbers to the GitHub issue numbers, so that the GitHub issue numbers are automatically linked to the issues by pkgdown's auto-linking feature. Issue numbers have been rewritten based on the following correspondence. Also, the pkgdown settings have been changed and updated to link to GitHub. I generated the Changelog page using the `pkgdown::build_news()` function and verified that the links work correctly. --- ARROW-6338 #5198 ARROW-6364 #5201 ARROW-6323 #5169 ARROW-6278 #5141 ARROW-6360 #5329 ARROW-6533 #5450 ARROW-6348 #5223 ARROW-6337 #5399 ARROW-10850 #9128 ARROW-10624 #9092 ARROW-10386 #8549 ARROW-6994 #23308 ARROW-12774 #10320 ARROW-12670 #10287 ARROW-16828 #13484 ARROW-14989 #13482 ARROW-16977 #13514 ARROW-13404 #10999 ARROW-16887 #13601 ARROW-15906 #13206 ARROW-15280 #13171 ARROW-16144 #13183 ARROW-16511 #13105 ARROW-16085 #13088 ARROW-16715 #13555 ARROW-16268 #13550 ARROW-16700 #13518 ARROW-16807 #13583 ARROW-16871 #13517 ARROW-16415 #13190 ARROW-14821 #12154 ARROW-16439 #13174 ARROW-16394 #13118 ARROW-16516 #13163 ARROW-16395 #13627 ARROW-14848 #12589 ARROW-16407 #13196 ARROW-16653 #13506 ARROW-14575 #13160 ARROW-15271 #13170 ARROW-16703 #13650 ARROW-16444 #13397 ARROW-15016 #13541 ARROW-16776 #13563 ARROW-15622 #13090 ARROW-18131 #14484 ARROW-18305 #14581 ARROW-18285 #14615 * Closes: #33631 Authored-by: SHIMA Tatsuya <ts1s1andn@gmail.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>
…to GitHub issue numbers (apache#34260) Rewrite the Jira issue numbers to the GitHub issue numbers, so that the GitHub issue numbers are automatically linked to the issues by pkgdown's auto-linking feature. Issue numbers have been rewritten based on the following correspondence. Also, the pkgdown settings have been changed and updated to link to GitHub. I generated the Changelog page using the `pkgdown::build_news()` function and verified that the links work correctly. --- ARROW-6338 apache#5198 ARROW-6364 apache#5201 ARROW-6323 apache#5169 ARROW-6278 apache#5141 ARROW-6360 apache#5329 ARROW-6533 apache#5450 ARROW-6348 apache#5223 ARROW-6337 apache#5399 ARROW-10850 apache#9128 ARROW-10624 apache#9092 ARROW-10386 apache#8549 ARROW-6994 apache#23308 ARROW-12774 apache#10320 ARROW-12670 apache#10287 ARROW-16828 apache#13484 ARROW-14989 apache#13482 ARROW-16977 apache#13514 ARROW-13404 apache#10999 ARROW-16887 apache#13601 ARROW-15906 apache#13206 ARROW-15280 apache#13171 ARROW-16144 apache#13183 ARROW-16511 apache#13105 ARROW-16085 apache#13088 ARROW-16715 apache#13555 ARROW-16268 apache#13550 ARROW-16700 apache#13518 ARROW-16807 apache#13583 ARROW-16871 apache#13517 ARROW-16415 apache#13190 ARROW-14821 apache#12154 ARROW-16439 apache#13174 ARROW-16394 apache#13118 ARROW-16516 apache#13163 ARROW-16395 apache#13627 ARROW-14848 apache#12589 ARROW-16407 apache#13196 ARROW-16653 apache#13506 ARROW-14575 apache#13160 ARROW-15271 apache#13170 ARROW-16703 apache#13650 ARROW-16444 apache#13397 ARROW-15016 apache#13541 ARROW-16776 apache#13563 ARROW-15622 apache#13090 ARROW-18131 apache#14484 ARROW-18305 apache#14581 ARROW-18285 apache#14615 * Closes: apache#33631 Authored-by: SHIMA Tatsuya <ts1s1andn@gmail.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>
This PR:
voltrondata-labs-datasets
bucket.gs_bucket()
function to R, which parallels the design of thes3_bucket()
function.fs.Rmd
vignette to also discuss GCS. Left discussion about authentication for a follow-up, partly because I am still confused how GCP auth works for non-application use.