Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ARROW-17188: [R] Update news for 9.0.0 #13726

Merged
merged 3 commits into from
Jul 28, 2022
Merged

ARROW-17188: [R] Update news for 9.0.0 #13726

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
d18702d
doc: write news
wjones127 Jul 24, 2022
9cc9caa
docs: cleanup
wjones127 Jul 27, 2022
2084df2
docs: reorder and consolidate
wjones127 Jul 27, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
47 changes: 41 additions & 6 deletions r/NEWS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,19 +19,54 @@

# arrow 8.0.0.9000
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't this be 9.0.0?

Suggested change
# arrow 8.0.0.9000
# arrow 9.0.0

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, I see this is supposed to be done on the utils-prepare.sh script as with the other versions.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct, we don't make this change manually


* The `arrow.dev_repo` for nightly builds of the R package and prebuilt
libarrow binaries is now https://nightlies.apache.org/arrow/r/.
* `lubridate::parse_date_time()` datetime parser:
* `orders` with year, month, day, hours, minutes, and seconds components are supported.
* the `orders` argument in the Arrow binding works as follows: `orders` are transformed into `formats` which subsequently get applied in turn. There is no `select_formats` parameter and no inference takes place (like is the case in `lubridate::parse_date_time()`).
## Arrays and Tables

* Table and RecordBatch `$num_rows()` method returns a double (previously integer), avoiding integer overflow on larger tables. (ARROW-14989, ARROW-16977)

## Reading and Writing

* New functions `read_ipc_file()` and `write_ipc_file()` are added.
These functions are almost the same as `read_feather()` and `write_feather()`,
but differ in that they only target IPC files (Feather V2 files), not Feather V1 files.
* `read_arrow()` and `write_arrow()`, deprecated since 1.0.0 (July 2020), have been removed.
Instead of these, use the `read_ipc_file()` and `write_ipc_file()` for IPC files, or,
`read_ipc_stream()` and `write_ipc_stream()` for IPC streams.
* `write_parquet()` now defaults to writing Parquet format version 2.4 (was 1.0). Previously deprecated arguments `properties` and `arrow_properties` have been removed; if you need to deal with these lower-level properties objects directly, use `ParquetFileWriter`, which `write_parquet()` wraps.
* `write_parquet()` now defaults to writing Parquet format version 2.4 (was 1.0). Previously deprecated arguments `properties` and `arrow_properties` have been removed; if you need to deal with these lower-level properties objects directly, use `ParquetFileWriter`, which `write_parquet()` wraps. (ARROW-16715)
* UnionDatasets can unify schemas of multiple InMemoryDatasets with varying
schemas. (ARROW-16085)
* `write_dataset()` preserves all schema metadata again. In 8.0.0, it would drop most metadata, breaking packages such as sfarrow. (ARROW-16511)
* Reading and writing functions (such as `write_csv_arrow()`) will automatically (de-)compress data if the file path contains a compression extension (e.g. `"data.csv.gz"`). This works locally as well as on remote filesystems like S3 and GCS. (ARROW-16144)]
* `FileSystemFactoryOptions` can be provided to `open_dataset()`, allowing you to pass options such as which file prefixes to ignore. (ARROW-15280)
* By default, `S3FileSystem` will not create or delete buckets. To enable that, pass the configuration option `allow_bucket_creation` or `allow_bucket_deletion`. (ARROW-15906)
* `GcsFileSystem` and `gs_bucket()` allow connecting to Google Cloud Storage. (ARROW-13404, ARROW-16887)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe lead with this one? We should sort the section based on relevance/priority

* Removed `read_arrow()` and `write_arrow()` functions. They have been deprecated for several versions in favor of corresponding "ipc" and "feather" functions. (ARROW-16268)

## Arrow dplyr queries

* Bugfixes:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Likewise let's lead with major new features (new dplyr verbs, then new functions) and put bug fixes at the end

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, though I was putting these at the top because they seemed like critical bugfixes.

* Count distinct now gives correct result across multiple row groups. (ARROW-16807)
* Aggregations over partition columns return correct results. (ARROW-16700)
* `dplyr::union` and `dplyr::union_all` are supported. (ARROW-15622)
* `dplyr::glimpse` is supported. (ARROW-16776)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we say more than just "supported"?

* `show_exec_plan()` can be added to the end of a dplyr pipeline to show the underlying plan, similar to `dplyr::show_query`. (ARROW-15016)
Copy link
Contributor

@dragosmg dragosmg Jul 27, 2022 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want to add here that both dplyr::show_query() and dplyr::explain() also work?

We now support namespacing (pkg:: prefixing) => something like the chunk below works.

mtcars %>% 
  mutate(make_model = rownames(.)) %>% 
  arrow_table() %>% 
  mutate(name_length = stringr::str_length(make_model)) %>% 
  collect()

* User-defined functions are supported in queries. Use `register_scalar_function()` to create them. (ARROW-16444)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should go higher up. Also should discuss map_batches() alongside this since they're both kinds of UDF

* `lubridate::parse_date_time()` datetime parser: (ARROW-14848, ARROW-16407)
* `orders` with year, month, day, hours, minutes, and seconds components are supported.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are some orders not supported?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, #13506 adds the remaining formats/ orders. So far the focus was on supporting the orders that would enable the higher level parsers (e.g. ymd_hms()).

* the `orders` argument in the Arrow binding works as follows: `orders` are transformed into `formats` which subsequently get applied in turn. There is no `select_formats` parameter and no inference takes place (like is the case in `lubridate::parse_date_time()`).
* `lubridate::ymd()` and related string date parsers supported. (ARROW-16394). Month (`ym`, `my`) and quarter (`yq`) resolution parsers are also added. (ARROW-16516)
* lubridate family of `ymd_hms` datetime parsing functions are supported. (ARROW-16395)
* `lubridate::fast_strptime()` supported. (ARROW-16439)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure we need a separate bullet point for every function that just says "supported". We can group them as is relevant, and we don't need to include all of the JIRA issue ids.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I second ☝🏻 that.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I'll consolidate. Wrote those bullets in a hurry :)

I include the JIRA IDs because it makes it way easier for me to revise the news. We can strip them at the end if people don't want them shown publicly.

* `lubridate::floor_date()`, `lubridate::ceiling_date()`, and `lubridate::round_date()` are supported. (ARROW-14821)
* `strptime()` supports the `tz` argument to pass timezones. (ARROW-16415)
* added `lubridate::qday()` (day of quarter)
* `map_batches()` returns a `RecordBatchReader` and requires that the function it maps returns something coercible to a `RecordBatch` through the `as_record_batch()` S3 function. It can also run in streaming fashion if passed `.lazy = TRUE`. (ARROW-15271, ARROW-16703)
* `exp()` and `sqrt()` supported in dplyr queries. (ARROW-16871)

## Packaging

* The `arrow.dev_repo` for nightly builds of the R package and prebuilt
libarrow binaries is now https://nightlies.apache.org/arrow/r/.
* Brotli and BZ2 are shipped with MacOS binaries. BZ2 is shipped with Windows binaries. (ARROW-16828)

# arrow 8.0.0

Expand Down