GH-38916: [R] Simplify dataset and table print output #38917

thisisnic · 2023-11-28T10:45:51Z

Rationale for this change

When printing objects with data with lots of rows, the output is long and unwieldy.

What changes are included in this PR?

Truncates long schema print output and adds the number of columns to dataset print output.
Add number of columns to output so it's clear how many there are in total

Are these changes tested?

Yes

Are there any user-facing changes?

Yes

Before:

library(arrow)
x <- tibble::tibble(!!!letters, .rows = 5)
InMemoryDataset$create(x)
#> InMemoryDataset
#> "a": string
#> "b": string
#> "c": string
#> "d": string
#> "e": string
#> "f": string
#> "g": string
#> "h": string
#> "i": string
#> "j": string
#> "k": string
#> "l": string
#> "m": string
#> "n": string
#> "o": string
#> "p": string
#> "q": string
#> "r": string
#> "s": string
#> "t": string
#> "u": string
#> "v": string
#> "w": string
#> "x": string
#> "y": string
#> "z": string
arrow_table(x)
#> Table
#> 5 rows x 26 columns
#> $"a" <string>
#> $"b" <string>
#> $"c" <string>
#> $"d" <string>
#> $"e" <string>
#> $"f" <string>
#> $"g" <string>
#> $"h" <string>
#> $"i" <string>
#> $"j" <string>
#> $"k" <string>
#> $"l" <string>
#> $"m" <string>
#> $"n" <string>
#> $"o" <string>
#> $"p" <string>
#> $"q" <string>
#> $"r" <string>
#> $"s" <string>
#> $"t" <string>
#> $"u" <string>
#> $"v" <string>
#> $"w" <string>
#> $"x" <string>
#> $"y" <string>
#> $"z" <string>
record_batch(x)
#> RecordBatch
#> 5 rows x 26 columns
#> $"a" <string>
#> $"b" <string>
#> $"c" <string>
#> $"d" <string>
#> $"e" <string>
#> $"f" <string>
#> $"g" <string>
#> $"h" <string>
#> $"i" <string>
#> $"j" <string>
#> $"k" <string>
#> $"l" <string>
#> $"m" <string>
#> $"n" <string>
#> $"o" <string>
#> $"p" <string>
#> $"q" <string>
#> $"r" <string>
#> $"s" <string>
#> $"t" <string>
#> $"u" <string>
#> $"v" <string>
#> $"w" <string>
#> $"x" <string>
#> $"y" <string>
#> $"z" <string>

After:

library(arrow)

x <- tibble::tibble(!!!letters, .rows = 5)
InMemoryDataset$create(x)
#> InMemoryDataset
#> 26 columns 
#> "a": string
#> "b": string
#> "c": string
#> "d": string
#> "e": string
#> "f": string
#> "g": string
#> "h": string
#> "i": string
#> "j": string
#> "k": string
#> "l": string
#> "m": string
#> "n": string
#> "o": string
#> "p": string
#> "q": string
#> "r": string
#> "s": string
#> "t": string
#> ...
#> Use `schema()` to see entire schema
arrow_table(x)
#> Table
#> 5 rows x 26 columns
#> $"a" <string>
#> $"b" <string>
#> $"c" <string>
#> $"d" <string>
#> $"e" <string>
#> $"f" <string>
#> $"g" <string>
#> $"h" <string>
#> $"i" <string>
#> $"j" <string>
#> $"k" <string>
#> $"l" <string>
#> $"m" <string>
#> $"n" <string>
#> $"o" <string>
#> $"p" <string>
#> $"q" <string>
#> $"r" <string>
#> $"s" <string>
#> $"t" <string>
#> ...
#> Use `schema()` to see entire schema
record_batch(x)
#> RecordBatch
#> 5 rows x 26 columns
#> $"a" <string>
#> $"b" <string>
#> $"c" <string>
#> $"d" <string>
#> $"e" <string>
#> $"f" <string>
#> $"g" <string>
#> $"h" <string>
#> $"i" <string>
#> $"j" <string>
#> $"k" <string>
#> $"l" <string>
#> $"m" <string>
#> $"n" <string>
#> $"o" <string>
#> $"p" <string>
#> $"q" <string>
#> $"r" <string>
#> $"s" <string>
#> $"t" <string>
#> ...
#> Use `schema()` to see entire schema

Closes: [R] Simplify dataset and table print output #38916

github-actions · 2023-11-28T10:46:22Z

⚠️ GitHub issue #38916 has been automatically assigned in GitHub to PR creator.

paleolimbot · 2023-12-05T14:08:37Z

I see this still has some tests failing...just making sure you weren't waiting on any of us for a review!

thisisnic · 2023-12-07T17:06:08Z

I see this still has some tests failing...just making sure you weren't waiting on any of us for a review!

Nah, life has just been busy, but thanks for checking in on it. I'll ping you once I get time to take a look at it again!

thisisnic · 2024-01-10T16:41:37Z

@paleolimbot This is now ready for review; mind giving this a look over when you get the chance?

r/R/schema.R

amoeba · 2024-03-08T20:54:06Z

r/R/schema.R

+print_schema_fields <- function(s, truncate = FALSE, max_fields = 20) {
+  if (truncate && length(s$fields) > max_fields) {
+    fields_out <- paste(map_chr(s$fields[1:max_fields], ~ .$ToString()), collapse = "\n")
+    fields_out <- paste0(fields_out, "\n...\n")


Do we know how many fields we're not showing here? Rather than ..., giving a count of hidden fields would be a nice improvement. I'm thinking about how pillar does it:

> starwars # A tibble: 87 × 14 name height mass hair_color skin_color eye_color birth_year sex gender homeworld species films vehicles <chr> <int> <dbl> <chr> <chr> <chr> <dbl> <chr> <chr> <chr> <chr> <lis> <list> 1 Luke Skywalk… 172 77 blond fair blue 19 male mascu… Tatooine Human <chr> <chr> 2 C-3PO 167 75 NA gold yellow 112 none mascu… Tatooine Droid <chr> <chr> 3 R2-D2 96 32 NA white, bl… red 33 none mascu… Naboo Droid <chr> <chr> 4 Darth Vader 202 136 none white yellow 41.9 male mascu… Tatooine Human <chr> <chr> 5 Leia Organa 150 49 brown light brown 19 fema… femin… Alderaan Human <chr> <chr> 6 Owen Lars 178 120 brown, gr… light blue 52 male mascu… Tatooine Human <chr> <chr> 7 Beru Whitesu… 165 75 brown light blue 47 fema… femin… Tatooine Human <chr> <chr> 8 R5-D4 97 32 NA white, red red NA none mascu… Tatooine Droid <chr> <chr> 9 Biggs Darkli… 183 84 black light brown 24 male mascu… Tatooine Human <chr> <chr> 10 Obi-Wan Keno… 182 77 auburn, w… fair blue-gray 57 male mascu… Stewjon Human <chr> <chr> # ℹ 77 more rows # ℹ 1 more variable: starships <list> # ℹ Use `print(n = ...)` to see more rows

Great idea!

r/R/schema.R

amoeba · 2024-03-08T21:00:24Z

Thanks @thisisnic. I see this as a nice improvement and don't see any issues with it. I left a few notes.

jonkeane

This looks good, I have a few minor comments, but nothing major.

I'm not at all opposed to merging this since it's an improvement, but I think #32110 will be the big win here for most end users.

r/R/record-batch-reader.R

r/tests/testthat/test-schema.R

thisisnic · 2024-03-10T15:42:14Z

Thanks @jonkeane, I agree we should take a look at #32110 soon, but merge this as an interim thing in the meantime.

conbench-apache-arrow · 2024-03-14T14:56:14Z

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit ac1708c.

There was 1 benchmark result with an error:

Commit Run on ursa-i9-9960x at 2024-03-14 05:40:53Z
- dataframe-to-table (R) with dataset=chi_traffic_2020_Q1, language=R

There was 1 benchmark result indicating a performance regression:

Commit Run on ursa-i9-9960x at 2024-03-14 05:40:53Z
- tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-07, scale_factor=1

The full Conbench report has more details. It also includes information about 19 possible false positives for unstable benchmarks that are known to sometimes produce them.

thisisnic requested a review from paleolimbot as a code owner November 28, 2023 10:45

github-actions bot added Component: R awaiting committer review Awaiting committer review labels Nov 28, 2023

thisisnic force-pushed the GH-38916_print_output branch from 789d104 to 9fb4e27 Compare January 10, 2024 15:57

github-actions bot added the Component: Documentation label Jan 10, 2024

amoeba reviewed Mar 8, 2024

View reviewed changes

r/R/schema.R Outdated Show resolved Hide resolved

amoeba reviewed Mar 8, 2024

View reviewed changes

r/R/schema.R Outdated Show resolved Hide resolved

github-actions bot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels Mar 8, 2024

thisisnic requested review from assignUser, kou, raulcd, lidavidm, kevingurney, domoritz, trxcllnt, zeroshade, CurtHagenlocher, wgtmac and westonpace as code owners March 8, 2024 23:13

github-actions bot added Component: Java Component: Parquet Component: Go Component: C++ labels Mar 8, 2024

github-actions bot removed the Component: Swift label Mar 8, 2024

thisisnic removed the Component: Documentation label Mar 8, 2024

thisisnic removed request for kou, trxcllnt, lidavidm, zeroshade, domoritz, raulcd, CurtHagenlocher, westonpace, wgtmac, kevingurney and assignUser March 8, 2024 23:15

github-actions bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Mar 8, 2024

jonkeane approved these changes Mar 9, 2024

View reviewed changes

r/R/record-batch-reader.R Outdated Show resolved Hide resolved

r/tests/testthat/test-schema.R Show resolved Hide resolved

github-actions bot added awaiting merge Awaiting merge and removed awaiting changes Awaiting changes labels Mar 9, 2024

Factor out repetetition into function and ensure that max_fields > 0

c9ba314

github-actions bot added Component: Documentation awaiting changes Awaiting changes and removed awaiting merge Awaiting merge labels Mar 10, 2024

Capitalise 'B'

39acb97

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Mar 10, 2024

Fix linting error

f7d8f93

thisisnic merged commit ac1708c into apache:main Mar 13, 2024
10 checks passed

thisisnic removed the awaiting change review Awaiting change review label Mar 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH-38916: [R] Simplify dataset and table print output #38917

GH-38916: [R] Simplify dataset and table print output #38917

thisisnic commented Nov 28, 2023 •

edited

github-actions bot commented Nov 28, 2023

paleolimbot commented Dec 5, 2023

thisisnic commented Dec 7, 2023

thisisnic commented Jan 10, 2024

amoeba Mar 8, 2024

thisisnic Mar 8, 2024

thisisnic Mar 8, 2024

amoeba commented Mar 8, 2024

jonkeane left a comment

thisisnic commented Mar 10, 2024

conbench-apache-arrow bot commented Mar 14, 2024

GH-38916: [R] Simplify dataset and table print output #38917

GH-38916: [R] Simplify dataset and table print output #38917

Conversation

thisisnic commented Nov 28, 2023 • edited

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

github-actions bot commented Nov 28, 2023

paleolimbot commented Dec 5, 2023

thisisnic commented Dec 7, 2023

thisisnic commented Jan 10, 2024

amoeba Mar 8, 2024

Choose a reason for hiding this comment

thisisnic Mar 8, 2024

Choose a reason for hiding this comment

thisisnic Mar 8, 2024

Choose a reason for hiding this comment

amoeba commented Mar 8, 2024

jonkeane left a comment

Choose a reason for hiding this comment

thisisnic commented Mar 10, 2024

conbench-apache-arrow bot commented Mar 14, 2024

thisisnic commented Nov 28, 2023 •

edited