[R] Support for csv options on open_dataset

[asfimport]

There's a lot of gotchas created around heterogeneity in arrow's support for csv parsing options beween read_csv_arrow() and open_dataset() (and further issues arising from migrating from readr::read_csv()).  Not sure if it's more helpful to report these in one place or as separate issues, but here's a few that keep tripping me up:

 

• "na" (defining the na-character choices) is not implemented on open_dataset(), though it is on read_csv_arrow()

• somewhat confusingly, open_dataset does support null_strings though, which appears to play the same roll.   The docs however suggest that open_dataset() ... options are passed to dataset_factory().  I think those docs should link to https://arrow.apache.org/docs/r/reference/CsvReadOptions.html .  https://arrow.apache.org/docs/r/reference/FileFormat.html suggests that null_strings is not one of the recognized CsvReadOptions, but it seems that it now is.  I appreciate the challenge of supporting both the readr-like options and the native arrow option names here, but the functionality and documentation remains very confusing!

  Also another gotcha: in arrow 6.0 release, if we supply an arrow schema, open_dataset assumes the first line of the csv is data and not column headers, so we have to do skip=1.  I see the logic (the schema names the columns anyway, so assuming we're going with those names why parse the names from the csv), but it's surprising since reading without the schema we do not use skip=1, and it's natural to want to go and declare column types while preserving csv column names.  The error messages on doing so aren't helpful, since if you forget skip=1, you are just told that any column that is not a string is "the incorrect type".  The open_dataset() docs imply that we can use read_csv_arrow() options, which suggest that we could provide types using col_types() instead of schema, but this appears not to be the case.  Also

Reporter: Carl Boettiger / @cboettig

_{Note: This issue was originally created as ARROW-15088. Please see the migration documentation for further details.}

[asfimport]

Nicola Crane / @thisisnic:
Thanks for this bug report @cboettig - will take a look at those items 1 by 1 so we can see what's covered already and what might want new tickets opening for:

• " in arrow 6.0 release, if we supply an arrow schema, open_dataset assumes the first line of the csv is data and not column headers, so we have to do skip=1....".. "error messages on doing so aren't helpful" - I agree; we have now added in some better error handling in ARROW-15123 and ARROW-14744

• When you say "The open_dataset() docs imply that we can use read_csv_arrow() options", which bits do you mean? There might be something in there we need to rephrase in there.

[asfimport]

Nicola Crane / @thisisnic:

• "somewhat confusingly, open_dataset does support null_strings though"

  Please can you give me a reprex of this working? It's not working for me and I am not seeing it in the code, but given the ellipses arguments get passed to a lot of places, I might be missing something.

[asfimport]

Nicola Crane / @thisisnic:

• " "na" (defining the na-character choices) is not implemented on open_dataset()"

Have opened ARROW-15470

[asfimport]

Carl Boettiger / @cboettig:
Sorry for the confusion.  Here's some reproducible examples:

"na is recognized in read_csv_arrow but  not implemented on open_dataset". Witness:

library(arrow)
library(readr)
csv <- readr_example("mtcars.csv")
read_csv_arrow(csv, na="-99")  # Works
open_dataset(csv, na="-99") # Error:

# Error in ParquetFragmentScanOptions$create(...) : 
# unused argument (na = "-99") 

• Re the docs, I'm explicitly referring to the docstring about the ... option in open_dataset, which links to dataset_factory, which says:

  Additional format-specific options, passed to {}FileFormat$create(){}. For CSV options, note that you can specify them either with the Arrow C++ library naming ("delimiter", "quoting", etc.) or the {}readr{}-style naming used in [read_csv_arrow()|https://arrow.apache.org/docs/r/reference/read_delim_arrow.html] ("delim", "quote", etc.). Not all readr options are currently supported; please file an issue if you encounter one that arrow should support.

   

  Re null_strings, I'm stumped there too – I see null_values documented in https://arrow.apache.org/docs/r/reference/CsvReadOptions.html still, but I cannot use it with either open_dataset() or read_csv_arrow().  Can you point me an reproducible example that uses that option?  Am I just misunderstanding that those options documented at that link can be used in open_dataset() etc and read_csv_arrow()

   

[asfimport]

Nicola Crane / @thisisnic:
Hi Carl,
I've put together a short reproducible example here:

library(arrow)
library(readr)
library(dplyr)

mtcars[1,1] = -99
readr::write_csv(mtcars, "mtcars_na.csv")
read_csv_arrow("mtcars_na.csv", na="-99")  # Works
open_dataset("mtcars_na.csv", na="-99", format = "csv") 
# Error: The following option is supported in "read_delim_arrow" functions but not yet supported here: "na"
open_dataset("mtcars_na.csv", null_values="-99", format = "csv") %>% collect() # Also works

In short, what is happening is that different bits of the Arrow C++ code are used when reading in data via open_dataset() versus read_csv_arrow(). We've done some work in read_csv_arrow() to hook up the readr style arguments with their Arrow equivalents, but there are cases like these where these arguments are not yet supported for datasets or simply haven't been hooked up.

In the example above, I've used the null_values argument to do the same thing that the na argument does in read_csv_arrow(), but from a UX perspective, I think it'd be great it we could just use the na argument to achieve the same thing and if I have time will look at getting ARROW-15470 done ahead of our next release.

[github-actions]

This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.