ARROW-6231: [C++] Allow generating CSV column names

[pitrou]

Add an option that, if enabled, will autogenerate column names
for the target table rather than read them from the CSV file.

[pitrou]

@nealrichardson @jorisvandenbossche thoughts?

[nealrichardson]

If true, column names will be of the form "f0", "f1"...

Why "f"? FTR, here's how it looks in R:

> cat("1,2,3\n4,5,6\n", file="test.csv")
> read.csv("test.csv", header=FALSE)
  V1 V2 V3
1  1  2  3
2  4  5  6
> readr::read_csv("test.csv", col_names=FALSE)
Parsed with column specification:
cols(
  X1 = col_double(),
  X2 = col_double(),
  X3 = col_double()
)
# A tibble: 2 x 3
     X1    X2    X3
  <dbl> <dbl> <dbl>
1     1     2     3
2     4     5     6

[pitrou]

"f" for field.

[wesm]

Does this patch address the usability problem with header_rows=0? I think it might make more sense to assign names by default in such case than to raise an exception.

[pitrou]

header_rows doesn't exist anymore. Instead we have an unrelated skip_rows.

[nealrichardson]

Yeah, the "usability problem" that came up on Stack Overflow was already addressed, just after 0.14 was released.

[wesm]

Okay, so in R we have this code

read.csv("test.csv", header=FALSE)

and in pandas

In [14]: buf = io.StringIO('1,2,3\n4,5,6')                                                                                                                                                     

In [15]: pandas.read_csv(buf, header=None)                                                                                                                                                     
Out[15]: 
   0  1  2
0  1  2  3
1  4  5  6

Do we want to have a header option here or only use the autogenerate option?

[pitrou]

What kind of "header option" are you looking for? If the user wants to change the autogenerated column names, they can probably do it on the table.

[pitrou]

Or are we talking about API ergonomics? Feel free to propose a different API, as I'm not sure what it would look like :-)

[jorisvandenbossche]

As far as I understand, a header option would be exactly the same as what you implemented, but just a different name?
header=True would indicate the file contains a header and the column names are read from the first line (after skip_rows), and header=False would mean that the file does not contain a header and the column names are auto-generated.

Given the usage of "header" in R (and in pandas, although with a slightly different meaning) is maybe a reason to use that instead of autogenerate_column_names. Although autogenerate_column_names is more explicit.

[pitrou]

I prefer the more explicit name personally.

[nealrichardson]

I don't have a strong opinion about the argument name. I expect that the R, Python, and whatever other libraries would expose an interface that matches the expectations of their users, and internally map the arguments to whatever they're called in the C++ library.

I do think the "f"-for-field prefix is unexpected, but 🤷‍♂

[pitrou]

I do think the "f"-for-field prefix is unexpected, but

What would be more expected, according to you?

[nealrichardson]

Well, in the R examples I pasted, the prefix character is V (for variable) or X, so that's what I expect, just based on what I know. I won't deny that that's arbitrary.

"f" doesn't resonate for me because I don't think of the columns in the table as "fields", though I understand where you're taking that from. Looking at ARROW-4511, "field" doesn't show up in the terminology section. And "record batch" later is explained as "a collection of independent arrays each having the same length as one another but potentially different data types." So "a" for array might be a better choice, or "v" for vector since the doc says that array and vector are interchangeable (and "v" is a variable prefix used elsewhere).

I'm curious what others think.

[codecov-io]

Codecov Report

Merging #5206 into master will decrease coverage by 22.93%.
The diff coverage is 100%.

@@             Coverage Diff             @@
##           master    #5206       +/-   ##
===========================================
- Coverage   88.71%   65.77%   -22.94%     
===========================================
  Files         934      554      -380     
  Lines      121320    71118    -50202     
  Branches     1437        0     -1437     
===========================================
- Hits       107627    46778    -60849     
- Misses      13331    24340    +11009     
+ Partials      362        0      -362

Impacted Files	Coverage Δ
cpp/src/arrow/csv/options.h	100% <ø> (ø)	⬆️
cpp/src/arrow/csv/reader.cc	94.82% <100%> (+0.23%)	⬆️
python/pyarrow/tests/test_csv.py	99.1% <100%> (+0.04%)	⬆️
python/pyarrow/_csv.pyx	99.28% <100%> (+0.02%)	⬆️
cpp/src/arrow/util/memory.h	0% <0%> (-100%)	⬇️
cpp/src/gandiva/date_utils.h	0% <0%> (-100%)	⬇️
cpp/src/arrow/util/memory.cc	0% <0%> (-100%)	⬇️
cpp/src/gandiva/decimal_type_util.h	0% <0%> (-100%)	⬇️
cpp/src/arrow/filesystem/util_internal.cc	0% <0%> (-100%)	⬇️
cpp/src/arrow/compute/logical_type.h	0% <0%> (-100%)	⬇️
... and 630 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cef82b4...cc7fdc6. Read the comment docs.

[jorisvandenbossche]

Personally, I wouldnt use "v", as we don't use the term vector in the C++ project. I would also not use "a" as a column/field is more than just the array. So I would rather go for "f" (field) or "c" (column) (with a preference for "c" as column is a more familiar term), or for something generic without meaning like "x".

[bkietz]

If there's this much disagreement, perhaps the cromulent solution is to provide autogenerate_column_names_prefix (default "f")

[pitrou]

"cromulent" simply should be the hardcoded prefix.

[pitrou]

Does anyone else have a strong feeling here? Like @jorisvandenbossche I think it should either be "f", "c" or "x". No personal preference.

[fsaintjacques]

I vote for 'c', then 'f'.

[fsaintjacques]

I also think that autogenerate_column_names_prefix would close this debate and let everyone be happy.

[pitrou]

Creating dubious preferences to make everyone happy is a UI antipattern, though.
See https://ometer.com/preferences.html

In this case anyone can rename the columns after the fact if they want to. The primary goal here is to avoid failing.

[fsaintjacques]

Then keep the 'f' as-is, and let's move on :). I can merge now if you're fine with it.

[pitrou]

I'm fine with it personally :-)