-
Notifications
You must be signed in to change notification settings - Fork 141
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Types and values of quoted strings #64
Comments
Hmmmm.....we should actually be accounting for quoted strings/other types during type detection. Can you post a specific case (file) where this isn't happening? Definitely a bug. |
It's attenu.csv.gz
|
I've reported this issue and fixed it in #65, but in the master the issue is present once again (the "quoted numbers detected as string column" test that I provided was also modified to expect that the 2nd (quoted) column is Int). Somewhat similar issue is that during type detection quoted nulls ("NA") are detected as nulls. |
@alyst, sorry for what's happened here. With the port to Nulls, I took the time to do quite a number of large refactorings, involving initial type detection, parsing, streaming, etc. In the process, there were some two dozen other issues closed, and I tried to make sure all existing issues stayed resolved, but I was worried that something would regress. In this case, the behavior that has changed, and that I'd like to support is that quoted fields are not automatically treated as Strings. The reasoning here is that I've personally encountered several different csv sources where, for some reason or another, a system chooses to quote all fields, regardless of being a string or not, or containing characters needing escaping or not. In the case of the julia> df = CSV.read(joinpath(dir, "attenu.csv"); null="NA", types=Dict(3=>Union{Null, String})) i.e. it's pretty easy to manually specify that the 3rd column should be Strings with null values as "NA". Does that all make sense? Sorry again if this has messed anything up at all. |
The mode that doesn't automatically treat quoted columns as strings definitely makes sense. But for me it's rather an indication of the problem with the |
Re |
FWIW in R, when we remove from |
This should be fixed on master with the switch to Note for now, you can get a NamedTuple of Vectors on master by doing |
AFAIU automatic type detection does not take into account whether the extracted string was quoted or not, which leads to incorrect predictions for the column containing quoted numbers as the first N elements.
Checking for quotes should also be done when checking for
null
strings: quoted string should be automatically regarded as non-null.Actually, encountered this while trying to convert
RDatasets
into usingDataTables
(datasets/attenu.csv.gz
).RDataset
has a nice test set of .csv files.The text was updated successfully, but these errors were encountered: