Repository navigation
Registration of custom column types and parsers #865
Description
This is a feature request that should not impact current readr's behaviour, but that would open up the possibility of enhancement from third-party packages (cc @edzer).
Motivation
Consider the cases in which a rectangular data source contains, e.g.,
- Quantities with units and/or errors.
1.53(2) m/s. - Spatial data:
POINT(0, 1).
readr::read_csv("
quantity,point
1.53(3) m/s,\"POINT(0,1)\"
5.21(1) m/s,\"POINT(1,5)\"
")
#> # A tibble: 3 x 2
#> quantity point
#> <chr> <chr>
#> 1 quantity point
#> 2 1.53(3) m/s POINT(0,1)
#> 3 5.21(1) m/s POINT(1,5)Currently, those data types are read as character (and other use cases may be stripped to numbers), and the user needs to convert them. The idea would be to allow packages to register custom column types and parsers into readr so that, in this example, if packages quantities and sf were loaded, readr would have automatically generated columns with quantities and sf objects respectively.
Changes required
I could be missing something, but the general changes needed for this would be:
- Move things to
inst/includeto expose theCollectorclass, so that other packages can link toreadrand derive safely from this class. - Some mechanism to register
- (C++) the custom collector into the list of available subclasses (and the means to insert it in a specific position in the chain?) and a parser (guesser) function.
- (R) custom
col_*andparse_*.
The only drawback I can think of is that a package may, e.g., register a parser that catches everything and messes things up. To avoid this issue, readr::read_* may gain a flag to enable external parsers, so that using them requires an action from the user.
If this enhancement is considered, I would be more than happy to work on it.
Activity
msberends commentedon Nov 5, 2018
I'll second this! Would be very welcome if we could extend the functionality of
readrwith other packages.Our AMR package is all about antimicrobial resistance (AMR). The new class
rsimakes sure columns only contain valid antimicrobial interpretations: resistant (R), susceptible (S) or intermediate (I). This can be forced upon a vector withas.rsi.Anyway, if data are read from a microbiological laboratory system (in a hospital), all columns with results of antbiotics will contain just R, S or I values. Suchcolumns could be parsed as
rsiwhen the AMR package is loaded 😄strazto commentedon Feb 6, 2019
Consider this third-ed !
Some notes:
Imo, allow users to supply parsers as a named list, and maybe to be able to override builtin parsers (Throw a warning if they try to do this with an option to suppress it, maybe).
With col_types, extend the abbreviation string assignment to allow users to instead use the key for their input parser.
msberends commentedon Nov 4, 2019
This would still be awesome to have 🙄😄
bart1 commentedon Dec 17, 2021
I also have a use for this. In my case values are written to a character field that are probably best represented by a matrix in R. The CSV records look like this:
These are measurements of acceleration made in a 5 hz burst over 2 axis. Alternatively If i code them as raw vectors I can considerably reduce the memory consumption (66 % reduction).
Robsteranium commentedon Aug 2, 2022
I also have a use case for this in
{csvwr}where I need to handle lots of different parsing options per the CSVW specification (which extends the XML Schema Datatypes standard). For example, I'd like to be able to configurecol_logicalto parse e.g. "yes" & "no". I'm also wondering if I could use collectors to do some validation (e.g. that a string matches a regex or a number is within some bounds).Contrary to the posts above, I'd suggest that parsers be specified explicitly with a
cols()specification rather than having them registered as a side-effect of loading a package. As convenient as the latter might be for authors of individual packages it won't be obvious to users and offers no way to reconcile conflicts between packages.Likewise I wouldn't extend this to the compact string representation as that will quickly lead to collisions. We've already used a quarter of the (lower/ upper case alphabet) namespace between
readrand this thread!Indeed I'd expect that some parsers will benefit from configuration (as do
col_dateetc).Instead I'd suggest this work with the
col_typesargument e.g.One thing that strikes me looking at the source is that the collectors are implemented in C++, presumably for speed.
Would it be possible to modify
Collector::createsuch that it can resolve third-party collectors at runtime? All the possible collectors won't be available when this class is compiled so there would need to be some way to register/ find collectors.This would still require that packages implement their collectors in C++. An alternative might be to allow people to provide collectors in R code, accepting that this might dramatically slow down parsing. This would lower the barriers to extension without prohibiting package authors from re-implementing their parsers in C++ if it proved too slow in R.