Skip to content

qio 0.1.2

Latest

Choose a tag to compare

@github-actions github-actions released this 14 Nov 13:42
· 3 commits to main since this release
477fb59

Version 0.1.2 as submitted to CRAN.

Changes

New features

  • write_parquet() writes a list of raw vectors as a binary (BYTE_ARRAY)
    column, with NULL for a missing value, and parquet_schema() declares it
    as "BINARY", "GEOMETRY" or "GEOGRAPHY". The geospatial types hold
    well-known binary (WKB) with a crs, stored as given, and for GEOGRAPHY
    an edge algorithm.
  • qio records each geospatial column chunk's statistics as the Parquet
    specification defines them. The geometry types are the top-level ones only:
    a MultiPolygon is listed, its polygons are not. GEOMETRY chunks get a
    bounding box, and a NaN coordinate is left out of its own dimension only.
    GEOGRAPHY chunks get no bounding box, because a geodesic edge can leave
    the box of its vertices. A chunk holding a value that is not valid WKB
    records no geospatial statistics, so readers cannot skip it by mistake.
  • geospatial_statistics() reports the bounding box and geometry types of
    every GEOMETRY and GEOGRAPHY column chunk, from qio's files or any
    other writer's, so readers can choose row groups by box before reading
    data.

Bug fixes

  • Fixed write_parquet(append = TRUE) leaving the original file unreadable
    when the append failed with an error after writing its first new row group
    (#34). An error during an append now leaves the file as it was, and
    "bytes"-encoded strings in columns, names, metadata or crs are refused
    before any file is touched.
  • bloom_filter_may_contain() no longer reports FALSE, "definitely
    absent", for values that are present. bit64::integer64 values are hashed
    exactly, and unsigned columns are checked by their stored bit pattern. A
    fraction, an out-of-range value, or a classed value such as a factor or
    POSIXct is now an error instead of a FALSE, so code that relied on
    that FALSE must convert its values first (#43).
  • An explicit INT32 column no longer writes -2147483648 as a missing value.
    That value is R's integer NA, so it is now refused like any other
    out-of-range value, and the error names the row and the value (#38).
  • schema() reports a GEOMETRY or GEOGRAPHY CRS whole however long it is.
    Files that put inline PROJJSON there, as DuckDB does, had it cut to 127
    bytes.