feat(r): Add arrow package integration

[paleolimbot]

This adds an arrow::ExtensionType implementation and a number of S3 methods to enable hopefully seamless interactions between the arrow and sf. There are almost certainly other methods that need to be added (maybe some for wk_wkt and friends in the wk package) but this should be a start.

library(arrow, warn.conflicts = FALSE)
library(geoarrow)
library(sf)
#> Linking to GEOS 3.11.0, GDAL 3.5.3, PROJ 9.1.0; sf_use_s2() is TRUE

nc <- sf::read_sf(system.file("shape/nc.shp", package = "sf"))
(tbl <- as_arrow_table(nc))
#> Table
#> 100 rows x 15 columns
#> $AREA <double>
#> $PERIMETER <double>
#> $CNTY_ <double>
#> $CNTY_ID <double>
#> $NAME <string>
#> $FIPS <string>
#> $FIPSNO <double>
#> $CRESS_ID <int32>
#> $BIR74 <double>
#> $SID74 <double>
#> $NWBIR74 <double>
#> $BIR79 <double>
#> $SID79 <double>
#> $NWBIR79 <double>
#> $geometry: geoarrow.multipolygon <CRS <{>
#> $  "$schema" <"https://pro...>
#> 
#> See $metadata for additional Schema metadata
tbl$geometry$type
#> GeometryExtensionType
#> geoarrow.multipolygon <CRS: {
#>   "$schema": "https://pro...
sf::st_as_sf(tbl)
#> Simple feature collection with 100 features and 14 fields
#> Geometry type: MULTIPOLYGON
#> Dimension:     XY
#> Bounding box:  xmin: -84.32385 ymin: 33.88199 xmax: -75.45698 ymax: 36.58965
#> Geodetic CRS:  NAD27
#> # A tibble: 100 × 15
#>     AREA PERIMETER CNTY_ CNTY_ID NAME  FIPS  FIPSNO CRESS_ID BIR74 SID74 NWBIR74
#>    <dbl>     <dbl> <dbl>   <dbl> <chr> <chr>  <dbl>    <int> <dbl> <dbl>   <dbl>
#>  1 0.114      1.44  1825    1825 Ashe  37009  37009        5  1091     1      10
#>  2 0.061      1.23  1827    1827 Alle… 37005  37005        3   487     0      10
#>  3 0.143      1.63  1828    1828 Surry 37171  37171       86  3188     5     208
#>  4 0.07       2.97  1831    1831 Curr… 37053  37053       27   508     1     123
#>  5 0.153      2.21  1832    1832 Nort… 37131  37131       66  1421     9    1066
#>  6 0.097      1.67  1833    1833 Hert… 37091  37091       46  1452     7     954
#>  7 0.062      1.55  1834    1834 Camd… 37029  37029       15   286     0     115
#>  8 0.091      1.28  1835    1835 Gates 37073  37073       37   420     0     254
#>  9 0.118      1.42  1836    1836 Warr… 37185  37185       93   968     4     748
#> 10 0.124      1.43  1837    1837 Stok… 37169  37169       85  1612     1     160
#> # ℹ 90 more rows
#> # ℹ 4 more variables: BIR79 <dbl>, SID79 <dbl>, NWBIR79 <dbl>,
#> #   geometry <MULTIPOLYGON [°]>

^{Created on 2023-12-06 with reprex v2.0.2}

[paleolimbot]

@anthonynorth if you have the bandwidth I'd be grateful for any of the thoughts you have time to write here!

[anthonynorth]

Does arrow::as_array_array() copy if we have a single chunk and we're avoiding that?

[paleolimbot]

I am pretty sure that it does (which should be fixed in arrow, but I haven't gotten there yet!). I only found out recently that concat_arrays() is the only way to force a copy of an Arrow thing.

[anthonynorth]

Should this be using indices?

[paleolimbot]

😬 definitely! I fixed this and added a test.

[anthonynorth]

We're using this conversion pattern in as_geoarrow_vctr() also. Should we extract it into a utility, or just convert x with as_geoarrow_vctr()? E.g.

Suggested change

	if (is.null(type)) {
	type <- arrow::as_data_type(attr(x, "schema", exact = TRUE))
	chunks <- attr(x, "chunks", exact = TRUE)
	} else {
	stream <- as_nanoarrow_array_stream(x, schema = as_nanoarrow_schema(type))
	chunks <- nanoarrow::collect_array_stream(stream, validate = FALSE)
	type <- arrow::as_data_type(type)
	}
	schema <- as_nanoarrow_schema(type)
	if (!is.null(type)) {
	array <- arrow::as_chunked_array(as_geoarrow_vctr(x, schema = as_nanoarrow_schema(type)))
	return(array)
	}
	schema <- attr(x, "schema", exact = TRUE)
	type <- arrow::as_data_type(schema)
	chunks <- attr(x, "chunks", exact = TRUE)

[paleolimbot]

The fact that slices are involved changed this a bit! I think there is more room for optimization (in particular, Arrow C++ doesn't import/export chunked arrays as array streams, which would help a lot!).

[anthonynorth]

I assume the following isn't equivalent? My lack of knowledge of arrow and nanoarrow internals is probably obvious with this question.

arrow::chunked_array(!!!chunks, type = type)

[paleolimbot]

We added chunked_array() pretty recently but it's definitely the suggested constructor!

[anthonynorth]

I don't understand what these lines are for? Can we just create the array from chunks?

geoarrow-c/r/geoarrow/R/arrow-compat.R

Lines 26 to 33 in a9293f3

	arrays <- vector("list", length(chunks))
	for (i in seq_along(arrays)) {
	tmp_schema <- nanoarrow::nanoarrow_allocate_schema()
	nanoarrow::nanoarrow_pointer_export(schema, tmp_schema)
	tmp_array <- nanoarrow::nanoarrow_allocate_array()
	nanoarrow::nanoarrow_pointer_export(chunks[[i]], tmp_array)
	arrays[[i]] <- arrow::Array$import_from_c(tmp_array, tmp_schema)
	}

[anthonynorth]

Do we need a tibble?

Suggested change

	df <- tibble::as_tibble(x)
	df <- as.data.frame(x)

[paleolimbot]

Good catch! I forgot that Arrow doesn't "depend" on tibble (although it returns them quite a lot of the time)

[anthonynorth]

Looks good. A couple of very minor things that might be improvements (see comments)