string limitation for very large geojson

[sericson0]

I am trying to load the Microsoft building footprint data for Texas (https://github.com/Microsoft/USBuildingFootprints) which is 2.1 GB. However, when i run the code:

footprint_data = "(path to folder containing data)./Texas.geojson"
texas_buildings = geojson_sf(footprint_data)

I get the following error

Error in rcpp_read_sf_file(normalizePath(geojson), expand_geometries) : 
  R character strings are limited to 2^31-1 bytes

The same code works for smaller files in the Microsoft data, so it seems like the error is due to the size of the file. Any suggestions would be greatly appreciated :-)

---

TODO

• change buffer_string to use rapidjson::FileReadStream directly
• handle fopen() for windows
• correct buffer size char writeBuffer[65536]; ?

[dcooley]

Indeed - the solution may be to implement FileReadStream rather than, or as well as the buffer - to - StringVector implementation

[SymbolixAU]

working prototype

sf <- geojsonsf:::rcpp_read_sf_file_stream(normalizePath("~/Downloads/Texas.geojson"), FALSE)
sf
# Simple feature collection with 9891540 features and 0 fields
# geometry type:  POLYGON
# dimension:      XY
# bbox:           xmin: -106.646 ymin: 25.85305 xmax: -93.52176 ymax: 36.50031
# epsg (SRID):    4326
# proj4string:    +proj=longlat +datum=WGS84 +no_defs
# First 10 features:
#                          geometry
# 1  POLYGON ((-95.61533 30.7471...
# 2  POLYGON ((-95.59383 30.7382...
# 3  POLYGON ((-95.56662 30.7327...
# 4  POLYGON ((-95.55648 30.7391...
# 5  POLYGON ((-95.57037 30.7253...
# 6  POLYGON ((-95.57031 30.7267...
# 7  POLYGON ((-95.57383 30.7276...
# 8  POLYGON ((-95.5707 30.72964...
# 9  POLYGON ((-95.57335 30.7297...
# 10 POLYGON ((-95.569 30.72455,...

[dcooley]

timings

system.time({
  sf <- geojsonsf::geojson_sf("~/Downloads/Texas.geojson", buffer_size = 1024) ## default
})

# user  system elapsed 
# 126.185  16.056 189.778

system.time({
  sf <- geojsonsf::geojson_sf("~/Downloads/Texas.geojson", buffer_size = 2048)
})
rm(list=ls());gc()

# user  system elapsed 
# 91.989  20.256 127.668

system.time({
  sf <- geojsonsf::geojson_sf("~/Downloads/Texas.geojson", buffer_size = 65536)
})
rm(list=ls());gc()

# user  system elapsed 
# 132.908  13.697 191.704 

[DarioBoh]

working prototype

sf <- geojsonsf:::rcpp_read_sf_file_stream(normalizePath("~/Downloads/Texas.geojson"), FALSE)
sf
# Simple feature collection with 9891540 features and 0 fields
# geometry type:  POLYGON
# dimension:      XY
# bbox:           xmin: -106.646 ymin: 25.85305 xmax: -93.52176 ymax: 36.50031
# epsg (SRID):    4326
# proj4string:    +proj=longlat +datum=WGS84 +no_defs
# First 10 features:
#                          geometry
# 1  POLYGON ((-95.61533 30.7471...
# 2  POLYGON ((-95.59383 30.7382...
# 3  POLYGON ((-95.56662 30.7327...
# 4  POLYGON ((-95.55648 30.7391...
# 5  POLYGON ((-95.57037 30.7253...
# 6  POLYGON ((-95.57031 30.7267...
# 7  POLYGON ((-95.57383 30.7276...
# 8  POLYGON ((-95.5707 30.72964...
# 9  POLYGON ((-95.57335 30.7297...
# 10 POLYGON ((-95.569 30.72455,...

geojsonsf:::rcpp_read_sf_file_stream
Error in get(name, envir = asNamespace(pkg), inherits = FALSE) : 
  object 'rcpp_read_sf_file_stream' not found

sessionInfo

─ Session info  
setting  value                       
 version  R version 3.6.3 (2020-02-29)
 os       macOS Mojave 10.14.6        
 system   x86_64, darwin15.6.0        
 ui       RStudio                     
 language (EN)                        
 collate  en_US.UTF-8                 
 ctype    en_US.UTF-8                 
 tz       America/New_York            
 date     2020-03-24      
 geojsonsf     1.3.3   2020-03-18 [1] CRAN (R 3.6.0)

[SymbolixAU]

The rcpp_read_sf_file_stream was my working prototype I made during development. You should now be using the R code geojsonsf::geojson_sf()