Heads-up, GDAL 2.2 coming #303

rsbivand · 2017-04-14T11:01:42Z

edzer · 2017-04-14T15:08:50Z

Thanks! Building it in a container now; I noticed that rgdal breaks on 2.2; shall I look at it, or did you address it already?

edzer · 2017-04-14T22:03:45Z

I made one small commit to rgdal which makes it compile with 2.2, but check gives several errors; see details below.

* checking examples ... ERROR
Running examples in 'rgdal-Ex.R' failed
The error most likely occurred in:

> ### Name: writeOGR
> ### Title: Write spatial vector data using OGR
> ### Aliases: writeOGR
> ### Keywords: spatial
> 
> ### ** Examples
> 
> cities <- readOGR(system.file("vectors", package = "rgdal")[1], "cities")
OGR data source with driver: ESRI Shapefile 
Source: "/rgdal/rgdal.Rcheck/rgdal/vectors", layer: "cities"
with 606 features
It has 4 fields
Integer64 fields read as strings:  POPULATION 
> is.na(cities$POPULATION) <- cities$POPULATION == -99
> summary(cities$POPULATION)
 1270000  1550000  1140000  1190000  1225000  1250000  1300000  1400000 
       5        4        3        3        3        3        3        3 
 1600000  2050000   320000   600000   700000   800000  1025000  1060000 
       3        3        3        3        3        3        2        2 
 1100000 11100000  1170000  1185000  1325000  1460000  1500000  1525000 
       2        2        2        2        2        2        2        2 
 1800000   185000  1875000   203000  2250000  2325000  2400000  2950000 
       2        2        2        2        2        2        2        2 
 3025000   310000  3800000   400000   465000   535000   625000  6450000 
       2        2        2        2        2        2        2        2 
  670000   675000   770000   890000  9300000   960000   100000  1005000 
       2        2        2        2        2        2        1        1 
 1010000     1012 10150000  1020000  1024940  1029731  1030000  1037700 
       1        1        1        1        1        1        1        1 
 1040000  1045000  1050000  1055000  1061000   106416  1065000  1069727 
       1        1        1        1        1        1        1        1 
 1070000 10750000  1088624   109000   109382   109486   110153  1104209 
       1        1        1        1        1        1        1        1 
 1110000   111914  1120000  1144000   114500  1149401  1150000   116000 
       1        1        1        1        1        1        1        1 
 1160000  1165000  1175000    11753  1179000    12000   120000  1200000 
       1        1        1        1        1        1        1        1 
 1205000   121265  1216000  1227200    12426   124400   125263  1255000 
       1        1        1        1        1        1        1        1 
 1272400  1275000  (Other)     NA's 
       1        1      415       30 
> td <- file.path(tempdir(), "rgdal_examples"); dir.create(td)
> # BDR 2016-12-15 (MapInfo driver fails writing to directory with ".")
> if(nchar(Sys.getenv("OSGEO4W_ROOT")) > 0) {
+     OLDPWD <- getwd()
+     setwd(td)
+     td <- "."
+ }
> writeOGR(cities, td, "cities", driver="ESRI Shapefile")
> try(writeOGR(cities, td, "cities", driver="ESRI Shapefile"))
Error in writeOGR(cities, td, "cities", driver = "ESRI Shapefile") : 
  layer exists, use a new layer name
> writeOGR(cities, td, "cities", driver="ESRI Shapefile", overwrite_layer=TRUE)
> cities2 <- readOGR(td, "cities")
OGR data source with driver: ESRI Shapefile 
Source: "/tmp/RtmpPPj0YN/rgdal_examples", layer: "cities"
with 606 features
It has 4 fields
> summary(cities2$POPULATION)
          1270000  1550000  1140000  1190000  1225000  1250000  1300000 
      30        5        4        3        3        3        3        3 
 1400000  1600000  2050000   320000   600000   700000   800000  1025000 
       3        3        3        3        3        3        3        2 
 1060000  1100000 11100000  1170000  1185000  1325000  1460000  1500000 
       2        2        2        2        2        2        2        2 
 1525000  1800000   185000  1875000   203000  2250000  2325000  2400000 
       2        2        2        2        2        2        2        2 
 2950000  3025000   310000  3800000   400000   465000   535000   625000 
       2        2        2        2        2        2        2        2 
 6450000   670000   675000   770000   890000  9300000   960000   100000 
       2        2        2        2        2        2        2        1 
 1005000  1010000     1012 10150000  1020000  1024940  1029731  1030000 
       1        1        1        1        1        1        1        1 
 1037700  1040000  1045000  1050000  1055000  1061000   106416  1065000 
       1        1        1        1        1        1        1        1 
 1069727  1070000 10750000  1088624   109000   109382   109486   110153 
       1        1        1        1        1        1        1        1 
 1104209  1110000   111914  1120000  1144000   114500  1149401  1150000 
       1        1        1        1        1        1        1        1 
  116000  1160000  1165000  1175000    11753  1179000    12000   120000 
       1        1        1        1        1        1        1        1 
 1200000  1205000   121265  1216000  1227200    12426   124400   125263 
       1        1        1        1        1        1        1        1 
 1255000  1272400  1275000  (Other) 
       1        1        1      415 
> if ("SQLite" %in% ogrDrivers()$name) {
+   tf <- tempfile()
+   try(writeOGR(cities, tf, "cities", driver="SQLite", layer_options="LAUNDER=NO"))
+ }
> if ("GeoJSON" %in% ogrDrivers()$name) {
+   js <- '{
+     "type": "MultiPolygon",
+     "coordinates": [[[[102.0, 2.0], [103.0, 2.0], [103.0, 3.0], [102.0, 3.0],
+     [102.0, 2.0]]], [[[100.0, 0.0], [101.0, 0.0], [101.0, 1.0], [100.0, 1.0],
+     [100.0, 0.0]]]]
+   }'
+   spdf <- readOGR(js, layer='OGRGeoJSON')
+   in1_comms <- sapply(slot(spdf, "polygons"), comment)
+   print(in1_comms)
+   tf <- tempfile()
+   writeOGR(spdf, tf, "OGRGeoJSON", driver="GeoJSON")
+   spdf1 <- readOGR(tf, "OGRGeoJSON")
+   in2_comms <- sapply(slot(spdf1, "polygons"), comment)
+   print(in2_comms)
+   print(isTRUE(all.equal(in1_comms, in2_comms)))
+ }
OGR data source with driver: GeoJSON 
Source: "{
    "type": "MultiPolygon",
    "coordinates": [[[[102.0, 2.0], [103.0, 2.0], [103.0, 3.0], [102.0, 3.0],
    [102.0, 2.0]]], [[[100.0, 0.0], [101.0, 0.0], [101.0, 1.0], [100.0, 1.0],
    [100.0, 0.0]]]]
  }", layer: "OGRGeoJSON"
with 1 features
It has 0 fields
[1] "0 0"
Error in ogrInfo(dsn = dsn, layer = layer, encoding = encoding, use_iconv = use_iconv,  : 
  Cannot open layer
Calls: readOGR ... suppressMessages -> withCallingHandlers -> ogrInfo -> .Call
Execution halted
* checking for unstated dependencies in 'tests' ... OK
* checking tests ...
  Running 'test_proj.R'
  Comparing 'test_proj.Rout' to 'test_proj.Rout.save' ... OK
  Running 'tests.R'
  Comparing 'tests.Rout' to 'tests.Rout.save' ... OK
  Running 'tripup.R'
 ERROR
Running the tests in 'tests/tripup.R' failed.
Last 13 lines of output:
  + writeOGR(SpatialPolygonsDataFrame(aMPh, data=df3), tf, "OGRGeoJSON",
  +  driver=driver, verbose=TRUE)
  + rP <- as(readOGR(tf, "OGRGeoJSON", verbose=FALSE), "SpatialPolygons")
  + cat("aMPh with driver:", driver, "\n")
  + unlink(paste(tf, "*", sep=""), recursive=driver == "ESRI Shapefile")
  + cat(" coords", isTRUE(all.equal(lapply(slot(aMPh, "polygons"), function(x) lapply(slot(x, "Polygons"), slot, "coords")), lapply(slot(rP, "polygons"), function(x) lapply(slot(x, "Polygons"), slot, "coords")), check.attributes=FALSE)), "\n")
  + cat(" holes", isTRUE(all.equal(lapply(slot(aMPh, "polygons"), function(x) lapply(slot(x, "Polygons"), slot, "hole")), lapply(slot(rP, "polygons"), function(x) lapply(slot(x, "Polygons"), slot, "hole")), check.attributes=FALSE)), "\n")
  + }
  Object initially classed as: wkbPolygon
  SFS comments in Polygons objects
  Writing 9 wkbPolygon objects
  Error in ogrInfo(dsn = dsn, layer = layer, encoding = encoding, use_iconv = use_iconv,  : 
    Cannot open layer
  Calls: as ... suppressMessages -> withCallingHandlers -> ogrInfo -> .Call
  Execution halted
* checking for unstated dependencies in vignettes ... OK
* checking package vignettes in 'inst/doc' ... WARNING
Package vignette without corresponding PDF/HTML:
   'OGR_shape_encoding.Rnw'

* checking running R code from vignettes ...
   'OGR_shape_encoding.Rnw' using 'UTF-8' ... OK
 OK
* checking re-building of vignette outputs ... SKIPPED
* DONE

Status: 2 ERRORs, 1 WARNING
See
  '/rgdal/rgdal.Rcheck/00check.log'
for details.

edzer · 2017-04-14T22:12:52Z

sf check against gdal 2.2beta1 results below. Looks like meuse's proj4string changed again, a few other funny CRS things (see also silcence test commit above) that need checking out.

* checking tests ...
  Running 'cast.R'
  Comparing 'cast.Rout' to 'cast.Rout.save' ...4c4
< Linking to GEOS 3.5.1, GDAL 2.2.0beta1, proj.4 4.9.2, lwgeom 2.3.2 r15302
---
> Linking to GEOS 3.5.0, GDAL 2.1.0, proj.4 4.9.2
  Running 'crs.R'
  Comparing 'crs.Rout' to 'crs.Rout.save' ...15c15,16
<   GDAL Error 6: EPSG PCS/GCS code -1 not found in EPSG support files.  Is this a valid EPSG coordinate system?
---
>   GDAL Error 6: EPSG PCS/GCS code -1 not found in EPSG support files.  Is this a valid
> EPSG coordinate system?
27c28,29
<   GDAL Error 6: EPSG PCS/GCS code 999999 not found in EPSG support files.  Is this a valid EPSG coordinate system?
---
>   GDAL Error 6: EPSG PCS/GCS code 999999 not found in EPSG support files.  Is this a valid
> EPSG coordinate system?
45c47
< [1] "+proj=sterea +lat_0=52.15616055555555 +lon_0=5.38763888888889 +k=0.9999079 +x_0=155000 +y_0=463000 +ellps=bessel +towgs84=565.2369,50.0087,465.658,-0.406857,0.350733,-1.87035,4.0812 +units=m +no_defs"
---
> [1] "+proj=sterea +lat_0=52.15616055555555 +lon_0=5.38763888888889 +k=0.9999079 +x_0=155000 +y_0=463000 +ellps=bessel +towgs84=565.4171,50.3319,465.5524,-0.398957,0.343988,-1.87740,4.0725 +units=m +no_defs"
  Running 'dist.R'
  Comparing 'dist.Rout' to 'dist.Rout.save' ... OK
  Running 'dplyr.R'
  Comparing 'dplyr.Rout' to 'dplyr.Rout.save' ...73c73
< # A tibble: 4 x 3
---
> # A tibble: 4 × 3
  Running 'empty.R'
  Comparing 'empty.Rout' to 'empty.Rout.save' ... OK
  Running 'gdal_geom.R'
  Comparing 'gdal_geom.Rout' to 'gdal_geom.Rout.save' ...91c91
< GDAL has 77 drivers
---
> GDAL has 75 drivers
  Running 'geos.R'
  Comparing 'geos.Rout' to 'geos.Rout.save' ... OK
  Running 'graticule.R'
  Comparing 'graticule.Rout' to 'graticule.Rout.save' ... OK
  Running 'grid.R'
  Comparing 'grid.Rout' to 'grid.Rout.save' ... OK
  Running 'plot.R'
  Comparing 'plot.Rout' to 'plot.Rout.save' ... OK
  Running 'read.R'
  Comparing 'read.Rout' to 'read.Rout.save' ...153c153
< Updating /sf.Rcheck/tests/x.gpkg
---
> Updating /home/edzer/git/sf.Rcheck/tests/x.gpkg
  Running 'roundtrip.R'
  Comparing 'roundtrip.Rout' to 'roundtrip.Rout.save' ...71c71
<  Ah     :39   1st Qu.:  80.0   epsg:NA      :  0  
---
>  Ah     :39   1st Qu.:  80.0   epsg:28992   :  0  
87c87
<  epsg:NA      :   0  
---
>  epsg:28992   :   0  
95c95
<  epsg:NA      :0  
---
>  epsg:28992   :0  
100c100
<  epsg:NA      :0  
---
>  epsg:28992   :0  
105c105
<  epsg:NA      :0  
---
>  epsg:28992   :0  
118c118
<  epsg:NA      :   0  
---
>  epsg:28992   :   0  
133c133
<  epsg:NA      :   0  
---
>  epsg:28992   :   0  
  Running 'sfc.R'
  Comparing 'sfc.Rout' to 'sfc.Rout.save' ...369c369
<      "3.5.1" "2.2.0beta1"      "4.9.2" 
---
> "3.5.1" "2.1.3" "4.9.2" 
  Running 'sfg.R'
  Comparing 'sfg.Rout' to 'sfg.Rout.save' ... OK
  Running 'testthat.R'
  Comparing 'testthat.Rout' to 'testthat.Rout.save' ...7c7
< Reading layer `nc.gpkg' from data source `/sf.Rcheck/sf/gpkg/nc.gpkg' using driver `GPKG'
---
> Reading layer `nc.gpkg' from data source `/home/edzer/git/sf.Rcheck/sf/gpkg/nc.gpkg' using driver `GPKG'
30c30
< OK: 431 SKIPPED: 9 FAILED: 0
---
> OK: 509 SKIPPED: 1 FAILED: 0
  Running 'wkb.R'
  Comparing 'wkb.Rout' to 'wkb.Rout.save' ... OK
 OK
* checking for unstated dependencies in vignettes ... OK
* checking package vignettes in 'inst/doc' ... WARNING
Package vignettes without corresponding PDF/HTML:
   'sf1.Rmd'
   'sf2.Rmd'
   'sf3.Rmd'

* checking running R code from vignettes ...
   'sf1.Rmd' using 'UTF-8' ... OK
   'sf2.Rmd' using 'UTF-8' ... OK
   'sf3.Rmd' using 'UTF-8' ... OK
 NONE
* checking re-building of vignette outputs ... SKIPPED
* DONE

Status: 1 WARNING, 1 NOTE
See
  '//sf.Rcheck/00check.log'
for details.

edzer · 2017-04-15T12:55:25Z

rgdal on r-forge now passes checks on both gdal < 2.2 and 2.2beta1.

rsbivand · 2017-04-15T14:25:41Z

I hadn't had time to look, will try to look after Tuesday - thanks for looking!

rsbivand · 2017-04-21T07:53:31Z

Regression: #309

rsbivand · 2017-04-21T07:54:53Z

Looks like RFC 67: add null field state for OGR features, in addition to unset fields

edzer · 2017-04-21T08:07:46Z

Done here.

rsbivand · 2017-04-21T08:14:39Z

My post to gdal-dev:

To: gdal-dev@lists.osgeo.org
Subject: 2.2.0beta1 regression NULL values

See #309 - < 2.2.0 we could write
"missing values" in vector drivers and read them back. In 2.2.0beta1, we see
REAL "missing values" written out OK, but read back as numeric zero. We've
only checked shapefiles and GPKG so far.

Is this RFC 67? Until now, rgdal code (and likely sf) has used:

  case OFTReal:
    if (poFeature->IsFieldSet(iField))
      REAL(ans)[iRow]=poFeature->GetFieldAsDouble(iField);
    else REAL(ans)[iRow]=NA_REAL;
    break;

and the same for other field and list field types. Should we be using:

int CPL_DLL OGR_F_IsFieldSetAndNotNull( OGRFeatureH, int );

and should we be writing:

void CPL_DLL OGR_F_SetFieldNull( OGRFeatureH, int );

instead of leaving the field unset with for example:

         if (!ISNA(NUMERIC_POINTER(VECTOR_ELT(ldata, j))[i]))
             poFeature->SetField( CHAR(STRING_ELT(fld_names, j)),
                 NUMERIC_POINTER(VECTOR_ELT(ldata, j))[i] );

that is jumping over features for which the field value coming from R is
missing?

We'd need to condition on GDAL version here (too), I guess.

Was this regression intended?

rsbivand · 2017-04-21T09:56:46Z

Even answered here. I followed up on same thread.

edzer · 2017-04-23T09:37:00Z

I think we're all set for 2.2; I added a docker file for testing with GDAL 2.2.

edzer added a commit that referenced this issue Apr 14, 2017

add gdal check for NULL fields, address #192, #303

ec5a48d

edzer added a commit that referenced this issue Apr 14, 2017

silence several tests against gdal 2.2, #303

795d0a9

edzer closed this as completed Apr 23, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Heads-up, GDAL 2.2 coming #303

Heads-up, GDAL 2.2 coming #303

rsbivand commented Apr 14, 2017

edzer commented Apr 14, 2017

edzer commented Apr 14, 2017

edzer commented Apr 14, 2017

edzer commented Apr 15, 2017

rsbivand commented Apr 15, 2017

rsbivand commented Apr 21, 2017

rsbivand commented Apr 21, 2017

edzer commented Apr 21, 2017

rsbivand commented Apr 21, 2017

rsbivand commented Apr 21, 2017

edzer commented Apr 23, 2017

Heads-up, GDAL 2.2 coming #303

Heads-up, GDAL 2.2 coming #303

Comments

rsbivand commented Apr 14, 2017

edzer commented Apr 14, 2017

edzer commented Apr 14, 2017

edzer commented Apr 14, 2017

edzer commented Apr 15, 2017

rsbivand commented Apr 15, 2017

rsbivand commented Apr 21, 2017

rsbivand commented Apr 21, 2017

edzer commented Apr 21, 2017

rsbivand commented Apr 21, 2017

rsbivand commented Apr 21, 2017

edzer commented Apr 23, 2017