Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature: facilitate a wfs 2 as a backend #58

Closed
pvgenuchten opened this issue Jul 18, 2018 · 7 comments

Comments

4 participants
@pvgenuchten
Copy link
Contributor

commented Jul 18, 2018

so pygeoapi can be used on the many wfs 2 implementations out there

@jorgejesus jorgejesus self-assigned this Jul 19, 2018

@jorgejesus

This comment has been minimized.

Copy link
Member

commented Jul 19, 2018

Use WFS2.0 as data source for pygeoapi

@tomkralidis

This comment has been minimized.

Copy link
Member

commented Jul 19, 2018

+1 great feature! Considerations:

  • we can use OWSLib’s WFS functionality to interact with WFS 1 or 2 sources
  • we can also use OWSLib’s Filter XML functionality to convert WFS 3 property filters into WFS 1/2 Filter XML. This may get tricky depending on the underlying WFS 1/2 implementation
  • the devil will be in the WFS 1/2 GetFeature responses. OWSLib simply returns string/XML object representations which would need to be converted into GeoJSON. Suggest we start with common WFS 1/2 implementations (MapServer, GeoServer, others?)
@jorgejesus

This comment has been minimized.

Copy link
Member

commented Sep 12, 2018

@jorgejesus jorgejesus added enhancement and removed inprogress labels Nov 17, 2018

@justb4

This comment has been minimized.

Copy link
Member

commented Apr 15, 2019

Think this can be solved in a compact way with a single OGRProvider: a PyGeoAPI Provider that uses the GDAL/OGR Python bindings. Did some tests with both GeoServer and MapServer WFS v2 backends like from PDOK (Dutch National SDI). Many of the WFS-FES capabilities can be supported directly (if backend supports WFS v2):

  • result type hits: feature count
  • paging: translates to WFS paging with configurable pagesize
  • spatial filtering: e.g. bounding box or any other
  • attribute filtering, including get-feature-by-id: e.g. get feature by (gml) id

Advantages are:

  • no need for GML request construction (like Filters) and response parsing and GML-GeoJSON mapping
  • support not just WFS v2 but (m)any OGR source-types like GPKG, Shapefile, FGDB etc
  • Proxy support for URL-based backends
  • multiple auth methods support for URL-based backends

Things to consider:

  • OGR WFS Driver may issue WFS GetCapabilities request on each Open() call (need investigation)
  • the regular WFS v1, v2 performance considerations like complex filters bringing a WFS down

Software Design consideration:

  • Develop a single OGRProvider as a one-in-all Provider, or as an ABC with specialized OGRProvider-derived classes like OGRWFSProvider, OGRShapefileProvider etc.

I would aim first for OGRProvider with specialized Driver/OGRSource-type-specific helper-classes.
We do something very similar within a Stetl OGRInput ETL class. Main difference (per Source-type) is really on per-OGR-Driver-specific Source options (next to standard GDAL/OGR options). We can catch this within the PGA config YAML like:

                        .
                        .
    provider:
        name: OGR
        data:
            source_type: WFS
            source: WFS:http://geodata.nationaalgeoregister.nl/rdinfo/wfs?
            source_options:
                VERSION: 2.0.0
                OGR_WFS_PAGING_ALLOWED: YES
                OGR_WFS_LOAD_MULTIPLE_LAYER_DEFN: NO
             gdal_ogr_options:
                GDAL_CACHEMAX: 64
                GDAL_HTTP_PROXY: (optional proxy)
                GDAL_PROXY_AUTH: (optional auth for remote WFS)
                CPL_DEBUG: NO
        id_field: gml_id
        layer: rdinfo:stations
                        .
                        .

    provider:
        name: OGR
        data:
            source_type: ESRI Shapefile
            source: tests/data/rdinfo.shp
            source_options:
                        .
            gdal_ogr_options:
                GDAL_CACHEMAX: 64
        id_field: fid
        layer: stations

I can take stab at this, would be nice to have something running to e.g. wrap all PDOK WFS-es.

refs:

@tomkralidis

This comment has been minimized.

Copy link
Member

commented Apr 15, 2019

Thanks @justb4. This approach works. I'm glad the plugin framework is flexible enough to encompass an OGR driver. In one way, OGR can replace every other provider that implements the same thing. It will be good to have them all, and let them mature/evolve over time.

@justb4

This comment has been minimized.

Copy link
Member

commented Apr 25, 2019

update: OGR Provider can even do (paged) fetch from ESRI FeatureServers!

@justb4 justb4 self-assigned this May 13, 2019

@justb4 justb4 moved this from To do to In progress in OSGeo Community Sprint 2019 May 13, 2019

tomkralidis added a commit that referenced this issue May 15, 2019

Introduce OGRProvider: WFS backend support (#119)
* #89 fix check_format function

* #89 fix check_format function - fix default return

* #58 first version OGRProvider with working WFS tests/config

* #58 second version, ogr2ogr-like reprojection support - more to follow

* #58 3rd version, support Sourcetypes SHP and GPKG with tests+travis stuff

* #58 fixed #95 for OGR Provider and its tests

* #58 use more performant backend-WFS in pygeoapi-config.yml

* #58 4th version: WFS backend ok, including OGR Python paging gotchas fixed ready for PR

* #58 fix Travis build for GDAL Python bindings

* #58 fix Travis build: Unit tests failed: missing Shapefile .zip now added

* #58 #119 rework from PR comments: config, tests, quotes

@justb4 justb4 moved this from In progress to Done in OSGeo Community Sprint 2019 May 15, 2019

@justb4

This comment has been minimized.

Copy link
Member

commented May 15, 2019

Think we're done with OGRProvider for WFS2 backends. Tested with GeoServer, MapServer and deegree WFSs (see tests), including bbox queries. Other OGR backends (Drivers) work as well like GPKG, ESRI Shapefile, ESRI FeatureServer, except for paging (will need to use OGR SQL for most). Will open separate issue(s) for those.

So closing here, re-open if still issues with WFS2 OGR backends.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.