CITE Test Upgrade
The OGC Compliance Program provides the CITE tool for testing standards compliance. Our build process makes heavy use of this tool ... ensuring that each and every release of GeoServer correctly implements WMS, WFS, and WCS.
There are four problems with this:
- We last updated our CITE build tool 2012 (so our testing is quite out of date).
- geoserver.org website can only say we are a "compliant implementation", rather than a "certified implementation" of the WCS, WMS and WFS standards.
- New services such as WPS and CSW are not "blackbox" tested.
- Finally the transition to build.geoserver.org and subsequent rebuild means all CITE tests are currently failing
The OGC provide the CITE Team Engine both as a web service and as an open source project that can be used to check a single instance of GeoServer. This is a little awkward as we would like to integrate these tests into our build server; as such our build server stands up a GeoServer instance, performance the tests, and publishes the results here. These tests also require a database - and modify that database over the course of the tests (so our build server needs a PostGIS running).
Armed with a positive test result we can as an OSGeo Project ask for OGC certification, allowing us to update our website. Open source projects also have the option to be listed as a "reference implementation" (providing an instance of the server is available online for potential clients to test against).
- 2017 Code sprint - design and approached discussed proof of concept of git submodule approach
- FOSS4G NA 2014 CITE Sprint - foss4gna activity to get developers set up to run cite tests locally
- TEAM Engine
- geoserver-cite-tools - Justin's fork from 2013
- geoserver-cite-tools te-2015 - Andrea's
- geoserver data - cite data directory configurations
How to fix this?
This activity is difficult enough that it has not attracted a contributor, and our previous 2012 CITE Test automation was good enough that no funding a replacement has been forthcoming.
This is intended as a sponsored activity:
- Gathering sponsorship from interested parties
- GeoServer Budget 2019 identified this as a priority
Propose the following:
- A short term contract to setup the test harness and build scripts to automat cite tests on build.geosever.org.
A team code sprint to pass the tests once infrastructure support is in place
Work with OSGeo to achieve certification once tests results are available.
Work with OSGeo SAC or OGC to host GeoServer configurations to act as a reference implementation.
Test Harness and Build Automation
Use git submodule to to manage CITE data directory configurations
Setup CITE TEAM Engine as a submodule
Control which revision exactly we run against using this approach. Avoid making our own fork and contribute any and all test fixes to the upstream project.
Build support scripts
- scripts to manage temporary GeoServer in build directory
- scripts to manage CITE Test Database (assume PostGIS but hsql possible)
Alternatives for running tests:
Option: Take advantage of the CITE TEAM Engine's new REST API to run tests
Option: CITE TEAM Engine may be available to run as a Docker container
Option: Run using a script (same as 2012 approach)
Setup a short-term contract, to set up the following:
Upgrade the test harness
- Use https://github.com/opengeospatial/teamengine as a git submodule
Set up build jobs to test existing tests
- WCS 1.0 - previously certified
- WCS 1.1
- WFS 1.0 - previously certified
- WFS 1.1
- WMS 1.1 - previously certified
- WMS 1.3
Code Sprint on Certification
In the latter part of the year set up a code sprint to work on passing the tests:
Initial focus on passing build jobs above
- WCS 1.0
- WCS 1.1
- WFS 1.0
- WFS 1.1
- WMS 1.1
- WMS 1.3
Add new test that we are missing
- WFS 2.0 - passed in isolation
- WCS 2.0
- WMTS 1.0 - passed in isolation
- WPS 1.0
- Maybe GML and KML and GeoPackage
- Submit required paperwork, and test results, update website
- Stand up a reference server for OGC to use (if OSGeo or OGC provide hardware) with one test per workspace/virtual service?