WFS Filters based on WFS indexing with SOLR

Date	August 03 2015	Contacts	Florent Gravin, Francois Prunayre
Status		Release	3.2.0
Resources	Available	Ticket #	Pull request
Source code	https://github.com/geonetwork/core-geonetwork/tree/solr-wfs-indexing
Funding	Ifremer

Overview

Index the data to improve search and data visualization.

All datasets described in metadata records and accessible through a download services (ie WFS) can be analyzed to improve search and data visualisation.

This proposal add:

Workers to analyze WFS datasets
Enhance the mapviewer to easily browse datasets
Monitoring interface to trigger analysis

Table view with facet filters

CFV 31 May 2016

+1 Florent, Francois, Jeroen, Jose, Simon

Proposal

The current mapviewer only allows to query object on a location using WMS GetFeatureInfo request. In order to provide more advanced filtering, WFS datasource can be used. Once a metadata record reference the dataset in a WFS, the service will be used to analyze the data and then provide filtering to the end-user.

Get features, index and analyze

To build the filter interface, the following elements are required:

the list of fields: Those are provided by the WFS DescribeFeatureType operation
the features: Those are provided by the WFS GetFeature operation

In order to provide simple and easier filtering, a facet mechanism is built on top of the WFS service. For this part, spatial features are retrieved from the WFS and indexed into a SOLR collection which will index and provide facetted search.

analysis overview

Instead of manually computing facets, SOLR scenario was preferred for the following reasons:

automatically build and compute facet
support a large set of objects
dynamic fields can be used to properly index different types of fields

Index structure

The SOLR schema will contains dynamic fields in order to easily host a variety of attribute tables and data types (text, dates, numbers). While indexing based on the DescribeFeatureType information (column names and types), proper schema fields will be used.

Multivalued fields could also be indexed. Configuration also allows user to tokenize field containing list of values (eg. list of parameters).

The geometry will also be indexed in SOLR. Dataview could be limited to current map extent.

Heatmap facets will provides an overview of feature localization.

Retrieving all features may be a relatively long process and will be designed like a harvester. The main application can trigger harvesting task by sending messages (JMS). The harvester are based on Camel routing engine and can be installed on a separate server.

Manual and/or automatic scan task

Indexing all features from all WFS feature type being a long process, the following approach are foreseen:

add a button to request the index of the layer
add a CRON that index every night
add admin ui to manage WFS indexing

After indexing, feature filtering will be available on the map.

Designing, applying filters and rendering results

Filters will be created based on a facetted search like approaches. Once the filter selected by the user:

a list of selected filters are displayed
a table will display selected features (retrieved from SOLR)
map visualization
WMS rendering: a server component will generate an SLD file which will be propagated to all getMap requests to only display filtered features

Note: One desired feature of SLD generation is to base the style on SLD style used by the WMS server. This means that the SLD needs to be downloaded from the WMS server, parsed and updated with the filters. For example consider a SLD that styles all highways with one style and all other road types with another style. When we generate a style we want to download the style from the WMS and modify it by adding user filter to each rule.

The algorithm is as follows:

The user will submit the desired filters
The server will check capabilities of the various WMS versions of the WMS Server to try and find a version that supports GetStyles. (GetStyles is introduced at version 1.1.0)
If GetStyles is not supported:
Geometry type will be retrieved from SOLR
A Base SLD will be chose based on the Geometry type
If GetStyles is supported
GetStyles will be used to download the SLD from the WMS Server
The downloaded SLD will be used as the Base SLD
Each rule in the SLD will be updated by modifying the filter of the rule:
If a rule does not have a filter then the filter will be set on the rule
If a rule has a filter then the user filter will be ANDed with the rule filter

Tests

This work has been tested with:

MapServer
QGISServer
GeoServer

Contribution to libraries

GeoTools / Add GetStyles operation https://github.com/geotools/geotools/pull/1055

Related work

[GN Solr migration plan] (https://docs.google.com/a/camptocamp.com/document/d/1wajKcNzQa-WDXa9I5S50s0ciWd75aIuXCmU1YFVSNDs/edit?usp=sharing)
[Data indexing using Tika] (https://github.com/titellus/daobs/blob/master/tasks/data-indexer/src/main/resources/META-INF/spring/daobs-task-routes.xml#L137)

Proposal Type:

Type: Widget
Module: Community

Voting History

Vote Proposed:

Future work

Feature will be associated with metadata record. When Solr will be used as the backend for metadata, this will allows combined search between data and metadata for better search results.