Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WFS indexing / Align with sextant #3205

Merged
merged 15 commits into from
Nov 26, 2018
Merged

Conversation

fxprunayre
Copy link
Member

@fxprunayre fxprunayre commented Oct 27, 2018

Changes:

  • Add support ES 6.4.x
  • Add support for custom index name and specify type names which are GeoNetwork specific (ES 6 does not allow multiple typenames in index)
  • Add support for ES cluster and alias
  • Limit proxy to GN index (and not the full ES cluster)
  • Send bulk in asynch mode for better performances
  • Better GT logging
  • Configuration for GeoJSON geometry precision (defining a custom precision model preserving topology instead of the current 4 decimals only applied which may lead to invalid geometry construction)
  • Handle CDATA in feature attributes

Those changes are backported from Ifremer Sextant.

Future tasks:

  • @jahow could you check some of the FIXME's client side to see if you've better fix and check that all (most) of Sextant features are also here (I'm not aware of all the changes in facets, date, graphs, ...). Thanks

@fxprunayre fxprunayre added this to the 3.6.0 milestone Oct 27, 2018
Copy link
Collaborator

@cmangeat cmangeat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clarification around ac049ee are needed in my opinion.

@jahow
Copy link
Contributor

jahow commented Nov 5, 2018

I'll take a look ASAP.

Basically we want the files to be the same as in sextant, which is where most of the WFS filter/indexing work was done, right?

@fxprunayre
Copy link
Member Author

Basically we want the files to be the same as in sextant, which is where most of the WFS filter/indexing work was done, right?

Yep.

@fxprunayre fxprunayre changed the title [WIP] WFS indexing / Align with sextant WFS indexing / Align with sextant Nov 13, 2018
cmangeat and others added 12 commits November 22, 2018 15:44
workarounds
refine error messaging when report saved
do not go further when failure at es report initialisation time
infer wfs version from geotools sources rather than from parameters
default to GML3 for datasources, not taken into account when wfs 1.0.0
leverage wfs datasources timeout to 3000s
… of the default 4 applies by GeometryJSON) and optionnaly apply a precision model to reduce the geometry complexity and avoid to build invalid geometry when removing decimals. This may create a simplified geometry above the equivalent displayed in WMS. Set those values depending on the WFS harvested.
…operty file on startup. Not filtered on build.
This was introduced by https://github.com/camptocamp/sextant-geonetwork/commit/e5ede11b419e4f0fcc0bf8c4e420aabb9dd8ff60

featureAttributeToDocumentFieldNames was shared in a EsWFSFeatureIndexer
singleton. Changing scope to prototype to have one instance on each
harvest.
* Add support for index type
* Limit proxy to the index we are using (and not all the ES cluster)
* WFS indexing / Add support for CDATA
@fxprunayre fxprunayre force-pushed the feature/wfsharvester/align-with-sextant branch from 1fbbd5c to d94d719 Compare November 22, 2018 14:46
Filtering is done by adding an 'include' criteria on the bucket aggregation for the facet.
+ fix to labels for fields only in app profile
Copy link
Contributor

@jahow jahow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I've added a few commits that I felt were missing but nothing much. Thanks

@jahow jahow merged commit b190d05 into master Nov 26, 2018
fxprunayre added a commit that referenced this pull request Dec 17, 2018
Related to #3205 which now encode URL and feature type name to support ES 6.4+ and to point to the correct index containing features (and not to the ES cluster root only).
fxprunayre added a commit that referenced this pull request Dec 17, 2018
Related to #3205 which now encode URL and feature type name to support ES 6.4+ and to point to the correct index containing features (and not to the ES cluster root only).
@fxprunayre fxprunayre deleted the feature/wfsharvester/align-with-sextant branch December 18, 2018 08:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants