GeoSPARQL Compliance Benchmark

This is the GeoSPARQL Compliance Benchmark, integrated into the HOBBIT Platform.

The GeoSPARQL Compliance Benchmark aims to evaluate the GeoSPARQL compliance of RDF storage systems. The benchmark uses 206 SPARQL queries to test the extent to which the benchmarked system supports the 30 requirements defined in the GeoSPARQL standard.

As a result, the benchmark provides two metrics:

• Correct answers: The number of correct answers out of all GeoSPARQL queries, i.e. tests.
• GeoSPARQL compliance percentage: The percentage of compliance with the requirements of the GeoSPARQL standard.

Results

You can find a set of results from the latest experiments on the hosted instance of the HOBBIT Platform (log in as Guest). [last update: 07.07.2021]

If you want your RDF triplestore tested, you can add it as a system to the HOBBIT Platform, and then run an experiment using the hosted instance of the HOBBIT Platform.

Publications

• Milos Jovanovik, Timo Homburg, Mirko Spasić. "A GeoSPARQL Compliance Benchmark". ISPRS International Journal of Geo-Information 10(7):487, 2021.
• Milos Jovanovik, Timo Homburg, Mirko Spasić. "Software for the GeoSPARQL Compliance Benchmark". Software Impacts 8:100071, 2021.
• (preprint) Milos Jovanovik, Timo Homburg, Mirko Spasić. "A GeoSPARQL Compliance Benchmark". arXiv:2102.06139.

Mapping Requirements to Queries

Req.	Set of corresponding queries	Description
Core component (CORE)
R1	Q01.rq	Selecting of the first triple where geometry A is the subject
R2	Q02.rq	Selecting of the first entity of type geo:SpatialObject
R3	Q03.rq	Selecting of the first entity of type geo:Feature
Topology vocabulary extension (TOP)
R4	Q04-1.rq, Q04-2.rq, Q04-3.rq, Q04-4.rq, Q04-5.rq, Q04-6.rq, Q04-7.rq, Q04-8.rq	Testing the properties geo:sfEquals, geo:sfDisjoint, geo:sfIntersects, geo:sfTouches, geo:sfCrosses, geo:sfWithin, geo:sfContains and geo:sfOverlaps
R5	Q05-1.rq, Q05-2.rq, Q05-3.rq, Q05-4.rq, Q05-5.rq, Q05-6.rq, Q05-7.rq, Q05-8.rq	Testing the properties geo:ehEquals, geo:ehDisjoint, geo:ehMeet, geo:ehOverlap, geo:ehCovers, geo:ehCoveredBy, geo:ehInside and geo:ehContains
R6	Q06-1.rq, Q06-2.rq, Q06-3.rq, Q06-4.rq, Q06-5.rq, Q06-6.rq, Q06-7.rq, Q06-8.rq	Testing the properties geo:rcc8eq, geo:rcc8dc, geo:rcc8ec, geo:rcc8po, geo:rcc8tppi, geo:rcc8tpp, geo:rcc8ntpp and geo:rcc8ntppi
Geometry extension (GEOEXT)
R7	Q07.rq	Selecting of all entities of type geo:Geometry
R8	Q08-1.rq, Q08-2.rq	Selecting of the value of geometry A denoted by the properties geo:hasGeometry and geo:hasDefaultGeometry
R9	Q09-1.rq, Q09-2.rq, Q09-3.rq, Q09-4.rq, Q09-5.rq, Q09-6.rq	Selecting of the value of geometry A denoted by the properties geo:dimension, geo:coordinateDimension, geo:spatialDimension, geo:isEmpty, geo:isSimple and geo:hasSerialization
R10	Q10.rq	Checking of the datatype of a correctly defined WKT literal from the dataset
R11	Q11.rq	Checking of the equality of two geometries from the dataset
R12	Q12.rq	Checking if the system interprets the axis order within a point geometry according to the spatial reference system being used
R13	Q13-1.rq, Q13-2.rq	Checking if an empty RDFS Literal of type geo:wktLiteral is interpreted as an empty geometry
R14	Q14.rq	Checking of the geo:asWKT value of geometry A against the expected literal value
R15	Q15.rq	Checking whether all the values of the geo:asGML property contain a valid GM_Object subtype in it and whether its datatype is geo:gmlLiteral
R16	Q16-1.rq, Q16-2.rq	Checking if an empty geo:gmlLiteral is interpreted as an empty geometry
R17	---	---
R18	Q18.rq	Checking the geo:asGML value of geometry A against the expected literal value
R19	Q19-1-1.rq, Q19-1-2.rq, Q19-1-3.rq, Q19-1-4.rq, Q19-2-1.rq, Q19-2-2.rq, Q19-3-1.rq, Q19-3-2.rq, Q19-4-1.rq, Q19-4-2.rq, Q19-4-3.rq, Q19-4-4.rq, Q19-5-1.rq, Q19-5-2.rq, Q19-5-3.rq, Q19-5-4.rq, Q19-6-1.rq, Q19-6-2.rq, Q19-6-3.rq, Q19-6-4.rq, Q19-7-1.rq, Q19-7-2.rq, Q19-7-3.rq, Q19-7-4.rq, Q19-8-1.rq, Q19-8-2.rq, Q19-9-1.rq, Q19-9-2.rq	Checking a support of the geospatial functions geof:distance, geof:buffer, geof:convexHull, geof:intersection, geof:union, geof:difference, geof:symDifference, geof:envelope and geof:boundary
R20	Q20-1.rq, Q20-2.rq	Checking a support of the geospatial function geof:getSRID
Geometry topology extension (GTOP)
R21	Q21-1.rq, Q21-2.rq, Q21-3.rq, Q21-4.rq	Checking a support of the geospatial operator geof:relate
R22	Q22-1-1.rq, Q22-1-2.rq, Q22-1-3.rq, Q22-1-4.rq, Q22-2-1.rq, Q22-2-2.rq, Q22-2-3.rq, Q22-2-4.rq, Q22-3-1.rq, Q22-3-2.rq, Q22-3-3.rq, Q22-3-4.rq, Q22-4-1.rq, Q22-4-2.rq, Q22-4-3.rq, Q22-4-4.rq, Q22-5-1.rq, Q22-5-2.rq, Q22-5-3.rq, Q22-5-4.rq, Q22-6-1.rq, Q22-6-2.rq, Q22-6-3.rq, Q22-6-4.rq, Q22-7-1.rq, Q22-7-2.rq, Q22-7-3.rq, Q22-7-4.rq, Q22-8-1.rq, Q22-8-2.rq, Q22-8-3.rq, Q22-8-4.rq	Checking a support of the geospatial functions geof:sfEquals, geof:sfDisjoint, geof:sfIntersects, geof:sfTouches, geof:sfCrosses, geof:sfWithin, geof:sfContains and geof:sfOverlaps
R23	Q23-1-1.rq, Q23-1-2.rq, Q23-1-3.rq, Q23-1-4.rq, Q23-2-1.rq, Q23-2-2.rq, Q23-2-3.rq, Q23-2-4.rq, Q23-3-1.rq, Q23-3-2.rq, Q23-3-3.rq, Q23-3-4.rq, Q23-4-1.rq, Q23-4-2.rq, Q23-4-3.rq, Q23-4-4.rq, Q23-5-1.rq, Q23-5-2.rq, Q23-5-3.rq, Q23-5-4.rq, Q23-6-1.rq, Q23-6-2.rq, Q23-6-3.rq, Q23-6-4.rq, Q23-7-1.rq, Q23-7-2.rq, Q23-7-3.rq, Q23-7-4.rq, Q23-8-1.rq, Q23-8-2.rq, Q23-8-3.rq, Q23-8-4.rq	Checking a support of the geospatial functions geof:ehEquals, geof:ehDisjoint, geof:ehMeet, geof:ehOverlap, geof:ehCovers, geof:ehCoveredBy, geof:ehInside and geof:ehContains
R24	Q24-1-1.rq, Q24-1-2.rq, Q24-1-3.rq, Q24-1-4.rq, Q24-2-1.rq, Q24-2-2.rq, Q24-2-3.rq, Q24-2-4.rq, Q24-3-1.rq, Q24-3-2.rq, Q24-3-3.rq, Q24-3-4.rq, Q24-4-1.rq, Q24-4-2.rq, Q24-4-3.rq, Q24-4-4.rq, Q24-5-1.rq, Q24-5-2.rq, Q24-5-3.rq, Q24-5-4.rq, Q24-6-1.rq, Q24-6-2.rq, Q24-6-3.rq, Q24-6-4.rq, Q24-7-1.rq, Q24-7-2.rq, Q24-7-3.rq, Q24-7-4.rq, Q24-8-1.rq, Q24-8-2.rq, Q24-8-3.rq, Q24-8-4.rq	Checking a support of the geospatial functions geof:rcc8eq, geof:rcc8dc, geof:rcc8ec, geof:rcc8po, geof:rcc8tppi, geof:rcc8tpp, geof:rcc8ntpp and geof:rcc8ntppi
RDFS entailment extension (RDFSE)
R25	Q25-1.rq, Q25-2.rq, Q25-3.rq	Checking if the system supports selecting both materialized RDF triples, as well as inferred RDF triples based on the RDFS Entailment Regime
R26	Q26-1.rq, Q26-2.rq	Checking if the system supports selecting both materialized RDF triples, as well as inferred RDF triples based on the RDFS/OWL class hierarchy of geometry types from Simple Features
R27	Q27.rq	Checking if the system supports selecting both materialized RDF triples, as well as inferred RDF triples based on the RDFS/OWL class hierarchy of geometry types of the GML schema
Query rewrite extension (QRW)
R28	Q28-1.rq, Q28-2.rq, Q28-3.rq, Q28-4.rq, Q28-5.rq, Q28-6.rq, Q28-7.rq, Q28-8.rq	Testing the properties geor:sfEquals, geor:sfDisjoint, geor:sfIntersects, geor:sfTouches, geor:sfCrosses, geor:sfWithin, geor:sfContains and geor:sfOverlaps using both materialized RDF triples and inferred RDF triples
R29	Q29-1.rq, Q29-2.rq, Q29-3.rq, Q29-4.rq, Q29-5.rq, Q29-6.rq, Q29-7.rq, Q29-8.rq	Testing the properties geor:ehEquals, geor:ehDisjoint, geor:ehMeet, geor:ehOverlap, geor:ehCovers, geor:ehCoveredBy, geor:ehInside and geor:ehContains using both materialized RDF triples and inferred RDF triples
R30	Q30-1.rq, Q30-2.rq, Q30-3.rq, Q30-4.rq, Q30-5.rq, Q30-6.rq, Q30-7.rq, Q30-8.rq	Testing the properties geor:rcc8eq, geor:rcc8dc, geor:rcc8ec, geor:rcc8po, geor:rcc8tppi, geor:rcc8tpp, geor:rcc8ntpp and geor:rcc8ntppi using both materialized RDF triples and inferred RDF triples

Acknowledgement

The benchmark has been developed as part of the HOBBIT and SAGE research projects.