Home

Introduction

OBIS Schema Extractor (v2) is a Java-based web application for data schema extraction from SPARQL endpoints.

OBIS Schema Extractor artifact is a packaged executable JAR file and needs Java 17 to run.

For running OBIS Schema Extractor locally:

1. install Java
2. copy build/schema-extractor-exec.jar, together with the entire build folder contents, to any folder
3. run executable JAR file java -jar schema-extractor-exec.jar
4. navigate to http://server:port/swagger-ui.html (default URL - http://localhost:8080/swagger-ui.html)
5. configure the parameters, as described on the browser page, or use create the configuration file (a sample configuration file is available in the build folder)

Example Execution

We provide a resulting Schema file, a Configuration file and the Execution log for extracting the schema from Nobel Prizes SPARQL API endpoint (overall schema extraction time is below 3 minutes).

Further possible use of the schema is to upload it into Data Shape Server that can be set up either independently, or as a part of ViziQuer Tools container system. The schema upload is done by DSS Schema import utility.

Extractor Parameters

The parameters available in Swagger-based extractor execution are most conveniently accessible from the browser page for running the extractor.

The specifications of all parameters, including the ones currently hidden (@ApiParam(hidden = true)) from the browser UI (but available for use from the configuration file), are available in the parameter description code.

An example configuration file is provided along with the extractor, as well (when extracting all classes and/or properties from an endpoint, set the includedClasses and/or includedProperties parameters to []).

The Extractor extracts the schema from a SPARQL Endpoint by means of (possibly long) series of SPARQL SELECT queries issued against it. The query templates are provided in a separate Source File.

The following table describes some important parameters

Parameter Name	Datatype / Enum	Description
endpointUrl	String	SPARQL Endpoint URL, for example, http://localhost:8890/sparql. Use URL that accepts GET or POST API requests.
graphName	String	Named Graph (optional). If no graph name is provided, the search will involve all graphs from the endpoint
calculateSubClassRelations	Boolean	Calculate subclass relation (strongly recommended)
calculateMultipleInheritanceSuperclasses	Boolean	Include multiple inheritance check, strongly recommended. If set to false, no more than a single direct super-class for a class shall be found. Use this option only for very large endpoints, if the processing of subclass relations is stuck in the extractor for a longer time.
minimalAnalyzedClassSize	Integer	Minimal analyzed class size. The classes below the specified size shall be included in the schema, still they shall not be considered as possible super-classes of other classes and their links to properties shall not be analyzed individually. If 1, all classes should be analyzed. Values above 1, e.g. 10 or 100, may be essential for endpoints with more than several thousand classes.
requireClasses	Boolean	At least one class is required for schema extraction. If no classes are found, extractor work is aborted (default = true)
calculatePropertyPropertyRelations	Boolean	Calculate property-property adjacency: following properties, same subject, same object. Important for visual and textual query auto-completion use case; is used in schema visualization only if property sources and targets are considered as potential nodes (work in progress). Can be time-consuming for larger schemas, consider appropriate propertyPropertyLinkCheckBackupMode (default = true)
propertyPropertyLinkCheckBackupMode	none, limits, limitsPlus, limitsPlusSimple, details	Set backup validation queries to calculate property-property relations in the case of a failing property co-occurrence query for a given base property. details, limitsPlus and limitsPlusSimple guarantee trying a single-line pattern of two property co-occurrence, if larger patterns fail. details gives exact statistics for two property co-occurrence, whenever possible. limitsPlus attempts at least a limit-based estimate, limitsPlusSimple can stay with yes/no answer for co-occurrence for performance reasons (with some count information imputed at later schema import steps). none and limits are faster, but can result in partially incomplete information. (default = limitsPlusSimple)
calculateSourceAndTargetPairs	Boolean	Calculate pairs of source and target classes for properties. Creates finer-grained schemas and is used as a source of statistics in visual schema diagrams (default = true)
calculateDomainsAndRanges	Boolean	Calculate domain and range classes for properties (in the strict, ontological sense) (default = true)
calculateImportanceIndexes	basic, unionBased, classCoverage, no	Calculate ascription points / principal classes for properties. Strongly recommended if schema diagrams are envisaged; allows avoiding ascription of a property both to a subclass and a superclass. Use classCoverage if all class-to-property connections should be marked for the class itself or for some superclass/subclass. unionBased allows not ascribing a property to a class, if the class instances that have the property are all in the union of classes where the property is already ascribed to; basic looks for a single such class (default = basic)
calculateClosedClassSets	Boolean	Check, if the property source and target class set sets cover all non-literal property subjects and objects; essential for future SHACL export, can be used in advanced schema diagrams (default = true)
calculateCardinalities	none, propertyLevelOnly, propertyLevelAndClassContext	Calculate minimum and maximum cardinalities for properties (default = propertyLevelAndClassContext). May consider propertyLevelOnly for larger schemas.
calculateDataTypes	none, propertyLevelOnly, propertyLevelAndClassContext	Calculate data types for attributes (default = propertyLevelAndClassContext). May consider propertyLevelOnly for larger schemas.
sampleLimitForDataTypeCalculation	Long	Instance sample size for datatype calculation. It is recommended to provide a constraint (e.g., 100K or 1M) for larger data sets as the exact data set information can be less essential in the schemas and attempts to obtain exact data type usage statistics may take unreasonably long time.
calculateInstanceNamespaces	no, detailed, overview	Experimental feature: calculate instance namespace URIs (default = no)
sampleLimitForInstanceNamespacesCalculation	Long	Limit of sample instance count to use in namespace calculation. No value or 0 means all data will be used (default = 1000)
addedLabels	List	Properties for class and property labels. rdfs:label and skos:prefLabel are assumed by default. For labels in specific languages use @{en;de} notation after the label property IRI or short form. To exclude a default labeling property, use property@{-} notation
addIntersectionClasses	yes, no	Add intersection class information into the extracted schema (default = yes)
excludedNamespaces	List	List of excluded namespaces, e.g. http://www.openlinksw.com/schemas/virtrdf#
includedClassesFile	CSV file	Valid CSV file with the list of included classes. To be used, if the created schema is to be restricted to the explicitly listed classes. If not specified, all classes will be analyzed.
includedPropertiesFile	CSV file	Valid CSV file with the list of included properties. To be used, if the created schema is to be restricted to the explicitly listed classes. If not specified, all properties will be analyzed
enableLogging	Boolean	Enable SPARQL query logging to file on the server (default = true)
saveThisConfig	Boolean	Save this configuration to file on the server (default = true)