Home

Introduction

OBIS Schema Extractor (v2) is a Java-based web application for data schema extraction from SPARQL endpoints.

OBIS Schema Extractor artifact is a packaged executable JAR file and needs Java 17 to run.

For running OBIS Schema Extractor locally:

1. install Java
2. copy build/schema-extractor-exec.jar, together with the entire build folder contents, to any folder
3. run executable JAR file java -jar schema-extractor-exec.jar
4. navigate to http://server:port/swagger-ui.html (default URL - http://localhost:8080/swagger-ui.html)
5. configure the parameters, as described on the browser page, or use create the configuration file (a sample configuration file is available in the build folder)

Example Execution

We provide a resulting Schema file, a Configuration file and the Execution log for extracting the schema from Nobel Prizes SPARQL API endpoint (overall schema extraction time is below 3 minutes).

Further possible use of the schema is to upload it into Data Shape Server that can be set up either independently, or as a part of ViziQuer Tools container system. The schema upload is done by DSS Schema import utility.

Extractor Parameters

The parameters available in Swagger-based extractor execution are most conveniently accessible from the browser page for running the extractor.

The specifications of all parameters, including the ones currently hidden (@ApiParam(hidden = true)) from the browser UI (but available for use from the configuration file), are available in the parameter description code.

An example configuration file is provided along with the extractor, as well (when extracting all classes and/or properties from an endpoint, set the includedClasses and/or includedProperties parameters to []).

The Extractor extracts the schema from a SPARQL Endpoint by means of (possibly long) series of SPARQL SELECT queries issued against it. The query templates are provided in a separate Source File.

The following table describes some important parameters

Parameter Name	Datatype / Enum	Description
endpointUrl	String	SPARQL Endpoint URL, for example, http://localhost:8890/sparql. Use URL that accepts GET or POST API requests.
graphName	String	Named Graph (optional). If no graph name is provided, the search will involve all graphs from the endpoint
calculateSubClassRelations	Boolean	Calculate subclass relation (strongly recommended)
calculateMultiple InheritanceSuperclasses	Boolean	Include multiple inheritance check, strongly recommended. If set to false, no more than a single direct super-class for a class shall be found. Set false only for very large endpoints, if the processing of subclass relations is stuck in the extractor for a longer time.
minimalAnalyzedClassSize	Integer	Minimal analyzed class size. The classes below the specified size shall be included in the schema, still they shall not be considered as possible super-classes of other classes and their links to properties shall not be analyzed individually. If 1, all classes should be analyzed. Values above 1, e.g. 10 or 100, may be essential for endpoints with more than several thousand classes.
requireClasses	Boolean	At least one class is required for schema extraction. If no classes are found, extractor work is aborted (default = true)
calculatePropertyPropertyRelations	Boolean	Calculate property-property adjacency: following properties, same subject, same object. Important for visual and textual query auto-completion use case; is used in schema visualization only if property sources and targets are considered as potential nodes (work in progress). Can be time-consuming for larger schemas, consider appropriate propertyPropertyLinkCheckBackupMode (default = true)
propertyPropertyLink CheckBackupMode	none, limits, limitsPlus, limitsPlusSimple, details	Set backup validation queries to calculate property-property relations in the case of a failing property co-occurrence query for a given base property. details, limitsPlus and limitsPlusSimple guarantee trying a single-line pattern of two property co-occurrence, if larger patterns fail. details gives exact statistics for two property co-occurrence, whenever possible. limitsPlus attempts at least a limit-based estimate, limitsPlusSimple can stay with yes/no answer for co-occurrence for performance reasons (with some count information imputed at later schema import steps). none and limits are faster, but can result in partially incomplete structure information. (default = limitsPlusSimple)
calculateSourceAndTargetPairs	Boolean	Calculate pairs of source and target classes for properties. Creates finer-grained schemas and is used as a source of statistics in visual schema diagrams (default = true)
calculateDomainsAndRanges	Boolean	Calculate domain and range classes for properties (in the strict, ontological sense) (default = true)
calculateImportanceIndexes	basic, unionBased, classCoverage, no	Calculate ascription points / principal classes for properties. Strongly recommended if schema diagrams are envisaged; allows avoiding ascription of a property both to a subclass and a superclass. Use classCoverage if all class-to-property connections should be marked for the class itself or for some superclass/subclass. unionBased allows not ascribing a property to a class, if the class instances that have the property are all in the union of classes where the property is already ascribed to; basic looks for a single such class (default = basic)
calculateClosedClassSets	Boolean	Check, if the property source and target class set sets cover all non-literal property subjects and objects; essential for future SHACL export, can be used in advanced schema diagrams (default = true)
calculateCardinalities	none, propertyLevelOnly, propertyLevelAndClassContext	Calculate minimum and maximum cardinalities for properties (default = propertyLevelAndClassContext). May consider propertyLevelOnly for larger schemas.
calculateDataTypes	none, propertyLevelOnly, propertyLevelAndClassContext	Calculate data types for attributes (default = propertyLevelAndClassContext). May consider propertyLevelOnly for larger schemas.
sampleLimitForDataTypeCalculation	Long	Instance sample size for datatype calculation. It is recommended to provide a constraint (e.g., 100K or 1M) for larger data sets as the exact data set information can be less essential in the schemas and attempts to obtain exact data type usage statistics may take unreasonably long time.
calculateInstanceNamespaces	no, detailed, overview	Experimental feature: calculate instance namespace URIs (default = no)
sampleLimitFor InstanceNamespacesCalculation	Long	Limit of sample instance count to use in namespace calculation. No value or 0 means all data will be used (default = 1000)
addedLabels	List	Properties for class and property labels. rdfs:label and skos:prefLabel are assumed by default. For labels in specific languages use @{en;de} notation after the label property IRI or short form. To exclude a default labeling property, use property@{-} notation
addIntersectionClasses	yes, no	Record intersection class information in the extracted schema (default = yes)
excludedNamespaces	List	List of excluded namespaces, e.g. http://www.openlinksw.com/schemas/virtrdf#
includedClassesFile	CSV file	Valid CSV file with the list of included classes. To be used, if the created schema is to be restricted to the explicitly listed classes. If not specified, all classes will be analyzed.
includedPropertiesFile	CSV file	Valid CSV file with the list of included properties. To be used, if the created schema is to be restricted to the explicitly listed properties. If not specified, all properties will be analyzed
enableLogging	Boolean	Enable SPARQL query logging to file on the server (default = true)
saveThisConfig	Boolean	Save this configuration to a file on the server (default = true)

The following parameters can be set using the configuration file (the aformentioned parameters can be used from the configuration file, as well, except includedClassesFile is replaced by includedClasses and includedPropertiesFile is replaced by includedProperties.

Parameter Name	Datatype / Enum	Description
validateClassesList	Boolean	Validate the class list using simple and statistics queries (default = true)
validatePropertiesList	Boolean	Validate the property list using simple and statistics queries (default = true)
includeDistinctSubjectsAndObjects	no, propertyLevel, yes	Calculate distinct subjects and objects either not at all, at property level only, or also in source/target class context (default = propertyLevel)
computeBlankNodeStatistics	no, targetsOnly, full	Calculate blank node statistics. targetsOnly checks blank-node objects for property triples only; full also checks blank-node subjects for properties, as well as blank node class instances (default = no)
addIntersectionClasses	yes, no	Add calculated intersection classes to the output schema (default = yes)
exactCountCalculations	yes, no	Controls whether exact/DISTINCT count queries are used. Using DISTINCT in queries may give finer results, if the same triples are provided by the endpoint more that once (e.g., from several named graphs). Using DISTINCT may lead to dramatic performance and quality decrease for larger endpoints, therefore it is not recommended (consider obtaining distinct instances/triples in simpler settings instead) (default = no)
calculateDistinctTriples	Boolean	Calculate distinct class instances/property triples separately when exact count calculations are not enabled (default = true)
principalClassificationProperties	List	Main classification properties defining class structure. If omitted in config, defaults to full rdf:type URI: http://www.w3.org/1999/02/22-rdf-syntax-ns#type
classificationPropertiesWithConnectionsOnly	List	Additional classification properties whose objects may define subsets of principal classifiers and are checked for property connections (default = empty list). Limited use.
simpleClassificationProperties	List	Simple classifier properties whose objects may define subsets of principal classifiers, but are not checked as sources/targets for other properties (default = empty list). Can be recommended for specifying properties with classifier values that are expected to enrich the schema.
includedLabels	List	Label properties to use for classes and properties. Each object has labelPropertyFullOrPrefix and languages. In config files, include rdfs:label and skos:prefLabel explicitly if they are wanted
includedLabels.labelPropertyFullOrPrefix	String	Full IRI or known prefix form of a label property, e.g. rdfs:label, skos:prefLabel, or http://www.w3.org/2004/02/skos/core#altLabel
includedLabels.languages	List	Optional list of language tags for a label property, e.g. ["en"]. Empty list means no language restriction
includedClasses	List	Optional list of classes to include. If omitted or empty, classes are discovered from the endpoint
includedClasses.className	String	Class URI to include
includedClasses.instanceCount	String (integer)	Optional known instance count for the class. If absent or not positive, the count is queried from the endpoint
includedProperties	List	Optional list of properties to include. If omitted or empty, properties are discovered from the endpoint
includedProperties.propertyName	String	Property URI to include
includedProperties.instanceCount	String (integer)	Optional known triple/property instance count. If absent or not positive, the count is queried from the endpoint
excludedNamespaces	List	List of namespaces or prefixes to exclude. Matching is done as URI/string prefix matching
largeQueryTimeout	Long	Timeout for large SPARQL queries, in seconds. 0, null, or negative means no timeout (default = 0). Some typical values can be 600 or 1200.
smallQueryTimeout	Long	Timeout for small SPARQL queries, in seconds. 0, null, or negative means no timeout (default = 0). A typical value can be 60.
delayOnFailure	Long	Delay after query/endpoint failure, in seconds. 0, null, or negative means no delay (default = 0). Can be useful in rare occasions, if the endpoint responds positively to Health check query (cached), but not real data queries.
waitingTimeForEndpoint	Long	Waiting time for endpoint recovery, in minutes. 0 or null means wait forever (default = 0)
logNoClassesForProperty	yes, no, sourcesOnly	Controls logging when a property has no source or target (for properties with non-literal objects) classes (default = no)
crossCheckTargetClassesOn NonLiteralPropertyObjectCheckFailure	Boolean	If non-literal property object check fails, still cross-check target classes (default = false)