LDES validation client

This simple LDES client allows validating server implementations with a SHACL file. As an LDES stream consists of multiple pages linked to each other, SHACL validation of individual pages is insufficient to guarantee consistency.

The client follows all tree:relations from the starting url, and adds all fragments/tree:nodes to one big graph. It does so by applying object embedding: All triples contained in a certain graph (page) are anonymised and nested inside a page object.

Object embedding

The object embedding algorithm assumes that the graph operated on is an object graph, and can be represented as a tree: only named object occur at the 'root', and blank nodes can only occur once in the object position.

Given a graph containing the following triples:

food:stew a schema:Recipe ;
	rdfs:label "Stoofvlees"
	schema:step [
		schema:position "1" ;
		schema:text "Vlees aanbakken" .
	] ;
	schema:step [
		schema:position "2" ;
		schema:text "Pan deglaceren met St. Bernardus 12" .
	] .
food:maccheese a schema:Recipe ;
	rdfs:label "Mac and Cheese" .

After object embedding this becomes:

[] a oe:RDFDocument;
	oe:hasObject [
		rdf:subject food:stew ;
		a schema:Recipe ;
		rdfs:label "Stoofvlees"
		schema:step [
			schema:position "1" ;
			schema:text "Vlees aanbakken" .
		] ;
		schema:step [
			schema:position "2" ;
			schema:text "Pan deglaceren met St. Bernardus 12" .
		] .
	] .
	oe:hasObject [
		rdf:subject food:maccheese ;
		a schema:Recipe ;
		rdfs:label "Mac and Cheese" .
	] .

Install

• checkout repository
• cd to folder
• python3 -m venv env
• . ./env/bin/activate
• pip install pipreqs
• pip install -r requirements.txt

Freeze dependencies

• pipreqs

How to run

• Set following required parameters in the class TreeSpider of tree.py. Please refer to the example gipod.py.
  • name
    • e.g. name = "gipod"
  • allowed_domains
    • e.g. allowed_domains = ["private-api.gipod.beta-vlaanderen.be"]
  • start_urls
    • e.g. start_urls = ["https://private-api.gipod.beta-vlaanderen.be/api/v1/ldes/mobility-hindrances"]
• Under the directory ./crawldf/, run scrapy crawl [name], e.g.scrapy crawl gipod.
  or create a main.py under the directory ./crawldf/ with follow content, please modify it accordingly:

    from scrapy.cmdline import execute  
    execute(["scrapy","crawl","gipod"])

Output

• items.rdf file is generated under the directory ./crawldf/