Python implementation of ShEx 2.0
This package is a reasonably literal implementation of the Shape Expressions Language 2.0. It can parse and "execute" ShExC and ShExJ source.
- 0.2.dev3 -- added SchemaEvaluator and other tweaks. There are still some unit tests that fail -- beware
- 0.3.0 -- Fix several issues. Still does not pass all unit tests -- see
- 0.4.0 -- Added sparql_slurper capabilities.
- 0.4.1 -- Resolves several issues with reactome and disease test cases
- 0.4.2 -- Fix issues #13 (missing start) and #14 (Inconsistent shape causes loop)
- 0.4.3 -- Fix issues #16 and #15 and some refactoring
- 0.5.0 -- First cut at returning fail reasons... some work still needed
- 0.5.1 -- Update shexc parser to include multi-line comments and bug fixes
- 0.5.2 -- Issue with installer - missed the parse_tree package
- 0.5.3 -- make sparql_slurper a dependency
- 0.5.4 -- Fixed long recursion issue with blood pressure example
- 0.5.5 -- Fixed zero cardinality issue (#20)
- 0.5.6 -- Added CLI entry point and cleaned up error reporting
- 0.5.7 -- Throw an error on an invalid focus node (#23)
- 0.5.9 -- Candidate for ShEx 2.1
- 0.5.10 -- Fixed evaluator to load files, strings, etc. as ShEx
- 0.5.11 -- Added Collections Flattening graph option to evaluator.
pip install PyShEx
Note: If you need to escape single quotes in RDF literals, you will need to install the bleeding edge of rdflib:
pip uninstall rdflib pip install git+https://github.com/rdflib/rdflib
rdflib-jsonld is NOT compatible with the bleeding edge rdflib, so you can't use a json-ld parser in this situation.
> shexeval -h usage: shexeval [-h] [-f FORMAT] [-s START] [-fn FOCUS] [-d] [-ss] [-cf] rdf shex positional arguments: rdf Input RDF file or SPARQL endpoint if slurper option set shex ShEx specification optional arguments: -h, --help show this help message and exit -f FORMAT, --format FORMAT Input RDF Format -s START, --start START Start shape -fn FOCUS, --focus FOCUS RDF focus node -d, --debug Add debug output -ss, --slurper Use SPARQL slurper graph -cf, --flattener Use RDF Collections flattener graph
pyshex package is subdivided into:
- shape_expressions_language - implementation of the various sections in Shape Expressions Language 2.0. As an example, 3. Terminology is implemented in p3_terminology.py, 5.2 Validation Definition in p5_2_validation_definition.py, etc.
- shapemap_structure_and_language - implementation of ShapeMap Structure and Language (as well as we can understand it)
- sparql11_query - required sections from SPARQL 1.1 Query Language section 17.2
- utils - supporting utilities
The ShEx schema definitions for this package come from ShExJSG
We are trying to keep the python as close as possible to the (semi-)formal specification. As an example, the statement:
Se is a ShapeAnd and for every shape expression se2 in shapeExprs, satisfies(n, se2, G, m)
is implemented in Python as:
... if isinstance(se, ShExJ.ShapeAnd): return satisfiesShapeAnd(cntxt, n, se) ... def satisfiesShapeAnd(cntxt: Context, n: nodeSelector, se: ShExJ.ShapeAnd) -> bool: return all(satisfies(cntxt, n, se2) for se2 in se.shapeExprs)
This package is built using:
- ShExJSG -- an object representation of the ShEx AST as defined by ShEx.jsg and compiled through the PyJSG compiler.
- The python ShExC compiler -- which transforms the Shape Expressions Language into ShExJSG images.
Performance has been improved, but our current implementation of the
sparql_slurper is entirely too fine-grained. Our
next steps include:
- Get non-conformance reasons into the responses
- Improve diagnostic and debugging tools
- Add a time-out to catch really long evaluations
- Adjust the slurper to pull larget chunks as needed and then refine on the retrieval end
This implementation passes all of the tests in the master branch of validation/manifest.ttl with the following exceptions:
At the moment, there are 1077 tests, of which:
- 970 pass
- 107 are skipped - reasons:
- (52) sht:toldBNode, sht:LexicalBNode and sht:BNodeShapeLabel test non-blank blank nodes (
rdflibdoes not preserve bnode "identity")
- (24) sht:OutsideBMP -- test uses multi byte unicode (two aren't tagged)
- (16) Uses ShEx 2.1 IMPORT feature -- not yet implemented (three aren't tagged)
- (3) Focus is a Literal -- not yet implemented
- (5) Uses ShEx 2.1 INCLUDE feature -- not yet implemented
- (3) Uses manifest shapemap feature -- not yet implemented
- (2) sht:relativeIRI -- this isn't a real problem, but we havent taken time to deal with this in the test harness
rdflibhas a parsing error when escaping single quotes. (Issue submitted, awaiting release)
As mentioned above, at the moment this is as literal an implementation of the specification as was sensible. This means, in particular, that we are less than clever when it comes to partition management.
manifest_tester.py is the current testing tool. Once we get through the complete set of tests we'll create a command line tool and a UI
Note: At the moment we're just returning pass/fail. We need to find documentation about what the return document should look like before we start returning detailed reports.