Additional SHACL support to Ontospy/gendocs #107

balon · 2021-12-09T00:45:28Z

This patch series is intended to bring Ontospy's current SHACL support up to the level of displaying Property Shapes
on class pages and Shape Properties on shape pages.

A new module, ontospy/core/shacl_helper.py, returns SHACL property constraints, is attached to each OntoClass object,
once the shapes are built in Ontospy.

A few notes on changes:

The modifications made in 4a6d48c method of linking nodes in tree-diagram were done to ensure the diagram was still properly linking to other shapes (this is achieved by using a django filter and 'forwarding' the slugify function from core/utils)
The modifications made in 1750e87 will only display if an ontology uses SHACL

Please advise if there is a better approach to increasing the support for SHACL into Ontospy, or if any modifications are
needed to the current patch presented.

This patch is the first in a series to bring Ontospy's current state of SHACL support up to the level where the Shapes pages are built. Serialization will be handled in a later patch. ontospy/core/shacl_helper.py: * Returns dictionary of SHACL property constraints, given an Ontospy object ontospy/core/ontospy.py: * Imports shacl_helper to build SHACL property constraints * Amends call to build SHACL property constraints at end of build_shapes method - Modifies logic to allow all_shapes to be built Co-authored-by: Dr. Stanley A Bernsteen <sbernste@mitre.org> Signed-off-by: TJ Balon <tbalon@mitre.org>

This patch modifies the browser_shapeinfo.html file to adjust the formatting and provide documentation. The patch has no functional changes. Modifications: * Addition of comments to note logic for panels * Reformat HTML to conventional tag indentation * Remove excess whitespace Co-authored-by: Dr. Stanley A Bernsteen <sbernste@mitre.org> Signed-off-by: TJ Balon <tbalon@mitre.org>

This modifies the browser_shapeinfo.html file to display the SHACL shape properties based on the OntoShape object that is rendered on the HTML page. Co-authored-by: Dr. Stanley A Bernsteen <sbernste@mitre.org> Signed-off-by: TJ Balon <tbalon@mitre.org>

This patch adds functionality to the HTML rendering portion of the ontodocs, and includes an additional namespace for rdflib. ontospy/ontodocs/viz/viz_html_multi.py: * Adds function to identify OWL Classes which define themselves as SHACL Node Shapes - shape and class URIs must match - must be of RDF.type: SH.NodeShape - must be of RDF.type: OWL.Class ontospy/core/utils.py: * Imports SH namespace for rdflib Co-authored-by: Alex Nelson <alexander.nelson@nist.gov> Signed-off-by: TJ Balon <tbalon@mitre.org>

This patch replaces the tree-diagram in the browser_shapeinfo.html file if the shape is a SHACL NodeShape and OWL Class (with matching URIs). ontospy/ontodocs/viz_factory.py: * Imports Library for django.template * Adds filter to call ontospy.core.utils.slugify() from a django template ontospy/ontodocs/viz/viz_html_multi.py: * Checks shapes to see if they are defined as OWL.Class & SH.NodeShape - Updates parents/children of the entity to the matching class's parents/children for the tree-diagram during template rendering ontospy/../browser/browser_shapeinfo.html: * Replaces hyperlinks in tree-diagram to generalize if a shape/class diagram is build built - links are prefixed with "shape-" - slugs are replaced with qname and a call to the slugify filter Signed-off-by: TJ Balon <tbalon@mitre.org>

This patch modifies the browser_classinfo.html file to adjust the formatting and provide documentation. The patch has no functional changes. Modifications: * Addition of comments to note logic for panels * Reformat HTML to conventional tag indentation * Remove excess whitespace Co-authored-by: Dr. Stanley A Bernsteen <sbernste@mitre.org> Signed-off-by: TJ Balon <tbalon@mitre.org>

This patch modifies the browser_classinfo.html file to display a panel for classes which have associated SHACL property shapes. If an ontology does not implement SHACL, there are no functional changes. Co-authored-by: Dr. Stanley A Bernsteen <sbernste@mitre.org> Signed-off-by: TJ Balon <tbalon@mitre.org>

lambdamusic · 2021-12-09T09:21:31Z

Thanks. I'll be able to review it over the next few days.
If you could attach/reference a sample rdf file for testing all new features that'd be great.

This patch addresses a use case where an ontology defines no or few rdfs:domain statements for its properties, instead relating properties to classes via SHACL shapes. If a property is associated to classes via SHACL property shapes within an ontology, it does not seem appropriate to treat the property as universally applicable. Co-authored-by: Alex Nelson <alexander.nelson@nist.gov> Signed-off-by: TJ Balon <tbalon@mitre.org>

balon · 2021-12-10T17:57:13Z

Attached is uco_monolithic.ttl, which can be used for testing the features in this patch series. The file could also be constructed by running the makefile found in the ucoProject/UCO repository.

Thanks!

uco_monolithic.ttl.zip

This patch should have no functional impact, except possibly some speedup from a braces encapsulation. Signed-off-by: Alex Nelson <alexander.nelson@nist.gov>

Review of the SHACL specification showed the sh:targetClass is not explicitly required under certain conditions, cited in getPropsApplicableByShapes. Signed-off-by: Alex Nelson <alexander.nelson@nist.gov>

The SHACL specification, section 2.1.3.3 "Implicit Class Targets", indicates a reflexive sh:targetClass triple is implied for a SHACL Class that is also a Shape. The function updated in this patch reviewed NodeShapes for being classes based on an explicit sh:targetClass statement. This patch changes the check to rely on that triple being implied under the SHACL specification conditions. Additionally, the "Class" definition is adjusted to being a SHACL Class, which covers RDFS Classes in addition to the prior implementation of only OWL classes. References: * https://www.w3.org/TR/shacl/#implicit-targetClass Signed-off-by: Alex Nelson <alexander.nelson@nist.gov>

ajnelson-nist · 2021-12-16T16:08:37Z

This PR has received an in-flight update, for the same reason noted over on PR 108.

lambdamusic · 2021-12-18T21:01:48Z

First of all, thanks! This looks like a terrific addition to Ontospy.

I just released a minor new version v 1.9.9.4 that should simplify merging this functionality (it's in master), so would you please update your changes accordingly?

Here's a few comments to get started.

I am not too familiar with Shacl patterns - but I do understand that it is customary to add two types to the same URI (a class and a shape). Ontospy always took the naive approach of a unique-URI-per-entity-type, so I can see why you changed this line (..if not test_existing_cl:.... That shouldn't be necessary anymore with the just released v 1.9.9.4, I've added a more specific method self.get_shapes that allows to safely index classes/shapes with the same URI.
For some reason the documentation generation is backward incompatible. See for example this documentation page for SciGraph - RDF source here - previously I was able to generate a 'Shapes' page for SciGraph, but with your patch I get none. Any idea why?
More generally, re the templates: I don't fully understand how/where you would like to organise the information about Shapes in the HTML templates .. ideally for me any changes should be backward compatible. I'll be spending more time on this after points 1 and 2 above are sorted.

Hope this makes sense..

PS Just FYI I was planning to clean up the whole codebase (admittedly a bit dated and messy!) and release a V2 of Ontospy in the near future, with better docs etc.... I'll hold off doing that until we complete this PR!

tobiasschweizer · 2022-01-06T13:43:53Z

Hi

I added a feature request in #104 and I think your PR does what I strive for :-)

I did the following:

I cloned https://github.com/casework/Ontospy/tree/feature-SHACL-gendocs
I ran python setup.py install and installed pygments and django<4(Possible Problem with Django Version 4 #112)
I ran ontospy gendocs uco_monolithic.ttl from my venv

What I get as HTML looks correct in terms of property shapes on the class page. This is really nice!

However, I tried this with my own ontology which consists of

an ontology part for schema.org classes
a SHACL part that targets schema.org classes via sh:targetClass

As observed in #107 (comment) I don't see the Shapes page anymore.

Could this be because classes and node shapes do not use the same IRI? For some more details about the ontology I use, please see #104

Thanks a lot!

ajnelson-nist · 2022-01-07T19:16:40Z

Thank you both for the feedback! We'll be getting back to you next week.

balon · 2022-01-12T17:51:50Z

Thank you for the feedback. We had a discussion today about a timeline to updating the PR as requested. We're both currently tied up on a few tasks, but plan to address this early next week with plans to have something ready to push by next Friday.

We will also look at the issues presented involving the missing shapes page on some ontologies.

lambdamusic · 2022-01-12T20:28:53Z

Sounds good, thanks. If there are any updates at my end, I'll let you know.

ajnelson-nist · 2022-01-20T21:39:46Z

We're working through the remarks now. We've made a catch-up merge to integrate master's current state, and have converted the branch to a draft state while we step through the other remarks.

@lambdamusic , re:

For some reason the documentation generation is backward incompatible. See for example this documentation page for SciGraph - RDF source here - previously I was able to generate a 'Shapes' page for SciGraph, but with your patch I get none. Any idea why?

Could you explain what behavior you're seeing here? After our catch-up merge, we regenerated the page, and saw these effects:

https://lambdamusic.github.io/ontospy-examples/scigraph-core-ontologyttl/index.html - this page's "Metrics" table lists 17 data shapes, which matches the current deployed state on ontospy-examples.
https://lambdamusic.github.io/ontospy-examples/scigraph-core-ontologyttl/entities-tree-shapes.html - this page lists 17 shapes.
https://lambdamusic.github.io/ontospy-examples/scigraph-core-ontologyttl/shape-shapesaffiliation.html - this page does not list a properties table. However, we believe that is because we search for properties based on sh:path being defined in the sh:PropertyShape. (Our added queries do not search for an explicit class declaration of sh:PropertyShape; we instead assume that based on sh:property linking the blank node and implying it's a property shape.)

I suspect we've actually resolved your second issue, but that's mainly from our failure to reproduce what you saw.

ajnelson-nist · 2022-01-20T21:44:16Z

On further review, sh:predicate is not defined in SHACL. Did the scigraph RDF source intend to use sh:path?

ajnelson-nist · 2022-01-20T21:55:29Z

On the shapes page, we intend to list a table of how properties associate with a class. There's a conceptual hop there, which is probably something we need to talk through to come to a good design consensus.

The user interface complexity we're trying to hide is going to apply to SHACL-based schemes that don't define owl:Restrictions. In our use case, we don't have owl:Restrictions, but we do have sh:PropertyShapes. So, our property associations would be listed on the Shapes HTML page. But, to see things about other classes in the parent/child hierarchy, the user would hop over to the Class page, where no properties would be listed.

So, we tried to apply some redundancy - having the properties associated via SHACL be listed on the class page as well. This saves the user two clicks (oscillating between shapes and class pages) per level of the class hierarchy they want to review.

We probably wouldn't go for this redundancy if SHACL had a concept of "SubNodeShape," but they don't.

That's the feedback we could get to today, we're happy to discuss this further.

lambdamusic · 2022-03-11T18:57:08Z

Hi folks - first of all, wish I had time to get back to you sooner, sorry about that and thanks a lot for the patience.
I reviewed the PO after the merging, have three main comments.

1) Shapes HTML page for scigraph is ok

@ajnelson-nist as you pointed out, the scigraph ontology 'shapes' page is not missing anymore. Don't know what's changed, I do recall noticing the difference, but of course I might be wrong. So all's good there. Backward compatibility is fine.

2) New ontospy tests error

I do get an error when running the test suite (./tools/run-all-tests.sh). The error is triggered when trying to scan the schema test RDF model. You can reproduce simply by typing $ ontospy scan ontospy/tests/rdf/schema/ (on this branch, of course). Here's the traceback:

...
  File "/Users/michele.pasin/code/python/ontospy/ontospyProject/ontospy/cli.py", line 181, in scan
    action_analyze(sources, endpoint, print_opts, verbose, extra, raw, individuals, format)
  File "/Users/michele.pasin/code/python/ontospy/ontospyProject/ontospy/core/actions.py", line 123, in action_analyze
    g = Ontospy(
  File "/Users/michele.pasin/code/python/ontospy/ontospyProject/ontospy/core/ontospy.py", line 134, in __init__
    self.build_all(
  File "/Users/michele.pasin/code/python/ontospy/ontospyProject/ontospy/core/ontospy.py", line 249, in build_all
    self.build_shapes()
  File "/Users/michele.pasin/code/python/ontospy/ontospyProject/ontospy/core/ontospy.py", line 635, in build_shapes
    shacl_constraints = build_shacl_constraints(self)
  File "/Users/michele.pasin/code/python/ontospy/ontospyProject/ontospy/core/shacl_helper.py", line 255, in build_shacl_constraints
    add_property_constraints_from_shape_triples(lineage_class.uri, property_constraints, shape.triples)
  File "/Users/michele.pasin/code/python/ontospy/ontospyProject/ontospy/core/shacl_helper.py", line 298, in add_property_constraints_from_shape_triples
    raise Exception('Something is very wrong.  Class URI not in triples.')
Exception: Something is very wrong.  Class URI not in triples.

The error is triggered by line line 297. Not really sure I fully understand the logic there, so maybe you can shed some light.

Depending on how bad the cause of the error is, I'd suggest issuing a warning instead and/or failing silently.
Ontospy is not really a schema validator, so I'd prefer to render an incomplete output rather than failing completely.

3) SciGraph shapes error

You are totally right @ajnelson-nist, sh:predicate should be sh:path. Not sure what happened there, but I suppose when we built that model a few years back we may have used an older Shacl specification which is now deprecated. Who knows..

Anyhow, good that you spotted it. I've updated the test RDF with sh:path within a new folder scigraph2 and pushed it to this branch. Tried to run ontospy on it, but also here I'm getting the error above:

  File "/Users/michele.pasin/code/python/ontospy/ontospyProject/ontospy/core/shacl_helper.py", line 298, in add_property_constraints_from_shape_triples
    raise Exception('Something is very wrong.  Class URI not in triples.')
Exception: Something is very wrong.  Class URI not in triples.

So, even more curious about how to get rid of that.

To sum up - I believe that after addressing that validation error above, we're good to go :-)

The UI issues you mentioned are important, but maybe better tackled outside this PR and by using more test input data.

…on review This patch starts a series of type system reviews to diagnose a few similarly-appearing bugs. Signed-off-by: Alex Nelson <alexander.nelson@nist.gov>

ajnelson-nist · 2022-04-11T22:01:52Z

First, @tobiasschweizer , you were right to give a poke, thanks for that.

@lambdamusic , thank you for the test case. We stumbled on a similar issue. @balon and I have started a type review of the shacl_helper.py script on observing that there were a few errant assumptions about nodes being required to be blank nodes or rdflib.URIRefs.

A recent release of UCO triggered a documentation build failure, localized to something in `shacl_helper.py`. Review of the code found a few missed characteristics of the SHACL specification. The build failure came from an assumption that a SHACL PropertyShape would always be a blank node. While in UCO this was previously true, a named SHACL Property Shape was introduced, defeating the assumption in `shacl_helper.py`. The issue proved difficult to diagnose. As an aid, the file received type signature additions, to help separate RDFLib types, from OntoSpy types, from types local to `shacl_helper.py`. Some class-level variables also got promoted to module-level variables, because (1) their usage behavior was effectively as a module-level variable, and (2) the design pattern can lead to nefariously difficult behavior to debug. See [1], searching the page for the phrase "mistaken use of a class variable". For the sake of improving code comprehensibility, some coding patterns that were using nested `DefaultDict`s to assist with RDF triple iteration were replaced with using temporary `rdflib.Graph`s, for the `graph.triples()` iterator. It's fair to note this trade for comprehensibility may have come at some runtime expense. With this patch, `shacl_helper.py` independently passes static type review by running `mypy` (version 0.942). Type errors are reported from other files that were reviewed only as deep as was necessary to review `shacl_helper.py`'s type goals (especially `entities.py`), but we consider reviewing type signatures in those files as out of scope of this pull request. Also with this patch, UCO 0.8.0 documentation once again builds. References: * [1] https://docs.python.org/3/tutorial/classes.html Acked-by: TJ Balon <tbalon@mitre.org> Signed-off-by: Alex Nelson <alexander.nelson@nist.gov>

ajnelson-nist · 2022-04-27T23:18:45Z

@lambdamusic : @balon and I think the "Something is very wrong" exception has been addressed. I think we've covered the tests you've mentioned above; ./tools/run-all-tests.sh and nosetests both pass. If there's nothing else, this might be good to merge.

lambdamusic · 2022-05-26T15:50:43Z

Thanks @ajnelson-nist @balon this is good to go!

lambdamusic · 2022-05-26T16:56:45Z

Available in https://github.com/lambdamusic/Ontospy/releases/tag/2.0.0-alpha

ajnelson-nist · 2022-05-26T17:03:08Z

Thank you!

tobiasschweizer · 2022-05-30T06:01:51Z

Hi there,

That is great news, I'll try this asap.

balon and others added 7 commits December 8, 2021 17:11

balon and others added 2 commits December 9, 2021 16:09

Merge branch 'hotfix-SHACL-shapes-usage' into feature-SHACL-gendocs

153d5fa

lambdamusic added new-feature gendocs labels Dec 15, 2021

ajnelson-nist added 5 commits December 16, 2021 10:09

Fix whitespace nit reported by Git

141fe7a

Match new query's spelling to getSKOSDirectSupers

ae42fbf

This patch should have no functional impact, except possibly some speedup from a braces encapsulation. Signed-off-by: Alex Nelson <alexander.nelson@nist.gov>

Rely on implied sh:targetClass

6fa1b3b

Review of the SHACL specification showed the sh:targetClass is not explicitly required under certain conditions, cited in getPropsApplicableByShapes. Signed-off-by: Alex Nelson <alexander.nelson@nist.gov>

Merge branch 'hotfix-SHACL-shapes-usage' into feature-SHACL-gendocs

1d7e289

lambdamusic mentioned this pull request Dec 22, 2021

How to Document Properties and Cardinalities Defined in SHACL Shapes? #104

Open

tobiasschweizer mentioned this pull request Jan 19, 2022

Integrate SHACL Support for Docs on GitHub Pages Connectome-Implementation-Team/rescs_shacl_shapes#3

Open

Merge branch 'master' into feature-SHACL-gendocs

805d237

balon marked this pull request as draft January 20, 2022 21:34

lambdamusic marked this pull request as ready for review March 11, 2022 17:39

scigraph2 rdf test

c509dd8

tobiasschweizer mentioned this pull request Apr 8, 2022

Problem when validating xsd:float RDFLib/pySHACL#140

Closed

ajnelson-nist added 3 commits April 11, 2022 17:19

Rename variable to prevent package name conflict; start type annotati…

9c3961b

…on review This patch starts a series of type system reviews to diagnose a few similarly-appearing bugs. Signed-off-by: Alex Nelson <alexander.nelson@nist.gov>

Restore whitespace

8217329

Flag problem spot

b4dd695

lambdamusic merged commit a7f0506 into lambdamusic:master May 26, 2022

ajnelson-nist deleted the feature-SHACL-gendocs branch May 26, 2022 17:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Additional SHACL support to Ontospy/gendocs #107

Additional SHACL support to Ontospy/gendocs #107

balon commented Dec 9, 2021 •

edited

Loading

lambdamusic commented Dec 9, 2021

balon commented Dec 10, 2021

ajnelson-nist commented Dec 16, 2021

lambdamusic commented Dec 18, 2021

tobiasschweizer commented Jan 6, 2022 •

edited

Loading

ajnelson-nist commented Jan 7, 2022

balon commented Jan 12, 2022

lambdamusic commented Jan 12, 2022

ajnelson-nist commented Jan 20, 2022

ajnelson-nist commented Jan 20, 2022

ajnelson-nist commented Jan 20, 2022

lambdamusic commented Mar 11, 2022

ajnelson-nist commented Apr 11, 2022

ajnelson-nist commented Apr 27, 2022

lambdamusic commented May 26, 2022

lambdamusic commented May 26, 2022

ajnelson-nist commented May 26, 2022

tobiasschweizer commented May 30, 2022

Additional SHACL support to Ontospy/gendocs #107

Additional SHACL support to Ontospy/gendocs #107

Conversation

balon commented Dec 9, 2021 • edited Loading

lambdamusic commented Dec 9, 2021

balon commented Dec 10, 2021

ajnelson-nist commented Dec 16, 2021

lambdamusic commented Dec 18, 2021

tobiasschweizer commented Jan 6, 2022 • edited Loading

ajnelson-nist commented Jan 7, 2022

balon commented Jan 12, 2022

lambdamusic commented Jan 12, 2022

ajnelson-nist commented Jan 20, 2022

ajnelson-nist commented Jan 20, 2022

ajnelson-nist commented Jan 20, 2022

lambdamusic commented Mar 11, 2022

1) Shapes HTML page for scigraph is ok

2) New ontospy tests error

3) SciGraph shapes error

ajnelson-nist commented Apr 11, 2022

ajnelson-nist commented Apr 27, 2022

lambdamusic commented May 26, 2022

lambdamusic commented May 26, 2022

ajnelson-nist commented May 26, 2022

tobiasschweizer commented May 30, 2022

balon commented Dec 9, 2021 •

edited

Loading

tobiasschweizer commented Jan 6, 2022 •

edited

Loading