Ability to know in a validation report which shapes actually matched some targets, and which didn't

[tfrancart]

In a validation report, it is impossible to distinguish between 2 situations:

1. A shape found some targets, and all targets conforms to the shape (hence, no validation results triggered by that shape)
2. The shape did not found any target (hence, no validation results triggered by that shape)

This can lead to (very) misleading situations where the validation report is empty, not because the data is valid, but because no targets were matched (e.g. a wrong identifier used in a sh:targetClass)

An idea to implement this would be to have an extra option when computing the validation report, e.g. "includeTargets" or something similar, that would add an explicit link (e.g. sh:hasTarget) between shapes and their targets in the validation report:

# this shape found some targets
ex:MyShape a sh:NodeShape
  sh:hasTarget ex:node1, ex:node2 ;
.

# this one didn't
ex:MyShape a sh:NodeShape .

(could be also in the other direction ex:node1 sh:isTargetOf ex:MyShape)

[tpluscode]

What if you reverse that to look from the Focus Node? Like code coverage. Maybe it's more important to find that my data was not matched by any shapes than to learn which shapes were not used.

[tfrancart]

What if you reverse that to look from the Focus Node? Like code coverage. Maybe it's more important to find that my data was not matched by any shapes than to learn which shapes were not used.

Absolutely - you were too quick, I separated this question in another issue at #326
But definitely yes, we need something like "shape coverage".
Historically, this use case was the very first we encountered, but we more recently stumbled upon #326.

[tpluscode]

Yea, a minute later I saw notifications from the other issue.

I wonder if it isn't two sides of the same coin, but #326 would be more useful. And then you can always query from a coverage report to see which shapes were involved (matched their targets) and which were not

[tfrancart]

A note on how we tackle this in SHACL Play:

1. We resolve shapes targets by analyzing target definitions : https://github.com/sparna-git/shacl-play/blob/master/shacl-validator/src/main/java/fr/sparna/rdf/shacl/targets/ShapeFocusNodesResolver.java
   (probably not very well named, should be ShapesTargetResolver and not ShapeFocusNodeResolver, as shapes can validate focus nodes without having target definitions)
2. We add an extra explicit link in the validation report and in the data graph to link shapes to nodes: shp:hasFocusNode (here again, name of the property subject to change) : https://github.com/sparna-git/shacl-play/blob/master/shacl-validator/src/main/java/fr/sparna/rdf/shacl/targets/StoreHasFocusNodeListener.java
3. Then we do a second pass of validation on this enhanced data graph + shapes graph with a custom shape that checks that every shape that has a target must have at least one shp:hasFocusNode : https://github.com/sparna-git/shacl-play/blob/master/shacl-validator/src/main/resources/shapes-coverage-shapes.ttl - this is merged into the final validation report
4. And we do another pass to check that every subject in the data graph was targeted at least once: https://github.com/sparna-git/shacl-play/blob/master/shacl-validator/src/main/resources/closed-graph-shapes.ttl - this is merged into the final validation report

[tfrancart]

I wonder if it isn't two sides of the same coin

It is indeed. See the implementation note above, where we compute shapes targets, and then from this explicit link, we can validate which shape didn't had any targets, and which triple subject wasn't validated by anything.

[ajnelson-nist]

I feel like this is similar in goal to a general performance-profiling mechanism. I thought this was on an open Issue somewhere, but I can't find it.

Performance-profiling would care about how long it takes to "Finish" evaluating a shape against all its targets, and an important piece of context for that timing metric is knowing how many targets it evaluated.

A natural side-effect of performance profiling should be realizing, "Oh, wait, this shape didn't hit anything."

I appreciate that's just half of the concerns of this Issue. Performance-profiling applied to a node, rather than a shape, could help understand how complex each node is to validate against a shapes graph, though I suspect that would mostly be interesting from a SHACL design-symmetry perspective. However, the side-effect of realizing "Oh, wait, this node wasn't checked by anything" would be in line with the premise of this Issue.

[tfrancart]

Note : we might need to distinguish in the discussion between targets and focus nodes. Resolving targets of shapes is easy, but resolving every actual focus nodes of every shape could only be done by logging each focused node while validating.
For the use-case described here and in #326, we are really interested by the targets only, not by the focus nodes of every (node and property) shapes.

[tpluscode]

I feel like this is similar in goal to a general performance-profiling mechanism.

This and similar functionalities could be solved by adding informational, non-violation instances of sh:Result. Other than predefining possible specialised result subtypes (re #192), I don't think there is anything blocking even in the current SHACL spec.

[afs]

There are probably many use cases so the standards could aim to at find the building blocks that tools can use this to check shapes to target mapping, coverage (of data graph, of shapes graph, of imports even).

Thought: maybe instead of putting this in the validation report, we could have a separate output (which can go in the same graph or a different graph as the validation report) which is the trace of a validation run.

Tools could then use this trace to do various analysis.

[tfrancart]

There are probably many use cases so the standards could aim to at find the building blocks that tools can use this to check shapes to target mapping, coverage (of data graph, of shapes graph, of imports even).

I think the necessary building block here are:

• a shape to focus node property : sh:hasFocusNode
• a shape to target node mapping property : sh:hasTargetNode (it needs to be in this direction since targets can be literal), which would be subproperty of sh:hasFocusNode, since targets are special ways to resolve to focus nodes

Thought: maybe instead of putting this in the validation report, we could have a separate output (which can go in the same graph or a different graph as the validation report) which is the trace of a validation run.

I like the idea very much

[tpluscode]

Thought: maybe instead of putting this in the validation report, we could have a separate output (which can go in the same graph or a different graph as the validation report) which is the trace of a validation run.

What is the reason not to use specialised types of sh:Result for this within the validation report itself?

[TallTed]

trace of a validation run

I am strongly in favor of this idea, wherever it might get written.

[afs]

What is the reason not to use specialised types of sh:Result for this within the validation report itself?

"which can go in the same graph"

The question is whether it must be in the validation report.

As the validation report and the trace serve different purposes they may go to different tools.

The trace may be large - and a conforming validation report is very small.

[TallTed]

The question is whether it must be in the validation report.
[...]
The trace may be large - and a conforming validation report is very small.

Presumably, the trace is a controllable option, so a user chooses when it gets written (and maybe where), similar to traces and other debug output from other tools.

I would be against such a trace/log being automatically produced from every run.