Fix protocol manifest

[gkellogg]

This is a fair rewrite of the protocol manifest (Turtle, HTML and now JSON-LD) based on the protocol_validator.

Information used in the CGI script has been extracted into a Ruby script used to generate the manifests. It should be at least as good as what was up previously, and conform to the semantics embedded in the CGI script (other than scripted result validation). It uses the http and cnt ontologies to encode the sequence of HTTP interactions, taking some liberties in the response fields to allow for matching results. For example:

:query_get a mf:ProtocolTest;
  dawgt:approval dawgt:Approved;
  dawgt:approvedBy <http://www.w3.org/2009/sparql/meeting/2012-11-20#resolution_3>;
  mf:action [
    a ht:Connection;
    ht:connectionAuthority "www.example";
    ht:requests ([
        a ht:Request;
        ht:absolutePath "/sparql?query=ASK%20%7B%7D&default-graph-uri=http%3A%2F%2Fkasei.us%2F2009%2F09%2Fsparql%2Fdata%2Fdata0.rdf";
        ht:httpVersion "1.1";
        ht:methodName "GET";
        ht:resp [
          a ht:Response;
          ht:body [
            a cnt:ContentAsText;
            cnt:characterEncoding "UTF-8";
            cnt:chars true
          ];
          ht:headers ([
              a ht:RequestHeader;
              ht:fieldName "Content-Type";
              ht:fieldValue "application/sparql-results+xml, application/sparql-results+json";
              ht:headerElements ([
                  a ht:HeaderElement;
                  ht:elementName "application/sparql-results+xml"
                ] [
                  a ht:HeaderElement;
                  ht:elementName "application/sparql-results+json"
                ])
            ]);
          ht:statusCodeValue "2XX",
            "3XX"
        ]
      ])
  ];
  mf:name "query via GET" .

cc/ @kasei

[gkellogg]

Another thing that might be useful is to remove the fuzzy results comparison, and instead use a specific result set (.srx, .srj, or .ttl) to compare that the result set or graph is isomorphic with the expected result. From the manifest description, it would just mean adding a mf:result which isn't currently used. Then, the ht:body within the ht:resp could be eliminated. Might benefit from desiring a negative version of mf:ProtocolTest, but this could be implied by the expected status code.

[kasei]

Overall I think this looks good. I'm going to try to take a closer look at the specifics of the protocol test definitions this coming week.

I'm unfamiliar with all the ruby/rake/haml stuff in here. I see that those predate this PR, but I would find it helpful to have a description of what's going on here in the PR (and maybe also somewhere in the repo) – what files are source of truth, what are supporting code to generate other files, which files are committed to git but are just generated from other files, etc.

[gkellogg]

The Ruby/Rake/Haml stuff is just there to produce the manifests. It happens to be what I'm familiar with, but the data structure used in the gen_manifest.rb should be pretty familiar to anyone using Python or Javascript. The key is the content of the mf:action which is produced.

[gkellogg]

An interesting project might be to re-write the CGI to use the interaction descriptions from the manifest, rather than as baked in. I essentially did this for my own tests (here and here), although they use a mock testing structure.

[gkellogg]

I would find it helpful to have a description of what's going on here in the PR (and maybe also somewhere in the repo) – what files are source of truth, what are supporting code to generate other files, which files are committed to git but are just generated from other files, etc.

Fair enough, as this mechanism is used pervasively throughout the repo. I'll make that the focus of another PR, but basically, the strategy is to generate JSON-LD based manifests from the Turtle source using a common Frame, and then use the resulting manifest.jsonld to generate the HTML. The template treats the JSON-LD as plain JSON and just iterates through the data-structures to generate HTML. There is more specific code for protocol to generate the manifest.ttl, but generally the code to generate the manifest.jsonld and index.html is common and run from the Rakefile in the root directory.

In principle, this could be done easily enough in another language, as code to parse and frame JSON-LD from Turtle is standard, and Haml is one of many different templating languages (e.g. Shpaml.

[gkellogg]

@kasei ping.

[joka921]

@kasei @gkellogg
We are currently working on a tool that uses the manifest.ttl files to run the tests. Especially for the protocol tests this is somewhat hacky, as th manifests are a little underspecified. From our experience, every solution should use the manifest (publicly available, readable to humans and machines alike) as the sole source of ground truth (specify the inputs for the default and named graphs, specify the expected result, and only the input is not a query (like for the other test types) but a literal protocol string.

Then you can still build an official binary evaluation via CGI or something else, but it is easier to also integrate these tests and extend them into other tools.

[gkellogg]

Rebased to lastest, although some more follow-on may be necessary.

[w3cbot]

This was discussed during the #rdf-star meeting on 03 July 2025.

View the transcript

w3c/rdf-tests#79

gregg: 79 is created by me
… nothing is in manifest that drove the expected behaviour
… depended on running a script that gregg wiliams managed
… not relating to work that needs to be done soon, we should consider if that is the approach we want to do

ora: would be ok to check if people want to merge

andy: agree, worth it for moving things forward

adrian: 193 and 198 seem to be ok, on concepts doc

pchampin: will take care about this by end of week

---