W3 test reorg #1891

ghost · 2022-05-07T16:30:56Z

Summary of changes

Migrated the various test_*_w3c.py test files into test/test_w3c_spec/
Removed the DAWG test suite because it's a legacy, un-updated version of the W3C sparql11 test suite (the latter not currently included as an RDFLib test suite)
Added the W3C sparql11 test suite to test/data/suites/w3c as a replacement for the old DAWG tests, renamed test_dawg.py as test_sparql_w3c.py and relocated as mentioned above.
Migrated the contained-within-DAWG RDFLib-specific SPARQL test suite (rdflib) to test/data/rdflib-sparql (with the intention of later breaking out the test separately from its current inclusion at the end of test/test_w3c_spec/test_sparql_w3c.py)
Changed the content of the (unused) test/data/suites/n3 (which isn't an element of the published W3C RDF test suite) to that of the W3C N3 test suite because it's an updated version of the same test suite, cloned test_turtle_w3c.py as test_n3_w3c.py, adjusted it to run and added the failures as a pro tem internal, marked-as-xfail "skiplist".
Replaced the elderly version of the Turtle test suite included in the W3C RDF test suite (as turtle) with the updated version of the same tests in the N3 test suite TurtleTests subfolder, fixing the incorrect reference in manifest.ttl as necessary to get it to work.
Pythonised the skiptests.list as test/skipped_sparql_tests.py to make it more tractable and adjusted test/test_w3c_spec/test_sparql_w3c.py to suit.
Added four (marked-as-xfail) unicode numeric codepoint escape tests to the RDFLib-specific SPARQL test suite test/data/rddlib-sparql
Added a couple of TurtleTest URIRefs to test/manifest.py and enabled the execution of DAWG-proposed tests, pro tem
Renamed the existing W3C RDF test suite rdfxml subdirectory as rdf-xml to update it to current W3C structure
Transcribed buried, commented-out tests in DAWG (e.g.

rdflib/test/DAWG/data-r2/basic/manifest.ttl

Line 33 in 6ed2ef4

# :term-6 # Not SPARQL 1.1

) as entries in skiplist.

(Or, the above can be implemented as half-a-dozen separate PRs if desired)

Checklist

Checked that there aren't other open pull requests for the same change.
Added tests for any changes that have a runtime impact.
Checked that all tests and type checking passes.
Considered granting push permissions to the PR branch,
so maintainers can fix minor issues and keep your PR up to date.

coveralls · 2022-05-07T16:58:19Z

Coverage remained the same at 88.248% when pulling 766ccc3 on gjhiggins:w3-test-reorg into b99d022 on RDFLib:master.

aucampia · 2022-05-07T19:00:34Z

@gjhiggins mind if I split this up a bit just to make it easier to review?

Also, if I run the fetcher the following files change - not sure why, I think maybe because you are using the test suites from github, if so we should correct the URLs but maybe it is best to consider updating the test suites independently also.

$ git log -1
commit 20188357204f8c687654fc03545d63f95abed0df (HEAD -> w3-test-reorg, gjhiggins/w3-test-reorg)
Author: Graham Higgins <gjh@bel-epa.com>
Date:   Sat May 7 18:19:52 2022 +0100

    use pubiished SPARQL test suite data
$ git status
On branch w3-test-reorg
Your branch is up to date with 'gjhiggins/w3-test-reorg'.

nothing to commit, working tree clean
$ task test:data:fetch 
task: [test:data:fetch] .venv/bin/python test/data/fetcher.py 
2022-05-07T20:58:25.953 168177 139962706229056 020:INFO     root         fetcher:87:fetch fetching https://github.com/w3c/N3/archive/c44d123c5958ca04117e28ca3769e2c0820f72e6.zip to temp file /tmp/tmpcv22er67/i6qH988zpZ.zip
2022-05-07T20:58:27.326 168177 139962706229056 020:INFO     root         fetcher:122:fetch Downloaded https://github.com/w3c/N3/archive/c44d123c5958ca04117e28ca3769e2c0820f72e6.zip and extracted files matching re.compile('^[^\\/]+[\\/]tests[\\/](.+)$') to /home/iwana/sw/d/github.com/iafork/rdflib.cleanish/test/data/suites/w3c/n3
2022-05-07T20:58:27.708 168177 139962706229056 020:INFO     root         fetcher:87:fetch fetching https://www.w3.org/2013/TurtleTests/TESTS.tar.gz to temp file /tmp/tmpcv22er67/LdC9B1dlIa.tar.gz
2022-05-07T20:58:27.930 168177 139962706229056 020:INFO     root         fetcher:122:fetch Downloaded https://www.w3.org/2013/TurtleTests/TESTS.tar.gz and extracted files matching re.compile('^[^\\/]+[\\/](.+)$') to /home/iwana/sw/d/github.com/iafork/rdflib.cleanish/test/data/suites/w3c/turtle
2022-05-07T20:58:28.298 168177 139962706229056 020:INFO     root         fetcher:87:fetch fetching https://www.w3.org/2013/N-QuadsTests/TESTS.tar.gz to temp file /tmp/tmpcv22er67/S1MSjxntZM.tar.gz
2022-05-07T20:58:28.332 168177 139962706229056 020:INFO     root         fetcher:122:fetch Downloaded https://www.w3.org/2013/N-QuadsTests/TESTS.tar.gz and extracted files matching re.compile('^(.+)$') to /home/iwana/sw/d/github.com/iafork/rdflib.cleanish/test/data/suites/w3c/nquads
2022-05-07T20:58:28.705 168177 139962706229056 020:INFO     root         fetcher:87:fetch fetching https://www.w3.org/2013/N-TriplesTests/TESTS.tar.gz to temp file /tmp/tmpcv22er67/JPjAizxL0A.tar.gz
2022-05-07T20:58:28.722 168177 139962706229056 020:INFO     root         fetcher:122:fetch Downloaded https://www.w3.org/2013/N-TriplesTests/TESTS.tar.gz and extracted files matching re.compile('^(.+)$') to /home/iwana/sw/d/github.com/iafork/rdflib.cleanish/test/data/suites/w3c/ntriples
2022-05-07T20:58:29.107 168177 139962706229056 020:INFO     root         fetcher:87:fetch fetching https://www.w3.org/2013/TrigTests/TESTS.tar.gz to temp file /tmp/tmpcv22er67/hQ2aGXyl1D.tar.gz
2022-05-07T20:58:29.333 168177 139962706229056 020:INFO     root         fetcher:122:fetch Downloaded https://www.w3.org/2013/TrigTests/TESTS.tar.gz and extracted files matching re.compile('^(.+)$') to /home/iwana/sw/d/github.com/iafork/rdflib.cleanish/test/data/suites/w3c/trig
2022-05-07T20:58:29.575 168177 139962706229056 020:INFO     root         fetcher:52:fetch Downloaded http://www.w3.org/2000/01/rdf-schema to /home/iwana/sw/d/github.com/iafork/rdflib.cleanish/test/data/rdfs.ttl
$ git status
On branch w3-test-reorg
Your branch is up to date with 'gjhiggins/w3-test-reorg'.

Changes not staged for commit:
  (use "git add/rm <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
	modified:   test/data/suites/w3c/n3/manifest.ttl
	deleted:    test/data/suites/w3c/turtle/IRI-resolution-01.nt
	deleted:    test/data/suites/w3c/turtle/IRI-resolution-01.ttl
	deleted:    test/data/suites/w3c/turtle/IRI-resolution-02.nt
	deleted:    test/data/suites/w3c/turtle/IRI-resolution-02.ttl
	deleted:    test/data/suites/w3c/turtle/IRI-resolution-07.nt
	deleted:    test/data/suites/w3c/turtle/IRI-resolution-07.ttl
	deleted:    test/data/suites/w3c/turtle/IRI-resolution-08.nt
	deleted:    test/data/suites/w3c/turtle/IRI-resolution-08.ttl
	modified:   test/data/suites/w3c/turtle/README
	deleted:    test/data/suites/w3c/turtle/blankNodePropertyList_as_object_containing_objectList.nt
	deleted:    test/data/suites/w3c/turtle/blankNodePropertyList_as_object_containing_objectList.ttl
	deleted:    test/data/suites/w3c/turtle/blankNodePropertyList_as_object_containing_objectList_of_two_objects.nt
	deleted:    test/data/suites/w3c/turtle/blankNodePropertyList_as_object_containing_objectList_of_two_objects.ttl
	deleted:    test/data/suites/w3c/turtle/localName_with_PN_CHARS_BASE_character_boundaries.nt
	deleted:    test/data/suites/w3c/turtle/localName_with_PN_CHARS_BASE_character_boundaries.ttl
	modified:   test/data/suites/w3c/turtle/localName_with_assigned_nfc_PN_CHARS_BASE_character_boundaries.nt
	modified:   test/data/suites/w3c/turtle/localName_with_assigned_nfc_PN_CHARS_BASE_character_boundaries.ttl
	modified:   test/data/suites/w3c/turtle/manifest.ttl
	deleted:    test/data/suites/w3c/turtle/predicateObjectList_with_blankNodePropertyList_as_object.nt
	deleted:    test/data/suites/w3c/turtle/predicateObjectList_with_blankNodePropertyList_as_object.ttl
	deleted:    test/data/suites/w3c/turtle/test-38.nt
	deleted:    test/data/suites/w3c/turtle/test-38.ttl
	deleted:    test/data/suites/w3c/turtle/turtle-eval-lists-01.nt
	deleted:    test/data/suites/w3c/turtle/turtle-eval-lists-01.ttl
	deleted:    test/data/suites/w3c/turtle/turtle-eval-lists-02.nt
	deleted:    test/data/suites/w3c/turtle/turtle-eval-lists-02.ttl
	deleted:    test/data/suites/w3c/turtle/turtle-eval-lists-03.nt
	deleted:    test/data/suites/w3c/turtle/turtle-eval-lists-03.ttl
	deleted:    test/data/suites/w3c/turtle/turtle-eval-lists-04.nt
	deleted:    test/data/suites/w3c/turtle/turtle-eval-lists-04.ttl
	deleted:    test/data/suites/w3c/turtle/turtle-eval-lists-05.nt
	deleted:    test/data/suites/w3c/turtle/turtle-eval-lists-05.ttl
	modified:   test/data/suites/w3c/turtle/turtle-subm-01.nt
	modified:   test/data/suites/w3c/turtle/turtle-subm-27.nt
	modified:   test/data/suites/w3c/turtle/turtle-subm-27.ttl
	modified:   test/data/suites/w3c/turtle/turtle-syntax-bad-n3-extras-06.ttl
	deleted:    test/data/suites/w3c/turtle/turtle-syntax-blank-label.nt
	deleted:    test/data/suites/w3c/turtle/turtle-syntax-ln-colons.nt
	deleted:    test/data/suites/w3c/turtle/turtle-syntax-ln-dots.nt
	deleted:    test/data/suites/w3c/turtle/turtle-syntax-pname-dots.ttl

Untracked files:
  (use "git add <file>..." to include in what will be committed)
	test/data/suites/w3c/turtle/turtle-syntax-lists-01.ttl
	test/data/suites/w3c/turtle/turtle-syntax-lists-02.ttl
	test/data/suites/w3c/turtle/turtle-syntax-lists-03.ttl
	test/data/suites/w3c/turtle/turtle-syntax-lists-04.ttl
	test/data/suites/w3c/turtle/turtle-syntax-lists-05.ttl

no changes added to commit (use "git add" and/or "git commit -a")

aucampia · 2022-05-07T19:08:34Z

Also, if I run the fetcher the following files change - not sure why, I think maybe because you are using the test suites from github, if so we should correct the URLs but maybe it is best to consider updating the test suites independently also.

I'm guessing it is because of:

Replaced the elderly version of the Turtle test suite included in the W3C RDF test suite (as turtle) with the updated version of the same tests in the N3 test suite TurtleTests subfolder, fixing the incorrect reference in manifest.ttl as necessary to get it to work.

I would rather not do that, if w3c wants to release a new test suite it should be published here IMO: https://www.w3.org/TR/rdf11-testcases/ - and if we have to modify tests then the test suite is anyway problematic.

ghost · 2022-05-07T19:37:04Z

Hack it about as you will, I'll follow closely and review promptly.

The turtle directory in W3C's RDF test suite is 4 years old and is a frozen dupe of TurtleTests in the W3C N3 suite. Otherwise, I was aiming for replicating a checkout of the W3C RDF suite and just ignoring the turtle subdirectory in favour of the current one in the N3 test suite.

The "issue" is confined to manifest files, https://github.com/w3c/N3/blob/c44d123c5958ca04117e28ca3769e2c0820f72e6/tests/manifest.ttl#L13 is incorrect, there is no manifest,ttl in N3Tests so the RDFLib test suite execution produces an Exception --- however, there is a manifest-syntax.ttl present, so that's what I updated it to use. (I would imagine that Gregg Kellogg would appreciate a PR).

aucampia · 2022-05-07T20:32:13Z

The turtle directory in W3C's RDF test suite is 4 years old and is a frozen dupe of TurtleTests in the W3C N3 suite.

But it the official test suite, it is even the same as in https://github.com/w3c/rdf-tests/tree/main/ntriples I believe. It may be frozen but so is the spec, and without some official statement accompanying it's release I don't think we should change it. More tests may be fine but if we want to report compliance with the ntriples test suite the suite we should be reporting compliance to is the official test suite.

ghost · 2022-05-07T22:26:26Z

But it the official test suite

I don't have any kind of agenda here, I just discovered that test/data/suites/w3c/n3 was completely unused by RDFLib tests and it isn't part of the current W3C RDF test suite so I went looking for an "official" W3C N3 test suite and found https://github.com/w3c/N3/tree/master/tests and recruited the NTriples tests.

it is even the same as in https://github.com/w3c/rdf-tests/tree/main/ntriples I believe.

Apart from the turtle tests, the W3C N3 test suite contents don't seem to have any other correlates in the W3C RDF test suite.

It may be frozen but so is the spec, and without some official statement accompanying it's release I don't think we should change it.

Sure, I have no problem with that stance.

I suspect the difference between the fetcher results and the current RDFLib w3c content is due to the fact that the RDFLib w3c hasn't been updated since 2015.

ghost · 2022-05-08T10:00:07Z

Just to summarise:

Move test/data/suites/DAWG/rdflib up to test/data/suites/rdflib-sparql and adjust the reference in test_dawg.py accordingly.
Delete test/data/suites/DAWG/
Pull sparql11 into test/data/suites/w3c, adjust reference in test_dawg.py
Rename test/data/suites/w3c/rdfxml to test/data/suites/w3c/rdf-xml, in prep for refurbishing RDFLib's copy of the W3C RDF test suite, adjust test/data/suites/DAWG/data-r2 and test/data/suites/DAWG/data-sparql11 references in test_dawg.py accordingly.
Ohhh, _now I see ... I hadn't grokked that you'd only just added test/data/suites/w3c/n3 --- 'cos I couldn't find anything that actually tested it, I had it confused with the old n3 directory (which was renamed to n3roundtrip) and that's why I made the error of replacing your d/l of the W3C N3 test suite with my d/l of the same files. Anyway, you might want to keep the n3 test I added as test_n3_w3c.py, although the main test manifest is still borked as discussed.

aucampia · 2022-05-08T15:34:56Z

Will start splitting off things from here either today or later in W19.

And yes, the n3 test suite data was only added recently in #1860 because I'm using it in #1858 (which still needs 1 review) and I did not want to make the fix PR too big to keep it easy to review.

aucampia · 2022-05-10T21:45:42Z

Will merge and resolve tomorrow - also hoping for this to merge w3c/N3#99.

ghost · 2022-05-10T21:56:38Z

Will merge and resolve tomorrow - also hoping for this to merge w3c/N3#99.

Worth waiting for, Gregg Kellogg's clearly on the case.

aucampia · 2022-05-11T21:43:40Z

Still busy slicing this up, in this PR you moved test/data/suites/DAWG/data-r2 to test/data/suites/w3c/sparql11/data-r2 - data-r2 is however the test suite for SPARQL 1.0, so I don't think it should be under sparql11, I'm going to instead rather use the name test/data/suites/w3c/dawg-data-r2

aucampia · 2022-05-11T22:23:38Z

@gjhiggins I split off some more PRs but there is still some work left here, will likely only finish this next week as I have other priorities also.

This is now in `test/data/suites/w3c/data-r2/`: - #1908

We should be testing with unaltered official test data, issues with the test data should be reported upstream and accomodations for issues should be made in code.

aucampia · 2022-05-12T15:27:37Z

I restored the turtle test suite to match the upstream version now, basically what is left is:

Switch from test/data/suites/DAWG/data-r2 to test/data/suites/w3c/dawg-data-r2
Replace test/skiptests.list with test/skipped_sparql_tests.py
Move test/data/suites/DAWG/rdflib to test/data/rdflib-sparql/
... (some more stuff I don't want to write down now)

aucampia · 2022-05-13T19:40:29Z

Two more PRs split off:

After these all that is left is removing test/data/suites/DAWG/data-r2 (the old pre-release DAWG test suite for SPARQL 1.0)

aucampia · 2022-05-14T12:26:22Z

I will try finish this today still, these are all very useful changes @gjhiggins - so thanks you, hard to deal with them in one PR but still good they are done.

ghost · 2022-05-14T13:40:35Z

hard to deal with them in one PR but still good they are done.

/me takes note ffr

This is not defined by DAWG and not used.

This file is now colocated with the test file

aucampia · 2022-05-14T17:51:48Z

Final two PRs split off from this:

After that this is essentially closed.

aucampia

Only a one line fix to prefix left, all the rest are merged as separate PRs.

Graham Higgins added 4 commits May 7, 2022 16:03

continuing re-org of w3c tests

9a8b5db

mark nt-specific fails as such

de68ea6

Merge remote-tracking branch 'origin/master' into w3-test-reorg

8a7f1bf

update to use new test utils structure

12de6a8

use pubiished SPARQL test suite data

2018835

ghost self-requested a review May 7, 2022 17:55

aucampia mentioned this pull request May 10, 2022

Added functionalities for sparql queries in the form "WITH <graph> DELETE/INSERT DATA { . . . }" #1903

Closed

3 tasks

aucampia self-assigned this May 10, 2022

aucampia mentioned this pull request May 10, 2022

Migrated the various test_*_w3c.py test files into test/test_w3c_spec/ #1904

Merged

3 tasks

Merge remote-tracking branch 'upstream/master' into w3-test-reorg

73df402

aucampia mentioned this pull request May 11, 2022

Add n3 test suite runner #1906

Merged

3 tasks

This was referenced May 11, 2022

Move DAWG/data-sparql11 to w3c/sparql11/data-sparql11 #1907

Merged

Issue 166 in SPARQLWrapper solved #1900

Closed

Rename test/data/suites/DAWG/data-r2-1.0 #1908

Merged

aucampia added 3 commits May 12, 2022 17:14

Merge remote-tracking branch 'upstream/master' into w3-test-reorg

5a9818e

Remove test/data/suites/w3c/sparql11/data-r2/

6344aa2

This is now in `test/data/suites/w3c/data-r2/`: - #1908

Reverted changes to N3 and Turtle test suites

374c65c

We should be testing with unaltered official test data, issues with the test data should be reported upstream and accomodations for issues should be made in code.

fix dawg-tests-r2 path

19fb3e1

This was referenced May 13, 2022

Move test/data/suites/DAWG/rdflib #1924

Merged

Switch to latest DAWG test suite #1925

Merged

aucampia added 3 commits May 14, 2022 19:35

Merge remote-tracking branch 'upstream/master' into w3-test-reorg

ace54e7

Remove reference to DAWG.Proposed

8403d68

This is not defined by DAWG and not used.

Removed skipped_sparql_tests

55794a3

This file is now colocated with the test file

This was referenced May 14, 2022

Added SPARQL unicode numeric codepoint escape tests #1929

Merged

Removed old and unused test/data/suites/DAWG/data-r2 #1930

Merged

aucampia added 2 commits May 14, 2022 20:07

Fix URI

68a116a

Merge branch 'master' into w3-test-reorg

766ccc3

aucampia approved these changes May 14, 2022

View reviewed changes

aucampia requested a review from a team May 14, 2022 21:53

aucampia added the review wanted This indicates that the PR is ready for review label May 14, 2022

aucampia merged commit aa6cde3 into RDFLib:master May 14, 2022

ghost deleted the w3-test-reorg branch May 15, 2022 10:53

ghost removed the review wanted This indicates that the PR is ready for review label Jun 4, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

W3 test reorg #1891

W3 test reorg #1891

ghost commented May 7, 2022 •

edited by ghost

coveralls commented May 7, 2022 •

edited

aucampia commented May 7, 2022

aucampia commented May 7, 2022

ghost commented May 7, 2022

aucampia commented May 7, 2022

ghost commented May 7, 2022

ghost commented May 8, 2022

aucampia commented May 8, 2022

aucampia commented May 10, 2022

ghost commented May 10, 2022

aucampia commented May 11, 2022

aucampia commented May 11, 2022

aucampia commented May 12, 2022

aucampia commented May 13, 2022 •

edited

aucampia commented May 14, 2022

ghost commented May 14, 2022

aucampia commented May 14, 2022 •

edited

aucampia left a comment

W3 test reorg #1891

W3 test reorg #1891

Conversation

ghost commented May 7, 2022 • edited by ghost

Summary of changes

Checklist

coveralls commented May 7, 2022 • edited

aucampia commented May 7, 2022

aucampia commented May 7, 2022

ghost commented May 7, 2022

aucampia commented May 7, 2022

ghost commented May 7, 2022

ghost commented May 8, 2022

aucampia commented May 8, 2022

aucampia commented May 10, 2022

ghost commented May 10, 2022

aucampia commented May 11, 2022

aucampia commented May 11, 2022

aucampia commented May 12, 2022

aucampia commented May 13, 2022 • edited

aucampia commented May 14, 2022

ghost commented May 14, 2022

aucampia commented May 14, 2022 • edited

aucampia left a comment

Choose a reason for hiding this comment

ghost commented May 7, 2022 •

edited by ghost

coveralls commented May 7, 2022 •

edited

aucampia commented May 13, 2022 •

edited

aucampia commented May 14, 2022 •

edited