Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: rework SPARQL test suite #1988

Merged

Conversation

aucampia
Copy link
Member

@aucampia aucampia commented Jun 5, 2022

Summary of changes

This patch reworks the SPARQL test suite to use the new DAWG manifest
processor. The actual checker is also more accurate, correctly reporting
negative tests and more correctly checking the results of positive
tests.

Some of the tests that were not executed before are still
being skipped, but they will now correctly report as skipped.

Other changes:

  • Fixed test/data/suites/rdflib/sparql/unicode.srx. This file did not
    have the correct content for the query. I verified this with Apache Jena.

Checklist

  • Checked that there aren't other open pull requests for
    the same change.
  • Checked that all tests and type checking passes.
  • Considered granting push permissions to the PR branch,
    so maintainers can fix minor issues and keep your PR up to date.

@aucampia aucampia force-pushed the iwana-20220605T1943-rework_sparql_test branch from 2aef2ea to 3267043 Compare June 5, 2022 22:54
@coveralls
Copy link

coveralls commented Jun 5, 2022

Coverage Status

Coverage increased (+0.03%) to 88.555% when pulling c6a0a09 on aucampia:iwana-20220605T1943-rework_sparql_test into 60ff2fd on RDFLib:master.

@aucampia
Copy link
Member Author

aucampia commented Jun 5, 2022

Check that EARL report matches pytest report:

$ task test -- test/test_w3c_spec/test_sparql11_w3c.py --tb=native
task: [test]  .venv/bin/python -m pytest  test/test_w3c_spec/test_sparql11_w3c.py '--tb=native'
============================================================================ test session starts ============================================================================
platform linux -- Python 3.10.4, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/iwana/sw/d/github.com/iafork/rdflib, configfile: pyproject.toml
plugins: subtests-0.7.0, cov-3.0.0
collected 547 items                                                                                                                                                         

test/test_w3c_spec/test_sparql11_w3c.py ...............x.xxxx............x...................................................x.x..................................... [ 22%]
........x...x....x...xxxxxxx.xxx.....xxxxx...........xx......xxxx..xxxxxxxxxxxxxxxx.....x.x..............................................................xx.......... [ 53%]
.........................................xsssssss......................................................xxx.............xxx..x........................................ [ 83%]
..........................xx.....xxx.x.................sssssssssssssssssssssssssssssssssssss                                                                          [100%]

============================================================================= warnings summary ==============================================================================
test/test_w3c_spec/test_sparql11_w3c.py: 50 warnings
  /home/iwana/sw/d/github.com/iafork/rdflib/rdflib/term.py:1139: DeprecationWarning: NotImplemented should not be used in a boolean context
    return not self.__gt__(other) and not self.eq(other)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
========================================================= 437 passed, 44 skipped, 66 xfailed, 50 warnings in 7.43s ==========================================================
$ sparql --results=CSV \
> $(find test/data/suites/w3c/sparql11/ -name 'manifest*' -printf '--data %p ') --data test_reports/rdflib_w3c_sparql11-latest.ttl \
> --query <(echo '
> PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
> PREFIX mf: <http://www.w3.org/2001/sw/DataAccess/tests/test-manifest#>
> PREFIX earl: <http://www.w3.org/ns/earl#>
> SELECT DISTINCT ?entry ?outcome WHERE {
>   ?manifest rdf:type mf:Manifest.
>   ?manifest mf:entries ?entries.
>   ?entries rdf:rest*/rdf:first ?entry.
>   OPTIONAL {
>     ?assertion earl:test ?entry.
>     ?assertion earl:result / earl:outcome ?outcome.
>   }
> }') \
> | sed 1d | gawk -F, '{ print $2 }' | sort | uniq -c
     66 http://www.w3.org/ns/earl#failed
    437 http://www.w3.org/ns/earl#passed
     44 http://www.w3.org/ns/earl#untested

@aucampia aucampia force-pushed the iwana-20220605T1943-rework_sparql_test branch 2 times, most recently from e7b4dac to 87e4578 Compare June 5, 2022 23:09
This patch reworks the SPARQL test suite to use the new DAWG manifest
processor. The actual checker is also more accurate, correctly reporting
negative tests and more correctly checking the resutls of positive
tests.

There are still some tests from the SPARQL test suite that are not executed,
but checkers for these will be implemented later.

Other changes:
- Fixed `test/data/suites/rdflib/sparql/unicode.srx`. This file did not
  have the correct content for the query. I verified this with Apache Jena.
@aucampia aucampia force-pushed the iwana-20220605T1943-rework_sparql_test branch from 87e4578 to c6a0a09 Compare June 5, 2022 23:10
@aucampia aucampia marked this pull request as ready for review June 5, 2022 23:12
@aucampia aucampia requested a review from a team June 5, 2022 23:28
@aucampia aucampia added the review wanted This indicates that the PR is ready for review label Jun 5, 2022
@aucampia aucampia changed the title Rework SPARQL test suite test: rework SPARQL test suite Jun 5, 2022
Copy link

@ghost ghost left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rockin'

@aucampia aucampia merged commit d263009 into RDFLib:master Jun 8, 2022
@@ -0,0 +1,128 @@
"""
Runs the SPARQL 1.0 test suite from.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@aucampia - whoops, looks like a citation was truncated.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AFAICT, the relevant (resolvable) URL is https://www.w3.org/2001/sw/DataAccess/tests/data-r2

@aucampia aucampia deleted the iwana-20220605T1943-rework_sparql_test branch June 23, 2022 19:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
review wanted This indicates that the PR is ready for review
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants