Added SPARQL 1.1 compliance tests (incomplete)

[k00ni]

What do you do, if you have a cold and must stay at home? You write tests for ARC2!

W3C test suite

There is a test suite for SPARQL 1.1 compliance provided by the W3C here: https://www.w3.org/2009/sparql/docs/tests/. It contains many tests for different types of queries. They range from simple SELECT queries over function-usage to graph management.

Goal and motivation

I want to know the current state of SPARQL 1.0/1.1 support of ARC2. Instead of using custom queries, i decided to use the official test suite. Their tests showed that ARC2 lacks major parts of SPARQL 1.0/1.1. Please do not understand that as blaming, but as a first step to define a basement for further development. With these tests, further changes on the query engine can be made without worrying about adding bugs or change expected behavior.

Tested ARC2 component

These new tests check the result from ARC2_Store::query for a given query.

Coverage

I picked mostly cases, which cover major functionality. In the following related folders from the test suite:

• aggregates
• construct
• delete
• drop
• exists
• grouping
• move
• syntax-update-1

Additionally, there a some cases which have exotic kinds of configuration (manifest.ttl). A few of those are also part of the list.

7 folders are covered, 22 folders are not covered yet.

Important: How to interpret the new tests

Don't get fooled, there are 2 different kinds of the tests:

• First kind of tests actually test nothing, because i manually checked if the functionality is available. That is the case for instance for the MOVE command. Based on this wiki page. Nevertheless, i ran some tests, but they approved the wiki page information. I marked those tests as skipped on purpose, to signal lacks in the SPARQL support. If all these would be successful, one could get the impression, ARC2 has no lacks in SPARQL support.
• The other kind of tests check the functionality. They setup the test environment based on a given manifest.ttl, load data etc. and run a W3C-provided query. A test is approved, if the expected result (given by the W3C) matches the actual result (returned from ARC2).

Every skipped test provides meaningful information. For instance Tests\sparql11\AggregatesTest::test_agg_avg_01 outputs:

Test skipped, because of rounding bug in AVG function. 
See https://github.com/semsol/arc2/issues/99

This way one can see the limitations of ARC2, sometimes also the reason. If one decides to extend/fix ARC2 SPARQL support, they can take the related test and change it.

Current output in PHPUnit

New PHPUnit output below. To see a full overview, you can look here.

.SS.SSSSSSS.SSS..S..SS.S....S.....SSSSSSSSSSSSSSSSSSSSSSS.SSSSS  63 / 171 ( 36%)
..SS...SSSSSSS..SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS......... 126 / 171 ( 73%)
.S.S.........................................                   171 / 171 (100%)


Time: 32 seconds, Memory: 12.00MB

There were 94 skipped tests:

1) Tests\sparql11\AggregatesTest::test_agg_avg_01

Test skipped, because of rounding bug in AVG function. 
See https://github.com/semsol/arc2/issues/99
/home/travis/build/k00ni/arc2/tests/sparql11/AggregatesTest.php:29
/home/travis/build/k00ni/arc2/vendor/phpunit/phpunit/phpunit:52


2) Tests\sparql11\AggregatesTest::test_agg_avg_02

This kind of query is currently not supported. ARC2_Store::query returns 0 for query: 
PREFIX : <http://www.example.org/>
SELECT ?s (AVG(?o) AS ?avg)
FROM <http://arc/data/> WHERE { ?s ?p ?o
}
GROUP BY ?s
HAVING (AVG(?o) <= 2.0)

/home/travis/build/k00ni/arc2/tests/sparql11/AggregatesTest.php:41
/home/travis/build/k00ni/arc2/vendor/phpunit/phpunit/phpunit:52

[...]

Further changes

Besides these new files in tests/sparql11, i changed the following files:

• .travis.yml - added PHP 7.2 as further backend
• composer.json - Changed some require-dev entries to fix failing tests in Travis.
• phpunit.xml - Separated test suites, in case, one wants to run tests without compliance tests.
• tests/ARC2_TestCase.php - Small refinements and further comments.
• tests/bootstrap.php - Explicit force error_reporting(E_ALL)

Related issues

During this process, i opened the following issues:

• ARC2_Store::query: function AVG fails sometimes, because it is rounding up #99
• ARC2_Store::query: function SUM fails sometimes, because it is rounding #100

[k00ni]

Ping @semsol :)

[semsol]

Cheers :-)