Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SMW 2.4 with Blazegraph SparqlStore #1583

Closed
lloonngg opened this issue May 14, 2016 · 27 comments

Comments

Projects
None yet
3 participants
@lloonngg
Copy link

commented May 14, 2016

Setup and configuration

  • SMW version: 2.4-alpha (43acf66) 04:19, 29 April 2016
  • MW version: 1.26.2
  • DB (MySQL etc.): 10.0.20-MariaDB-1~trusty-log
  • Blazegraph: 2.1.0

Issue

I'm testing out the latest Blazegraph 2.1.0 (running on Nano Sparql Server) as a new SparqlStore for SMW 2.4. I put into LocalSettings the following SMW settings; by the way there is a typo in the $smwgSparqlQueryEndpoint url; stray " character). I changed 'localhost:9999' to the ip where my Blazegraph service is running (http://192.168.1.21:9999) and kept everything else as is.

$smwgDefaultStore = 'SMWSparqlStore';
$smwgSparqlDatabaseConnector = 'blazegraph';
$smwgSparqlQueryEndpoint = 'http://localhost:9999/bigdata/namespace/kb/sparql';
$smwgSparqlUpdateEndpoint = 'http://localhost:9999/bigdata/namespace/kb/sparql';
$smwgSparqlDataEndpoint = '';

When I go to Special:SMWAdmin to Initialize/Upgrade Tables, the output shows "Selected storage engine is "SMWSQLStore3" (or an extension thereof)" rather than blazegraph as the storage engine. The old SMW tables in MariaDB shows up as already existing.

When I go back to Special:SMWAdmin to check the configuration settings, it correctly shows:

"smwgDefaultStore": "SMWSparqlStore",
"smwgSparqlDatabaseConnector": "blazegraph",
"smwgSparqlDatabase": "SMWSparqlDatabase",
"smwgSparqlQueryEndpoint": "http://192.168.1.21:9999/bigdata/namespace/kb/sparql",
"smwgSparqlUpdateEndpoint": "http://192.168.1.21:9999/bigdata/namespace/kb/sparql",
"smwgSparqlDataEndpoint": "",
"smwgSparqlDefaultGraph": "",

update.php and runjobs run just fine to update the templates, pages, properties, etc. But after refreshing the SMW data, I cannot see the SMW data which I was able to see before when the storage engine was indeed SMWSQLStore3.

Is there some other config in LocalSettings required to point SMW to the Blazegraph sparql store?

Thanks,

Long

@mwjames

This comment has been minimized.

Copy link
Contributor

commented May 15, 2016

I'm testing out the latest Blazegraph 2.1.0 (running on Nano Sparql Server) as a new SparqlStore for SMW 2.4.

Our Travis environment runs Blazegraph 1.5.2 with:

and passes our test suite therefore I know Blazegraph works [0]. 2.1 shouldn't cause any issues but I'm a bit too buys to make a switch to 2.1 for the test setup.

update.php and runjobs run just fine to update the templates, pages, properties, etc. But after refreshing the SMW data,
Is there some other config in LocalSettings required to point SMW to the Blazegraph sparql store?

I'm guessing you did also run the rebuidlData.phpscript or in case of Special:SMWadmin the data repair since "update.php and runjobs" alone don't refresh the data (== sending the RDF export to the endpoint).

image

For a general check-up on your installation you can simply run composer phpunit[1] from the cmd line via the SMW root directory where you should get a similar output as in [0].

[0] https://s3.amazonaws.com/archive.travis-ci.org/jobs/129857107/log.txt
[1] https://github.com/SemanticMediaWiki/SemanticMediaWiki/tree/master/tests#running-tests

@mwjames

This comment has been minimized.

Copy link
Contributor

commented May 22, 2016

@lloonngg Any update on this issue?

@lloonngg

This comment has been minimized.

Copy link
Author

commented May 23, 2016

Sorry, I was tied up recently. I tried composer phpunit and found quite a few failures with my installation. I reverted to sqlstore3 and still got errors. I'm planning to rebuild my installation and using the smw unit testing as I go along. Any reason to think that the errors are due to MW 1.26.2?

@mwjames

This comment has been minimized.

Copy link
Contributor

commented May 23, 2016

I tried composer phpunit and found quite a few failures with my installation. I reverted to sqlstore3 and still got errors.

SQLStore or SPARQLStore are interchangeable except for the fact that data have to be rebuild, functionality and interface wise it is not expected that a user can tell any difference (the same goes for running the tests with 10 or so are specifically targeted at the SPARQLStore but all others are universal) .

As for the errors, maybe you can put them in a pastebin (or gist) and link it to this issue.

Any reason to think that the errors are due to MW 1.26.2?

No. Some Travis jobs run with 1.26.3 others with 1.24* and we still have a job that tests against 1.19*.

PS: When I said "I know Blazegraph works" with a persistence on unit (integration) tests then it is because we invested quite amount of time to provide 3600+ tests and have Travis (our CI environment provider) constantly run against the major triple-stores.

@lloonngg

This comment has been minimized.

Copy link
Author

commented Jun 2, 2016

I created a new stack with the following:

MediaWiki 1.27.0-rc.0
HHVM 3.13.1 (srv)
MariaDB 10.1.14-MariaDB-1~trusty
ICU 52.1
SMW 2.4-alpha (20a7082)

Just testing out sqlstore3 first with phpunit testing, I got the following output and error. I also tried MW 1.26.3 and 1.25.6 with the same SMW 2.4alpha extension and get similar errors during phpunit testing. Am I missing a dependency? Only SMW was installed as an extension with this clean build.

php ../../tests/phpunit/phpunit.php -c phpunit.xml.dist
Using HHVM 3.13.1 (5.6.99-hhvm)

Semantic MediaWiki: 2.4-alpha (SMWSQLStore3, mysql)
MediaWiki: 1.27.0-rc.0 (MediaWiki vendor autoloader)
Site language: en

Execution time: 2016-06-02 02:25
Xdebug: Disabled (or not installed)

PHPUnit 4.8.24 by Sebastian Bergmann and contributors.

Runtime: HHVM 3.13.1
Configuration: /var/www/html/mediawiki2/extensions/SemanticMediaWiki/phpunit.xml.dist

............................................................. 61 / 3661 ( 1%)
............................................................. 122 / 3661 ( 3%)
............................................................. 183 / 3661 ( 4%)
............................................................. 244 / 3661 ( 6%)
............................................................. 305 / 3661 ( 8%)
............................................................. 366 / 3661 ( 9%)
............................................................. 427 / 3661 ( 11%)
............................................................. 488 / 3661 ( 13%)
............................................................. 549 / 3661 ( 14%)
............................................................. 610 / 3661 ( 16%)
............................................................. 671 / 3661 ( 18%)
............................................................. 732 / 3661 ( 19%)
............................................................. 793 / 3661 ( 21%)
............................................................. 854 / 3661 ( 23%)
............................................................. 915 / 3661 ( 24%)
............................................................. 976 / 3661 ( 26%)
............................................................. 1037 / 3661 ( 28%)
............................................................. 1098 / 3661 ( 29%)
............................................................. 1159 / 3661 ( 31%)
............................................................. 1220 / 3661 ( 33%)
............................................................. 1281 / 3661 ( 34%)
............................................................. 1342 / 3661 ( 36%)
............................................................. 1403 / 3661 ( 38%)
............................................................. 1464 / 3661 ( 39%)
............................................................. 1525 / 3661 ( 41%)
............................................................. 1586 / 3661 ( 43%)
............FF............................................... 1647 / 3661 ( 44%)
............................................................. 1708 / 3661 ( 46%)
............................................................. 1769 / 3661 ( 48%)
............................................................. 1830 / 3661 ( 49%)
............................................................. 1891 / 3661 ( 51%)
............................................................. 1952 / 3661 ( 53%)
............................................................. 2013 / 3661 ( 54%)
............................................................. 2074 / 3661 ( 56%)
............................................................. 2135 / 3661 ( 58%)
............................................................. 2196 / 3661 ( 59%)
............................................................. 2257 / 3661 ( 61%)
............................................................. 2318 / 3661 ( 63%)
............................................................. 2379 / 3661 ( 64%)
............................................................. 2440 / 3661 ( 66%)
............................................................. 2501 / 3661 ( 68%)
............................................................. 2562 / 3661 ( 69%)
............................................................. 2623 / 3661 ( 71%)
............................................................. 2684 / 3661 ( 73%)
............................................................. 2745 / 3661 ( 74%)
............................................................. 2806 / 3661 ( 76%)
............................................................. 2867 / 3661 ( 78%)
............................................................. 2928 / 3661 ( 79%)
............................................................. 2989 / 3661 ( 81%)
.......................................................
Fatal error: Call to undefined method MediaWiki\MediaWikiServices::resetServiceForTesting() in /var/www/html/mediawiki2/extensions/SemanticMediaWiki/tests/phpunit/TestEnvironment.php on line 102
Script php ../../tests/phpunit/phpunit.php -c phpunit.xml.dist handling the phpunit event returned with an error

[RuntimeException]
Error Output:
Fatal error: Call to undefined method MediaWiki\MediaWikiServices::resetServiceForTesting() in /var/www/html/mediawiki2/e
xtensions/SemanticMediaWiki/tests/phpunit/TestEnvironment.php on line 102

@mwjames

This comment has been minimized.

Copy link
Contributor

commented Jun 2, 2016

FF............................................... 1647 / 3661 ( 44%)

Those two failures are caused by the #1456 HHVM bug.

Fatal error: Call to undefined method MediaWiki\MediaWikiServices::resetServiceForTesting() in /var/www/html/mediawiki2/extensions/SemanticMediaWiki/tests/phpunit/TestEnvironment.php on line 102

A change was necessary due to #1615 but it turned out that not all of the 1.28.* code has access to this method therefore required 9e3f9a8, yet 1.27.0-rc.0 behaves different again which means (argggg....) it requires e6b2c50.

Please pull a new version and try again.

@lloonngg

This comment has been minimized.

Copy link
Author

commented Jun 2, 2016

I'm still getting those two failures with the latest SMW pull.

MediaWiki 1.27.0-rc.0
HHVM 3.13.1 (srv)
MariaDB 10.1.14-MariaDB-1~trusty
ICU 52.1
SMW 2.4-alpha (b56a8fa) 06:46, 2 June 2016

php ../../tests/phpunit/phpunit.php -c phpunit.xml.dist
Using HHVM 3.13.1 (5.6.99-hhvm)

Semantic MediaWiki: 2.4-alpha (SMWSQLStore3, mysql)
MediaWiki: 1.27.0-rc.0 (MediaWiki vendor autoloader)
Site language: en

Execution time: 2016-06-02 11:29
Xdebug: Disabled (or not installed)

PHPUnit 4.8.24 by Sebastian Bergmann and contributors.

Runtime: HHVM 3.13.1
Configuration: /var/www/html/mediawiki2/extensions/SemanticMediaWiki/phpunit.xml.dist

............................................................. 61 / 3661 ( 1%)
............................................................. 122 / 3661 ( 3%)
............................................................. 183 / 3661 ( 4%)
............................................................. 244 / 3661 ( 6%)
............................................................. 305 / 3661 ( 8%)
............................................................. 366 / 3661 ( 9%)
............................................................. 427 / 3661 ( 11%)
............................................................. 488 / 3661 ( 13%)
............................................................. 549 / 3661 ( 14%)
............................................................. 610 / 3661 ( 16%)
............................................................. 671 / 3661 ( 18%)
............................................................. 732 / 3661 ( 19%)
............................................................. 793 / 3661 ( 21%)
............................................................. 854 / 3661 ( 23%)
............................................................. 915 / 3661 ( 24%)
............................................................. 976 / 3661 ( 26%)
............................................................. 1037 / 3661 ( 28%)
............................................................. 1098 / 3661 ( 29%)
............................................................. 1159 / 3661 ( 31%)
............................................................. 1220 / 3661 ( 33%)
............................................................. 1281 / 3661 ( 34%)
............................................................. 1342 / 3661 ( 36%)
............................................................. 1403 / 3661 ( 38%)
............................................................. 1464 / 3661 ( 39%)
............................................................. 1525 / 3661 ( 41%)
............................................................. 1586 / 3661 ( 43%)
............FF............................................... 1647 / 3661 ( 44%)
............................................................. 1708 / 3661 ( 46%)
............................................................. 1769 / 3661 ( 48%)
............................................................. 1830 / 3661 ( 49%)
............................................................. 1891 / 3661 ( 51%)
............................................................. 1952 / 3661 ( 53%)
............................................................. 2013 / 3661 ( 54%)
............................................................. 2074 / 3661 ( 56%)
............................................................. 2135 / 3661 ( 58%)
............................................................. 2196 / 3661 ( 59%)
............................................................. 2257 / 3661 ( 61%)
............................................................. 2318 / 3661 ( 63%)
............................................................. 2379 / 3661 ( 64%)
............................................................. 2440 / 3661 ( 66%)
............................................................. 2501 / 3661 ( 68%)
............................................................. 2562 / 3661 ( 69%)
............................................................. 2623 / 3661 ( 71%)
............................................................. 2684 / 3661 ( 73%)
............................................................. 2745 / 3661 ( 74%)
............................................................. 2806 / 3661 ( 76%)
............................................................. 2867 / 3661 ( 78%)
............................................................. 2928 / 3661 ( 79%)
............................................................. 2989 / 3661 ( 81%)
............................................................. 3050 / 3661 ( 83%)
........
Fatal error: Cannot access protected property SMWQuantityValue::$m_unitin
Script php ../../tests/phpunit/phpunit.php -c phpunit.xml.dist handling the phpunit event returned with an error

[RuntimeException]
Error Output:
Fatal error: Cannot access protected property SMWQuantityValue::$m_unitin

@mwjames

This comment has been minimized.

Copy link
Contributor

commented Jun 2, 2016

Fatal error: Cannot access protected property SMWQuantityValue::$m_unitin

Is also mentioned in #1456 as HHVM bug caused by facebook/hhvm#5128.

Our Travis HHVM [0] uses HHVM 3.6.6 (5.6.99-hhvm) which isn't/wasn't an issue.

[0] https://travis-ci.org/SemanticMediaWiki/SemanticMediaWiki/jobs/134708474

@mwjames

This comment has been minimized.

Copy link
Contributor

commented Jun 2, 2016

Fatal error: Cannot access protected property SMWQuantityValue::$m_unitin

The line HHVM complains about is [0] assert( '$this->m_unitfactors[$this->m_unitin] != 0 /* Should be filtered by initConversionData() */' );

You can try removing ' from the assert -> as in assert( $this->m_unitfactors[$this->m_unitin] != 0 /* Should be filtered by initConversionData() */ );

[0] https://github.com/SemanticMediaWiki/SemanticMediaWiki/blob/master/includes/datavalues/SMW_DV_Quantity.php#L51

@lloonngg

This comment has been minimized.

Copy link
Author

commented Jun 2, 2016

I tried the editing SMW_DV_Quantity.php as you suggested and it gave the same errror and others. I downgraded to HHVM 3.6.6 and still see the failures. The test makes it to the end now and shows a third failure. If the errors are mainly due to testing and not functionality of SMW, is it safe to proceed with Blazegraph for the backend?

php ../../tests/phpunit/phpunit.php -c phpunit.xml.dist
Using HHVM 3.6.6 (5.6.99-hhvm)

Semantic MediaWiki: 2.4-alpha (SMWSQLStore3, mysql)
MediaWiki: 1.27.0-rc.0 (MediaWiki vendor autoloader)
Site language: en

Execution time: 2016-06-02 02:57
Xdebug: Disabled (or not installed)

PHPUnit 4.8.24 by Sebastian Bergmann and contributors.

Runtime: HHVM 3.6.6
Configuration: /var/www/html/mediawiki/extensions/SemanticMediaWiki/phpunit.xml.dist

............................................................. 61 / 3661 ( 1%)
............................................................. 122 / 3661 ( 3%)
............................................................. 183 / 3661 ( 4%)
............................................................. 244 / 3661 ( 6%)
............................................................. 305 / 3661 ( 8%)
............................................................. 366 / 3661 ( 9%)
............................................................. 427 / 3661 ( 11%)
............................................................. 488 / 3661 ( 13%)
............................................................. 549 / 3661 ( 14%)
............................................................. 610 / 3661 ( 16%)
............................................................. 671 / 3661 ( 18%)
............................................................. 732 / 3661 ( 19%)
............................................................. 793 / 3661 ( 21%)
............................................................. 854 / 3661 ( 23%)
............................................................. 915 / 3661 ( 24%)
............................................................. 976 / 3661 ( 26%)
............................................................. 1037 / 3661 ( 28%)
............................................................. 1098 / 3661 ( 29%)
............................................................. 1159 / 3661 ( 31%)
............................................................. 1220 / 3661 ( 33%)
............................................................. 1281 / 3661 ( 34%)
............................................................. 1342 / 3661 ( 36%)
............................................................. 1403 / 3661 ( 38%)
............................................................. 1464 / 3661 ( 39%)
............................................................. 1525 / 3661 ( 41%)
............................................................. 1586 / 3661 ( 43%)
............FF............................................... 1647 / 3661 ( 44%)
............................................................. 1708 / 3661 ( 46%)
............................................................. 1769 / 3661 ( 48%)
............................................................. 1830 / 3661 ( 49%)
............................................................. 1891 / 3661 ( 51%)
............................................................. 1952 / 3661 ( 53%)
............................................................. 2013 / 3661 ( 54%)
............................................................. 2074 / 3661 ( 56%)
............................................................. 2135 / 3661 ( 58%)
............................................................. 2196 / 3661 ( 59%)
............................................................. 2257 / 3661 ( 61%)
............................................................. 2318 / 3661 ( 63%)
............................................................. 2379 / 3661 ( 64%)
............................................................. 2440 / 3661 ( 66%)
............................................................. 2501 / 3661 ( 68%)
............................................................. 2562 / 3661 ( 69%)
............................................................. 2623 / 3661 ( 71%)
............................................................. 2684 / 3661 ( 73%)
............................................................. 2745 / 3661 ( 74%)
............................................................. 2806 / 3661 ( 76%)
............................................................. 2867 / 3661 ( 78%)
............................................................. 2928 / 3661 ( 79%)
............................................................. 2989 / 3661 ( 81%)
............................................................. 3050 / 3661 ( 83%)
............................................................. 3111 / 3661 ( 84%)
.....F....................................................... 3172 / 3661 ( 86%)
...........................................................S. 3233 / 3661 ( 88%)
..............................................SSSSSSS........ 3294 / 3661 ( 89%)
............................................................. 3355 / 3661 ( 91%)
............................................................. 3416 / 3661 ( 93%)
............................................................. 3477 / 3661 ( 94%)
............................................................. 3538 / 3661 ( 96%)
..........................................SSS................ 3599 / 3661 ( 98%)
............................................................. 3660 / 3661 ( 99%)
.

Time: 2.7 minutes, Memory: 323.84MB

There were 3 failures:

  1. SMW\Tests\IntlTimeFormatterTest::testFormat with data set #3 ('2/1300/11/02/12/03/25.888499949', 'en', 'Y-m-d H:i:s.u', '1300-11-02 12:03:25.888500')
    Failed asserting that two strings are equal.
    --- Expected
    +++ Actual
    @@ @@
    -'1300-11-02 12:03:25.888500'
    +'1300-11-02 12:03:25.888499'

/var/www/html/mediawiki/extensions/SemanticMediaWiki/tests/phpunit/Unit/IntlTimeFormatterTest.php:49

  1. SMW\Tests\IntlTimeFormatterTest::testFormat with data set #4 ('2/1300/11/02/12/03/25.888499949', 'en', 'H:i:s.u', '12:03:25.888500')
    Failed asserting that two strings are equal.
    --- Expected
    +++ Actual
    @@ @@
    -'12:03:25.888500'
    +'12:03:25.888499'

/var/www/html/mediawiki/extensions/SemanticMediaWiki/tests/phpunit/Unit/IntlTimeFormatterTest.php:49

  1. SMW\Tests\Integration\ByJsonScript\ByJsonScriptFixtureTestCaseRunnerTest::executeTestCasesFor with data set "p-0414.json" ('/var/www/html/mediawiki/exten...4.json')
    Failed on #13.888499949is being rounded to.888500`` for
     Has dateHas date
    Example/P0414/78 July 2012 11:14:1511:14:15.888499

with (StringContains) [ 11:14:15.888500 ]
Failed asserting that 1 matches expected 2.

/var/www/html/mediawiki/extensions/SemanticMediaWiki/tests/phpunit/Utils/Validators/StringValidator.php:76
/var/www/html/mediawiki/extensions/SemanticMediaWiki/tests/phpunit/Utils/Validators/StringValidator.php:27
/var/www/html/mediawiki/extensions/SemanticMediaWiki/tests/phpunit/Integration/ByJsonScript/ParserTestCaseProcessor.php:127
/var/www/html/mediawiki/extensions/SemanticMediaWiki/tests/phpunit/Integration/ByJsonScript/ParserTestCaseProcessor.php:64
/var/www/html/mediawiki/extensions/SemanticMediaWiki/tests/phpunit/Integration/ByJsonScript/ByJsonScriptFixtureTestCaseRunnerTest.php:206
/var/www/html/mediawiki/extensions/SemanticMediaWiki/tests/phpunit/Integration/ByJsonScript/ByJsonScriptFixtureTestCaseRunnerTest.php:188
/var/www/html/mediawiki/extensions/SemanticMediaWiki/tests/phpunit/ByJsonTestCaseProvider.php:129
/var/www/html/mediawiki/extensions/SemanticMediaWiki/tests/phpunit/MwDBaseUnitTestCase.php:125

There were 11 skipped tests:

  1. SMW\Tests\Integration\MediaWiki\SearchInPageDBIntegrationTest::testSearchForGeographicCoordinateValueAsTerm
    Requires 'Geographic coordinate' to be a supported data type (see Semantic Maps)

/var/www/html/mediawiki/extensions/SemanticMediaWiki/tests/phpunit/Integration/MediaWiki/SearchInPageDBIntegrationTest.php:65
/var/www/html/mediawiki/extensions/SemanticMediaWiki/tests/phpunit/MwDBaseUnitTestCase.php:125

  1. SMW\Tests\Integration\SPARQLStore\QueryResultLookupWithoutBaseStoreIntegrationTest::testQuerySubjects_afterUpdatingSemanticData
    Requires a SPARQLStore instance

/var/www/html/mediawiki/extensions/SemanticMediaWiki/tests/phpunit/Integration/SPARQLStore/QueryResultLookupWithoutBaseStoreIntegrationTest.php:45

  1. SMW\Tests\Integration\SPARQLStore\QueryResultLookupWithoutBaseStoreIntegrationTest::testQueryZeroResults_afterSubjectRemoval
    Requires a SPARQLStore instance

/var/www/html/mediawiki/extensions/SemanticMediaWiki/tests/phpunit/Integration/SPARQLStore/QueryResultLookupWithoutBaseStoreIntegrationTest.php:45

  1. SMW\Tests\Integration\SPARQLStore\QueryResultLookupWithoutBaseStoreIntegrationTest::testQuerySubjects_onNamspaceRestrictedCondition
    Requires a SPARQLStore instance

/var/www/html/mediawiki/extensions/SemanticMediaWiki/tests/phpunit/Integration/SPARQLStore/QueryResultLookupWithoutBaseStoreIntegrationTest.php:45

  1. SMW\Tests\Integration\SPARQLStore\QueryResultLookupWithoutBaseStoreIntegrationTest::testQuerySubobjects_afterUpdatingWithEmptyContainerAllAssociatedEntitiesGetRemovedFromGraph
    Requires a SPARQLStore instance

/var/www/html/mediawiki/extensions/SemanticMediaWiki/tests/phpunit/Integration/SPARQLStore/QueryResultLookupWithoutBaseStoreIntegrationTest.php:45

  1. SMW\Tests\Integration\SPARQLStore\RedirectLookupIntegrationTest::testRedirectTragetLookupForNonExistingEntry with data set #0 (SMW\Exporter\Element\ExpNsResource Object (...))
    Requires a SPARQLStore instance

/var/www/html/mediawiki/extensions/SemanticMediaWiki/tests/phpunit/Integration/SPARQLStore/RedirectLookupIntegrationTest.php:37

  1. SMW\Tests\Integration\SPARQLStore\RedirectLookupIntegrationTest::testRedirectTragetLookupForNonExistingEntry with data set #1 (SMW\Exporter\Element\ExpNsResource Object (...))
    Requires a SPARQLStore instance

/var/www/html/mediawiki/extensions/SemanticMediaWiki/tests/phpunit/Integration/SPARQLStore/RedirectLookupIntegrationTest.php:37

  1. SMW\Tests\Integration\SPARQLStore\RedirectLookupIntegrationTest::testRedirectTragetLookupForExistingEntry
    Requires a SPARQLStore instance

/var/www/html/mediawiki/extensions/SemanticMediaWiki/tests/phpunit/Integration/SPARQLStore/RedirectLookupIntegrationTest.php:37

  1. SMW\Tests\System\InstallationGlobalsProviderIntegrityTest::testNamespaceSettingOnExampleIfSet
    Skip test due to missing http://example.org/id/ setting

/var/www/html/mediawiki/extensions/SemanticMediaWiki/tests/phpunit/Integration/System/InstallationConfigurationIntegrityTest.php:60

  1. SMW\Tests\System\InstallationGlobalsProviderIntegrityTest::testNamespacesWithSemanticLinksOnTravisCustomNamespace with data set #0 ('GLOBALS', array(true, false, true, false, true, false, true, false, true, false, true, false, true, false, false, false, false, false, true, false, true, false))
    Test can only be executed with a specified NS_TRAVIS

/var/www/html/mediawiki/extensions/SemanticMediaWiki/tests/phpunit/Integration/System/InstallationConfigurationIntegrityTest.php:75

  1. SMW\Tests\System\InstallationGlobalsProviderIntegrityTest::testNamespacesWithSemanticLinksOnTravisCustomNamespace with data set #1 ('Settings', array(true, false, true, false, true, false, true, false, true, false, true, false, true, false, false, false, false, false, true, false, true, false))
    Test can only be executed with a specified NS_TRAVIS

/var/www/html/mediawiki/extensions/SemanticMediaWiki/tests/phpunit/Integration/System/InstallationConfigurationIntegrityTest.php:75

FAILURES!
Tests: 3661, Assertions: 11309, Failures: 3, Skipped: 11.
Script php ../../tests/phpunit/phpunit.php -c phpunit.xml.dist handling the phpunit event returned with an error

@mwjames

This comment has been minimized.

Copy link
Contributor

commented Jun 2, 2016

The test makes it to the end now and shows a third failure. If the errors are mainly due to testing and not functionality of SMW, is it safe to proceed with Blazegraph for the backend?

The three errors belong to the same category of HHVM having problems with edge handling of micro seconds as outlined earlier. So, nothing we can do from the SMW side unless of course you drop the test case.

is it safe to proceed with Blazegraph for the backend?

I'd say 👍 Since you encountered three of the expected three, switch the gear and plug-in the Blazegraph and we should see similar results.

@lloonngg

This comment has been minimized.

Copy link
Author

commented Jun 2, 2016

I have Blazegraph 2.1.0 installed and running. When running rebuildData.php under the SMW maintenance folder I get the following error. It is not the same as the one previously reported for curl and sparqlstore with the Chunked-Encoded data error [1]. I have tried changing the from curl version 7.35 to 7.36 or 7.43 without any improvement. Forcing the http version for curl also does not help [2].

Fatal error: Uncaught exception 'SMW\SPARQLStore\Exception\HttpDatabaseConnectionException' with message 'Failed to communicate to Endpoint: "http://localhost:9999/bigdata/namespace/kb/sparql
due to curl error: 1 (Protocol "http not supported or disabled in libcurl).
' in /.../extensions/SemanticMediaWiki/src/SPARQLStore/BadHttpResponseMapper.php:67
Stack trace:
#0 /.../extensions/SemanticMediaWiki/src/SPARQLStore/RepositoryConnector/GenericHttpRepositoryConnector.php(561): SMW\SPARQLStore\BadHttpResponseMapper->mapResponseToHttpRequest()
#1 /.../extensions/SemanticMediaWiki/src/SPARQLStore/RepositoryConnector/GenericHttpRepositoryConnector.php(425): SMW\SPARQLStore\RepositoryConnector\GenericHttpRepositoryConnector->mapHttpRequestError()
#2 /.../extensions/SemanticMediaWiki/src/SPARQLStore/RepositoryConnector/GenericHttpRepositoryConnector.php(143): SMW\SPARQLStore\RepositoryConnector\GenericHttpRepositoryConnector->doQuery()
#3 /.../extensions/SemanticMediaWiki/src/SPARQLStore/RedirectLookup.php(120): SMW\SPARQLStore\RepositoryConnector\GenericHttpRepositoryConnector->select()
#4 /.../extensions/SemanticMediaWiki/src/SPARQLStore/RedirectLookup.php(95): SMW\SPARQLStore\RedirectLookup->lookupResourceUriTargetFromDatabase()
#5 /.../extensions/SemanticMediaWiki/src/SPARQLStore/RedirectLookup.php(73): SMW\SPARQLStore\RedirectLookup->doLookupResourceUriTargetFor()
#6 /.../extensions/SemanticMediaWiki/src/SPARQLStore/TurtleTriplesBuilder.php(277): SMW\SPARQLStore\RedirectLookup->findRedirectTargetResource()
#7 /.../extensions/SemanticMediaWiki/src/SPARQLStore/TurtleTriplesBuilder.php(246): SMW\SPARQLStore\TurtleTriplesBuilder->expandUpdateExpResource()
#8 /.../extensions/SemanticMediaWiki/src/SPARQLStore/TurtleTriplesBuilder.php(337): SMW\SPARQLStore\TurtleTriplesBuilder->expandUpdateExpElement()
#9 /.../extensions/SemanticMediaWiki/src/SPARQLStore/TurtleTriplesBuilder.php(221): SMW\SPARQLStore\TurtleTriplesBuilder->expandUpdateExpData()
#10 /.../extensions/SemanticMediaWiki/src/SPARQLStore/TurtleTriplesBuilder.php(177): SMW\SPARQLStore\TurtleTriplesBuilder->prepareUpdateExpData()
#11 /.../extensions/SemanticMediaWiki/src/SPARQLStore/TurtleTriplesBuilder.php(85): SMW\SPARQLStore\TurtleTriplesBuilder->serializeToTurtleRepresentation()
#12 /.../extensions/SemanticMediaWiki/src/SPARQLStore/TurtleTriplesBuilder.php(156): SMW\SPARQLStore\TurtleTriplesBuilder->doBuild()
#13 /.../extensions/SemanticMediaWiki/src/SPARQLStore/SPARQLStore.php(208): SMW\SPARQLStore\TurtleTriplesBuilder->hasTriplesForUpdate()
#14 /.../extensions/SemanticMediaWiki/src/SPARQLStore/SPARQLStore.php(182): SMW\SPARQLStore\SPARQLStore->doSparqlFlatDataUpdate()
#15 /.../extensions/SemanticMediaWiki/src/SPARQLStore/SPARQLStore.php(230): SMW\SPARQLStore\SPARQLStore->doSparqlDataUpdate()
#16 /.../extensions/SemanticMediaWiki/includes/storage/SMW_Store.php(244): SMW\SPARQLStore\SPARQLStore->doDataUpdate()
#17 /.../extensions/SemanticMediaWiki/includes/storage/SMW_Store.php(263): SMW\Store->updateData()
#18 /.../extensions/SemanticMediaWiki/src/MediaWiki/Jobs/UpdateJob.php(92): SMW\Store->clearData()
#19 /.../extensions/SemanticMediaWiki/src/MediaWiki/Jobs/UpdateJob.php(56): SMW\MediaWiki\Jobs\UpdateJob->doUpdate()
#20 /.../extensions/SemanticMediaWiki/src/SQLStore/ByIdDataRebuildDispatcher.php(191): SMW\MediaWiki\Jobs\UpdateJob->run()
#21 /.../extensions/SemanticMediaWiki/src/Maintenance/DataRebuilder.php(307): SMW\SQLStore\ByIdDataRebuildDispatcher->dispatchRebuildFor()
#22 /.../extensions/SemanticMediaWiki/src/Maintenance/DataRebuilder.php(266): SMW\Maintenance\DataRebuilder->doExecuteFor()
#23 /.../extensions/SemanticMediaWiki/src/Maintenance/DataRebuilder.php(167): SMW\Maintenance\DataRebuilder->doRebuildAll()
#24 /.../extensions/SemanticMediaWiki/maintenance/rebuildData.php(150): SMW\Maintenance\DataRebuilder->rebuild()
#25 /.../maintenance/doMaintenance.php(103): SMW\Maintenance\RebuildData->execute()
#26 /.../extensions/SemanticMediaWiki/maintenance/rebuildData.php(197): include()
#27 {main}
Exception encountered, of type "SMW\SPARQLStore\Exception\HttpDatabaseConnectionException"
@mwjames

This comment has been minimized.

Copy link
Contributor

commented Jun 2, 2016

have tried changing the from curl version 7.35 to 7.36 or 7.43 without any improvement. Forcing the http version for curl also does not help

cUrl needs to work out-of-the box otherwise the handler can't establish a connection [0]. This seems more of a low-level, install, ubuntu issue.

Fatal error: Uncaught exception 'SMW\SPARQLStore\Exception\HttpDatabaseConnectionException' with message 'Failed to communicate to Endpoint: "http://localhost:9999/bigdata/namespace/kb/sparql
due to curl error: 1 (Protocol "http not supported or disabled in libcurl).

This leads me to https://curl.haxx.se/docs/faq.html#Protocol_xxx_not_supported_or_di.

[0] https://www.semantic-mediawiki.org/wiki/Help:SPARQLStore/Exception/HttpDatabaseConnectionException

@mwjames

This comment has been minimized.

Copy link
Contributor

commented Jun 2, 2016

I recognized 2.1.0 changed the url pattern to http://localhost:9999/bigdata according to [0], are we sure namespace/kb/sparql remained the same?

Is the http://localhost:8080/bigdata/status reachable? [1] is referencing to some different urls such ashttp://localhost:8080/bigdata/sparql.

In contrast, 1.5.2 uses http://localhost:9999/bigdata/namespace/kb/sparql as endpoint.

[0] https://wiki.blazegraph.com/wiki/index.php/NanoSparqlServer
[1] https://wiki.blazegraph.com/wiki/index.php/NanoSparqlServer#Active_URLs

@lloonngg

This comment has been minimized.

Copy link
Author

commented Jun 2, 2016

I figured out the problem. The documentation on how to set up the Blazegraph endpoint for sparqlstore has a stray " in the QueryEndpoint url. Removing it and leaving the url as is (http://localhost:9999/bigdata/namespace/kb/sparql) got rid of the libcurl error during rebuildData.

Unfortunately, when I run the phpunit test, there are a ton of failures (see attached). So maybe Blazegraph 2.1.0 may not be compatible out of the box. I'm going to try 1.5.2 to see if performs like sqlstore3 with phpunit testing.

MW1.27rc0_SMW2.4alpha_Blazegraph2.1.0_phpunit_testing.txt

@mwjames

This comment has been minimized.

Copy link
Contributor

commented Jun 3, 2016

Unfortunately, when I run the phpunit test, there are a ton of failures (see attached)

  1. SMW\Tests\System\InstallationGlobalsProviderIntegrityTest::testNamespaceSettingOnExampleIfSet
    Skip test due to missing http://example.org/id/ setting

Did you set something like enableSemantics( 'http://example.org/id/', true ); in your LocalSettings.php file?

@mwjames

This comment has been minimized.

Copy link
Contributor

commented Jun 3, 2016

So maybe Blazegraph 2.1.0 may not be compatible out of the box

.......................................................FF.F.. 3050 / 3661 ( 83%)
.F.FFFFF..F.FFFFF..FFFF..F..F..FFFFFF.F.FFF.F.FFFFFFFFFF..... 3111 / 3661 ( 84%)
F.FFFFF..F..FF.FFF.F...F..FFF.FF.FFFF..F.SF.FF..FFFF.F.FFFF.. 3172 / 3661 ( 86%)
.....F......................F.............................FSF 3233 / 3661 ( 88%)
FF.FFFFFFFFFFFFF.FF...FF..FFFFF...F............FFF.F......... 3294 / 3661 ( 89%)
...............................F......
SMW\Tests\Integration\MediaWiki\Import\Maintenance\RebuildDataMaintenanceTest testRebuildData ran for 10.537 seconds
....................... 3355 / 3661 ( 91%)
............................................................. 3416 / 3661 ( 93%)
............................................................. 3477 / 3661 ( 94%)
............................................................. 3538 / 3661 ( 96%)
..........................................SSS................ 3599 / 3661 ( 98%)
............................................................. 3660 / 3661 ( 99%)

Those F's indicate a failure during the integration tests [0].

The integration tests simulate a "human" input with an active DB/Triple store connection which is compared against an expected output and in case it fails, the data storage and retrieval process was unsuccessful.

[0] https://www.semantic-mediawiki.org/wiki/Help:Integration_tests

@mwjames

This comment has been minimized.

Copy link
Contributor

commented Jun 3, 2016

You did run it with something like java -server -Xmx4g -Dbigdata.propertyFile=$BASE_PATH/tests/travis/blazegraph-store.properties ... [0] ?

In the beginning we had some issues with Blazgraph but the developers helped us [1] therefore it is run with the [0] option.

[0] https://github.com/SemanticMediaWiki/SemanticMediaWiki/blob/master/tests/travis/blazegraph-store.properties
[1] https://sourceforge.net/p/bigdata/mailman/message/34072978/

@lloonngg

This comment has been minimized.

Copy link
Author

commented Jun 3, 2016

I do have enableSemantics set in LocalSettings.php.

I deployed and launch Blazegraph as instructed here:

git clone -b BLAZEGRAPH_RELEASE_2_1_0 --single-branch https://github.com/blazegraph/database.git BLAZEGRAPH_RELEASE_2_1_0
cd BLAZEGRAPH_RELEASE_2_1_0
./scripts/mavenInstall.sh
./scripts/startBlazegraph.sh

So I ended up changing the sparql endpoint url. Using this:

$smwgSparqlQueryEndpoint = 'http://localhost:9999/blazegraph/namespace/kb/sparql';
$smwgSparqlUpdateEndpoint = 'http://localhost:9999/blazegraph/namespace/kb/sparql';

Yielded this with phpunit testing: BG2.1.0.phpunit_testing.txt

Using this:

$smwgSparqlQueryEndpoint = 'http://localhost:9999/blazegraph/sparql';
$smwgSparqlUpdateEndpoint = 'http://localhost:9999/blazegraph/sparql';

Yielded this with phpunit testing: BG2.1.0.phpunit_testing2.txt

Both showed some http 500 error, so not sure what to make of it. The first endpoint setting gave 6 failures. There appears to be 5 failures with the second endpoint setting which did not specify a namespace.

Any of these errors serious? Seems like we are getting somewhere.

@mwjames

This comment has been minimized.

Copy link
Contributor

commented Jun 4, 2016

FAILURES!
Tests: 3660, Assertions: 11005, Errors: 7, Failures: 5, Skipped: 6.

Seems like we are getting somewhere.

Yes, this is definitely looking better. If you study the error then you can see that the DELETE is getting refused.

Any of these errors serious?

PREFIX wiki: <http://172.20.174.215/index.php/Special:URIResolver/>
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
PREFIX owl: <http://www.w3.org/2002/07/owl#>
PREFIX swivt: <http://semantic-mediawiki.org/swivt/1.0#>
PREFIX property: <http://172.20.174.215/index.php/Special:URIResolver/Property-3A>
PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>
WITH <http://example.org/phpunit-testrun> DELETE { <http://172.20.174.215/index.php/Special:URIResolver/Example/P0414/2> ?p ?o } WHERE { <http://172.20.174.215/index.php/Special:URIResolver/Example/P0414/2> ?p ?o }
PREFIX wiki: <http://172.20.174.215/index.php/Special:URIResolver/>
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
PREFIX owl: <http://www.w3.org/2002/07/owl#>
PREFIX swivt: <http://semantic-mediawiki.org/swivt/1.0#>
PREFIX property: <http://172.20.174.215/index.php/Special:URIResolver/Property-3A>
PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>
WITH <http://example.org/phpunit-testrun> DELETE { ?s ?p ?o } WHERE { ?s ?p ?o }

As noted in [0], in case of a BadHttpDatabaseResponseException or Error: Query refused it is best to copy the SPARQL directly into the query/update endpoint and see the specific error since we don't have more information as to why a query got refused by the time we receive the response from the repository.

As side note, when looking at [1] you can see we are running with com.bigdata.rdf.store.AbstractTripleStore.quads=true not sure whether your instance is in quads mode.

##
## Setup for QUADS mode without the full text index.
##
com.bigdata.rdf.sail.truthMaintenance=false
com.bigdata.rdf.store.AbstractTripleStore.quads=true
com.bigdata.rdf.store.AbstractTripleStore.statementIdentifiers=false
com.bigdata.rdf.store.AbstractTripleStore.textIndex=false
com.bigdata.rdf.store.AbstractTripleStore.axiomsClass=com.bigdata.rdf.axioms.NoAxioms

[0] https://www.semantic-mediawiki.org/wiki/Help:SPARQLStore/Exception/BadHttpDatabaseResponseException
[1] https://github.com/SemanticMediaWiki/SemanticMediaWiki/blob/master/tests/travis/blazegraph-store.properties#L31-L38

@lloonngg

This comment has been minimized.

Copy link
Author

commented Jun 5, 2016

As side note, when looking at [1] you can see we are running with com.bigdata.rdf.store.AbstractTripleStore.quads=true not sure whether your instance is in quads mode.

Setup for QUADS mode without the full text index.

com.bigdata.rdf.sail.truthMaintenance=false
com.bigdata.rdf.store.AbstractTripleStore.quads=true
com.bigdata.rdf.store.AbstractTripleStore.statementIdentifiers=false
com.bigdata.rdf.store.AbstractTripleStore.textIndex=false
com.bigdata.rdf.store.AbstractTripleStore.axiomsClass=com.bigdata.rdf.axioms.NoAxioms

Yes I confirmed that my instance of Blazegraph 2.1.0 is running quads mode. My blazegraphstore.properties does not contain the very last setting when compared to:

[1] https://github.com/SemanticMediaWiki/SemanticMediaWiki/blob/master/tests/travis/blazegraph-store.properties#L31-L38

# To avoid java.lang.IllegalStateException
com.bigdata.rdf.store.AbstractTripleStore.inlineDateTimes=false

Adding this last setting in did not seem to get rid of the errors.

Running this sparql query in the Blazegraph query interface:

PREFIX wiki: <http://172.20.174.215/index.php/Special:URIResolver/>
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
PREFIX owl: <http://www.w3.org/2002/07/owl#>
PREFIX swivt: <http://semantic-mediawiki.org/swivt/1.0#>
PREFIX property: <http://172.20.174.215/index.php/Special:URIResolver/Property-3A>
PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>
WITH <http://example.org/phpunit-testrun> DELETE { <http://172.20.174.215/index.php/Special:URIResolver/Example/P0414/2> ?p ?o } WHERE { <http://172.20.174.215/index.php/Special:URIResolver/Example/P0414/2> ?p ?o }

Shows this error:

ERROR: SPARQL-QUERY: queryStr=PREFIX wiki: 
PREFIX rdf: 
PREFIX rdfs: 
PREFIX owl: 
PREFIX swivt: 
PREFIX property: 
PREFIX xsd: 
WITH  DELETE {  ?p ?o } WHERE {  ?p ?o }
java.util.concurrent.ExecutionException: org.openrdf.query.MalformedQueryException: Encountered " "with" "WITH "" at line 8, column 1.
Was expecting one of:
    "base" ...
    "prefix" ...
    "select" ...
    "construct" ...
    "describe" ...
    "ask" ...

    at java.util.concurrent.FutureTask.report(FutureTask.java:122)
    at java.util.concurrent.FutureTask.get(FutureTask.java:188)
    at com.bigdata.rdf.sail.webapp.BigdataServlet.submitApiTask(BigdataServlet.java:281)
    at com.bigdata.rdf.sail.webapp.QueryServlet.doSparqlQuery(QueryServlet.java:636)
    at com.bigdata.rdf.sail.webapp.QueryServlet.doPost(QueryServlet.java:263)
    at com.bigdata.rdf.sail.webapp.RESTServlet.doPost(RESTServlet.java:269)
    at com.bigdata.rdf.sail.webapp.MultiTenancyServlet.doPost(MultiTenancyServlet.java:192)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:769)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1125)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1059)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
    at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
    at org.eclipse.jetty.server.Server.handle(Server.java:497)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:248)
    at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:610)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:539)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.openrdf.query.MalformedQueryException: Encountered " "with" "WITH "" at line 8, column 1.
Was expecting one of:
    "base" ...
    "prefix" ...
    "select" ...
    "construct" ...
    "describe" ...
    "ask" ...

    at com.bigdata.rdf.sail.sparql.Bigdata2ASTSPARQLParser.parseQuery2(Bigdata2ASTSPARQLParser.java:400)
    at com.bigdata.rdf.sail.webapp.BigdataRDFContext.getQueryTask(BigdataRDFContext.java:2367)
    at com.bigdata.rdf.sail.webapp.QueryServlet$SparqlQueryTask.call(QueryServlet.java:713)
    at com.bigdata.rdf.sail.webapp.QueryServlet$SparqlQueryTask.call(QueryServlet.java:653)
    at com.bigdata.rdf.task.ApiTaskForIndexManager.call(ApiTaskForIndexManager.java:68)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    ... 1 more
Caused by: com.bigdata.rdf.sail.sparql.ast.ParseException: Encountered " "with" "WITH "" at line 8, column 1.
Was expecting one of:
    "base" ...
    "prefix" ...
    "select" ...
    "construct" ...
    "describe" ...
    "ask" ...

    at com.bigdata.rdf.sail.sparql.ast.SyntaxTreeBuilder.generateParseException(SyntaxTreeBuilder.java:9718)
    at com.bigdata.rdf.sail.sparql.ast.SyntaxTreeBuilder.jj_consume_token(SyntaxTreeBuilder.java:9585)
    at com.bigdata.rdf.sail.sparql.ast.SyntaxTreeBuilder.Query(SyntaxTreeBuilder.java:337)
    at com.bigdata.rdf.sail.sparql.ast.SyntaxTreeBuilder.QueryContainer(SyntaxTreeBuilder.java:212)
    at com.bigdata.rdf.sail.sparql.ast.SyntaxTreeBuilder.parseQuery(SyntaxTreeBuilder.java:32)
    at com.bigdata.rdf.sail.sparql.Bigdata2ASTSPARQLParser.parseQuery2(Bigdata2ASTSPARQLParser.java:336)
    ... 8 more
@mwjames

This comment has been minimized.

Copy link
Contributor

commented Jun 5, 2016

WITH DELETE { ?p ?o } WHERE { ?p ?o }
java.util.concurrent.ExecutionException: org.openrdf.query.MalformedQueryException: Encountered " "with" "WITH "" at line 8, column 1.
Was expecting one of:
"base" ...
"prefix" ...
"select" ...
"construct" ...
"describe" ...
"ask" ...

WITH DELETE { ?p ?o } WHERE { ?p ?o } I'm missing the graph after WITH because the example is send as:

WITH <http://example.org/phpunit-testrun> DELETE { <http://172.20.174.215/index.php/Special:URIResolver/Example/P0414/2> ?p ?o } WHERE { <http://172.20.174.215/index.php/Special:URIResolver/Example/P0414/2> ?p ?o }

The construct is per [0] legitimate and I would try asking on [1] why that would cause a java.util.concurrent.ExecutionException: org.openrdf.query.MalformedQueryException: Encountered " "with" "WITH "" at line 8, column 1.

DELETE per [0] (SPARQL 1.1) is defined as:

WITH <http://example/addresses>
DELETE { ?person ?property ?value } 
WHERE { ?person ?property ?value ; foaf:givenName 'Fred' } 

[0] https://www.w3.org/TR/sparql11-update/#delete
[1] https://lists.sourceforge.net/lists/listinfo/bigdata-developers

@lloonngg

This comment has been minimized.

Copy link
Author

commented Jun 14, 2016

So I looked at the problematic query (I replaced "example.org" with my ip address) again:

PREFIX wiki: <http://172.20.174.215/index.php/Special:URIResolver/>
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
PREFIX owl: <http://www.w3.org/2002/07/owl#>
PREFIX swivt: <http://semantic-mediawiki.org/swivt/1.0#>
PREFIX property: <http://172.20.174.215/index.php/Special:URIResolver/Property-3A>
PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>
WITH <http://172.20.174.215/phpunit-testrun> DELETE { <http://172.20.174.215/index.php/Special:URIResolver/Example/P0414/2> ?p ?o } WHERE { <http://172.20.174.215/index.php/Special:URIResolver/Example/P0414/2> ?p ?o }

And realized that I had tested it as a "query" on the Blazegraph query interface (yielding the "with delete" error above) when it should have been an "update."

Putting that delete sparql statement in the correct update form on the interface seems to yield a proper response without errors:

totalElapsed=24ms, elapsed=24ms, connFlush=0ms, batchResolve=0, whereClause=0ms, deleteClause=0ms, insertClause=0ms

I then thought that perhaps the sparql endpoint settings for query and update may be incorrect in LocalSettings.php, so I changed the SMW settings to this:

$smwgDefaultStore = 'SMWSparqlStore';
$smwgSparqlDatabaseConnector = 'blazegraph';
$smwgSparqlQueryEndpoint = 'http://localhost:9999/blazegraph/namespace/kb/sparql/query';
$smwgSparqlUpdateEndpoint = 'http://localhost:9999/blazegraph/namespace/kb/sparql/update';
$smwgSparqlDataEndpoint = '';
$smwgSparqlRepositoryConnectorForcedHttpVersion = true;

Updating the tables and rebuilding the SMW data yielded no errors after running jobs.

I reran the phpunit testing and noticed no errors from the Blazegraph service anymore (i.e., the "with delete" error from before). The sparql endpoints have changed with Blazegraph 2.1.0. From the standpoint of Blazegraph, I believe things are working. With the new query and update endpoint settings, I can store SMW properties and query for them, including subobjects.

There appears to be a few new errors and failures. however. Phpunit testing showed these
results.

The 7 errors are all related to:

SMW\Tests\Integration\ByJsonScript\ByJsonScriptFixtureTestCaseRunnerTest::executeTestCasesFor with data set "s-0001.json" ('/var/www/html/mediawiki/exten...1.json')
BadMethodCallException: Call to a member function setContext() on a non-object (NULL)

Failures 1 & 2 are perhaps related to the 7 errors above:

Call to a member function getTitle() on a non-object (NULL)

Not sure what failures 3 & 4 are.

Failures 5-7 are similar to what we saw before with HHVM microseconds handling.

Things seem to be much better with the corrected endpoint settings. Just need to deal with the new errors and failures. Any thoughts?

@mwjames

This comment has been minimized.

Copy link
Contributor

commented Jun 14, 2016

Things seem to be much better with the corrected endpoint settings.

Well done.

Failures 1 & 2 are perhaps related to the 7 errors above:

Exactly.

Call to a member function getTitle() on a non-object (NULL)
Not sure what failures 3 & 4 are.

plus

  1. SMW\Tests\DataValues\InfoLinksProviderTest::testGetInfolinkTextOnNumberValue and
  2. SMW\Tests\DataValues\InfoLinksProviderTest::testGetInfolinkTextOnStringValue relates to Special:SearchByProperty

are all caused by SpecialPageFactory::getPage( ... ) (MW function) being unable to create a SPECIAL title object and ultimately fails when trying to operate on it.

If we compare one of the early reports "BG2.1.0.phpunit_testing2.txt" then we can see those issues did not occur, so the question is why tests with SPECIAL are having a problem now.

Did you change the LocalSettings or added additional extensions between the first and the run now?

Just to be sure, when accessing the wiki itself SPECIAL pages (Browse, Ask) are available?

Locally on 1.27, I cannot replicate the issue and I have not seen the problem anywhere else. It could be caused by an internal cache [0] in SpecialPageFactory (but why only on your instance??) therefore would you mind temporarily adding the code to [1] and see if we can mitigate the problem for the 7 errors with BadMethodCallException.

        $queryParameters = isset( $case['special-page']['query-parameters'] ) ? $case['special-page']['query-parameters'] : array();

+       if ( method_exists( 'SpecialPageFactory', 'resetList' ) ) {
+           SpecialPageFactory::resetList();
+       }

[0] https://github.com/wikimedia/mediawiki/blob/master/includes/specialpage/SpecialPageFactory.php#L192-L200

[1] https://github.com/SemanticMediaWiki/SemanticMediaWiki/blob/master/tests/phpunit/Integration/ByJsonScript/SpecialPageTestCaseProcessor.php#L67

@lloonngg

This comment has been minimized.

Copy link
Author

commented Jun 14, 2016

No failures or errors! See phpunit testing results.

When you highlighted Special:SearchByProperty causing the errors/failures, I quickly realized that I had turned permission settings to restrict certain special pages to sysop only since the last set of testing. We run a private wiki and disable permissions for anon users. When I commented the special page restrictions out, the phpunit testing succeeded. Not sure why the HHVM microseconds failures are not there anymore. These are still my installation settings:

MediaWiki:  1.27.0-rc.0
HHVM:  3.6.6 (srv)
MariaDB:  10.1.14-MariaDB-1~trusty
ICU:  52.1

We are running a private wiki with ~184,000 pages and lots of triples. Very curious to see how Blazegraph performs. I will be migrating the system over to test over the next few weeks now that Blazegraph 2.1.0 is working with that latest MW and SMW.

I assume you can close this issue now. Thanks for all your help. The phpunit testing has been very helpful.

@mwjames

This comment has been minimized.

Copy link
Contributor

commented Jun 14, 2016

now that Blazegraph 2.1.0 is working with that latest MW and SMW.

Fantastic

Not sure why the HHVM microseconds failures are not there anymore. These are still my installation settings:

According to your "phpunit testing results" file, you run the test with "Using PHP 5.5.9-1ubuntu4.17" and not HHVM hence no issues with microseconds.

Using PHP 5.5.9-1ubuntu4.17

Semantic MediaWiki: 2.4-alpha (SMWSparqlStore, mysql, blazegraph)
MediaWiki:          1.27.0-rc.0 (MediaWiki vendor autoloader)
Site language:      en

Execution time:     2016-06-14 03:01
Xdebug:             2.2.3 (enabled)

PHPUnit 4.8.24 by Sebastian Bergmann and contributors.

Runtime:    PHP 5.5.9-1ubuntu4.17 with Xdebug 2.2.3
Configuration:  /var/www/html/mediawiki/extensions/SemanticMediaWiki/phpunit.xml.dist
@mwjames

This comment has been minimized.

Copy link
Contributor

commented Jun 23, 2016

We are running a private wiki with ~184,000 pages and lots of triples. Very curious to see how Blazegraph performs. I will be migrating the system over to test over the next few weeks now that Blazegraph 2.1.0 is working with that latest MW and SMW.

Maybe in future, you can share an experience report (switching cost, expected benefits vs actual ones, performance considerations pros and cons etc.).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.