SMW 2.4 with Blazegraph SparqlStore #1583
Comments
Our Travis environment runs Blazegraph 1.5.2 with:
and passes our test suite therefore I know Blazegraph works [0]. 2.1 shouldn't cause any issues but I'm a bit too buys to make a switch to 2.1 for the test setup.
I'm guessing you did also run the
For a general check-up on your installation you can simply run [0] https://s3.amazonaws.com/archive.travis-ci.org/jobs/129857107/log.txt |
|
@lloonngg Any update on this issue? |
|
Sorry, I was tied up recently. I tried composer phpunit and found quite a few failures with my installation. I reverted to sqlstore3 and still got errors. I'm planning to rebuild my installation and using the smw unit testing as I go along. Any reason to think that the errors are due to MW 1.26.2? |
As for the errors, maybe you can put them in a pastebin (or gist) and link it to this issue.
No. Some Travis jobs run with 1.26.3 others with 1.24* and we still have a job that tests against 1.19*. PS: When I said "I know Blazegraph works" with a persistence on unit (integration) tests then it is because we invested quite amount of time to provide 3600+ tests and have Travis (our CI environment provider) constantly run against the major triple-stores. |
|
I created a new stack with the following: MediaWiki 1.27.0-rc.0 Just testing out sqlstore3 first with phpunit testing, I got the following output and error. I also tried MW 1.26.3 and 1.25.6 with the same SMW 2.4alpha extension and get similar errors during phpunit testing. Am I missing a dependency? Only SMW was installed as an extension with this clean build.
Semantic MediaWiki: 2.4-alpha (SMWSQLStore3, mysql) Execution time: 2016-06-02 02:25 PHPUnit 4.8.24 by Sebastian Bergmann and contributors. Runtime: HHVM 3.13.1 ............................................................. 61 / 3661 ( 1%) [RuntimeException] |
Those two failures are caused by the #1456 HHVM bug.
A change was necessary due to #1615 but it turned out that not all of the 1.28.* code has access to this method therefore required 9e3f9a8, yet 1.27.0-rc.0 behaves different again which means (argggg....) it requires e6b2c50. Please pull a new version and try again. |
|
I'm still getting those two failures with the latest SMW pull. MediaWiki 1.27.0-rc.0
Semantic MediaWiki: 2.4-alpha (SMWSQLStore3, mysql) Execution time: 2016-06-02 11:29 PHPUnit 4.8.24 by Sebastian Bergmann and contributors. Runtime: HHVM 3.13.1 ............................................................. 61 / 3661 ( 1%) [RuntimeException] |
Is also mentioned in #1456 as HHVM bug caused by facebook/hhvm#5128. Our Travis HHVM [0] uses HHVM 3.6.6 (5.6.99-hhvm) which isn't/wasn't an issue. [0] https://travis-ci.org/SemanticMediaWiki/SemanticMediaWiki/jobs/134708474 |
The line HHVM complains about is [0] You can try removing |
|
I tried the editing SMW_DV_Quantity.php as you suggested and it gave the same errror and others. I downgraded to HHVM 3.6.6 and still see the failures. The test makes it to the end now and shows a third failure. If the errors are mainly due to testing and not functionality of SMW, is it safe to proceed with Blazegraph for the backend?
Semantic MediaWiki: 2.4-alpha (SMWSQLStore3, mysql) Execution time: 2016-06-02 02:57 PHPUnit 4.8.24 by Sebastian Bergmann and contributors. Runtime: HHVM 3.6.6 ............................................................. 61 / 3661 ( 1%) Time: 2.7 minutes, Memory: 323.84MB There were 3 failures:
/var/www/html/mediawiki/extensions/SemanticMediaWiki/tests/phpunit/Unit/IntlTimeFormatterTest.php:49
/var/www/html/mediawiki/extensions/SemanticMediaWiki/tests/phpunit/Unit/IntlTimeFormatterTest.php:49
with (StringContains) [ 11:14:15.888500 ] /var/www/html/mediawiki/extensions/SemanticMediaWiki/tests/phpunit/Utils/Validators/StringValidator.php:76 There were 11 skipped tests:
/var/www/html/mediawiki/extensions/SemanticMediaWiki/tests/phpunit/Integration/MediaWiki/SearchInPageDBIntegrationTest.php:65
/var/www/html/mediawiki/extensions/SemanticMediaWiki/tests/phpunit/Integration/SPARQLStore/QueryResultLookupWithoutBaseStoreIntegrationTest.php:45
/var/www/html/mediawiki/extensions/SemanticMediaWiki/tests/phpunit/Integration/SPARQLStore/QueryResultLookupWithoutBaseStoreIntegrationTest.php:45
/var/www/html/mediawiki/extensions/SemanticMediaWiki/tests/phpunit/Integration/SPARQLStore/QueryResultLookupWithoutBaseStoreIntegrationTest.php:45
/var/www/html/mediawiki/extensions/SemanticMediaWiki/tests/phpunit/Integration/SPARQLStore/QueryResultLookupWithoutBaseStoreIntegrationTest.php:45
/var/www/html/mediawiki/extensions/SemanticMediaWiki/tests/phpunit/Integration/SPARQLStore/RedirectLookupIntegrationTest.php:37
/var/www/html/mediawiki/extensions/SemanticMediaWiki/tests/phpunit/Integration/SPARQLStore/RedirectLookupIntegrationTest.php:37
/var/www/html/mediawiki/extensions/SemanticMediaWiki/tests/phpunit/Integration/SPARQLStore/RedirectLookupIntegrationTest.php:37
/var/www/html/mediawiki/extensions/SemanticMediaWiki/tests/phpunit/Integration/System/InstallationConfigurationIntegrityTest.php:60
/var/www/html/mediawiki/extensions/SemanticMediaWiki/tests/phpunit/Integration/System/InstallationConfigurationIntegrityTest.php:75
/var/www/html/mediawiki/extensions/SemanticMediaWiki/tests/phpunit/Integration/System/InstallationConfigurationIntegrityTest.php:75 FAILURES! |
The three errors belong to the same category of HHVM having problems with edge handling of micro seconds as outlined earlier. So, nothing we can do from the SMW side unless of course you drop the test case.
I'd say 👍 Since you encountered three of the expected three, switch the gear and plug-in the Blazegraph and we should see similar results. |
|
I have Blazegraph 2.1.0 installed and running. When running rebuildData.php under the SMW maintenance folder I get the following error. It is not the same as the one previously reported for curl and sparqlstore with the Chunked-Encoded data error [1]. I have tried changing the from curl version 7.35 to 7.36 or 7.43 without any improvement. Forcing the http version for curl also does not help [2]. |
cUrl needs to work out-of-the box otherwise the handler can't establish a connection [0]. This seems more of a low-level, install, ubuntu issue.
This leads me to https://curl.haxx.se/docs/faq.html#Protocol_xxx_not_supported_or_di. [0] https://www.semantic-mediawiki.org/wiki/Help:SPARQLStore/Exception/HttpDatabaseConnectionException |
|
I recognized 2.1.0 changed the url pattern to Is the In contrast, 1.5.2 uses [0] https://wiki.blazegraph.com/wiki/index.php/NanoSparqlServer |
|
I figured out the problem. The documentation on how to set up the Blazegraph endpoint for sparqlstore has a stray " in the QueryEndpoint url. Removing it and leaving the url as is (http://localhost:9999/bigdata/namespace/kb/sparql) got rid of the libcurl error during rebuildData. Unfortunately, when I run the phpunit test, there are a ton of failures (see attached). So maybe Blazegraph 2.1.0 may not be compatible out of the box. I'm going to try 1.5.2 to see if performs like sqlstore3 with phpunit testing. |
Did you set something like |
Those F's indicate a failure during the integration tests [0]. The integration tests simulate a "human" input with an active DB/Triple store connection which is compared against an expected output and in case it fails, the data storage and retrieval process was unsuccessful. [0] https://www.semantic-mediawiki.org/wiki/Help:Integration_tests |
|
You did run it with something like In the beginning we had some issues with Blazgraph but the developers helped us [1] therefore it is run with the [0] option. [0] https://github.com/SemanticMediaWiki/SemanticMediaWiki/blob/master/tests/travis/blazegraph-store.properties |
|
I do have enableSemantics set in LocalSettings.php. I deployed and launch Blazegraph as instructed here: So I ended up changing the sparql endpoint url. Using this: Yielded this with phpunit testing: BG2.1.0.phpunit_testing.txt Using this: Yielded this with phpunit testing: BG2.1.0.phpunit_testing2.txt Both showed some http 500 error, so not sure what to make of it. The first endpoint setting gave 6 failures. There appears to be 5 failures with the second endpoint setting which did not specify a namespace. Any of these errors serious? Seems like we are getting somewhere. |
Yes, this is definitely looking better. If you study the error then you can see that the
As noted in [0], in case of a As side note, when looking at [1] you can see we are running with [0] https://www.semantic-mediawiki.org/wiki/Help:SPARQLStore/Exception/BadHttpDatabaseResponseException |
Yes I confirmed that my instance of Blazegraph 2.1.0 is running quads mode. My blazegraphstore.properties does not contain the very last setting when compared to: Adding this last setting in did not seem to get rid of the errors. Running this sparql query in the Blazegraph query interface: Shows this error: |
The construct is per [0] legitimate and I would try asking on [1] why that would cause a DELETE per [0] (SPARQL 1.1) is defined as: [0] https://www.w3.org/TR/sparql11-update/#delete |
|
So I looked at the problematic query (I replaced "example.org" with my ip address) again: And realized that I had tested it as a "query" on the Blazegraph query interface (yielding the "with delete" error above) when it should have been an "update." Putting that delete sparql statement in the correct update form on the interface seems to yield a proper response without errors:
I then thought that perhaps the sparql endpoint settings for query and update may be incorrect in LocalSettings.php, so I changed the SMW settings to this: Updating the tables and rebuilding the SMW data yielded no errors after running jobs. I reran the phpunit testing and noticed no errors from the Blazegraph service anymore (i.e., the "with delete" error from before). The sparql endpoints have changed with Blazegraph 2.1.0. From the standpoint of Blazegraph, I believe things are working. With the new query and update endpoint settings, I can store SMW properties and query for them, including subobjects. There appears to be a few new errors and failures. however. Phpunit testing showed these The 7 errors are all related to: Failures 1 & 2 are perhaps related to the 7 errors above:
Not sure what failures 3 & 4 are. Failures 5-7 are similar to what we saw before with HHVM microseconds handling. Things seem to be much better with the corrected endpoint settings. Just need to deal with the new errors and failures. Any thoughts? |
Well done.
Exactly.
plus
are all caused by If we compare one of the early reports "BG2.1.0.phpunit_testing2.txt" then we can see those issues did not occur, so the question is why tests with SPECIAL are having a problem now. Did you change the Just to be sure, when accessing the wiki itself SPECIAL pages (Browse, Ask) are available? Locally on 1.27, I cannot replicate the issue and I have not seen the problem anywhere else. It could be caused by an internal cache [0] in |
|
No failures or errors! See phpunit testing results. When you highlighted Special:SearchByProperty causing the errors/failures, I quickly realized that I had turned permission settings to restrict certain special pages to sysop only since the last set of testing. We run a private wiki and disable permissions for anon users. When I commented the special page restrictions out, the phpunit testing succeeded. Not sure why the HHVM microseconds failures are not there anymore. These are still my installation settings: We are running a private wiki with ~184,000 pages and lots of triples. Very curious to see how Blazegraph performs. I will be migrating the system over to test over the next few weeks now that Blazegraph 2.1.0 is working with that latest MW and SMW. I assume you can close this issue now. Thanks for all your help. The phpunit testing has been very helpful. |
Fantastic
According to your "phpunit testing results" file, you run the test with "Using PHP 5.5.9-1ubuntu4.17" and not HHVM hence no issues with microseconds. |
Maybe in future, you can share an experience report (switching cost, expected benefits vs actual ones, performance considerations pros and cons etc.). |
Setup and configuration
Issue
I'm testing out the latest Blazegraph 2.1.0 (running on Nano Sparql Server) as a new SparqlStore for SMW 2.4. I put into LocalSettings the following SMW settings; by the way there is a typo in the $smwgSparqlQueryEndpoint url; stray " character). I changed 'localhost:9999' to the ip where my Blazegraph service is running (http://192.168.1.21:9999) and kept everything else as is.
When I go to Special:SMWAdmin to Initialize/Upgrade Tables, the output shows "Selected storage engine is "SMWSQLStore3" (or an extension thereof)" rather than blazegraph as the storage engine. The old SMW tables in MariaDB shows up as already existing.
When I go back to Special:SMWAdmin to check the configuration settings, it correctly shows:
update.php and runjobs run just fine to update the templates, pages, properties, etc. But after refreshing the SMW data, I cannot see the SMW data which I was able to see before when the storage engine was indeed SMWSQLStore3.
Is there some other config in LocalSettings required to point SMW to the Blazegraph sparql store?
Thanks,
Long
The text was updated successfully, but these errors were encountered: