Improve Store::getInProperty performance, refs 1234

[mwjames]

This PR is made in reference to: #1234

This PR addresses or contains:

• The following content was used to simulate some high data load
• Changes on pre/post to the query plan can be found here
• The PR in particular addresses the Store::getInProperties performance with a significant improvement in query execution time by extending the INNER JOIN with a subquery to pre-select and filter possible matches before being materialized. One example entity with > 100 incoming properties and an overall table size that covers ~1M triples demonstrated the following changes pre/post this PR.
  • MySQL ( 841.3639ms ↘ 39.1431ms)
  • Postgres (1282.6319ms ↘ 59.8922ms)
• The suggestion from SMWSQLStore3Readers::getInProperties performance #1234 on using something like ... WHERE smw_id IN (SELECT p_id FROM mediawiki."smw_di_wikipage" WHERE o_id='19683’); did not create the expected improvement on MySQL hence a different query was selected that showed favorable query plans on both MySQL and Postgres.

To eliminate filesort and other expensive operations we needed to:

• Create additional composite indexes to ensure the query planner choose the right one with the objective of using an index
• Eliminate ORDER BY as it is expected to returns all matches and sort the results during a post processing
• Filter possible matches before being materialized

SELECT DISTINCT smw_title,smw_sortkey,smw_iw FROM "smw_object_ids"
INNER JOIN (
  SELECT p_id FROM "smw_di_wikipage" WHERE o_id='266' GROUP BY p_id
 ) AS t1 ON t1.p_id=smw_id
WHERE smw_iw!=':smw' AND smw_iw!=':smw-delete'

Tested on:

MediaWiki	1.30.0-alpha (3d26a96)
PHP	7.1.1 (apache2handler)
MariaDB	10.1.21-MariaDB
ICU	57.1

MediaWiki	1.26.2
PHP	5.6.8 (apache2handler)
PostgreSQL	9.3.12
ICU	54.1

This PR includes:

• Tests (unit/integration)
• CI build passed

[mwjames]

This requires either update.php or setupStore.php to be run.

@kghbln One thing to be noted is due to the added indices, the update.php might take a bit longer the first time as those are created and depend largely on the size of the DB.

[kghbln]

@mwjames This surely will come out with a bang! The performance improvements seen are quite impressive! WOW!

One thing to be noted is due to the added indices, the update.php might take a bit longer the first time as those are created and depend largely on the size of the DB.

We should note this is the RELEASE NOTES in any case.

[kghbln]

@mwjames Shouldn't this say SMW_EL_INPROP?

[kghbln]

See #2758

[mwjames]

Well, the setting is internal in case something seems fishy in order to easily switch back and compare results without reworking the whole code from the inside. I hope that by the time of SMW 3.0 the setting can vanish and yes, the default value is different.

…

On 7/12/17, Karsten Hoffmeyer ***@***.***> wrote: kghbln commented on this pull request. > @@ -1495,4 +1495,18 @@ 'smwgEntityCollation' => 'identity', ## + ## + # Entity lookup specific features + # + # - SMW_EL_NONE applies no query or schema changes + # + # - SMW_EL_INPROP enables a new query form for selecting incoming properties + # (#1234) + # + # @SInCE 3.0 + # @default false @mwjames Shouldn't this say `SMW_EL_INPROP` -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #2461 (review)

[kghbln]

I hope that by the time of SMW 3.0 the setting can vanish and yes, the default value is different.

You mean the whole smwgEntityLookupFeatures parameter or just "false" as the internal setting?

[kghbln]

Documented on wiki