Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve query performance and add caching support #107

Merged
merged 43 commits into from
Mar 13, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
93da26c
Improve query performance and add caching support
Universal-Omega Mar 9, 2022
94b447e
CI: lint code to MediaWiki standards
Mar 9, 2022
48801f6
Update extension.json
Universal-Omega Mar 10, 2022
e7d1ab4
Update Query.php
Universal-Omega Mar 10, 2022
6d546ea
Update Query.php
Universal-Omega Mar 10, 2022
a802730
Update extension.json
Universal-Omega Mar 10, 2022
7dc8497
Update Parse.php
Universal-Omega Mar 10, 2022
1c83862
Update Article.php
Universal-Omega Mar 10, 2022
5dc651a
Update Parse.php
Universal-Omega Mar 10, 2022
2dc1c1e
Update Article.php
Universal-Omega Mar 10, 2022
3aa6a79
Update Query.php
Universal-Omega Mar 10, 2022
b661935
Update Query.php
Universal-Omega Mar 10, 2022
39dd702
Update Query.php
Universal-Omega Mar 10, 2022
8b7c01f
Cleanup + variable rename
Universal-Omega Mar 10, 2022
d3c5ffe
Update Article.php
Universal-Omega Mar 10, 2022
0f83530
Update Parse.php
Universal-Omega Mar 10, 2022
96baede
Update Query.php
Universal-Omega Mar 10, 2022
02b33d1
Update .phpcs.xml
Universal-Omega Mar 10, 2022
640f863
Update Query.php
Universal-Omega Mar 10, 2022
096e716
Update Query.php
Universal-Omega Mar 10, 2022
475319a
Update LST.php
Universal-Omega Mar 10, 2022
b9e08c3
Update mediawiki-tests.yml
Universal-Omega Mar 10, 2022
024e4dc
Update extension.json
Universal-Omega Mar 10, 2022
ddea1ec
Update CHANGELOG.md
Universal-Omega Mar 10, 2022
d45740e
Update README.md
Universal-Omega Mar 10, 2022
abb1c9a
Update CHANGELOG.md
Universal-Omega Mar 10, 2022
1b4d9d8
Update README.md
Universal-Omega Mar 10, 2022
34932fd
Update Parse.php
Universal-Omega Mar 10, 2022
91b3ee1
Add poolcounter error
Universal-Omega Mar 12, 2022
df9f8c8
Update Query.php
Universal-Omega Mar 12, 2022
c0c066c
Update en.json
Universal-Omega Mar 12, 2022
610868c
Update Parse.php
Universal-Omega Mar 12, 2022
1269869
Update Parse.php
Universal-Omega Mar 12, 2022
2814701
Update DynamicPageListHooks.php
Universal-Omega Mar 12, 2022
61a5e69
Update Logger.php
Universal-Omega Mar 12, 2022
bdeabb6
Update Parse.php
Universal-Omega Mar 12, 2022
29a108e
Update en.json
Universal-Omega Mar 12, 2022
26783b9
Update DynamicPageListHooks.php
Universal-Omega Mar 12, 2022
fe9947d
Update en.json
Universal-Omega Mar 12, 2022
c5c85a6
Update mediawiki-tests.yml
Universal-Omega Mar 12, 2022
4636a59
Update Query.php
Universal-Omega Mar 12, 2022
55311f7
Update CHANGELOG.md
Universal-Omega Mar 13, 2022
c89816c
Update extension.json
Universal-Omega Mar 13, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 14 additions & 12 deletions .github/workflows/mediawiki-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -203,10 +203,10 @@ jobs:
stage: qunit

# Latest stable MediaWiki - PHP 7.3 (composer-test)
- mw: 'REL1_36'
- mw: 'REL1_37'
php: 7.3
php-docker: 73
experimental: true
experimental: false
stage: composer-test

runs-on: ubuntu-latest
Expand Down Expand Up @@ -351,17 +351,19 @@ jobs:
if [ -e composer.json ]; then
composer install --prefer-dist --no-progress --no-interaction
composer fix

rm composer.lock

git config --global user.name "github-actions"
git config --global user.email "github-actions@users.noreply.github.com"
git remote set-url origin "https://x-access-token:${GITHUB_TOKEN}@github.com/${GITHUB_REPOSITORY}.git"
git checkout -b ${GITHUB_HEAD_REF}
git add .
git commit -am "CI: lint code to MediaWiki standards" -m "Check commit and GitHub actions for more details" || echo "No changes to commit"
git pull origin ${GITHUB_HEAD_REF} --rebase
git push --set-upstream origin ${GITHUB_HEAD_REF}
if ! git diff --exit-code --quiet; then
git config --global user.name "github-actions"
git config --global user.email "github-actions@users.noreply.github.com"
git remote set-url origin "https://x-access-token:${GITHUB_TOKEN}@github.com/${GITHUB_REPOSITORY}.git"
git checkout -b ${GITHUB_HEAD_REF}
git add .
git commit -am "CI: lint code to MediaWiki standards" -m "Check commit and GitHub actions for more details"
git pull origin ${GITHUB_HEAD_REF} --rebase
git push --set-upstream origin ${GITHUB_HEAD_REF}
else
echo "No changes to commit"
fi
fi

- name: Main Test
Expand Down
1 change: 0 additions & 1 deletion .phpcs.xml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,6 @@
<exclude name="MediaWiki.NamingConventions.LowerCamelFunctionsName.FunctionName" />
<exclude name="MediaWiki.WhiteSpace.SpaceBeforeSingleLineComment.NewLineComment" />
<exclude name="PSR2.Methods.MethodDeclaration.Underscore" />
<exclude name="MediaWiki.ControlStructures.AssignmentInControlStructures.AssignmentInControlStructures" />
<exclude name="PEAR.Functions.ValidDefaultValue.NotAtEnd" />
</rule>
<arg name="bootstrap" value="./vendor/mediawiki/mediawiki-codesniffer/utils/bootstrap-ci.php"/>
Expand Down
44 changes: 23 additions & 21 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,11 @@

The **DynamicPageList3** extension is a reporting tool for MediaWiki, listing category members and intersections with various formats and details. For full documentation, see the [manual](https://help.fandom.com/Extension:DPL3/Manual).

When invoked with a basic set of selection parameters DPL displays a list of pages in one or more categories. Selections may also be based on factors such as author, namespace, date, name pattern, usage of templates, or references to other articles. Output takes a variety of forms, some of which incorporate elements of selected articles.
When invoked with a basic set of selection parameters DPL3 displays a list of pages in one or more categories. Selections may also be based on factors such as author, namespace, date, name pattern, usage of templates, or references to other articles. Output takes a variety of forms, some of which incorporate elements of selected articles.

This extension is invoked with the parser function <code>{{#dpl: .... }}</code> or parser tag <code>&lt;DPL&gt;</nowiki></code>. A [Wikimedia](https://www.mediawiki.org/wiki/Special:MyLanguage/Extension:DynamicPageList_(Wikimedia))-compatible implementation of certain features can be invoked with <code>&lt;DynamicPageList&gt;</code>.
This extension is invoked with the parser function <code>{{#dpl: .... }}</code> or parser tag <code>&lt;DPL&gt;</nowiki></code>. A [Wikimedia](https://www.mediawiki.org/wiki/Special:MyLanguage/Extension:DynamicPageList_(Wikimedia))-compatible implementation of certain features can be invoked with <code>&lt;DynamicPageList&gt;</code>.

Complex look ups can result in computationally expensive database queries. However, by default all output is cached for a period of one hour to reduce the need to rerun the query every page load. The [DPL:Parameters: Other Parameters](https://help.fandom.com/Extension:DPL3/Parameters:_Other_parameters#cacheperiod) manual page contains information on parameters that can be used to disable the cache and allow instant updates.
Complex look ups can result in computationally expensive database queries. However, by default all output is cached for a period of one hour to reduce the need to rerun the query every page load. The [DPL:Parameters: Other Parameters](https://help.fandom.com/Extension:DPL3/Parameters:_Other_parameters#cacheperiod) manual page contains information on parameters that can be used to disable the cache and allow instant updates.

* Manual and Complete Documentation: [Documentation](https://help.fandom.com/Extension:DPL3/Manual)
* Source Code: [Source code at GitHub](https://github.com/Universal-Omega/DynamicPageList3)
Expand All @@ -18,49 +18,51 @@ Complex look ups can result in computationally expensive database queries. Howe
Please see the [releases page](https://github.com/Universal-Omega/DynamicPageList3/releases) for the latest releases.

## Configuration
These are DPL's configuration settings and along with their default values. To change them make sure they are defined before including the extension on the wiki. More configuration information is available on the **[MediaWiki extension page](https://www.mediawiki.org/wiki/Special:MyLanguage/Extension:DynamicPageList3#Configuration)**.
These are DPL3's configuration settings and along with their default values. To change them make sure they are defined before including the extension on the wiki. More configuration information is available on the **[MediaWiki extension page](https://www.mediawiki.org/wiki/Special:MyLanguage/Extension:DynamicPageList3#Configuration)**.

**Note:** In release 3.0.4 the configuration variable name was changed from <code>$dplSettings</code> to <code>$wgDplSettings</code>. This was to faciliate compatibility with Mediawiki 1.25's extension registration change.
**Note:** In release 3.0.4 the configuration variable name was changed from <code>$dplSettings</code> to <code>$wgDplSettings</code>. This was to faciliate compatibility with Mediawiki 1.25's extension registration change.

| Setting | Default | Description |
|:--------------------------------------------|---------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| $wgDplSettings['allowedNamespaces'] | null | By default all existing namespaces are used when DPL initializes. Customize this setting with an array of namespace constants to restrict DPL to work only in those namespaces. |
| $wgDplSettings['allowedNamespaces'] | null | By default all existing namespaces are used when DPL3 initializes. Customize this setting with an array of namespace constants to restrict DPL3 to work only in those namespaces. |
| $wgDplSettings['allowUnlimitedCategories'] | false | Set this to true to ignore 'maxCategoryCount' and allow unlimited categories. Please note that large amounts of categories in a query can slow down or crash servers. |
| $wgDplSettings['allowUnlimitedResults'] | false | Set this to true to ignore 'maxResultCount' and allow unlimited results. Please note that large result sets may result in slow or failed page loads. |
| $wgDplSettings['behavingLikeIntersection'] | false | Set DPL to always behave like Extension:Intersection. |
| $wgDplSettings['behavingLikeIntersection'] | false | Set DPL3 to always behave like Extension:Intersection. |
| $wgDplSettings['categoryStyleListCutoff'] | 6 | Maximum number of items in a category list before being cut off. |
| $wgDplSettings['fixedCategories'] | [] | This does something with preventing DPL from "looking" at these categories. |
| $wgDplSettings['fixedCategories'] | [] | This does something with preventing DPL3 from "looking" at these categories. |
| $wgDplSettings['functionalRichness'] | 3 | Set the level of parameters available to end users. |
| $wgDplSettings['maxCategoryCount'] | 4 | Maximum number of categories to allow in queries. |
| $wgDplSettings['minCategoryCount'] | 0 | Minimum number of categories to allow in queries. |
| $wgDplSettings['maxResultCount'] | 500 | Maximum number of results to return from a query. |
| $wgDplSettings['recursiveTagParse'] | false | Do recursive tag parsing on <dpl> parser tags converting tags and functions such as magic words like {{PAGENAME}}. This is similar to the {{#dpl}} parser function call, but may not work exactly the same in all cases. |
| $wgDplSettings['runFromProtectedPagesOnly'] | false | Set this to true to allow DPL to run from protected pages only. This is recommend if wiki administrators are having issues with malicious users creating computationally intensive queries. |
| $wgDplSettings['handleSectionTag'] | false | Set this to true to have DPL handle <section> tags outside of DPL parser tags. |
| $wgDplSettings['recursiveTagParse'] | false | Do recursive tag parsing on <dpl> parser tags converting tags and functions such as magic words like {{PAGENAME}}. This is similar to the {{#dpl}} parser function call, but may not work exactly the same in all cases. |
| $wgDplSettings['runFromProtectedPagesOnly'] | false | Set this to true to allow DPL3 to run from protected pages only. This is recommend if wiki administrators are having issues with malicious users creating computationally intensive queries. |
| $wgDplSettings['handleSectionTag'] | false | Set this to true to have DPL3 handle <section> tags outside of the parser tags provided by DPL3. |
| $wgDplSettings['maxQueryTime'] | 10000 | Maximum allowed time for database queries in milliseconds. |
| $wgDplSettings['queryCacheTime'] | 0 | Can help with situations where you have a template with the same query used on a large number of pages all being refreshed at once. The query cache cannot be purged. Suggested value between 30 to 600. |

The global variable {{manual|$wgNonincludableNamespaces}} is automatically respected by DPL. It will prevent the contents of the listed namespaces from appearing in DPL's output.
The global variable <code>$wgNonincludableNamespaces</code> is automatically respected by DPL3. It will prevent the contents of the listed namespaces from appearing in DPL3's output.

**Note: <code>$wgDplSettings['maxResultCount']</code> is a LIMIT *on the SQL query itself*. Some DPL query parameters like <code>includematch</code> are applied *after* the SQL query, however, so results here may easily be misleading.**
**Note: <code>$wgDplSettings['maxResultCount']</code> is a LIMIT *on the SQL query itself*. Some DPL3 query parameters like <code>includematch</code> are applied *after* the SQL query, however, so results here may easily be misleading.**

### Functional Richness

DynamicPageList3 has many features which are unlocked based on the maximum functional richness level. There are some that can cause high CPU or database load and should be used sparingly.
DynamicPageList3 has many features which are unlocked based on the maximum functional richness level. There are some that can cause high CPU or database load and should be used sparingly.

* <code>$wgDplSettings['functionalRichness'] = 0</code> is equivalent to Wikimedia's [DynamicPageList](https://www.mediawiki.org/wiki/Special:MyLanguage/Extension:DynamicPageList_(Wikimedia))
* <code>$wgDplSettings['functionalRichness'] = 1</code> adds additional formatting parameters
* <code>$wgDplSettings['functionalRichness'] = 2</code> adds performance equivalent features for templates and pagelinks
* <code>$wgDplSettings['functionalRichness'] = 3</code> allows more-expensive page inclusion features and regular expression queries.
* <code>$wgDplSettings['functionalRichness'] = 4</code> permits exotic and potentially dangerous batch update and delete operations; not recommended for public websites. Includes debugging parameters for testing and development.
* <code>$wgDplSettings['functionalRichness'] = 4</code> permits exotic and potentially dangerous batch update and delete operations; not recommended for public websites. Includes debugging parameters for testing and development.


## Usage
### Extended DPL Functionality
Extended DPL is invoked by using the parser function <code>{{#dpl: .... }}</code>, or the parser extension tag <code><DPL> .... </DPL></code>.
### Extended DPL3 Functionality
Extended DPL3 is invoked by using the parser function <code>{{#dpl: .... }}</code>, or the parser extension tag <code>&lt;DPL&gt; .... &lt;/DPL&gt;</code>.

*See [Manual - **General Usage and Invocation Syntax**](https://help.fandom.com/Extension:DPL3/General_usage_and_invocation_syntax) and [DPL:Parameters: **Criteria for Page Selection**](https://help.fandom.com/Extension:DPL3/Parameters:_Criteria_for_page_selection)*

### Backwards Compatibility
Functionality compatible with Wikimedia's DPL extension can be invoked with <code>&lt;DynamicPageList&gt; .... &lt;/DynamicPageList&gt;</code>. Further information can be found on the [Compatibility manual page](https://help.fandom.com/Extension:DPL3/Compatibility).
Functionality compatible with Wikimedia's DPL extension (Intersection) can be invoked with <code>&lt;DynamicPageList&gt; .... &lt;/DynamicPageList&gt;</code>. Further information can be found on the [Compatibility manual page](https://help.fandom.com/Extension:DPL3/Compatibility).

## Usage Philosophy and Overview
With the assumption there are some articles writtne about *countries* those articles will typically have three things in common:
Expand All @@ -77,7 +79,7 @@ category=countries
}}
</nowiki></pre>

With DPL one could:
With DPL3 one could:
* Generate a list of all those articles (or a random sample)
* Show metadata of the articles (popularity, date of last update, ..)
* Show one or more chapters of the articles ('transclude' content)
Expand Down Expand Up @@ -132,8 +134,8 @@ With DPL one could:

## Considerations
### Performance
DPL's code execution and database access is typically fast for typical category and article look ups. However, using loose LIKE and REGEXP match parameters and/or requesting large data sets can result in long database access times. Parser time should also be kept in consideration. For example, having the query of image results go into a template that displays them will result in a parser media transform for each one. This can quickly eat up 2MBs of RAM per media transform.
DPL3's code execution and database access is typically fast for typical category and article look ups. However, using loose LIKE and REGEXP match parameters and/or requesting large data sets can result in long database access times. Parser time should also be kept in consideration. For example, having the query of image results go into a template that displays them will result in a parser media transform for each one. This can quickly eat up 2MBs of RAM per media transform.

## See Also
### Further Reading
DPL can do much more than we can explain here. A complete **[manual](https://help.fandom.com/Extension:DPL3/Manual)** is available with full parameter documentation.
DPL3 can do much more than we can explain here. A complete **[manual](https://help.fandom.com/Extension:DPL3/Manual)** is available with full parameter documentation.
4 changes: 3 additions & 1 deletion extension.json
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,9 @@
"recursiveTagParse": false,
"runFromProtectedPagesOnly": false,
"handleSectionTag": false,
"alwaysCacheResults": false
"alwaysCacheResults": false,
"maxQueryTime": 10000,
"queryCacheTime": 0
}
},
"TrackingCategories": [
Expand Down
2 changes: 2 additions & 0 deletions i18n/en.json
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
"intersection_toofewcats": "Error: Too few categories!",
"intersection_noresults": "Error: No results!",
"intersection_noincludecats": "Error: You need to include at least one category, or specify a namespace!",
"intersection_pcerror": "This list cannot be displayed because the servers are overloaded. Please try again later.",
"dpl-desc": "A highly flexible report generator for MediaWikis",
"dpl_log_1": "Error: Wrong '$1' parameter: '$2'! Help: <code>$1= <i>empty string</i> (Main)$3</code>.",
"dpl_log_2": "Error: Wrong '$1' parameter: '$2'! Help: <code>$1= <i>full pagename</i></code>.",
Expand All @@ -33,6 +34,7 @@
"dpl_log_20": "Warning: An infinite transclusion loop is created by page '$0'.",
"dpl_log_21": "Query: <code>$0</code>",
"dpl_log_22": "Warning: No parameter option supplied for '$1'. (Missing '=')",
"dpl_log_23": "Error: List cannot be displayed because the servers are overloaded. Please try again later.",
"dpl_articlecount": "There {{PLURAL:$1|is one article|are $1 articles}} in this heading.",
"action-dpl_param_update_rules": "to use the parameter 'updaterules'",
"action-dpl_param_delete_rules": "to use the parameter 'deleterules'",
Expand Down
Loading