WikiLambda provides for hosting, and letting users call evaluation, of functions written, managed, and evaluated on a central wiki, Wikifunctions.
From whatever directory you wish to set up your development environment, run:
git clone "https://gerrit.wikimedia.org/r/mediawiki/core" mediawiki
cd mediawiki
git clone --recurse-submodules --remote-submodules https://gerrit.wikimedia.org/r/mediawiki/extensions/WikiLambda extensions/WikiLambda
cd extensions/WikiLambda
npm run local:setupNote that all of the docker container locations mentioned here and in docker-compose.override.yml
have the mediawiki- prefix. This is derived from the name of the directory to which
you have cloned the mediawiki/core repository.
- Bring up a development environment for MediaWiki (e.g. Docker or Vagrant). Be sure to install docker compose v2 instead of v1.
- In your
mediawiki/extensions/subdirectory, clone the extension as follows:git clone --recurse-submodules --remote-submodules https://gerrit.wikimedia.org/r/mediawiki/extensions/WikiLambda - In your
mediawiki/extensions/subdirectory, also clone the WikimediaMessages and UniversalLanguageSelector extensions:The following extensions are also recommended installations in the same directory:git clone https://gerrit.wikimedia.org/r/mediawiki/extensions/WikimediaMessages git clone https://gerrit.wikimedia.org/r/mediawiki/extensions/UniversalLanguageSelectorgit clone https://gerrit.wikimedia.org/r/mediawiki/extensions/EventLogging git clone https://gerrit.wikimedia.org/r/mediawiki/extensions/EventBus - Extend MediaWiki's composer dependencies to use ours by adding a
composer.local.jsonfile in yourmediawiki/directory:{ "extra": { "merge-plugin": { "include": [ "extensions/WikiLambda/composer.json" ] } } } - Run
docker compose exec mediawiki composer updateor similar. - Add the following to your
LocalSettings.phpfile:wfLoadExtension( 'WikiLambda' ); $wgWikiLambdaEnableRepoMode = true; $wgWikiLambdaEnableClientMode = true; wfLoadExtension( 'WikimediaMessages' ); wfLoadExtension( 'UniversalLanguageSelector' ); - Run
php maintenance/run.php createAndPromote --custom-groups functioneer,functionmaintainer --force Admin(ordocker compose exec mediawiki php maintenance/run.php createAndPromote --custom-groups functioneer,functionmaintainer --force Adminif MediaWiki is setup through Docker) to give your Admin user the special rights for creating and editing ZObjects. - Run
php maintenance/run.php update(ordocker compose exec mediawiki php maintenance/run.php updateif MediaWiki is setup through Docker) to provision necessary schemas and initial content (this step could take around 20 minutes).
Done! Navigate to the newly created Z1 page on your wiki to verify that the extension is successfully installed.
WikiLambda uses two back-end services for running user-defined and built-in functions in a secure, scalable environment; a set of "evaluators" that run user-defined native code, and an "orchestrator" that receives execution requests and determines what to run.
On install, the extension will try to use the orchestrator and evaluator services of the Beta Cluster version of Wikifunctions. This default configuration will let you do rudimentary tests with the built-in objects, but not with custom on-wiki content (as they are pointed at the content of Beta Wikifunctions).
You can test your installation by running the PHPUnit test suite as described in the MediaWiki install instructions:
docker compose exec mediawiki composer phpunit:entrypoint -- extensions/WikiLambda/tests/phpunit/integration/ActionAPI/ApiFunctionCallTest.php
If the tests all pass, your installation has successfully called the configured function orchestrator with the calls, executed them, and got the expected results back. Congratulations!
You can evaluate an arbitrary function call by navigating to localhost:8080/wiki/Special:RunFunction, and selecting a function.
If you would like to use your own installation of the function orchestrator and evaluator services, please perform the following additional step:
- Copy the contents of the
servicesblock inmediawiki/extensions/WikiLambda/docker-compose.sample.ymlto the analogousservicesblock in yourmediawiki/docker-compose.override.yml. - If you want to use a different port or name for your orchestrator service, you will need to set the
$wgWikiLambdaOrchestratorLocationconfiguration from the default ofmediawiki_function-orchestrator_1:6254in yourLocalSettings.phpfile, e.g. tomediawiki-function-orchestrator-1:6254you would add:
$wgWikiLambdaOrchestratorLocation = "http://mediawiki-function-orchestrator-1:6254/1/v1/evaluate";
This will provide you with your own orchestrator and evaluator services, pointed at your wiki. You can now use this for local content as well as built-in content.
- If your wiki is not called 'mediawiki-web', e.g. because your checkout of MediaWiki is not in a directory called 'mediawiki', you will need to set
$wgWikiLambdaOrchestratorLocationin yourLocalSettings.phpand make similar edits to theenvironmentvariables you have set in yourmediawiki/docker-compose.override.ymlfile.- To find out the correct name for all the variables, run
docker compose psin your mediawiki directory. The output should be something similar to this:
NAME COMMAND SERVICE STATUS PORTS core-function-evaluator-javascript-1 "node server.js" function-evaluator-javascript running 0.0.0.0:6929->6927/tcp, :::6929->6927/tcp core-function-evaluator-python-1 "node server.js" function-evaluator-python running 0.0.0.0:6928->6927/tcp, :::6928->6927/tcp core-function-orchestrator-1 "node server.js" function-orchestrator running 0.0.0.0:6254->6254/tcp, :::6254->6254/tcp core-mediawiki-1 "/bin/bash /php_entr…" mediawiki running 9000/tcp core-mediawiki-jobrunner-1 "/bin/bash /entrypoi…" mediawiki-jobrunner running core-mediawiki-web-1 "/bin/bash /entrypoi…" mediawiki-web running 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp- Use the
SERVICEname for themediawiki-webcontainer for yourWIKI_API_URLvariable,http://<MEDIAWIKI WEB SERVICE NAME>:8080/w/api.php- E.g.
http://mediawiki-web:8080/w/api.php
- E.g.
- Use the container
NAMEof the javascript evaluator for the javascriptevaluatorUriin theORCHESTRATOR_CONFIGvariable,http://<JAVASCRIPT EVALUATOR CONTAINER NAME>:6927/1/v1/evaluate/- E.g.
http://core-function-evaluator-javascript-1:6927/1/v1/evaluate/
- E.g.
- Use the container
NAMEof the python evaluator for the pythonevaluatorUriin theORCHESTRATOR_CONFIGvariable,http://<PYTHON EVALUATOR CONTAINER NAME>:6927/1/v1/evaluate/- E.g.
http://core-function-evaluator-python-1:6927/1/v1/evaluate/
- E.g.
- Use the container
NAMEof the orchestrator for the$wgWikiLambdaOrchestratorLocationconfig variable inLocalSettings.phpfile, as specified above.- E.g.
http://core-function-orchestrator-1:6254/1/v1/evaluate/
- E.g.
- To find out the correct name for all the variables, run
- If you would like to avoid permissions checks when developing locally, navigate to
localhost:8080/wikiand log in (login: Admin, password: dockerpass) - If you would like to bypass the cache when developing locally, change the signature of the
orchestratefunction inincludes/OrchestratorRequest.php, setting$bypassCache = true:
public function orchestrate( $query, $bypassCache = true ) : string {
If you would instead like to develop changes to the function orchestrator or evaluators, you will need to use a locally-built version of the services. To do this for the orchestrator:
- In a directory outside of your MediaWiki checkout, clone the services via
git clone --recurse-submodules --remote-submodules https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator. - From the root of your function-orchestrator installation, run
docker build -f .pipeline/blubber.yaml --target development -t local-orchestrator . - Alter
mediawiki/docker-compose.override.ymlto comment outimage: docker-registry...in thefunction-orchestratorservice stanza and uncomment theimage: local-orchestrator:latestline instead.
To do this for the evaluator:
- In a directory outside of your MediaWiki checkout, clone the services via
git clone --recurse-submodules --remote-submodules https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator. - From the root of your function-evaluator installation:
- To build the javascript evaluator image, run:
docker build -f .pipeline/blubber.yaml --target development-javascript-all-wasm -t local-evaluator-js . - To build the python evaluator image, run:
docker build -f .pipeline/blubber.yaml --target development-python3-all-wasm -t local-evaluator-py .
- To build the javascript evaluator image, run:
- Alter
mediawiki/docker-compose.override.ymlto comment outimage: docker-registry...in thefunction-evaluatorjavascript and python services and uncomment theimage: local-evaluator-js:latestandimage: local-evaluator-py:latestlines instead.
To run the extension PHPunit tests, from the MediaWiki installation directory, run:
docker compose exec mediawiki composer phpunit:entrypoint extensions/WikiLambda/tests/phpunit/
To run the Jest unit and integration tests, from the WikiLambda directory, do:
# Install npm dependencies
npm ci
# Run tests
npm test
# Run linter
npm run lint:fix
# Run unit tests
npm run test:unit
From your extensions/WikiLambda directory run:
npm run local:seleniumA set of Selenium tests, used to run end-to-end tests of the application, is available within the project. The tests require an environment with specific versions of things to run, and so it is suggested you use "fresh-node" to run them locally without the need to modify your personal environment.
The tests need a specific set of environment variable to be available. Please see the following list on how to set this https://www.mediawiki.org/wiki/Selenium/How-to/Set_environment_variables
For information on how to run fresh-node and how to get started, see the following documentation: https://www.mediawiki.org/wiki/Selenium/Getting_Started/Run_tests_using_Fresh
After the environment variable and fresh node are both set locally, run the following commands:
// go to your WikiLambda extension
cd path/to/WikiLambda # you can start in the extension, no need for core
// Initialize Fresh node
fresh-node -env -net # you can start fresh in the extension folder, no need for core
// Set the variable inline (unless you are loading them from a .env file)
export MW_SERVER=http://localhost:8080
export MW_SCRIPT_PATH=/w
export MEDIAWIKI_USER=Admin
export MEDIAWIKI_PASSWORD=dockerpass
# for additional debugging, you can enable video recording
# see https://www.mediawiki.org/wiki/Selenium/How-to/Record_videos_of_test_runs
export DISPLAY=:1
Xvfb "$DISPLAY" -screen 0 1280x1024x24 &
# run the browser tests
npm run browser-test
NOTE: the tests will produce some snapshot after completition (both on failure and success). This can be found on "extensions/WikiLambda/tests/selenium/log"
WikiLambda uses PoolCounter to limit the number of concurrent function calls a user may have in flight at any given time. In order to set the concurrency limit, you need to add configuration for a WikiLambdaFunctionCall pool to $wgPoolCounterConf in LocalSettings.php.
The example below allows users to have at most two functions executing at a given time, placing any function calls that exceed the concurrency limit in a queue:
$wgPoolCounterConf = [
'WikiLambdaFunctionCall' => [
'class' => MediaWiki\PoolCounter\PoolCounterClient::class,
'timeout' => 1, // wait timeout in seconds
'workers' => 2, // maximum number of active threads in each pool
'maxqueue' => 5, // maximum number of total threads in each pool
]
];
WikiLambda provides the {{#function:…}} parser function, which lets you embed function calls inside wikitext documents which are then resolved at parse time.
It is currently very limited, allowing only a single direct function call to a function which both takes only strings as its inputs and emits a string as its output.
To use, simply edit any wikitext page. The target function is the zeroth parameter (named by ZID), e.g.:
{{#function:Z12345}}
If your function takes parameters, they can be passed in pipe-delimited, e.g.:
{{#function:Z12345|Hello|world|!}}
Much further functionality is to come.