Wikipedia 1.0 engine
This directory contains the code of Wikipedia 1.0 supporting software. More information about the Wikipedia 1.0 project can be found on the Wikipedia in English.
wp1 subdirectory includes code for updating the
database, specifically the
ratings table (but also other
tables). The library code itself isn't directly runnable, but instead
is loaded and run through scripts in the top level directory, such as
requirements.txt is a list of python dependencies in pip format that
need to be installed in a virtual env in order to run the code in this
wiki_test.*.sql files are rough
approximations of the schemas of the two databases that the library
interfaces with. They are used for unit testing.
This code is targetted to and tested on Python 3. It may work intermittently with Python 2.7, but that is not recommended.
Creating your virtualenv
With Python 3, creating a virtualenv is a single easy command. It is recommmended that you run this command in the top level directory:
python3 -m venv venv
Activating your virtualenv
To activate your virtualenv, run:
You should see your prompt change, with a
(venv) appended to the front.
To install the requirements, make sure you are in your virtualenv as explained above, then use the following command:
pip3 install -r requirements.txt
Running the tests
The tests expect a localhost MySQL instance on the default port, with
a user of 'root' and no password. You also need two databases:
enwikip_test. They can use default settings and be
If you have that, and you've already installed the requirements above, you should be able to simply run:
From this directory, to run the tests. If you'd like to use a
different MySQL user or non-default password for the tests, simply
Populating the credentials module
The script needs access to the enwiki_p replica database, as well as
its own toolsdb application database. The credentials for these are on
toolforge in the replica.my.cnf file in the home directory. They need
to be formatted in a way that is consumable by the library and
pymysql. Look at
credentials.py.example and create a copy called
credentials.py with the relevant information filled in. You will
also need english wikipedia API credentials.
Running an application script
To run a script, you can simply run:
From the appropriate directory. However, this assumes that you have followed the steps above and are inside the appropriate virtualenv. For far-flung invocations of this script, you will need to reference the python3 wrapper script in the virtualenv:
This project is configured to use git pre-commit hooks managed by the
pre-commit (website). Pre-
commit checks let us ensure that the code is properly formatted with
yapf amongst other things.
If you've installed the requirements for this repository, the pre-commit binary should be available to you. To install the hooks, use:
Then, when you try to commit a change that would fail pre-commit, you get:
(venv) host:wikimedia_wp1_bot audiodude$ git commit -am 'Test commit' Trim Trailing Whitespace.................................................Passed Fix End of Files.........................................................Passed yapf.....................................................................Failed hookid: yapf
From there, the pre-commit hook will have modified and thus unstaged some or all of the files you were trying to commit. Look through the changes to make sure they are sane, then re-add them with git add, before trying your commit again.
GPLv2 or later, see LICENSE for more details.