Needed for the core:
Python module pyyaml
Python module Jinja2
Python module psycopg
PostgreSQL extension libversion
Needed for fetching/parsing repository data:
- Python module requests
- Python module rubymarshal
- Python module lxml
- Python module rpm (comes with RPM package manager)
- Python module jsonslicer
- Python module pyparsing
- Python module sqlite3
- tclsh and tcllib
Needed for web-application:
Optional, for doing HTML validation when running tests:
Optional, for checking schemas of configuration files:
- Python module voluptuous
Optional, for python code linting:
- Python module flake8
- Python module flake8-builtins
- Python module flake8-import-order
- Python module flake8-quotes
- Python module mypy
Since repology rules live in separate repository you'll need to
clone it first. The location may be arbitrary, but
subdirectory is what default configuration file points to, so
using it is the most simple way.
git clone https://github.com/repology/repology-rules.git rules.d
First, you may need to tune settings which are shared by all repology
utilities, such as directory for storing downloaded repository state
or DSN (string which specifies how to connect to PostgreSQL database).
repology.conf.default for default values, create
in the same directory to override them (don't edit
or specify path to alternative config via
environment variable, or override settings via command line.
By default, repology uses
./_state directory for storing raw and parsed
repository data and
Creating the database
For the following steps you'll need to set up the database. Ensure PostgreSQL server is up and running, and execute the following commands to create the database for repology:
psql --username postgres -c "CREATE DATABASE repology" psql --username postgres -c "CREATE USER repology WITH PASSWORD 'repology'" psql --username postgres -c "GRANT ALL ON DATABASE repology TO repology" psql --username postgres --dbname repology -c "CREATE EXTENSION pg_trgm" psql --username postgres --dbname repology -c "CREATE EXTENSION libversion"
in the case you want to change the credentials, don't forget to add
actual ones to
Next you can create database schema (tables, indexes etc.) and at the same time test that the database is accessible with the following command:
Fetching/updating repository data
The database is now ready to be filled with data. Typical Repology update cycle consists of multiple steps, but in most cases you'll need to just run all of them:
./repology-update.py --fetch --fetch --parse --database --postupdate
--fetchtells the utility to fetch raw repository data (download files, scrape websites, clone git repos) into state directory. Note that it won't refetch (update) data unless it's specified twice.
--parseenables parsing downloaded data into internal format which is also saved into state directory.
--databasepushes processed package data into the database.
--postupdateruns additional database processing such as calculating summaries and updating feeds. It's separate from
--databasebecause it can be ran in background, parallelly to the following fetch/update cycle.
Running the webapp
Repology is a flask application, so as long as you've set up database and configuration, you may just run the application locally:
and point your browser to http://127.0.0.1:5000/ to view the site. This should be enough for personal use, experiments and testing.
Alternatively, you may deploy the application in numerous ways, including mod_wsgi, uwsgi, fastcgi and plain CGI application. See flask documentation on deployment for more info.
For instance, you can deploy with
uwsgi with the following command
uwsgi --mount /=repology-app:app --pythonpath=<path-to-repology-checkout>
See separate repository https://github.com/repology/repology-linkchecker