Skip to content
Repology backend service to update repository and package data
Python TSQL PLpgSQL Other
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
repology
repos.d
sql.d
testdata
.codecov.yml
.gitignore
.travis.yml
COPYING
Makefile
README.md
maintainers.yaml
repology-benchmark.py
repology-dump.py
repology-gensitemap.py
repology-schemacheck.py
repology-test.conf.default
repology-update.py
repology.conf.default
requirements-dev.txt
requirements.txt
setup.cfg
setup.py

README.md

Repology

Build Status Coverage Status codecov Codacy Badge

Repology is a service which monitors a lot of package repositories and other sources and aggregates data on software package versions, reporting new releases and packaging problems.

This repository contains Repology updater code, a backend service which updates the repository information. See also the web application code.

Dependencies

Needed for fetching/parsing repository data:

Development dependencies

Optional, for doing HTML validation when running tests:

Optional, for checking schemas of configuration files:

Optional, for python code linting:

Running

Preparing

Since repology rules live in separate repository you'll need to clone it first. The location may be arbitrary, but rules.d subdirectory is what default configuration file points to, so using it is the most simple way.

git clone https://github.com/repology/repology-rules.git rules.d

Configuration

First, you may need to tune settings which are shared by all repology utilities, such as directory for storing downloaded repository state or DSN (string which specifies how to connect to PostgreSQL database). See repology.conf.default for default values, create repology.conf in the same directory to override them (don't edit repology.conf.default!) or specify path to alternative config via REPOLOGY_SETTINGS environment variable, or override settings via command line.

By default, repology uses ./_state directory for storing raw and parsed repository data and repology/repology/repology database/user/password on localhost.

Creating the database

For the following steps you'll need to set up the database. Ensure PostgreSQL server is up and running, and execute the following commands to create the database for repology:

psql --username postgres -c "CREATE DATABASE repology"
psql --username postgres -c "CREATE USER repology WITH PASSWORD 'repology'"
psql --username postgres -c "GRANT ALL ON DATABASE repology TO repology"
psql --username postgres --dbname repology -c "CREATE EXTENSION pg_trgm"
psql --username postgres --dbname repology -c "CREATE EXTENSION libversion"

in the case you want to change the credentials, don't forget to add actual ones to repology.conf.

Next you can create database schema (tables, indexes etc.) and at the same time test that the database is accessible with the following command:

./repology-update.py --initdb

Fetching/updating repository data

The database is now ready to be filled with data. Typical Repology update cycle consists of multiple steps, but in most cases you'll need to just run all of them:

./repology-update.py --fetch --fetch --parse --database --postupdate
  • --fetch tells the utility to fetch raw repository data (download files, scrape websites, clone git repos) into state directory. Note that it won't refetch (update) data unless it's specified twice.
  • --parse enables parsing downloaded data into internal format which is also saved into state directory.
  • --database pushes processed package data into the database.
  • --postupdate runs additional database processing such as calculating summaries and updating feeds. It's separate from --database because it can be ran in background, parallelly to the following fetch/update cycle.

Documentation

Author

License

GPLv3 or later, see COPYING.

You can’t perform that action at this time.