start of work to add spack. It was difficult to develop, so I am also…

… add a simple docker compose setup

with a database and application container to handle installing dependencies
and more easily running a development server

Signed-off-by: vsoch <vsoch@users.noreply.github.com>

Dockerfile

@@ -0,0 +1,32 @@
+FROM python:3.9
+
+ENV DEBIAN_FRONTEND=noninteractive
+RUN apt-get update && apt-get install -y \
+      python3-pip \
+      postgresql \
+      postgresql-contrib \
+      pkg-config \
+      wget \
+      yajl-tools \
+      libyajl-dev \
+      cmake \
+      libpq-dev
+
+WORKDIR /opt
+RUN wget https://github.com/repology/libversion/archive/refs/tags/3.0.1.tar.gz && \
+    tar -xzvf 3.0.1.tar.gz && \
+    cd libversion-3.0.1 && \
+    mkdir build && \
+    cmake . && \
+    cd build && \
+    cmake --build ../ && \
+    cd ../ && \
+    make install && \
+    ldconfig
+
+WORKDIR /code
+COPY . /code
+RUN pip install -r requirements.txt && pip install -r requirements-dev.txt && \
+    # import rpm required, not clear which is needed
+    pip install rpm
+

README.md

@@ -81,7 +81,11 @@ on localhost.
 
 ### Creating the database
 
-For the following steps you'll need to set up the database. Ensure
+For the following steps you'll need to set up the database. 
+
+#### Local Database
+
+You can optionally deploy your own database, and first ensure
 PostgreSQL server is up and running, and execute the following
 commands to create the database for repology:
 
@@ -96,11 +100,61 @@ psql --username postgres --dbname repology -c "CREATE EXTENSION libversion"
 in the case you want to change the credentials, don't forget to add
 actual ones to `repology.conf`.
 
+#### Docker
+
+In the case that you want to create the database with Docker, a docker setup
+is provided to do so.
+
+ - [Dockerfile](Dockerfile): builds a base image with dependencies to run repology
+ - [docker/Dockerfile.db](docker/Dockerfile.db) creates the database with the above credentials
+ - [docker-compose.yml](docker-compose.yml): provides orchestration between the two.
+
+For this approach, both [Docker](https://docs.docker.com/get-docker/) and [docker-compose](https://docs.docker.com/compose/install/) are required to be installed.
+To build the containers:
+
+```bash
+$ docker-compose build
+```
+
+And bring them up:
+
+```bash
+$ docker-compose up -d
+```
+
+Check the status of containers:
+
+```shell
+$ docker-compose ps
+           Name                          Command              State    Ports  
+------------------------------------------------------------------------------
+repology-updater_db_1         docker-entrypoint.sh postgres   Up      5432/tcp
+repology-updater_repology_1   tail -f /dev/null               Up              
+```
+
+You will then want to shell into the repology container to interact with the
+database, discussed in the next step.
+
+```bash
+$ docker exec -it repology-updater_repology_1 bash
+```
+
+#### Initialize database
+
 Next you can create database schema (tables, indexes etc.) and at the
 same time test that the database is accessible with the following command:
 
 ```shell
+# If you are outside the container and the database is on localhost
 ./repology-update.py --initdb
+
+# Inside the container with the database at host "db"
+./repology-update.py --initdb --dsn "host=db dbname=repology user=repology password=repology"
+```
+```
+Aug 11 03:48:47 (re)initializing database schema
+Aug 11 03:48:47   committing changes
+Aug 11 03:48:47 total time taken: 0.22 seconds
 ```
 
 ### Fetching/updating repository data
@@ -110,7 +164,11 @@ update cycle consists of multiple steps, but in most cases you'll need
 to just run all of them:
 
 ```shell
+# localhost database
 ./repology-update.py --fetch --fetch --parse --database --postupdate
+
+# docker database
+./repology-update.py --fetch --fetch --parse --database --postupdate --dsn "host=db dbname=repology user=repology password=repology"
 ```
 
   - `--fetch` tells the utility to fetch raw repository data

docker-compose.yml

@@ -0,0 +1,24 @@
+version: "3.9"
+services:
+  db:
+    # Important! Do not use this in production without changing credentials
+    env_file:
+      - postgres.env
+
+    restart: always
+    build:
+      context: docker/
+      dockerfile: Dockerfile.db
+
+
+  repology:
+    entrypoint: ["tail", "-f", "/dev/null"]
+    build:
+      context: .
+      dockerfile: Dockerfile
+
+    # For development so changes to local files can be propograted with restart
+    volumes:
+      - ./:/code
+    links:
+      - db

docker/Dockerfile.db

@@ -0,0 +1,39 @@
+FROM postgres:13.3
+
+# docker build -t repology-db .
+
+# NOTE: this is for development only! These are the development credentials
+# defined in the README.md and default configuration file. You should
+# absolutely not use this in a production setting.
+
+RUN apt-get update && apt-get install -y git \
+      build-essential \
+      postgresql-server-dev-13 \
+      wget \
+      pkg-config \
+      cmake
+
+WORKDIR /opt
+RUN wget https://github.com/repology/libversion/archive/refs/tags/3.0.1.tar.gz && \
+    tar -xzvf 3.0.1.tar.gz && \
+    cd libversion-3.0.1 && \
+    mkdir build && \
+    cmake . && \
+    cd build && \
+    cmake --build ../ && \
+    cd ../ && \
+    make install && \
+    ldconfig
+
+WORKDIR /opt
+RUN git clone https://github.com/repology/postgresql-libversion && \
+    cd postgresql-libversion && \
+    make && \
+    make install && \
+    ldconfig
+
+WORKDIR /docker-entrypoint-initdb.d
+RUN echo 'psql --username ${POSTGRES_USER} --dbname ${POSTGRES_DB} -c "CREATE EXTENSION pg_trgm"' > extension1.sh && \
+    echo 'psql --username ${POSTGRES_USER} --dbname ${POSTGRES_DB} -c "CREATE EXTENSION libversion"' > extension2.sh && \
+    chmod +x extension*.sh
+

postgres.env

@@ -0,0 +1,3 @@
+POSTGRES_DB=repology
+POSTGRES_USER=repology
+POSTGRES_PASSWORD=repology

repology-schemacheck.py

@@ -106,6 +106,7 @@
     'scoop',
     'sisyphus',
     'slackbuilds',
+    'spack',
     'tinycore',
     'slackware',
     'slitaz',

repology/packagemaker/names.py

@@ -157,6 +157,8 @@ class NameType:
 
     STACKAGE_NAME: ClassVar[int] = GENERIC_GEN_NAME
 
+    SPACK_NAME: ClassVar[int] = GENERIC_GEN_NAME
+
     DISTROWATCH_NAME: ClassVar[int] = GENERIC_GEN_NAME
 
     FRESHCODE_NAME: ClassVar[int] = GENERIC_GEN_NAME

repology/parsers/parsers/spack.py

@@ -0,0 +1,43 @@
+# Copyright (C) 2021 Dmitry Marakasov <amdmi3@amdmi3.ru>
+#
+# This file is part of repology
+#
+# repology is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# repology is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with repology.  If not, see <http://www.gnu.org/licenses/>.
+
+from typing import Iterable
+
+from repology.packagemaker import NameType, PackageFactory, PackageMaker
+from repology.parsers import Parser
+from repology.parsers.json import iter_json_list
+from repology.transformer import PackageTransformer
+import json
+
+class SpackJsonParser(Parser):
+    def iter_parse(self, path: str, factory: PackageFactory, transformer: PackageTransformer) -> Iterable[PackageMaker]:
+        with open(path, 'r') as fd:
+            data = json.loads(fd.read())
+        for package_name, packagedata in data['packages'].items():
+            with factory.begin() as pkg:
+
+                # I'm not sure what this is? It's a number?
+                pkg.add_name(package_name, NameType.SPACK)
+                pkg.set_version(packagedata['version'])
+                # We don't currently have good keywords, could in the future!
+                # pkg.add_categories(packagedata['keywords'])
+                pkg.add_homepages(packagedata['homepages'])
+                pkg.add_downloads(packagedata['downloads'])
+                pkg.set_summary(packagedata['summary'])
+                pkg.add_maintainers(packagedata['maintainers'])
+                pkg.set_extra_field('dependencies', packagedata['dependencies'])
+                yield pkg

repos.d/spack.yaml

@@ -0,0 +1,33 @@
+###########################################################################
+# Spack
+###########################################################################
+- name: spack
+  type: repository
+  desc: Spack
+  family: spack
+  color: "0f3a80"
+  minpackages: 5000
+  default_maintainer: fallback-mnt-spack@repology
+  sources:
+    - name: repology.json
+      fetcher:
+        class: FileFetcher
+        url: https://raw.githubusercontent.com/spack/packages/main/data/repology.json
+        allow_zero_size: false
+      parser:
+        class: SpackJsonParser
+  repolinks:
+    - desc: Spack Catalog
+      url: https://spack.github.io/packages/
+    - desc: Spack GitHub repository
+      url: https://github.com/spack/spack
+  packagelinks:
+    - type: PACKAGE_HOMEPAGE
+      url: 'https://spack.github.io/packages/package.html?name={name}'
+    - type: PACKAGE_SOURCES
+      url: 'https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/{name}/package.py'
+    - type: PACKAGE_RECIPE
+      url: 'https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/{name}/package.py'
+    - type: PACKAGE_RECIPE_RAW
+      url: 'https://raw.githubusercontent.com/spack/spack/develop/var/spack/repos/builtin/packages/{name}/package.py'
+  tags: [ all, production, spack ]