From 9bf55def3b33cd119051e06e680f2a82a58ad254 Mon Sep 17 00:00:00 2001 From: Olaf Alders Date: Sun, 1 Sep 2024 19:47:56 -0400 Subject: [PATCH 1/2] Tidy README.md --- README.md | 209 ++++++++++++++++++++++++++++-------------------------- 1 file changed, 108 insertions(+), 101 deletions(-) diff --git a/README.md b/README.md index 31cd866..61df61a 100644 --- a/README.md +++ b/README.md @@ -6,41 +6,41 @@ -* [Running the MetaCPAN stack with Docker (via Docker Compose)](#running-the-metacpan-stack-with-docker-via-docker-compose) -* [Quick Start](#quick-start) -* [Working with Containers](#working-with-containers) - * [Building Containers](#building-containers) - * [Accessing Containers](#accessing-containers) - * [Accessing Services](#accessing-services) - * [`web`](#web) - * [`api`](#api) - * [`Elasticsearch`](#elasticsearch) - * [`PostgreSQL`](#postgresql) - * [`grep`](#grep) -* [System architecture](#system-architecture) - * [The `bin/metacpan-docker` script](#the-binmetacpan-docker-script) - * [`bin/metacpan init`](#binmetacpan-init) - * [`bin/metacpan localapi`](#binmetacpan-localapi) - * [`bin/metacpan-docker pull`](#binmetacpan-docker-pull) - * [`bin/metacpan-docker reset`](#binmetacpan-docker-reset) - * [`bin/metacpan-docker` build/up/down/start/stop/run/ps/top...](#binmetacpan-docker-buildupdownstartstoprunpstop) - * [Services](#services) - * [`web`](#web-1) - * [`api`](#api-1) - * [`grep`](#grep-1) - * [Setting up a partial CPAN in the `api` service](#setting-up-a-partial-cpan-in-the-api-service) - * [Bootstrapping the `elasticsearch` indices](#bootstrapping-the-elasticsearch-indices) - * [Putting the above all together](#putting-the-above-all-together) - * [elasticsearch and elasticsearch_test](#elasticsearch-and-elasticsearch_test) -* [Tips and tricks](#tips-and-tricks) - * [Running your own miniCPAN inside metacpan-docker](#running-your-own-minicpan-inside-metacpan-docker) - * [Running tests](#running-tests) - * [Updating Carton dependencies](#updating-carton-dependencies) - * [Updating the git repositories](#updating-the-git-repositories) - * [Running Kibana to peek into Elasticsearch data](#running-kibana-to-peek-into-elasticsearch-data) -* [Peeking Inside the Container](#peeking-inside-the-container) -* [To Do](#to-do) -* [See also](#see-also) +- [Running the MetaCPAN stack with Docker (via Docker Compose)](#running-the-metacpan-stack-with-docker-via-docker-compose) +- [Quick Start](#quick-start) +- [Working with Containers](#working-with-containers) + - [Building Containers](#building-containers) + - [Accessing Containers](#accessing-containers) + - [Accessing Services](#accessing-services) + - [`web`](#web) + - [`api`](#api) + - [`Elasticsearch`](#elasticsearch) + - [`PostgreSQL`](#postgresql) + - [`grep`](#grep) +- [System architecture](#system-architecture) + - [The `bin/metacpan-docker` script](#the-binmetacpan-docker-script) + - [`bin/metacpan init`](#binmetacpan-init) + - [`bin/metacpan localapi`](#binmetacpan-localapi) + - [`bin/metacpan-docker pull`](#binmetacpan-docker-pull) + - [`bin/metacpan-docker reset`](#binmetacpan-docker-reset) + - [`bin/metacpan-docker` build/up/down/start/stop/run/ps/top...](#binmetacpan-docker-buildupdownstartstoprunpstop) + - [Services](#services) + - [`web`](#web-1) + - [`api`](#api-1) + - [`grep`](#grep-1) + - [Setting up a partial CPAN in the `api` service](#setting-up-a-partial-cpan-in-the-api-service) + - [Bootstrapping the `elasticsearch` indices](#bootstrapping-the-elasticsearch-indices) + - [Putting the above all together](#putting-the-above-all-together) + - [elasticsearch and elasticsearch_test](#elasticsearch-and-elasticsearch_test) +- [Tips and tricks](#tips-and-tricks) + - [Running your own miniCPAN inside metacpan-docker](#running-your-own-minicpan-inside-metacpan-docker) + - [Running tests](#running-tests) + - [Updating Carton dependencies](#updating-carton-dependencies) + - [Updating the git repositories](#updating-the-git-repositories) + - [Running Kibana to peek into Elasticsearch data](#running-kibana-to-peek-into-elasticsearch-data) +- [Peeking Inside the Container](#peeking-inside-the-container) +- [To Do](#to-do) +- [See also](#see-also) @@ -67,16 +67,16 @@ access docker follow the This document assumes the post-installation steps have been followed for the current user. -You will also need Docker buildx, and to enable Docker BuildKit. They should -be set up by default when using Docker Desktop, but on Linux you may need to +You will also need Docker buildx, and to enable Docker BuildKit. They should be +set up by default when using Docker Desktop, but on Linux you may need to install them. buildx is the `docker-buildx` package on Debian based systems. Docker BuildKit can be enabled by following the [Getting Started](https://docs.docker.com/build/buildkit/#getting-started) instructions. -If you are running a Mac ARM64 system, you will need to manually tell docker -to use the x86_64 version of Elasticsearch 2.4. This can be done by running -the command: +If you are running a Mac ARM64 system, you will need to manually tell docker to +use the x86_64 version of Elasticsearch 2.4. This can be done by running the +command: docker pull elasticsearch:2.4 --platform=linux/x86_64 @@ -92,10 +92,10 @@ Then, clone this repo and set up the environment: The `bin/metacpan-docker init` command clones the source repositories for: -* `metacpan-web` -* `metacpan-api` -* `metacpan-grep-front-end` -* `metacpan-cpan-extracted-lite` +- `metacpan-web` +- `metacpan-api` +- `metacpan-grep-front-end` +- `metacpan-cpan-extracted-lite` These repositories are automatically mounted into the appropriate docker containers allowing the developer to use their preferred tools to work with the @@ -104,8 +104,8 @@ source code. The `docker-compose up` command on its own will bring up the entire stack in the foreground (logs will be displayed). -The `docker-compose up` command will also fetch the official container images from -[MetaCPAN Docker Hub](https://cloud.docker.com/u/metacpan/repository/list) +The `docker-compose up` command will also fetch the official container images +from [MetaCPAN Docker Hub](https://cloud.docker.com/u/metacpan/repository/list) repositories. This will build the Docker containers for MetaCPAN, PostgreSQL and Elasticsearch @@ -118,7 +118,7 @@ following command in a separate terminal to get yourself up to speed: docker-compose exec api index-cpan.sh This will prompt you to confirm removing old indices and setting up mappings on -the Elasticsearch service (say `YES`). It will then proceed to rsync a partial +the Elasticsearch service (say `YES`). It will then proceed to rsync a partial CPAN in `/CPAN` for its metadata to be imported. Once the above is done, you should be able to see your local partial CPAN data @@ -147,8 +147,8 @@ For further details, read on! ### Building Containers -You can (re)build arbitrary containers. For instance, if you want to rebuild -the `api` container: +You can (re)build arbitrary containers. For instance, if you want to rebuild the +`api` container: docker-compose build api @@ -176,12 +176,12 @@ To access the `psql` command line client in the PostgreSQL container: Each container is responsible for a different service. Some of these services are available in the developer environment via ports on the host system. -We are using [traefik][13] to manage the traffic between services. -The current configuration is: +We are using [traefik][13] to manage the traffic between services. The current +configuration is: -* api: [http://api.metacpan.localhost](http://api.metacpan.localhost) -* web: [http://web.metacpan.localhost](http://web.metacpan.localhost) -* grep: [http://grep.metacpan.localhost](http://grep.metacpan.localhost) +- api: [http://api.metacpan.localhost](http://api.metacpan.localhost) +- web: [http://web.metacpan.localhost](http://web.metacpan.localhost) +- grep: [http://grep.metacpan.localhost](http://grep.metacpan.localhost) In order to access to the localhost subdomains, you probably have to manually add these entries in you `/etc/hosts` file. @@ -202,13 +202,13 @@ You can access the dashboard configuration via: The local instance of the web front end is accessible via: -* [http://localhost:5001](http://localhost:5001) -* [http://web.metacpan.localhost](http://web.metacpan.localhost) +- [http://localhost:5001](http://localhost:5001) +- [http://web.metacpan.localhost](http://web.metacpan.localhost) #### `api` -* [http://localhost:5000](http://localhost:5000) -* [http://api.metacpan.localhost](http://api.metacpan.localhost) +- [http://localhost:5000](http://localhost:5000) +- [http://api.metacpan.localhost](http://api.metacpan.localhost) #### `Elasticsearch` @@ -231,7 +231,7 @@ The PostgreSQL service by default is only accessible from other containers. The grep metacpan front end is accessible via: -* [http://grep.metacpan.localhost](http://grep.metacpan.localhost) +- [http://grep.metacpan.localhost](http://grep.metacpan.localhost) Note: this is using a smaller, frozen version of `metacpan-cpan-extracted` via [metacpan-cpan-extracted-lite](https://github.com/metacpan/metacpan-cpan-extracted-lite). @@ -240,28 +240,29 @@ Note: this is using a smaller, frozen version of `metacpan-cpan-extracted` via The system consists of several services that live in docker containers: -* `web` — the web interface on [http://localhost:5001](http://localhost:5001) -* `api` — the main server on [http://localhost:5000](http://localhost:5000) -* `api_test` — the api server for running tests via `prove` -* `elasticsearch` — database for `api` -* `elasticsearch_test` — database for `api_test` -* `pgdb` - PostgreSQL database container -* `logspout` - Docker log interface to [honeycomb.io](https://honeycomb.io) -* `grep` - the web interface for grep.metacpan on [http://localhost:3001](http://localhost:3001) +- `web` — the web interface on [http://localhost:5001](http://localhost:5001) +- `api` — the main server on [http://localhost:5000](http://localhost:5000) +- `api_test` — the api server for running tests via `prove` +- `elasticsearch` — database for `api` +- `elasticsearch_test` — database for `api_test` +- `pgdb` - PostgreSQL database container +- `logspout` - Docker log interface to [honeycomb.io](https://honeycomb.io) +- `grep` - the web interface for grep.metacpan on + [http://localhost:3001](http://localhost:3001) These services use one or more Docker volumes: -* `metacpan_cpan`: holds the CPAN archive, mounted in `/CPAN` -* `metacpan_elasticsearch`: holds the Elasticsearch database files -* `metacpan_elasticsearch_test`: holds the Elasticsearch test database files -* `metacpan_api_carton` and `metacpan_web_carton`: holds the dependencies +- `metacpan_cpan`: holds the CPAN archive, mounted in `/CPAN` +- `metacpan_elasticsearch`: holds the Elasticsearch database files +- `metacpan_elasticsearch_test`: holds the Elasticsearch test database files +- `metacpan_api_carton` and `metacpan_web_carton`: holds the dependencies installed by [Carton][4] for the `api` and `web` services, respectively; mounted on `/carton` instead of `local`, to prevent clashing with the host user's Carton -* `metacpan_git_shared`: points to the git repo containing all extracted CPAN - versions. This is mounted in `/shared/metacpan_git`. - This can be either `metacpan-cpan-extracted` or `metacpan-cpan-extracted-lite`. - The volume is bound to the local repo at `${PWD}/src/metacpan-cpan-extracted`. +- `metacpan_git_shared`: points to the git repo containing all extracted CPAN + versions. This is mounted in `/shared/metacpan_git`. This can be either + `metacpan-cpan-extracted` or `metacpan-cpan-extracted-lite`. The volume is + bound to the local repo at `${PWD}/src/metacpan-cpan-extracted`. [4]: https://metacpan.org/pod/Carton @@ -283,7 +284,8 @@ The `init` subcommand basically clones the [metacpan-api][5] and them, in preparation for future `docker-compose` or `bin/metacpan-docker localapi` commands. -It also clones the `metacpan-grep-front-end` and `metacpan-cpan-extracted-lite` repositories. +It also clones the `metacpan-grep-front-end` and `metacpan-cpan-extracted-lite` +repositories. [5]: https://github.com/metacpan/metacpan-api [6]: https://github.com/metacpan/metacpan-web @@ -299,22 +301,22 @@ default `docker-compose.yml`. #### `bin/metacpan-docker pull` -This is used to update all the git repository in `src/*`. -This will stay on your current local branch. +This is used to update all the git repository in `src/*`. This will stay on your +current local branch. #### `bin/metacpan-docker reset` -This is used to reset all the git repositories in `src/*` to their -latest version on `upstream/master`. -This will fail if you have some uncommitted local changes. -You should then commit or cancel your changes before re-running the command. +This is used to reset all the git repositories in `src/*` to their latest +version on `upstream/master`. This will fail if you have some uncommitted local +changes. You should then commit or cancel your changes before re-running the +command. #### `bin/metacpan-docker` build/up/down/start/stop/run/ps/top... -As noted earlier, `bin/metacpan-docker` is a thin wrapper around `docker-compose`, -so commands like `up`, `down`, and `run` will work as expected from -`docker-compose`. See the [docker-compose docs][7] for an overview of available -commands. +As noted earlier, `bin/metacpan-docker` is a thin wrapper around +`docker-compose`, so commands like `up`, `down`, and `run` will work as expected +from `docker-compose`. See the [docker-compose docs][7] for an overview of +available commands. [7]: https://docs.docker.com/compose/reference/overview/#command-options-overview-and-help @@ -326,8 +328,9 @@ commands. The `web` service is a checkout of `metacpan-web`, built as a Docker image. Running this service alone is enough if you want to just hack on the frontend, since by default the service is configured to talk to -[https://fastapi.metacpan.org](https://fastapi.metacpan.org) for its backend; if this is what you want, then you -can simply invoke `docker-compose up` or `docker-compose up web`. +[https://fastapi.metacpan.org](https://fastapi.metacpan.org) for its backend; if +this is what you want, then you can simply invoke `docker-compose up` or +`docker-compose up web`. #### `api` @@ -340,8 +343,9 @@ additional commands in a separate terminal once #### `grep` -The `grep` service is a checkout of `metacpan-grep-front-end`, built as a Docker image. Note that this is using the `metacpan_git_shared` volume, which requires the git repo for -`metacpan-cpan-extracted` which can be initialized by running: +The `grep` service is a checkout of `metacpan-grep-front-end`, built as a Docker +image. Note that this is using the `metacpan_git_shared` volume, which requires +the git repo for `metacpan-cpan-extracted` which can be initialized by running: ./bin/metacpan-docker init @@ -351,9 +355,9 @@ Running bin/metacpan-docker localapi exec api partial-cpan-mirror.sh -will `rsync` modules from selected CPAN authors, plus the package and author indices, -into the `api` service's `/CPAN` directory. This is nearly equivalent to the -same script in the (now deprecated) [metacpan-developer][8] repository. +will `rsync` modules from selected CPAN authors, plus the package and author +indices, into the `api` service's `/CPAN` directory. This is nearly equivalent +to the same script in the (now deprecated) [metacpan-developer][8] repository. [8]: https://github.com/metacpan/metacpan-developer ##### Bootstrapping the `elasticsearch` indices @@ -380,7 +384,9 @@ instead will set it all up for you. The `elasticsearch` and `elasticsearch_test` services use the official [Elasticsearch Docker image][9], configured with settings and scripts taken from -the [metacpan-puppet][10] repository. The `api` service depends on the `elasticsearch` service and the `api_test` service depends on the `elasticsearch_test` services. +the [metacpan-puppet][10] repository. The `api` service depends on the +`elasticsearch` service and the `api_test` service depends on the +`elasticsearch_test` services. [9]: https://store.docker.com/images/elasticsearch [10]: https://github.com/metacpan/metacpan-puppet @@ -445,16 +451,17 @@ You can use `bin/metacpan-docker pull` to update all `src/*` directories. ### Running Kibana to peek into Elasticsearch data By default, the `docker-compose.localapi.yml` configures the `elasticsearch` -service to listen on the Docker host at [http://localhost:9200](http://localhost:9200), and is also -accessible via the Docker `default` network address of [http://172.17.0.1:9200](http://172.17.0.1:9200); -you can inspect it via simple `curl` or `wget` requests, or use a [Kibana][12] -container, e.g. +service to listen on the Docker host at +[http://localhost:9200](http://localhost:9200), and is also accessible via the +Docker `default` network address of +[http://172.17.0.1:9200](http://172.17.0.1:9200); you can inspect it via simple +`curl` or `wget` requests, or use a [Kibana][12] container, e.g. docker run --rm -p 5601:5601 -e ELASTICSEARCH_URL=http://172.17.0.1:9200 -it kibana:4.6 -Running the above will provide a Kibana container at [http://localhost:5601](http://localhost:5601), -which you can configure to have it read the `cpan*` index in the `elasticsearch` -service. +Running the above will provide a Kibana container at +[http://localhost:5601](http://localhost:5601), which you can configure to have +it read the `cpan*` index in the `elasticsearch` service. It is also certainly possible to run Kibana as part of the compose setup, by configuring e.g. a `kibana` service. From 4bb6cc2a881c5e0bffbae7b9c79dd9c1d051c1e9 Mon Sep 17 00:00:00 2001 From: Olaf Alders Date: Sun, 1 Sep 2024 20:25:50 -0400 Subject: [PATCH 2/2] s/docker-compose/docker compose/ in README --- README.md | 54 +++++++++++++++++++++++++++--------------------------- 1 file changed, 27 insertions(+), 27 deletions(-) diff --git a/README.md b/README.md index 61df61a..6d6daee 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ [![CircleCI](https://circleci.com/gh/metacpan/metacpan-docker.svg?style=svg)](https://circleci.com/gh/metacpan/metacpan-docker) -![docker-compose up](https://github.com/metacpan/metacpan-docker/workflows/docker-compose%20up/badge.svg?branch=master) +![docker compose up](https://github.com/metacpan/metacpan-docker/workflows/docker-compose%20up/badge.svg?branch=master) @@ -80,8 +80,8 @@ command: docker pull elasticsearch:2.4 --platform=linux/x86_64 -It is highly recommended that you alias `docker-compose` to `fig` (its original -name) and use it wherever `docker-compose` is used. You are going to have to +It is highly recommended that you alias `docker compose` to `fig` (its original +name) and use it wherever `docker compose` is used. You are going to have to type this command a lot. Then, clone this repo and set up the environment: @@ -101,10 +101,10 @@ These repositories are automatically mounted into the appropriate docker containers allowing the developer to use their preferred tools to work with the source code. -The `docker-compose up` command on its own will bring up the entire stack in the +The `docker compose up` command on its own will bring up the entire stack in the foreground (logs will be displayed). -The `docker-compose up` command will also fetch the official container images +The `docker compose up` command will also fetch the official container images from [MetaCPAN Docker Hub](https://cloud.docker.com/u/metacpan/repository/list) repositories. @@ -115,7 +115,7 @@ Docker) and run the services. Don't forget to seed the local `metacpan-api` with a partial CPAN; run the following command in a separate terminal to get yourself up to speed: - docker-compose exec api index-cpan.sh + docker compose exec api index-cpan.sh This will prompt you to confirm removing old indices and setting up mappings on the Elasticsearch service (say `YES`). It will then proceed to rsync a partial @@ -128,7 +128,7 @@ elsewhere. Alternatively, if you just want to hack on the web frontend, you can run this instead of all the above: - docker-compose up web + docker compose up web From here, you can proceed and hack on the MetaCPAN code at `src/metacpan-api` and/or `src/metacpan-web` directories, and saving edits will reload the @@ -137,7 +137,7 @@ corresponding apps automatically! When done hacking (or, more likely, when you need to rebuild/refresh your Docker environment) you can then run - docker-compose down + docker compose down in another terminal to stop all MetaCPAN services and remove the containers. @@ -150,26 +150,26 @@ For further details, read on! You can (re)build arbitrary containers. For instance, if you want to rebuild the `api` container: - docker-compose build api + docker compose build api ### Accessing Containers -Containers are accessible via the `docker-compose exec` command followed by the +Containers are accessible via the `docker compose exec` command followed by the container and then the command to execute. For example, to start a shell prompt in the `api` container: - docker-compose exec api /bin/bash + docker compose exec api /bin/bash Executing tests via `prove` inside the API container: - docker-compose exec api_test prove -lvr \ + docker compose exec api_test prove -lvr \ t/00_setup.t \ t/01_darkpan.t \ t/api/controller/cover.t To access the `psql` command line client in the PostgreSQL container: - docker-compose exec pgdb psql + docker compose exec pgdb psql ### Accessing Services @@ -217,11 +217,11 @@ directly. They are available via the `api` and `api_test` containers. You can query the `elasticsearch` container via: - docker-compose exec elasticsearch curl http://localhost:9200 + docker compose exec elasticsearch curl http://localhost:9200 You can query the `elasticsearch_test` container via: - docker-compose exec elasticsearch_test curl http://localhost:9200 + docker compose exec elasticsearch_test curl http://localhost:9200 #### `PostgreSQL` @@ -267,13 +267,13 @@ These services use one or more Docker volumes: [4]: https://metacpan.org/pod/Carton Docker Compose is used to, uh, _compose_ them all together into one system. -Using `docker-compose` directly is a mouthful, however, so putting this all +Using `docker compose` directly is a mouthful, however, so putting this all together is done via the `bin/metacpan-docker` script to simplify setup and usage (and to get you started hacking on the MetaCPAN sooner!) ### The `bin/metacpan-docker` script -`bin/metacpan-docker` is a thin wrapper around the `docker-compose` command, +`bin/metacpan-docker` is a thin wrapper around the `docker compose` command, providing the environment variables necessary to run a basic MetaCPAN environment. It provides these subcommands: @@ -281,7 +281,7 @@ environment. It provides these subcommands: The `init` subcommand basically clones the [metacpan-api][5] and [metacpan-web][6] repositories, and sets up the git commit hooks for each of -them, in preparation for future `docker-compose` or +them, in preparation for future `docker compose` or `bin/metacpan-docker localapi` commands. It also clones the `metacpan-grep-front-end` and `metacpan-cpan-extracted-lite` @@ -292,12 +292,12 @@ repositories. #### `bin/metacpan localapi` -The `localapi` subcommand adds the necessary configuration for `docker-compose` +The `localapi` subcommand adds the necessary configuration for `docker compose` to run both the `metacpan-web` and `metacpan-api` services, along with `elasticsearch` and Docker volumes. Under the hood, it customizes the `COMPOSE_FILE` and `COMPOSE_PROJECT_NAME` environment variables used by -`docker-compose` to use additional YAML configuration files aside from the -default `docker-compose.yml`. +`docker compose` to use additional YAML configuration files aside from the +default `docker compose.yml`. #### `bin/metacpan-docker pull` @@ -314,8 +314,8 @@ command. #### `bin/metacpan-docker` build/up/down/start/stop/run/ps/top... As noted earlier, `bin/metacpan-docker` is a thin wrapper around -`docker-compose`, so commands like `up`, `down`, and `run` will work as expected -from `docker-compose`. See the [docker-compose docs][7] for an overview of +`docker compose`, so commands like `up`, `down`, and `run` will work as expected +from `docker compose`. See the [docker compose docs][7] for an overview of available commands. [7]: @@ -329,8 +329,8 @@ The `web` service is a checkout of `metacpan-web`, built as a Docker image. Running this service alone is enough if you want to just hack on the frontend, since by default the service is configured to talk to [https://fastapi.metacpan.org](https://fastapi.metacpan.org) for its backend; if -this is what you want, then you can simply invoke `docker-compose up` or -`docker-compose up web`. +this is what you want, then you can simply invoke `docker compose up` or +`docker compose up web`. #### `api` @@ -396,7 +396,7 @@ the [metacpan-puppet][10] repository. The `api` service depends on the ### Running your own miniCPAN inside metacpan-docker Suppose you have a local minicpan in `/home/ftp/pub/CPAN`. If you would like to -use this in `metacpan-docker`, then edit the `docker-compose.localapi.yml` to +use this in `metacpan-docker`, then edit the `docker compose.localapi.yml` to change the `api` service's volume mounts to use your local minicpan as `/CPAN`, e.g.: @@ -450,7 +450,7 @@ You can use `bin/metacpan-docker pull` to update all `src/*` directories. ### Running Kibana to peek into Elasticsearch data -By default, the `docker-compose.localapi.yml` configures the `elasticsearch` +By default, the `docker compose.localapi.yml` configures the `elasticsearch` service to listen on the Docker host at [http://localhost:9200](http://localhost:9200), and is also accessible via the Docker `default` network address of