new: docker-compose #283

florianm · 2022-01-28T12:26:01Z

This is an initial commit with a working minimal implementation - edit - nearly complete and works nicely. Ready for feedback.
Closes Deployment via docker-compose #272 (some points outstanding)
Dockerfile for Build
docker-compose for Build and Postgres (but not yet build2xlsform)
ancillary scripts
updated docs (offline notice, deployment)
GitHub action to build and push Build Docker image to ghcr.io (needs ODK core team's PAT and actor name as repo secrets)
Dockerisation of build2xlsform Provide Dockerfile and GH action to build and push image build2xlsform#21

Outstanding discussion points:

Is this overall a good idea as deployment mechanism - keep source install or switch to docker-compose?
Keep db or use docker-compose's db?

If we want to go down the docker-compose road, this could close a whole raft of issues and discussion points:

Close Ops: need to implement logrotate #108
Close Ongoing maintenance tasks #248 (adds offline deployment steps, may replace ansible deployment)
Close Move deployment tree to different repo #202 (no more ansible)

As I find time I will add the remaining bits. Any feedback or governance welcome.

Impressions:
Running in portainer

Listed on the shell

Running in browser

* This is an initial commit with a working minimal implementation * Addresses getodk#272 partially * Dockerfile for Build * docker-compose for Build and Postgres (but not yet build2xlsform) * ancillary scripts * updated docs (offline notice, deployment)

florianm · 2022-01-28T12:48:14Z

BUILDING.md

@@ -29,11 +29,68 @@ This section contains package maintenance procedures in preparation for deployme
  git push --tags
  ```
 * Create a new release from the new tag on GitHub and let GitHub auto-generate release notes. Mark releases used for testing as pre-release. The release bundles the code into an archive, which we'll use for deployment.
+* Pushing a new tag will generate a new Docker image with the same tag.


...once the GH action is in place

* that starts with v .. or is it all tags?

florianm · 2022-01-28T12:49:45Z

contrib/wait-for-it.sh

@@ -0,0 +1,177 @@
+#!/usr/bin/env bash


Borrowed 1:1 from ODK Central

florianm · 2022-01-28T12:50:27Z

docker-compose.yml

+    volumes:
+      - /var/lib/postgresql/data
+    environment:
+      POSTGRES_USER: odkbuild


Credentials hard coded here and in config.yml - could be parameterised via ENV. Seems good enough for Central though.

florianm · 2022-01-28T12:51:02Z

server/views/index.erb

@@ -195,6 +195,7 @@
      </div>
      <div class="offlineNotice">
        Tired of logging in? Did you know there is a <a href="https://github.com/getodk/build/releases/latest" rel="external">downloadable version of ODK Build available?</a> Simply save your forms to file from here using the File menu and you can open them there.
+        Follow the instructions <a href="https://github.com/getodk/build/blob/master/BUILDING.md">here</a> to run Build on your machine with `docker-compose`.


I'm sure there is better wording for this

florianm · 2022-01-28T12:51:46Z

contrib/unicorn.rb

+stdout_path "/var/log/odkbuild/stdout.log"
+
+listen "/var/odkbuild/build-server.sock", :backlog => 64
+listen 9393, :tcp_nopush => true


Apart from this line, exactly taken from a working server

florianm · 2022-01-28T23:41:12Z

For the GH action, I'll need a repo secret "GH_PAT" with a GH Personal Access Token for my GH actor.
echo "${{ secrets.GH_PAT }}" | docker login ghcr.io -u ${{ secrets.GH_ACTOR }} --password-stdin

The image gets pushed to IMAGE_ID=ghcr.io/${{ github.repository_owner }}/$IMAGE_NAME.

We will need a core team member to create the secrets

GH_PAT containing a GH personal access token with the correct scope and
GH_ACTOR (e.g. yanokwa).

I will flail around on this downstream branch trying to get the action to work with my own credentials. Once working, the action should work upstream too.

Edit: oh look at that. https://github.com/orgs/dbca-wa/packages/container/package/odkbuild

* likely to fail on missing credentials * need core team to create secrets GH_PAT and GH_ACTOR

… to docker GHA

* env var to toggle build2xlsform/localhost as hostname for conversion service * two docker-compose files for dev (build master) and deploy (pull tagged image) * docs in building and compose files to describe how and when to use

.github/workflows/docker.yml

BUILDING.md

Dockerfile

docker-compose.yml

issa-tseng · 2022-02-12T00:20:09Z

i didn't look up every flag but it all looks fine. the biggest red flag is we should name the postgres volume so we don't end up in a nightmare situation like we have with central again. the rest are comments in there.

probably somebody with more recent experience actually running build in production should also take a quick peek.

florianm · 2022-02-12T02:06:02Z

Thanks for the valuable feedback! I will name the postgres data volume.

* naming suggestion: volume named `odkbuild` identical to GHA image name assuming that `build` is overloaded (hence prefixed) but there will not be other volumes (hence not postfixed with e.g. _pgdata * volume survives `docker-compose down` and is picked up again by next `docker-compose up`

* GHA: only build when a tag beginning with "v" is pushed, drop development builds on every push to master/main/this PR branch * docker-compose: use what I think could be the official GetODK image names, but final naming is up to the core team * docker image versions: currently referring to "latest" for ease of testing, should be pinned before used in prod

florianm · 2022-02-13T06:11:17Z

Addressing review comments

I've pushed the last bunch of commits now to prepare for merging (see commit messages):

Use named database volume - survives a docker-compose down, build notes updated, warning removed.
Clarify that GHA builds only when a tag beginning with "v" is pushed. This convention is distributed over a few files.
Chosen Docker image names: "odkbuild" and "odkbuild2xlsform" (again referenced from several files). Volume name: "odkbuild". As short and specific as I could make them. Open to suggestions.

Testing

The docker-compose file will not build as is, because the referenced official images are not published yet. See 19a52aa for working published images through my own downstream repos.

Next steps

The core team should review the naming of images and volumes.
The core team should make a decision on whether deployment via docker-compose is a good replacement of the current source install which replaced the older ansible install. Docker-compose also can also replace the offline branch.
Merging should be low risk, because this PR does not touch any working parts that are in production right now.
Once merged, GH secrets can be set up to get the images to build and publish at ghcr.io/getodk, and the deployment via docker-compose can be tested on the staging server. This could require some fine-tuning of the GH action or the docker-compose files.

* Proposed at getodk/build2xlsform#23 * Prefix negotiable * The image will only exist when getodk/build2xlsform#23 is merged

lognaturel · 2022-03-11T00:08:11Z

Dockerfile

@@ -0,0 +1,39 @@
+FROM ruby:2.7.2
+LABEL author="GetODK Inc."


@yanokwa? We usually say ODK or ODK Community, I think.

Changed to ODK for now.

Dockerfile

florianm · 2022-02-12T12:32:20Z

.github/workflows/docker.yml

+name: docker
+
+env:
+  IMAGE_NAME: odkbuild


What would be the best image name?

Since it is lower cased, I added a prefix odk to clarify that this image refers to the software ODK Build.
I've used getodk/odkbuild2xlsform on the other end.

BUILDING.md

@@ -29,11 +29,68 @@ This section contains package maintenance procedures in preparation for deployme
  git push --tags
  ```
 * Create a new release from the new tag on GitHub and let GitHub auto-generate release notes. Mark releases used for testing as pre-release. The release bundles the code into an archive, which we'll use for deployment.
+* Pushing a new tag will generate a new Docker image with the same tag.


florianm · 2022-02-13T05:59:33Z

server/views/index.erb

@@ -195,6 +195,7 @@
      </div>
      <div class="offlineNotice">
        Tired of logging in? Did you know there is a <a href="https://github.com/getodk/build/releases/latest" rel="external">downloadable version of ODK Build available?</a> Simply save your forms to file from here using the File menu and you can open them there.


This line will be obsolete once the docker-compose option is available.

florianm · 2022-03-11T00:59:12Z

Dockerfile

@@ -0,0 +1,39 @@
+FROM ruby:2.7.2
+LABEL author="GetODK Inc."


Changed to ODK for now.

lognaturel

I've taken a cursory look and mostly verified naming. I've also verified prior feedback was addressed. Seems fine to merge and then follow up with another PR if @yanokwa or @issa has comments.

new: docker-compose

4ce74db

* This is an initial commit with a working minimal implementation * Addresses getodk#272 partially * Dockerfile for Build * docker-compose for Build and Postgres (but not yet build2xlsform) * ancillary scripts * updated docs (offline notice, deployment)

florianm added infrastructure needs discussion labels Jan 28, 2022

florianm added this to the 0.4.4 docker milestone Jan 28, 2022

florianm self-assigned this Jan 28, 2022

florianm added this to In progress in Roadmap via automation Jan 28, 2022

merge upstream master

fae2b77

florianm commented Jan 28, 2022

View reviewed changes

contrib/wait-for-it.sh

@@ -0,0 +1,177 @@

#!/usr/bin/env bash

Copy link

Contributor Author

florianm Jan 28, 2022 •

edited

Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Borrowed 1:1 from ODK Central

florianm commented Jan 28, 2022

View reviewed changes

Florian Mayer added 3 commits January 28, 2022 20:52

fix: add newline to start-odkbuild.sh

208da3f

fix: add newline to Dockerfile

c7c5adf

merge upstream master

5f8a262

Florian Mayer added 6 commits January 29, 2022 07:42

nes: add initial GH action to build and publish docker image

d4d70bb

* likely to fail on missing credentials * need core team to create secrets GH_PAT and GH_ACTOR

fix: fix GHA image name, get ghcr.io credentials from secrets

b41a08c

noop: touch GHA

acc94a4

noop: touch GHA

682c482

improve: add stub to use prebuilt image to docker-compose, debug note…

63fae57

… to docker GHA

improve: dev and deploy compose

19a52aa

* env var to toggle build2xlsform/localhost as hostname for conversion service * two docker-compose files for dev (build master) and deploy (pull tagged image) * docs in building and compose files to describe how and when to use

florianm marked this pull request as ready for review January 29, 2022 06:03

florianm requested a review from yanokwa January 29, 2022 06:06

florianm mentioned this pull request Jan 29, 2022

new: Dockerfile and GHA getodk/build2xlsform#23

Merged

florianm requested review from lognaturel and issa-tseng February 11, 2022 12:12

issa-tseng reviewed Feb 12, 2022

View reviewed changes

.github/workflows/docker.yml Outdated Show resolved Hide resolved

issa-tseng reviewed Feb 12, 2022

View reviewed changes

BUILDING.md Outdated Show resolved Hide resolved

issa-tseng reviewed Feb 12, 2022

View reviewed changes

Dockerfile Show resolved Hide resolved

issa-tseng reviewed Feb 12, 2022

View reviewed changes

docker-compose.yml Outdated Show resolved Hide resolved

Florian Mayer added 3 commits February 13, 2022 13:18

florianm requested a review from issa-tseng February 13, 2022 06:11

fix: link to Docker image odkbuild2xlsform

b4701ba

* Proposed at getodk/build2xlsform#23 * Prefix negotiable * The image will only exist when getodk/build2xlsform#23 is merged

lognaturel reviewed Mar 11, 2022

View reviewed changes

Dockerfile Outdated Show resolved Hide resolved

noop: Dockerfile author and desc wording

841c841

florianm commented Mar 11, 2022

View reviewed changes

Roadmap automation moved this from In progress to Reviewer approved Mar 11, 2022

lognaturel approved these changes Mar 11, 2022

View reviewed changes

florianm merged commit 6800570 into getodk:master Mar 11, 2022

Roadmap automation moved this from Reviewer approved to Done Mar 11, 2022

florianm deleted the 272-docker branch March 11, 2022 01:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

new: docker-compose #283

new: docker-compose #283

florianm commented Jan 28, 2022 •

edited

Loading

florianm Jan 28, 2022

issa-tseng Feb 12, 2022 •

edited

Loading

This comment was marked as outdated.

florianm Jan 28, 2022 •

edited

Loading

florianm Jan 28, 2022

florianm Jan 28, 2022

florianm Jan 28, 2022

florianm commented Jan 28, 2022 •

edited

Loading

issa-tseng commented Feb 12, 2022

florianm commented Feb 12, 2022

florianm commented Feb 13, 2022 •

edited

Loading

lognaturel Mar 11, 2022

florianm Mar 11, 2022

florianm Feb 12, 2022 •

edited

Loading

This comment was marked as outdated.

florianm Feb 13, 2022

florianm Mar 11, 2022

lognaturel left a comment

new: docker-compose #283

new: docker-compose #283

Conversation

florianm commented Jan 28, 2022 • edited Loading

Choose a reason for hiding this comment

issa-tseng Feb 12, 2022 • edited Loading

Choose a reason for hiding this comment

This comment was marked as outdated.

florianm Jan 28, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

florianm commented Jan 28, 2022 • edited Loading

issa-tseng commented Feb 12, 2022

florianm commented Feb 12, 2022

florianm commented Feb 13, 2022 • edited Loading

Addressing review comments

Testing

Next steps

Choose a reason for hiding this comment

Choose a reason for hiding this comment

florianm Feb 12, 2022 • edited Loading

Choose a reason for hiding this comment

This comment was marked as outdated.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lognaturel left a comment

Choose a reason for hiding this comment

florianm commented Jan 28, 2022 •

edited

Loading

issa-tseng Feb 12, 2022 •

edited

Loading

florianm Jan 28, 2022 •

edited

Loading

florianm commented Jan 28, 2022 •

edited

Loading

florianm commented Feb 13, 2022 •

edited

Loading

florianm Feb 12, 2022 •

edited

Loading