Skip to content

Commit

Permalink
Additional config for Docker and AWS hosting (#1298)
Browse files Browse the repository at this point in the history
* aws action

* aws action

* github actions

* github actions

* github actions

* github actions

* github actions

* github actions

* github actions

* github actions

* github actions

* github actions

* github actions

* github actions

* github actions

* github actions

* github actions

* github actions

* modified dockerfile

* modified requirements to include pytest libs also

* modified requirements to include pytest libs also

* added deploy script for dev

* no cache for compose build

* no cache for compose build

* updated coaster

* nginx path corrected

* makefile in assets

* coaster

* updated requirements

* updated requirements

* updated requirements

* updated requirements

* requirements file for tests updated

* requirements file for tests updated

* requirements file for tests updated

* disabled worker container in docker compose

* disabled worker container in docker compose

* added a new runtests version for docker called docker-tests.sh

* added a new runtests version for docker called docker-tests.sh

* added a new runtests version for docker called docker-tests.sh

* added a new runtests version for docker called docker-tests.sh

* added a new runtests version for docker called docker-tests.sh

* added a new runtests version for docker called docker-tests.sh

* added a container restart flag for postgres

* added a container restart flag for postgres

* added a container restart flag for postgres

* added a container restart flag for postgres

* added a container restart flag for postgres

* disabling dbconfig in tests

* disabling dbconfig in tests

* disabled tests

* disabled tests

* buildx fix

* buildx fix

* buildx fix

* ssh deploy

* ssh deploy

* ssh deploy

* ssh deploy

* ssh deploy

* ssh deploy

* ssh deploy

* ssh deploy

* ssh deploy

* ssh deploy

* ssh action added to yml

* ssh action added to yml

* tests added back

* tests added back

* Update testanddeploy.yml

reinstated tests

* hot reloading enabled + tests

* added credentials as secrets

* disabled tests until credentials are updated

* fixed yml error

* replaced s3 env with github secrets

* pre-commit fixes

* uwsgi+streamlining docker compose

* added dev docker-compose

* fixing staging deployment in github actions

* fixing staging deployment via github actions

* write permission on mount in staging deployment

* working on fixing the settings.py issue

* replaced s3 container env with github secrets

* fixing path on settings.py

* bind instance folder to host directory

* updating yml for mounting

* replacing docker run with docker-compose for flask db upgrade

* Update testanddeploy.yml

* enabled the autodeploy flow but disabled the tests

* precommit

* deploy

* staging deployment via fabric

* staging deployment via fabric

* staging deployment via fabric

* pre-commit fixes

* pre-commit fixes

* Legacy funnel redirects not relevant here; sort files

* Generate hashes for requirements

* Change Docker base image (untested)

* Update pre-commit config

* Add .dockerignore. Update Dockerfile for funnel. Add Dockerfile for funnel builder

* Build and run test containers

* Small changes to make runtests.sh self-sufficient

* Replace docker-compose-test.yml with docker-compose-ci.yml

* Removes entrypoint for CI tests, which was being used earlier to bypass failure in directly running pytest. docker-tests.sh is no longer required. Turns off TTY for CI tests, as CI output do not support it

* Switches image base to nikolaik/python-node

* Separate repo modules from local folder in isort (#1709)

* Change cookie serializer (#1710)

Funnel's cookie serializer used JWT, which was removed from itsdangerous, forcing us to pin the old version. We could not switch to another JWT implementation because our key rotation wrapper only works with itsdangerous. Rather than extend key rotation to another library, with fragile tests, we've opted to switch to a supported serializer in itsdangerous. This switch will invalidate all existing cookies, forcing a re-login for all users. A corresponding change in hasgeek/flask-lastuser#66 is required for all apps hosted in subdomains of Funnel.

In addition, RQ Dashboard now requires config on register and so has a new init function. Flask 2.3 no longer supports the `before_first_request` callback, so the blueprint must expect config from the app when first registered on it.

* Basic working dev setup on docker

* Remove watchdog

* Consolidated all docker operations in a single set of Dockerfile and docker-compose.yml files

* Backup to disable github workflow testanddeploy.yml

* Add PYTHONOPTIMIZE=2 for docker production and test; PYTHONDONTWRITEBYTECODE=1 PYTHONDEVMODE=1 for docker dev

* First attempt to run tests on GHA using docker

* Backup pytest.yml GHA, so as to not run it

* Attempt to run tests on docker

* Remove --link from all COPY commands in Dockerfile

* Enable usage of gha cache for docker builds on GHA

* Remove unnecessary docker compose up options and docker compose down action from docker-ci-test make target

* Do not restart funnel-test service on completion of run

* Try out local caching - one of the options in docker's gha caching documentation

* Replicate steps from the pytest github action, including publishing of coverage; try using gha cache for pip_
cache

* Set permissions for coverage folder

* Empty commit to re-run docker tests

* Basic setup to run funnel from production container using uwsgi; Remove PYTHONDONTWRITEBYTECODE=1 from dev container

* Remove obsolete files

* Remove PYTHONDEVMODE=1 and PYTHONOPTIMIZE=2 from Dockerfile

* Remove FLASK_REDIS_URL from sample.env

* Try new docker CI test setup on GHA

* Build image before using compose in Docker pytest GHA workflow

* Correction in ci.Dockerfile

* Added entrypoint for test container and made pulls quite in make target docker-ci-test

* Corrections in docker compose file

* Corrections in CI docker tests setup

* Undo version downgrades; restore pytest and drop testanddeploy in Github Actions

* Add Dockerfile linter to pre-commit checks

* Reformat Dockerfiles to pass linter check

* Add Hadolint for Dockerfile linting

* Update pre-commit hadolint hook

* Blanket changes for new docker CI setup

* Changes made to Dockerfile for bases

* Attempt to use resolve cache bind mounts' ownership issues for docker CI setup

* Remove obsolete files

* Remove OBJC_DISABLE_INITIALIZE_FORK_SAFETY from docker setup; do not run db migration in dev entrypoint

---------

Co-authored-by: vivekdurai <vivek@lexaude.com>
Co-authored-by: Mitesh Ashar <email@miteshashar.com>
  • Loading branch information
3 people committed Jun 14, 2023
1 parent d3291c7 commit 54a3697
Show file tree
Hide file tree
Showing 33 changed files with 870 additions and 96 deletions.
81 changes: 81 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
# Cache files
**/*.egg-info
**/*.pyc
.coverage
.mypy_cache
.pytest_cache
.ropeproject
**/.sass-cache
**/.webassets-cache
**/__pycache__
.webpack_cache
.cache

# MacOS file
**/.DS_Store

# Assets and static files
funnel/static/build
funnel/static/service-worker.js
funnel/static/service-worker.js.map
funnel/static/img/fa5-packed.svg
**/node_modules
**/*.packed.css
**/*.packed.js
**/baseframe-packed.*

# Download files
geoname_data

# Dependencies
build/dependencies

# Log and test files
error.log
error.log.*
ghostdriver.log
geckodriver.log
tests/cypress/screenshots
tests/cypress/videos
tests/screenshots
tests/unit/utils/markdown/data/output.html
coverage/

# Instance files that should not be checked-in
instance/development.py
instance/hasgeekapp.py
instance/production.py
instance/settings.py
instance/container_env

# Forbidden secrets
secrets.dev
secrets.test
newrelic.ini
.env
.env.*

# Local venvs (Idea creates them)
venv/
env/
.python-version

# Editors
.idea/
.vscode/
.project
.pydevproject
.settings
*.wpr
.vscode
*.sublime-workspace

# Versioning files
.git

# Local DBs
dump.rdb
test.db
*.bz2
*.gz
*.sql
71 changes: 71 additions & 0 deletions .github/workflows/docker-ci-tests.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
name: 'Pytest on docker'

on:
push:
branches: ['main']
pull_request:
branches: ['main']
paths:
- '**.py'
- '**.js'
- '**.scss'
- '**.jinja2'
- 'requirements/base.txt'
- 'requirements/test.txt'
- '.github/workflows/docker-ci-tests.yml'
- 'Dockerfile'
- 'pyproject.toml'
- '.eslintrc.js'
- 'docker-compose.yml'
- 'docker/compose/services.yml'
- 'docker/entrypoints/ci-test.sh'
- 'docker/initdb/test.sh'
- 'package.json'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
jobs:
test:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v3
- name: Cache npm
uses: actions/cache@v3
with:
path: .cache/.npm
key: docker-npm
- name: Cache node_modules
uses: actions/cache@v3
with:
path: node_modules
key: docker-node_modules-${{ hashFiles('package-lock.json') }}
- name: Cache pip
uses: actions/cache@v3
with:
path: .cache/pip
key: docker-pip
- name: Cache .local
uses: actions/cache@v3
with:
path: .cache/.local
key: docker-user-local
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2
- name: Build funnel-test image
id: build-funnel-test
uses: docker/build-push-action@v4
with:
context: .
file: ci.Dockerfile
tags: funnel-test:latest
load: true
push: false
- name: Run Tests
run: make docker-ci-test
- name: Upload coverage report to Coveralls
uses: coverallsapp/github-action@master
with:
github-token: ${{ secrets.github_token }}
path-to-lcov: coverage/funnel.lcov
flag-name: docker-3.11
9 changes: 4 additions & 5 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,13 @@
.pytest_cache
.ropeproject
.sass-cache
.sass-cache
.webassets-cache
.webpack_cache
__pycache__
.nox
monkeytype.sqlite3
.ci-cache
.cache

# MacOS file
.DS_Store
Expand Down Expand Up @@ -57,10 +59,7 @@ secrets.dev
secrets.test
newrelic.ini
.env
.env.testing
.env.staging
.env.development
.env.production
.env.*

# Local venvs (Idea creates them)
venv/
Expand Down
23 changes: 22 additions & 1 deletion .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,13 @@ default_stages: [commit]
# default_language_version:
# python: python3.9
ci:
skip: ['pip-audit', 'yesqa', 'no-commit-to-branch']
skip: [
'pip-audit',
'yesqa',
'no-commit-to-branch',
# 'hadolint-docker',
'docker-compose-check',
]
repos:
- repo: https://github.com/pre-commit-ci/pre-commit-ci-config
rev: v1.5.1
Expand Down Expand Up @@ -33,6 +39,8 @@ repos:
'PYSEC-2021-13', # https://github.com/pallets-eco/flask-caching/pull/209
'--ignore-vuln',
'PYSEC-2022-42969', # https://github.com/pytest-dev/pytest/issues/10392
'--ignore-vuln',
'PYSEC-2023-73', # https://github.com/RedisLabs/redisraft/issues/608
]
files: ^requirements/.*\.txt$
- repo: https://github.com/charliermarsh/ruff-pre-commit
Expand Down Expand Up @@ -181,3 +189,16 @@ repos:
hooks:
- id: reformat-gherkin
files: \.feature$
# - repo: https://github.com/Lucas-C/pre-commit-hooks-nodejs
# rev: v1.1.2
# hooks:
# - id: dockerfile_lint
# files: .*Dockerfile.*
# - repo: https://github.com/hadolint/hadolint
# rev: v2.12.1-beta
# hooks:
# - id: hadolint-docker
- repo: https://github.com/IamTheFij/docker-pre-commit
rev: v3.0.1
hooks:
- id: docker-compose-check
12 changes: 7 additions & 5 deletions .testenv
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,10 @@ FLASK_DEBUG_TB_ENABLED=false
FLASK_WTF_CSRF_ENABLED=true
# Use Redis cache so that rate limit validation tests work, with Redis db
FLASK_CACHE_TYPE=flask_caching.backends.RedisCache
FLASK_RQ_REDIS_URL=redis://localhost:6379/9
FLASK_RQ_DASHBOARD_REDIS_URL=redis://localhost:6379/9
FLASK_CACHE_REDIS_URL=redis://localhost:6379/9
REDIS_HOST=localhost
FLASK_RQ_REDIS_URL=redis://${REDIS_HOST}:6379/9
FLASK_RQ_DASHBOARD_REDIS_URL=redis://${REDIS_HOST}:6379/9
FLASK_CACHE_REDIS_URL=redis://${REDIS_HOST}:6379/9
# Disable logging in tests
FLASK_SQLALCHEMY_ECHO=false
FLASK_LOGFILE=/dev/null
Expand All @@ -34,8 +35,9 @@ FLASK_LASTUSER_COOKIE_DOMAIN='.funnel.test:3002'
FLASK_SITE_TITLE='Test Hasgeek'
FLASK_SITE_SUPPORT_EMAIL=test-support-email
FLASK_SITE_SUPPORT_PHONE=test-support-phone
FLASK_SQLALCHEMY_DATABASE_URI='postgresql+psycopg://localhost/funnel_testing'
FLASK_SQLALCHEMY_BINDS__geoname='postgresql+psycopg://localhost/geoname_testing'
DB_HOST=localhost
FLASK_SQLALCHEMY_DATABASE_URI='postgresql+psycopg://${DB_HOST}/funnel_testing'
FLASK_SQLALCHEMY_BINDS__geoname='postgresql+psycopg://${DB_HOST}/geoname_testing'
FLASK_TIMEZONE='Asia/Kolkata'
FLASK_BOXOFFICE_SERVER='http://boxoffice:6500/api/1/'
FLASK_IMGEE_HOST='http://imgee.test:4500'
Expand Down
129 changes: 98 additions & 31 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,45 +1,112 @@
FROM python:3.9-slim-bullseye
# syntax=docker/dockerfile:1.4

RUN apt-get -y update
FROM nikolaik/python-nodejs:python3.11-nodejs20-bullseye as base

# install curl
RUN apt-get -y install curl
# https://github.com/zalando/postgres-operator/blob/master/docker/logical-backup/Dockerfile
# https://stackoverflow.com/questions/68465355/what-is-the-meaning-of-set-o-pipefail-in-bash-script
SHELL ["/bin/bash", "-o", "pipefail", "-c"]

# get install script and pass it to execute:
RUN curl -sL https://deb.nodesource.com/setup_14.x | bash
LABEL Name=Funnel
LABEL Version=0.1

# and install node
RUN apt-get -y install nodejs git wget unzip build-essential make postgresql libpq-dev python-dev
USER pn
RUN \
mkdir -pv /home/pn/.cache/pip /home/pn/.npm /home/pn/tmp /home/pn/app /home/pn/app/coverage && \
chown -R pn:pn /home/pn/.cache /home/pn/.npm /home/pn/tmp /home/pn/app /home/pn/app/coverage
EXPOSE 3000
WORKDIR /home/pn/app

# We don't want to run our application as root if it is not strictly necessary, even in a container.
# Create a user and a group called 'app' to run the processes.
# A system user is sufficient and we do not need a home.
ENV PATH "$PATH:/home/pn/.local/bin"

RUN adduser --system --group --no-create-home app
FROM base as devtest_base
USER root
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked --mount=type=cache,target=/var/lib/apt,sharing=locked \
apt-get update -yqq && \
apt-get install -yqq --no-install-recommends lsb-release && \
sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list' && \
wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add - && \
apt-get update -yqq && apt-get upgrade -yqq && \
echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections && \
DEBIAN_FRONTEND=noninteractive apt-get install -yqq --no-install-recommends firefox-esr postgresql-client-15 && \
cd /tmp/ && \
curl -fsSL $(curl -fsSL https://api.github.com/repos/mozilla/geckodriver/releases/latest | grep browser_download_url | grep 'linux64.tar.gz\"'| grep -o 'http.*\.gz') > gecko.tar.gz && \
tar -xvzf gecko.tar.gz && \
rm gecko.tar.gz && \
chmod +x geckodriver && \
mv geckodriver /usr/local/bin && \
apt-get autoclean -yqq && \
apt-get autoremove -yqq && \
cd /home/pn/app
USER pn

# Make the directory the working directory for subsequent commands
WORKDIR app
FROM base as assets
COPY --chown=pn:pn package.json package.json
COPY --chown=pn:pn package-lock.json package-lock.json
RUN --mount=type=cache,target=/root/.npm \
--mount=type=cache,target=/home/pn/.npm,uid=1000,gid=1000 npm ci
COPY --chown=pn:pn ./funnel/assets ./funnel/assets
COPY --chown=pn:pn .eslintrc.js .eslintrc.js
COPY --chown=pn:pn webpack.config.js webpack.config.js
RUN --mount=type=cache,target=/root/.npm \
--mount=type=cache,target=/home/pn/.npm,uid=1000,gid=1000 npm run build

# Place the application components in a dir below the root dir
COPY . /app/
FROM base as dev_assets
COPY --chown=pn:pn package.json package.json
COPY --chown=pn:pn package-lock.json package-lock.json
RUN --mount=type=cache,target=/root/.npm \
--mount=type=cache,target=/home/pn/.npm,uid=1000,gid=1000 npm install
COPY --chown=pn:pn ./funnel/assets ./funnel/assets
COPY --chown=pn:pn .eslintrc.js .eslintrc.js
COPY --chown=pn:pn webpack.config.js webpack.config.js
RUN --mount=type=cache,target=/root/.npm \
--mount=type=cache,target=/home/pn/.npm,uid=1000,gid=1000 npx webpack --mode development --progress

RUN cd /app/funnel; make
FROM base as deps
COPY --chown=pn:pn Makefile Makefile
RUN make deps-editable
COPY --chown=pn:pn requirements/base.txt requirements/base.txt
RUN --mount=type=cache,target=/home/pn/.cache/pip,uid=1000,gid=1000 \
pip install --upgrade pip && \
pip install --use-pep517 -r requirements/base.txt

# Install from the requirements.txt we copied above
COPY requirements.txt /tmp
RUN pip install -r requirements.txt
COPY . /tmp/myapp
RUN pip install /tmp/myapp
FROM devtest_base as test_deps
COPY --chown=pn:pn Makefile Makefile
RUN make deps-editable
COPY --chown=pn:pn requirements/base.txt requirements/base.txt
COPY --chown=pn:pn requirements/test.txt requirements/test.txt
RUN --mount=type=cache,target=/home/pn/.cache/pip,uid=1000,gid=1000 pip install --use-pep517 -r requirements/test.txt

# Hand everything over to the 'app' user
RUN chown -R app:app /app
FROM devtest_base as dev_deps
COPY --chown=pn:pn Makefile Makefile
RUN make deps-editable
COPY --chown=pn:pn requirements requirements
RUN --mount=type=cache,target=/home/pn/.cache/pip,uid=1000,gid=1000 pip install --use-pep517 -r requirements/dev.txt
COPY --from=dev_assets --chown=pn:pn /home/pn/app/node_modules /home/pn/app/node_modules

# Subsequent commands, either in this Dockerfile or in a
# docker-compose.yml, will run as user 'app'
USER app
FROM deps as production
COPY --chown=pn:pn . .
COPY --chown=pn:pn --from=assets /home/pn/app/funnel/static /home/pn/app/funnel/static
ENTRYPOINT ["uwsgi", "--ini"]

FROM production as supervisor
USER root
RUN \
apt-get update -yqq && \
apt-get install -yqq --no-install-recommends supervisor && \
apt-get autoclean -yqq && \
apt-get autoremove -yqq && \
mkdir -pv /var/log/supervisor
COPY ./docker/supervisord/supervisord.conf /etc/supervisor/supervisord.conf
# COPY ./docker/uwsgi/emperor.ini /etc/uwsgi/emperor.ini
ENTRYPOINT ["/usr/bin/supervisord"]

# We are done with setting up the image.
# As this image is used for different
# purposes and processes no CMD or ENTRYPOINT is specified here,
# this is done in docker-compose.yml.
FROM test_deps as test
ENV PWD=/home/pn/app
COPY --chown=pn:pn . .

COPY --chown=pn:pn --from=assets /home/pn/app/funnel/static /home/pn/app/funnel/static
ENTRYPOINT ["/home/pn/app/docker/entrypoints/ci-test.sh"]
FROM dev_deps as dev
RUN --mount=type=cache,target=/home/pn/.cache/pip,uid=1000,gid=1000 cp -R /home/pn/.cache/pip /home/pn/tmp/.cache_pip
RUN mv /home/pn/tmp/.cache_pip /home/pn/.cache/pip
COPY --chown=pn:pn --from=dev_assets /home/pn/app/funnel/static /home/pn/app/funnel/static
Loading

0 comments on commit 54a3697

Please sign in to comment.