Skip to content

Dockerize application and add integration tests #890

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Sep 19, 2023
Merged

Dockerize application and add integration tests #890

merged 1 commit into from
Sep 19, 2023

Conversation

aequitas
Copy link
Collaborator

@aequitas aequitas commented Mar 8, 2023

Getting started documentation on setting up a developer environment and integration tests: https://github.com/internetstandards/Internet.nl/blob/docker/documentation/Docker-getting-started.md

Other documents can be found at: https://github.com/internetstandards/Internet.nl/blob/docker/documentation/Docker.md

Collective todo:

  • Test the setup with a production size dataset

Edit: to 'fix' the hundreds hidden items, Load more… a poor man's solution is this js:

function loadAllComments() {
    let needRescheduling = false;
    const buttons = Array.from(document.querySelectorAll('button'));
    buttons.forEach((button) => {
        if (button.classList.contains('ajax-pagination-btn')) {
            if(!button.hasAttribute('disabled') && button.innerText === 'Load more…') {
                console.log("found", button);
                needRescheduling = true;
                button.dispatchEvent(new MouseEvent('click', {
                    bubbles: true,
                    cancelable: true
                }));                
            }
            else if(button.hasAttribute('disabled') && button.innerText === 'Loading…') {
                console.log("waiting", button);
                needRescheduling = true;
            }
            else {
                console.log("unrecognized 'Load more' button", button);
            }
        }
    });
    if (needRescheduling) {
        setTimeout(loadAllComments, 200)
    }
    else {
        console.log("all comments loaded");
    }
}

loadAllComments();

@aequitas aequitas force-pushed the docker branch 30 times, most recently from 640c4f9 to 4b68438 Compare March 10, 2023 17:08
@aequitas aequitas force-pushed the docker branch 16 times, most recently from c57a266 to 5a515a7 Compare September 12, 2023 10:29
@aequitas aequitas force-pushed the docker branch 9 times, most recently from 560bfb8 to 11da151 Compare September 19, 2023 11:39
[1.7.x] Ref #941 - Quick fix for sectxt translation mixup

Manage external repositories through submodules and include other external sources in this repository

Refactor settings to be configurable by environment variables

Remove old docker implementation

Add Dockerfile for application and docker-compose.yml for application stack.

Add integration test suite

Miscellaneous cleanup

Github action for integration tests

Improve logging during tests

Allow local development in Docker Compose

Add unbound resolver for connection test

Add test target for domain and mail tests

Httpd webserver for testing against current configuration, to be replaced with nginx config

WIP: Split workers into multiple services

WIP: Use nginx as webserver instead of apache2/httpd

WIP: beat scheduler/batch?

WIP: testsuite

WIP: connection test

Add integration tests that test against live running instances instead of controlled environment.

WIP: temporary letsencrypt TLS certs

TODO: autoreload fix?

Use test target that complies with current main branch CSP header rules

WIP: disable advanced checks until test environment is fully simulated

Test targets healthcheck

Make Unbound use Docker resolver during tests

Adjust test scores

wip

Add public suffix list to integration test environment for mail test

TODO: why is this resolved during tests

Further isolate DNS in test environment to speed up testing

Disable forwarding of all DNS requests not explicitly handled by dnsmasq so tests won't wait for resolving. This speeds up the test suite from minutes to seconds.

Add target for testing all browsers

Used fixed ip for mail test target

Use .test tld for test targets

Unbound logging

wip

wip

Keep docker images locally so they don't have to be pulled in the next steps

Enable batch API

Documentation

Add monitoring

Statsd monitoring examples

Settings file for locohost.nl

Make integration tests code configurable

Remove the need for dist settings

Disable tracking on test/development

Fix connection test IPv6 resolving

Skip test we expect to fail to improve test suite speed

Add dependencies for integration test

Add live tests

TODO: fix unbound for batch on staging

Add security and robots .txt files

Add favicon

Split integration tests into multiple files

Split live tests into files

Cleanup testsuite

Deployment documentation

Make live/integration tests more similar

Split live tests

Enable linting on integration tests code

Move integration tests docker resources to docker folder and naming consistency

Deployment docs

IPv6 dev env docs

Fix batch by enabling scheduler which kicks of tasks

Add container logging to integration tests

Development docs

Switch from Apache to Nginx webserver, add letsencrypt, https in integration test environment

Reduce expected test score after rebase

Don't let build fail if Github has issues with their caching system

Split build from test and make concurrent

Split image pins in separate file, build versioned images in github actions

Make compose project names more distinct

Make reboot proof

Production settings

Celery exporter

Metrics documentation

Test connection test redirect, don't redirect for trailing slashes, redirect for subdomains as well

Fix race condition for certbot

Non conflicting address for staing unbound

Add worker healthcheck

Docs

Run integration tests without DEBUG, serve static files from Django in production, add static files caching and expires.

Solve redis connection leak issue, deprecate 3.7 build.

Add short commands

Move assets files out of python source directories, to improve docker build cache

Small improvements in docker image build speed

404's should return proper status code, because otherwise they might be cached

Work around precondition failed issue with celery workers and rabbitmq

Use correct path for development volume mounts

Fix grafana authenticaton nagging from websockets

Disable includeSubdomains on HSTS header, add docs.

Use Zone file notation for DNS example

Use Dockerfile as extension

Lower default healthcheck interval on production

Disable grafana connecting out

Docs

Improve stability

Add nowww class b complicance

Build images for monitoring so Git repo should not be needed for deployment

Silence warning

Remove unused Make targets

Version number

Matomo settings description

IP allow/deny lists and site wide basic auth

Container troubleshooting logging

Setuptools_scm version as tag

Serve connection test result over https

Fix nginx config issues

WIP: sign connection test zones

WIP: add test- commands

WIP: rework deployment docs without Git, Make and with env file split

Fix glue records for unbound zone

Docs

WIP: dev env volumes

Update test domains/email

Troubleshooting docs

Cleanup

typo fix

Signed-off-by: Sasha Romijn <github@mxsasha.eu>

Fix zones path for docker volume

Docs

Reduce port exposure

Firewall docs

Normalize env files

Document compose spec

Switch from gunicorn to uwsgi

Fix unbound config file interpolation after restart

Allow debug logging in test, even if DEBUG=False

Debugging docs

Devenv: autoreload and mounted sources as default

Store compose logs as artifacts for CI debugging

Integration test environment docs

Add test summary to CI

Disable trace logs by default

Add Docker linting and unittests, split workflows

Split docker and old CI

Add unit tests

Make CI test run more reliable

Remove need to generate secret key during deployment

Cleanup

Builtin unbound verbose logging

Allow shorthand 'env' for 'environment' in make commands

Pull all images to keep test run output clean

Consistency

Improve caching

Fix indenting

Document live testing, simplify live testing setup, add batch test

Docs

Add option for worker concurrency

Standalone live tests

Make celery exporter more reliable

Docs, env diff table, make dev env explicit

Move env files into docker directory

Fix unit test environment file

Improve when tasks are shown

Use layered approach for env files

Make rpki test more reliable when results are incomplete, document issues with test isolation.

Batch docs

Use static IP's for internal connections instead of relying on Docker's flaky internal resolver

Move logs capture lower to capture logs for all started services

Remove duplicate config

Make Hof seperately configurable

Fix race condition with internal resolver to make CI tests more reliable

Add test for development environment, split static files tests into existing and generated files, fix static files for develop environment.

Pytest colored output in Github

Suppres warnings in CI

Adjust test score

Add test report for live and develop tests

Add IPv6 documentation for macOS

All services should restart after 'reboot'

Add unit tests to summary as well

Default test-runner to live tests

Batch docs

Ignore test result parsing failures

Log failures of up command

Fix docs

Add test-all command

Cleanup host env file and improve batch docs

Build all images

Compare technical results for batch tests before asserting score

Remove outdated file

Run integration tests in ci verbose

Fix IPv6 docs

Make integration tests use unique id for each test to make correlating tests with logs easier

Make startup more reliable using start_period instead of many retries

Workaround for Docker not allowing intercontainer IPv6 traffic on 'internal' networks, will be fixed in Docker 25, see: moby/moby#45649

Fix allowed hosts after slimming down host-dist.env

Make issues with existing environment variables during deploy less likely

Make unbound verbose logging configurable

Test environment context

Revert "Pytest colored output in Github"

This reverts commit 43f3dbd.

WIP: Add second integration test with ip6tables enabled

Make rabbitmq healthcheck and startup more reliable

Remove outdated config file

Routinator basic healthcheck

Fix logging issues when bringing ci environment up

Update deployment docs

Disable routinator to pull in data during test and develop

Add website test to develop

Test development environment in CI using ipv6

Fix host-dist.env file

Improve instructions for DNSSEC reset

Expose DNS port on IPv6

Update live-tests docs for ipv6

Production and batch overview images

Move test-runner into integration-test

Architecture docs

Remove invalid TLSA from zone files

Add running message

Fix documentation for IPv6 on macOs, add HE tunnel mention, simplify Linux instructions

Disable waiting for healthy routinator as it takes to long of a time for routinator to start.

Remove outdated file

Allow configuring encrypted passwords

Document different test suites

Fix subdomain redirects in nginx (quick fix pending discussion)

Signed-off-by: Sasha Romijn <github@mxsasha.eu>

Fix nginx redirects, attempt 2

This reverts commit b959a54.

Fix conn domain letsencrypt

Add $IPV6_IP_PUBLIC to ALLOWED_HOSTS in host-dist.env

Add note about IPv6 addresses

Signed-off-by: Sasha Romijn <github@mxsasha.eu>

Improve resilience of certbot entrypoint

Documentation formatting fix

Signed-off-by: Sasha Romijn <github@mxsasha.eu>

Update integration test security.txt expectation ref #932

Add https scheme to CSP policy to meet #810

Set our Referrer-Policy to same-origin to meet #357

Make sure live tests examples always use latests

Remove old config file

Reduce complication in IPv6 setup for deployments

- Remove the need to assign and forward IPv6 subnets to the internal docker networks
- bind unbound ports explicitly to external addresses to not conflict with os resolvers on localhost
- update deployment documentation

Test conn. redirects

Revert IP in allowed hosts, add testcase

Fix develop environment ipv6 test

Use internal resolver for all queries made by app/worker

Rename test-resolver to mock-resolver for clarity

Work around issue with resolving race condition in CI

Fix documentation for nginx auth config

Simplify compose commands in docs

Pull test runner for develop test

There is no public IPv6 network on Github actions, so these jobs make no sense.

Enable DNSSEC validation for internal resolver

Reduce maximum test duration and retest timeout on development

Fix linting, fix linting errors

Keep celery exporter running after crash

Resign zones every time, but persist the keys, so we don't have to update DNS after IP/hostname changes.

Update docs

Split internal resolver in validating and permissive to both accomodate ldns-dane which requires validation and other queries which needs to be passed through to the client despite failing DNSSEC validation

Add live test for sites with known score

Don't search integration test directory during 'normal' tests

Add Gixy check for nginx configs

Nginx log vhost name, use chrome for playwright trace

Issue 894, disable hsts on https://conn. domain and return 404 not found for all non-configured domains

Use --project-name instead of --env-file for everything but up/pull/run

Add node exporter

Fix initial configuration for batch

Rewrite basic auth, require authentication when DEBUG is enabled

Integrate certbot with webserver service, auto reload nginx on certificate changes

Test-all

Set max tasks per child, to reduce memory issues on ipv6kaartmetingen.
@mxsasha mxsasha merged commit 21baea3 into main Sep 19, 2023
@mxsasha mxsasha deleted the docker branch September 19, 2023 12:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

5 participants