-
Notifications
You must be signed in to change notification settings - Fork 41
Dockerize application and add integration tests #890
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
640c4f9
to
4b68438
Compare
c57a266
to
5a515a7
Compare
560bfb8
to
11da151
Compare
[1.7.x] Ref #941 - Quick fix for sectxt translation mixup Manage external repositories through submodules and include other external sources in this repository Refactor settings to be configurable by environment variables Remove old docker implementation Add Dockerfile for application and docker-compose.yml for application stack. Add integration test suite Miscellaneous cleanup Github action for integration tests Improve logging during tests Allow local development in Docker Compose Add unbound resolver for connection test Add test target for domain and mail tests Httpd webserver for testing against current configuration, to be replaced with nginx config WIP: Split workers into multiple services WIP: Use nginx as webserver instead of apache2/httpd WIP: beat scheduler/batch? WIP: testsuite WIP: connection test Add integration tests that test against live running instances instead of controlled environment. WIP: temporary letsencrypt TLS certs TODO: autoreload fix? Use test target that complies with current main branch CSP header rules WIP: disable advanced checks until test environment is fully simulated Test targets healthcheck Make Unbound use Docker resolver during tests Adjust test scores wip Add public suffix list to integration test environment for mail test TODO: why is this resolved during tests Further isolate DNS in test environment to speed up testing Disable forwarding of all DNS requests not explicitly handled by dnsmasq so tests won't wait for resolving. This speeds up the test suite from minutes to seconds. Add target for testing all browsers Used fixed ip for mail test target Use .test tld for test targets Unbound logging wip wip Keep docker images locally so they don't have to be pulled in the next steps Enable batch API Documentation Add monitoring Statsd monitoring examples Settings file for locohost.nl Make integration tests code configurable Remove the need for dist settings Disable tracking on test/development Fix connection test IPv6 resolving Skip test we expect to fail to improve test suite speed Add dependencies for integration test Add live tests TODO: fix unbound for batch on staging Add security and robots .txt files Add favicon Split integration tests into multiple files Split live tests into files Cleanup testsuite Deployment documentation Make live/integration tests more similar Split live tests Enable linting on integration tests code Move integration tests docker resources to docker folder and naming consistency Deployment docs IPv6 dev env docs Fix batch by enabling scheduler which kicks of tasks Add container logging to integration tests Development docs Switch from Apache to Nginx webserver, add letsencrypt, https in integration test environment Reduce expected test score after rebase Don't let build fail if Github has issues with their caching system Split build from test and make concurrent Split image pins in separate file, build versioned images in github actions Make compose project names more distinct Make reboot proof Production settings Celery exporter Metrics documentation Test connection test redirect, don't redirect for trailing slashes, redirect for subdomains as well Fix race condition for certbot Non conflicting address for staing unbound Add worker healthcheck Docs Run integration tests without DEBUG, serve static files from Django in production, add static files caching and expires. Solve redis connection leak issue, deprecate 3.7 build. Add short commands Move assets files out of python source directories, to improve docker build cache Small improvements in docker image build speed 404's should return proper status code, because otherwise they might be cached Work around precondition failed issue with celery workers and rabbitmq Use correct path for development volume mounts Fix grafana authenticaton nagging from websockets Disable includeSubdomains on HSTS header, add docs. Use Zone file notation for DNS example Use Dockerfile as extension Lower default healthcheck interval on production Disable grafana connecting out Docs Improve stability Add nowww class b complicance Build images for monitoring so Git repo should not be needed for deployment Silence warning Remove unused Make targets Version number Matomo settings description IP allow/deny lists and site wide basic auth Container troubleshooting logging Setuptools_scm version as tag Serve connection test result over https Fix nginx config issues WIP: sign connection test zones WIP: add test- commands WIP: rework deployment docs without Git, Make and with env file split Fix glue records for unbound zone Docs WIP: dev env volumes Update test domains/email Troubleshooting docs Cleanup typo fix Signed-off-by: Sasha Romijn <github@mxsasha.eu> Fix zones path for docker volume Docs Reduce port exposure Firewall docs Normalize env files Document compose spec Switch from gunicorn to uwsgi Fix unbound config file interpolation after restart Allow debug logging in test, even if DEBUG=False Debugging docs Devenv: autoreload and mounted sources as default Store compose logs as artifacts for CI debugging Integration test environment docs Add test summary to CI Disable trace logs by default Add Docker linting and unittests, split workflows Split docker and old CI Add unit tests Make CI test run more reliable Remove need to generate secret key during deployment Cleanup Builtin unbound verbose logging Allow shorthand 'env' for 'environment' in make commands Pull all images to keep test run output clean Consistency Improve caching Fix indenting Document live testing, simplify live testing setup, add batch test Docs Add option for worker concurrency Standalone live tests Make celery exporter more reliable Docs, env diff table, make dev env explicit Move env files into docker directory Fix unit test environment file Improve when tasks are shown Use layered approach for env files Make rpki test more reliable when results are incomplete, document issues with test isolation. Batch docs Use static IP's for internal connections instead of relying on Docker's flaky internal resolver Move logs capture lower to capture logs for all started services Remove duplicate config Make Hof seperately configurable Fix race condition with internal resolver to make CI tests more reliable Add test for development environment, split static files tests into existing and generated files, fix static files for develop environment. Pytest colored output in Github Suppres warnings in CI Adjust test score Add test report for live and develop tests Add IPv6 documentation for macOS All services should restart after 'reboot' Add unit tests to summary as well Default test-runner to live tests Batch docs Ignore test result parsing failures Log failures of up command Fix docs Add test-all command Cleanup host env file and improve batch docs Build all images Compare technical results for batch tests before asserting score Remove outdated file Run integration tests in ci verbose Fix IPv6 docs Make integration tests use unique id for each test to make correlating tests with logs easier Make startup more reliable using start_period instead of many retries Workaround for Docker not allowing intercontainer IPv6 traffic on 'internal' networks, will be fixed in Docker 25, see: moby/moby#45649 Fix allowed hosts after slimming down host-dist.env Make issues with existing environment variables during deploy less likely Make unbound verbose logging configurable Test environment context Revert "Pytest colored output in Github" This reverts commit 43f3dbd. WIP: Add second integration test with ip6tables enabled Make rabbitmq healthcheck and startup more reliable Remove outdated config file Routinator basic healthcheck Fix logging issues when bringing ci environment up Update deployment docs Disable routinator to pull in data during test and develop Add website test to develop Test development environment in CI using ipv6 Fix host-dist.env file Improve instructions for DNSSEC reset Expose DNS port on IPv6 Update live-tests docs for ipv6 Production and batch overview images Move test-runner into integration-test Architecture docs Remove invalid TLSA from zone files Add running message Fix documentation for IPv6 on macOs, add HE tunnel mention, simplify Linux instructions Disable waiting for healthy routinator as it takes to long of a time for routinator to start. Remove outdated file Allow configuring encrypted passwords Document different test suites Fix subdomain redirects in nginx (quick fix pending discussion) Signed-off-by: Sasha Romijn <github@mxsasha.eu> Fix nginx redirects, attempt 2 This reverts commit b959a54. Fix conn domain letsencrypt Add $IPV6_IP_PUBLIC to ALLOWED_HOSTS in host-dist.env Add note about IPv6 addresses Signed-off-by: Sasha Romijn <github@mxsasha.eu> Improve resilience of certbot entrypoint Documentation formatting fix Signed-off-by: Sasha Romijn <github@mxsasha.eu> Update integration test security.txt expectation ref #932 Add https scheme to CSP policy to meet #810 Set our Referrer-Policy to same-origin to meet #357 Make sure live tests examples always use latests Remove old config file Reduce complication in IPv6 setup for deployments - Remove the need to assign and forward IPv6 subnets to the internal docker networks - bind unbound ports explicitly to external addresses to not conflict with os resolvers on localhost - update deployment documentation Test conn. redirects Revert IP in allowed hosts, add testcase Fix develop environment ipv6 test Use internal resolver for all queries made by app/worker Rename test-resolver to mock-resolver for clarity Work around issue with resolving race condition in CI Fix documentation for nginx auth config Simplify compose commands in docs Pull test runner for develop test There is no public IPv6 network on Github actions, so these jobs make no sense. Enable DNSSEC validation for internal resolver Reduce maximum test duration and retest timeout on development Fix linting, fix linting errors Keep celery exporter running after crash Resign zones every time, but persist the keys, so we don't have to update DNS after IP/hostname changes. Update docs Split internal resolver in validating and permissive to both accomodate ldns-dane which requires validation and other queries which needs to be passed through to the client despite failing DNSSEC validation Add live test for sites with known score Don't search integration test directory during 'normal' tests Add Gixy check for nginx configs Nginx log vhost name, use chrome for playwright trace Issue 894, disable hsts on https://conn. domain and return 404 not found for all non-configured domains Use --project-name instead of --env-file for everything but up/pull/run Add node exporter Fix initial configuration for batch Rewrite basic auth, require authentication when DEBUG is enabled Integrate certbot with webserver service, auto reload nginx on certificate changes Test-all Set max tasks per child, to reduce memory issues on ipv6kaartmetingen.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Getting started documentation on setting up a developer environment and integration tests: https://github.com/internetstandards/Internet.nl/blob/docker/documentation/Docker-getting-started.md
Other documents can be found at: https://github.com/internetstandards/Internet.nl/blob/docker/documentation/Docker.md
Collective todo:
Edit: to 'fix' the hundreds hidden items, Load more… a poor man's solution is this js: