[WIP]: Harden images and reorg #647

bstrdsmkr · 2024-07-05T18:04:31Z

Description

Harden frontend image
- rebase on https://github.com/nginxinc/docker-nginx-unprivileged
  - removes privileged port usage in base image
  - moves writeable directories out of user control
- Updates throughout to use relative links only so that images can be build once and used in any deployment
  - All configuration is now retrieved from the backend at runtime so that the image is fully static and immutable
Harden backend image
- move writable directories out of user control
  - run collectstatic during image build
- generate SBOM for backend at well-known location
Upgrade postgres to address vulnerabilities
Unify configuration to single .env file
Set sane defaults for config so that docker compose:
- runs with podman compose
- stands up fully functional stack out of the box
Configure gunicorn for backend automatically based on resources available
Fix broken links and warnings in docs

Type of change

Bug fix (non-breaking change which fixes an issue)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

How Has This Been Tested?

Local Tests
CI/CD Build

Checklist

My code follows the style guidelines of this project
I have performed a self-review of my own code
My changes generate no new warnings
If applicable - I have commented my code, particularly in hard-to-understand areas
If applicable - I have made corresponding changes to the documentation
If applicable - I have added tests that prove my fix is effective or that my feature works
If applicable - New and existing unit tests pass locally with my changes
Any dependent changes have been merged and published in downstream modules

sashakames · 2024-07-11T16:34:47Z

Hey, could the Docker Compose .yaml files be restored, please? We still use that as a single node alternative to K8S/Helm? Thanks!

bstrdsmkr · 2024-07-11T16:49:55Z

@sashakames first I just want to say that I know this is rough and needs a lot of work so you're not going to hurt my feelings, please give it a thorough scrub 😁

re: Docker compose -- my drive there was to consolidate the various pieces into a single Docker compose file that new users could stand up out-of-the-box and not need to configure anything. Then an overlay compose file could be used to override things that needed to be changed for production (in conjunction with the existing env vars). So, there should now be only two Docker compose related files in the root of the repo. (Fair warning that I can't fully test the Production setup so I'm hoping you can 😁 )

Does that address your concern?

bstrdsmkr

Self-review to help with explaining some rationale

bstrdsmkr · 2024-07-28T21:11:42Z

backend/.envs/.django

The *.env files were removed in favor of ensuring that all settings had a sane default out of the box though each service still respects settings loaded from .env files as before in order to preserve backwards compatibility

bstrdsmkr · 2024-07-28T21:12:55Z

backend/Dockerfile

@@ -1,4 +1,4 @@
-FROM python:3.9-slim-buster
+FROM python:3.9-slim


removing the -buster suffix allows automatic building on newer versions to avoid outdated packages

bstrdsmkr · 2024-07-28T21:15:19Z

backend/Dockerfile

-COPY ./docker/production/django/entrypoint /entrypoint
-RUN sed -i 's/\r$//g' /entrypoint
-RUN chmod +x /entrypoint
-RUN chown django /entrypoint


Files and directories owned by the running user are a security risk. Removing the entrypoint script in favor of running the gunicorn server directly allows avoiding the user chown

bstrdsmkr · 2024-07-28T21:16:08Z

backend/Dockerfile

-COPY --chown=django:django . /app
-
-USER django
+RUN python /app/manage.py collectstatic --noinput \


Running collectstatic during build allows faster startup and an immutable image

bstrdsmkr · 2024-07-28T21:18:47Z

backend/Dockerfile

-USER django
+RUN python /app/manage.py collectstatic --noinput \
+    && mkdir -p /app/staticfiles/.well-known \
+    && cyclonedx-py requirements requirements/base.txt --output-format json --outfile /app/staticfiles/.well-known/bom


Satisfies Executive Order 14028

bstrdsmkr · 2024-07-28T22:12:45Z

frontend/nginx.conf

+    # the production load balancer (ingress controller in k8s, Traefik on
+    # bare metal/vms) should route these paths before they reach this container.
+    location ~* ^/proxy|api|accounts|account-confirm-email|swagger|redoc|frontend-config\.js {
+        proxy_pass http://10.89.9.82:5000$request_uri;


using ipaddress instead of http://django as nginx's proxy_pass doesn't include docker's DNS resolver by default. We could add resolver 127.0.0.11 ipv6=off; to point it to docker compose's DNS resolver, but then it's hard coded and doesn't work for things like podman compose

bstrdsmkr · 2024-07-28T22:14:44Z

frontend/public/index.html

-      gtag('js', new Date());
-      gtag('config', '%REACT_APP_GOOGLE_ANALYTICS_TRACKING_ID%');
+    <!-- Load config from backend -->
+    <script type="text/javascript">


This is the most important frontend change -- load FRONTEND_SETTINGS from the backend endpoint and make it available on the window object

bstrdsmkr · 2024-07-28T22:15:31Z

frontend/src/api/index.test.ts

Could more be tested here?

bstrdsmkr · 2024-07-28T22:17:16Z

frontend/src/api/routes.ts

Make all routes relative to the page they're being served from so that the same image can be served from any url

bstrdsmkr · 2024-07-29T13:29:07Z

frontend/src/components/Globus/DatasetDownload.tsx

@@ -316,13 +313,9 @@ const DatasetDownloadForm: React.FC<React.PropsWithChildren<unknown>> = () => {
            }
          })
          .catch(async (error: ResponseError) => {
-            if (error.message !== '') {


Always log errors to console so they show up in test logs

sashakames · 2024-10-22T22:35:50Z

@bstrdsmkr You'll need to merge/rebase with the update to master done today, then we are making this priority #1

bstrdsmkr force-pushed the staticbuild branch 2 times, most recently from c7a0561 to 2106e20 Compare July 6, 2024 00:00

Harden images and reorg

bf6afce

bstrdsmkr force-pushed the staticbuild branch from 2106e20 to bf6afce Compare July 7, 2024 16:32

downiec self-requested a review July 9, 2024 01:17

sashakames marked this pull request as draft July 11, 2024 16:33

bstrdsmkr commented Jul 29, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP]: Harden images and reorg #647

[WIP]: Harden images and reorg #647

bstrdsmkr commented Jul 5, 2024

sashakames commented Jul 11, 2024

bstrdsmkr commented Jul 11, 2024

bstrdsmkr left a comment

bstrdsmkr Jul 28, 2024

bstrdsmkr Jul 28, 2024

bstrdsmkr Jul 28, 2024

bstrdsmkr Jul 28, 2024

bstrdsmkr Jul 28, 2024

bstrdsmkr Jul 28, 2024

bstrdsmkr Jul 28, 2024

bstrdsmkr Jul 28, 2024

bstrdsmkr Jul 28, 2024

bstrdsmkr Jul 29, 2024

sashakames commented Oct 22, 2024

		@@ -1,4 +1,4 @@
		FROM python:3.9-slim-buster
		FROM python:3.9-slim

[WIP]: Harden images and reorg #647

Are you sure you want to change the base?

[WIP]: Harden images and reorg #647

Conversation

bstrdsmkr commented Jul 5, 2024

Description

Type of change

How Has This Been Tested?

Checklist

sashakames commented Jul 11, 2024

bstrdsmkr commented Jul 11, 2024

bstrdsmkr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sashakames commented Oct 22, 2024