chore: docker builds optimizations #1795

raducristianpopa · 2023-08-28T11:48:47Z

Changes proposed in this pull request

Introduce multi-stage Docker builds to decrease image sizes.

Context

Current progress:

REPOSITORY                        TAG       IMAGE ID       CREATED          SIZE
rafiki-frontend                   latest    8dbaccf3fc7a   12 minutes ago   216MB
rafiki-mock-ase                   latest    fe5e820589f8   12 minutes ago   219MB
rafiki-backend                    latest    687f5d9e6eb7   13 minutes ago   231MB
rafiki-auth                       latest    733332ba4fc3   13 minutes ago   206MB
rafiki-postman-signatures         latest    b97b04b2feef   13 minutes ago   177MB
redis                             7         69bb36a0f47b   13 days ago      157MB
postgres                          15        ee56d70bcdf1   2 weeks ago      433MB
ghcr.io/tigerbeetle/tigerbeetle   0.13.57   34025e6f8272   7 weeks ago      20.2MB

Checklist

netlify · 2023-08-28T11:48:52Z

✅ Deploy Preview for brilliant-pasca-3e80ec canceled.

Name	Link
🔨 Latest commit	`f5a0e05`
🔍 Latest deploy log	https://app.netlify.com/sites/brilliant-pasca-3e80ec/deploys/65007296f296d90008ad64d5

raducristianpopa

I reckon we can still make the image sizes much smaller, but this is a good start. All node images come with npm and yarn pre-installed.

localenv/mock-account-servicing-entity/Dockerfile

raducristianpopa · 2023-08-28T13:21:26Z

localenv/mock-account-servicing-entity/Dockerfile

+RUN apk add --no-cache \
+    python3 \
+    make \
+    g++


node-gyp requirements.

raducristianpopa · 2023-08-28T13:38:33Z

localenv/mock-account-servicing-entity/Dockerfile

-RUN apt update
-RUN apt install -y curl xz-utils python3 build-essential
-
-# version in curl is not the version used. Dependent on the last command
 RUN corepack enable
 RUN corepack prepare pnpm@7.25.1 --activate


Any particular reason we are still using pnpm@v7 instead of v8?

I guess we forgot to update it?

Have you tried with pnpm v8?

I have tried with pnpm v8 when I was running into the httpbis-digest-headers issue. It should also work with v8 without any problems. As for the choice to stick with v7, I recall @mkurapov mentioning an edge case or an issue that was preventing us from upgrading to v8, but I'm not entirely sure about this.

raducristianpopa · 2023-08-30T07:01:45Z

https://github.com/interledger/rafiki/actions/runs/6021572573/job/16334677808?pr=1795#step:3:407

I'm currently running into some issues due to the httpbis-digest-headers library, which is neither published nor contains a dist/lib folder in its repository. This library is built dynamically upon installation (https://github.com/interledger/httpbis-digest-headers/blob/eb2e41c25cb6f2b8c610b74a853f36c6be8dbb4b/package.json#L11C8-L11C8). Our prod-deps stage, is set up to only install essential dependencies and exclude all devDependencies, which means we don't have TS available for building the library.

sabineschaller · 2023-09-05T08:15:13Z

'm currently running into some issues due to the httpbis-digest-headers library, which is neither published nor contains a dist/lib folder in its repository. This library is built dynamically upon installation (https://github.com/interledger/httpbis-digest-headers/blob/eb2e41c25cb6f2b8c610b74a853f36c6be8dbb4b/package.json#L11C8-L11C8). Our prod-deps stage, is set up to only install essential dependencies and exclude all devDependencies, which means we don't have TS available for building the library.

We did publish the package, so we should just swap to using it: https://www.npmjs.com/package/httpbis-digest-headers

raducristianpopa · 2023-09-05T10:49:35Z

We did publish the package, so we should just swap to using it: https://www.npmjs.com/package/httpbis-digest-headers

We also need to use the NPM registry for http-signature-utils (https://github.com/interledger/open-payments/blob/1379177f3cc3b74536905c162fd9f872a70b5f47/packages/http-signature-utils/package.json#L28) before removing

rafiki/packages/auth/Dockerfile

Line 23 in c815408

RUN pnpm add -g typescript

Do you want me to update the http-signature-utils library as well?

sabineschaller · 2023-09-05T11:26:15Z

Do you want me to update the http-signature-utils library as well?

Yes please. Thank you!

sabineschaller

This is fantastic! There is a lot of repetition across the dockerfiles. I don't know if we could somehow reduce that. But that was the reason I failed when trying to reduce the size so maybe we have to live with the repetition.

Thank you! ♥️

raducristianpopa · 2023-09-07T14:26:37Z

httpbis-digest-headers issue - solved ✅.

Let me know if we should move to pnpm v8.

This reverts commit 24db14d.

sabineschaller · 2023-09-07T14:50:34Z

Yeah, let's move to pnpm v8

raducristianpopa · 2023-09-07T15:40:55Z

.github/workflows/rafiki/env-setup/action.yml

-      with:
-        version: 7


This can be removed since I've added pnpm's version in the root package.json.

rafiki/package.json

Lines 7 to 11 in 4a72018

"engines": {

"pnpm": "^8.7.4",

"node": "18"

},

"packageManager": "pnpm@8.7.4",

sabineschaller

Beautiful, works for me. You just need to resolve merge conflicts.

raducristianpopa added 6 commits August 25, 2023 09:49

Auth package dockerfile optimization

647e974

Remove libc6-compat

e4dc1cf

Merge branch 'main' into rp--optimize-docker-builds

6a5e410

Merge branch 'main' into rp--optimize-docker-builds

eb1a2f8

Remove test compose file

fb4d02f

Backend dockerfile optimization

2b264c7

github-actions bot added pkg: backend Changes in the backend package. pkg: auth Changes in the GNAP auth package. labels Aug 28, 2023

Frontend and ASE Dockerfile optimizations

bbbfbaa

github-actions bot added pkg: frontend Changes in the frontend package. pkg: mock-ase labels Aug 28, 2023

raducristianpopa commented Aug 28, 2023

View reviewed changes

raducristianpopa added 5 commits August 29, 2023 08:23

Local HTTP Signatures app docker optimization

30cd458

[Test] Bump PNPM major version

024c693

Use offline mode only for the build step

3ea267b

Test out pnpm v8

6895e39

Add --prefer-offline option and revert pnpm to v7

d3ff7b6

raducristianpopa force-pushed the rp--optimize-docker-builds branch from 02fc2e0 to d3ff7b6 Compare August 29, 2023 06:06

raducristianpopa added 2 commits August 29, 2023 09:12

Try out cache busting

321ff2a

Remove CACHE_BUST arg

a0d8c84

raducristianpopa marked this pull request as ready for review August 29, 2023 06:22

raducristianpopa requested review from sabineschaller, mkurapov, BlairCurrey and njlie August 29, 2023 06:22

raducristianpopa added 2 commits August 30, 2023 08:08

Add prefer-offline options to all prod-deps stage

439e808

Remove --prefer-offline from auth and backend Dockerfile

e6c5fd7

raducristianpopa added 2 commits September 5, 2023 13:39

Merge branch 'main' into rp--optimize-docker-builds

292d063

Fetch httpbis-digest-headers from registry

8ed59c2

raducristianpopa mentioned this pull request Sep 6, 2023

chore(deps): pin httpbis-digest-headers version interledger/open-payments#277

Merged

sabineschaller previously approved these changes Sep 7, 2023

View reviewed changes

beniaminmunteanu previously approved these changes Sep 7, 2023

View reviewed changes

raducristianpopa added 6 commits September 7, 2023 16:59

Merge branch 'main' into rp--optimize-docker-builds

e5dd63c

Bump http-signature-utils version

8627793

Do not install TS globally when building prod deps

b6577a8

Make sure that http-signature-utils is bumped in every package

ba69900

Pin alpine version

455b164

Cache bust

24db14d

raducristianpopa dismissed stale reviews from beniaminmunteanu and sabineschaller via 24db14d September 7, 2023 14:21

Revert "Cache bust"

1bfa0eb

This reverts commit 24db14d.

raducristianpopa requested a review from sabineschaller September 7, 2023 14:27

Update pnpm to v8

4a72018

github-actions bot added the type: ci Changes to the CI label Sep 7, 2023

raducristianpopa commented Sep 7, 2023

View reviewed changes

sabineschaller previously approved these changes Sep 12, 2023

View reviewed changes

Merge branch 'main' into rp--optimize-docker-builds

f5a0e05

raducristianpopa dismissed sabineschaller’s stale review via f5a0e05 September 12, 2023 14:15

sabineschaller approved these changes Sep 13, 2023

View reviewed changes

raducristianpopa merged commit 05eaadd into main Sep 13, 2023
25 checks passed

raducristianpopa deleted the rp--optimize-docker-builds branch September 13, 2023 11:29

BlairCurrey mentioned this pull request Mar 18, 2024

chore: OpenAPI spec changes & other support for running localenv in docker windows #2530

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: docker builds optimizations #1795

chore: docker builds optimizations #1795

raducristianpopa commented Aug 28, 2023 •

edited

Loading

netlify bot commented Aug 28, 2023 •

edited

Loading

raducristianpopa left a comment

raducristianpopa Aug 28, 2023

raducristianpopa Aug 28, 2023

sabineschaller Sep 5, 2023

sabineschaller Sep 7, 2023

raducristianpopa Sep 7, 2023

raducristianpopa commented Aug 30, 2023 •

edited

Loading

sabineschaller commented Sep 5, 2023

raducristianpopa commented Sep 5, 2023

sabineschaller commented Sep 5, 2023

sabineschaller left a comment

raducristianpopa commented Sep 7, 2023

sabineschaller commented Sep 7, 2023

raducristianpopa Sep 7, 2023

sabineschaller left a comment

	"engines": {
	"pnpm": "^8.7.4",
	"node": "18"
	},
	"packageManager": "pnpm@8.7.4",

chore: docker builds optimizations #1795

chore: docker builds optimizations #1795

Conversation

raducristianpopa commented Aug 28, 2023 • edited Loading

Changes proposed in this pull request

Context

Checklist

netlify bot commented Aug 28, 2023 • edited Loading

✅ Deploy Preview for brilliant-pasca-3e80ec canceled.

raducristianpopa left a comment

Choose a reason for hiding this comment

raducristianpopa Aug 28, 2023

Choose a reason for hiding this comment

raducristianpopa Aug 28, 2023

Choose a reason for hiding this comment

sabineschaller Sep 5, 2023

Choose a reason for hiding this comment

sabineschaller Sep 7, 2023

Choose a reason for hiding this comment

raducristianpopa Sep 7, 2023

Choose a reason for hiding this comment

raducristianpopa commented Aug 30, 2023 • edited Loading

sabineschaller commented Sep 5, 2023

raducristianpopa commented Sep 5, 2023

sabineschaller commented Sep 5, 2023

sabineschaller left a comment

Choose a reason for hiding this comment

raducristianpopa commented Sep 7, 2023

sabineschaller commented Sep 7, 2023

raducristianpopa Sep 7, 2023

Choose a reason for hiding this comment

sabineschaller left a comment

Choose a reason for hiding this comment

raducristianpopa commented Aug 28, 2023 •

edited

Loading

netlify bot commented Aug 28, 2023 •

edited

Loading

raducristianpopa commented Aug 30, 2023 •

edited

Loading