Skip to content

test: reduce GitLab Puma memory use - #629

Merged
o1da merged 1 commit into
Roche:mainfrom
evrardj-roche:flaky-gitlab-test-fix
Aug 31, 2026
Merged

test: reduce GitLab Puma memory use#629
o1da merged 1 commit into
Roche:mainfrom
evrardj-roche:flaky-gitlab-test-fix

Conversation

@evrardj-roche

Copy link
Copy Markdown
Contributor

Puma reconfiguration cause temporary issues, documented in 1. We do not need reconfiguration during CI.

Closes: #628

@evrardj-roche
evrardj-roche marked this pull request as ready for review August 26, 2026 11:30
@evrardj-roche
evrardj-roche requested a review from defreng as a code owner August 26, 2026 11:30
@evrardj-roche

Copy link
Copy Markdown
Contributor Author

Let's do a few rounds of CI checks before merging to confirm it's the right fix.

Puma reconfiguration cause temporary issues, documented in [1].
We do not need reconfiguration during CI.

Closes: Roche#628

[1]: https://federal-support.gitlab.com/hc/en-us/articles/40212946732692-Frequent-502-Errors-and-Restarts-Due-to-Puma-Single-Mode-Configuration

Signed-off-by: Jean-Philippe Evrard <jean-philippe.evrard+rochepub@external.roche.com>
@evrardj-roche

evrardj-roche commented Aug 26, 2026

Copy link
Copy Markdown
Contributor Author

First pass OK.

@evrardj-roche

evrardj-roche commented Aug 26, 2026

Copy link
Copy Markdown
Contributor Author

Second pass failed on 502 (x1)
While this code might have helped reducing the 502 occurences, it's not enough to fix the flakiness.

@evrardj-roche

evrardj-roche commented Aug 26, 2026

Copy link
Copy Markdown
Contributor Author

Third pass OK

@evrardj-roche

Copy link
Copy Markdown
Contributor Author

Fourth pass fail (x1)

@evrardj-roche

Copy link
Copy Markdown
Contributor Author

Fifth pass OK

@evrardj-roche

evrardj-roche commented Aug 26, 2026

Copy link
Copy Markdown
Contributor Author

6th pass: Fail (x1)

@evrardj-roche evrardj-roche reopened this Aug 26, 2026
@evrardj-roche evrardj-roche reopened this Aug 26, 2026
@evrardj-roche
evrardj-roche marked this pull request as draft August 26, 2026 15:46
@evrardj-roche
evrardj-roche marked this pull request as ready for review August 26, 2026 15:46
@evrardj-roche

Copy link
Copy Markdown
Contributor Author

it seems CI is now b0rked.

@evrardj-roche

Copy link
Copy Markdown
Contributor Author

7th pass working.

@evrardj-roche

evrardj-roche commented Aug 27, 2026

Copy link
Copy Markdown
Contributor Author

8th pass: Fail (x1)

@evrardj-roche

Copy link
Copy Markdown
Contributor Author

9th pass: Success.

@evrardj-roche

Copy link
Copy Markdown
Contributor Author

10th pass: Fail.

This makes it a 50% success chance to merge, which is an improvement, but not by a large margin.

@evrardj-roche

Copy link
Copy Markdown
Contributor Author
#23 [linux/arm64 frontend-builder 4/5] RUN npm install
#23 140.0 npm error code ECONNRESET
#23 140.0 npm error network aborted
#23 140.0 npm error network This is a problem related to network connectivity.
#23 140.0 npm error network In most cases you are behind a proxy or have bad network settings.
#23 140.0 npm error network
#23 140.0 npm error network If you are behind a proxy, please make sure that the 'proxy' config is set properly.  See: 'npm help config'
#23 140.1 npm notice
#23 140.1 npm notice New major version of npm available! 11.17.0 -> 12.0.2
#23 140.1 npm notice Changelog: https://github.com/npm/cli/releases/tag/v12.0.2
#23 140.1 npm notice To update run: npm install -g npm@12.0.2
#23 140.1 npm notice
#23 140.1 npm error A complete log of this run can be found in: /root/.npm/_logs/2026-08-27T15_10_59_217Z-debug-0.log
#23 ERROR: process "/bin/sh -c npm install" did not complete successfully: exit code: 1

#39 [linux/arm64 backend-builder 3/7] RUN python -m pip install -U pip wheel
#39 CANCELED
------
 > [linux/arm64 frontend-builder 4/5] RUN npm install:
140.0 npm error network This is a problem related to network connectivity.
140.0 npm error network In most cases you are behind a proxy or have bad network settings.
140.0 npm error network
140.0 npm error network If you are behind a proxy, please make sure that the 'proxy' config is set properly.  See: 'npm help config'
140.1 npm notice
140.1 npm notice New major version of npm available! 11.17.0 -> 12.0.2
140.1 npm notice Changelog: https://github.com/npm/cli/releases/tag/v12.0.2
140.1 npm notice To update run: npm install -g npm@12.0.2
140.1 npm notice
140.1 npm error A complete log of this run can be found in: /root/.npm/_logs/2026-08-27T15_10_59_217Z-debug-0.log
------
WARNING: No output specified with docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load
Dockerfile:25
--------------------
  23 |     # Build the application
  24 |     WORKDIR /app
  25 | >>> RUN npm install
  26 |     RUN npm run build
  27 |     
--------------------
ERROR: failed to build: failed to solve: process "/bin/sh -c npm install" did not complete successfully: exit code: 1

This is an unrelated issue.

@evrardj-roche

Copy link
Copy Markdown
Contributor Author

Another issue appeared:

#41 [linux/arm64 frontend-builder 5/5] RUN npm run build
#41 78.54 The plugin "vite-tsconfig-paths" is detected. Vite now supports tsconfig paths resolution natively via the resolve.tsconfigPaths option. You can remove the plugin and set resolve.tsconfigPaths: true in your Vite config instead.
#41 79.01 vite v8.0.16 building client environment for production...
#41 79.21 
transforming...[PLUGIN_TIMINGS] Your build spent significant time in plugins. Here is a breakdown:
#41 91.93   - vite:css (28%)
#41 91.93   - vite:resolve-builtin:get-environment (17%)
#41 91.93 See https://rolldown.rs/options/checks#plugintimings for more details.
#41 91.93 
#41 101.7 [PLUGIN_TIMINGS] Your build spent significant time in plugins. Here is a breakdown:
#41 101.7   - vite:worker-import-meta-url (32%)
#41 101.7   - vite:asset-import-meta-url (21%)
#41 101.7   - vite:css (16%)
#41 101.7   - vite:resolve-builtin:get-environment (12%)
#41 101.7   - vite:worker (9%)
#41 101.7 See https://rolldown.rs/options/checks#plugintimings for more details.
#41 101.7 
#41 101.8 ✓ 1710 modules transformed.
#41 101.8 ✗ Build failed in 22.80s
#41 101.8 error during build:
#41 101.8 Build failed with 1 error:
#41 101.8 
#41 101.8 [plugin vite:css-post] /app/node_modules/normalize.css/normalize.css?inline
#41 101.8 Error: [lightningcss minify] Cannot find module '../lightningcss.linux-arm64-musl.node'
#41 101.8 Require stack:
#41 101.8 - /app/node_modules/lightningcss/node/index.js
#41 101.8     at Module._resolveFilename (node:internal/modules/cjs/loader:1564:15)
#41 101.8     at wrapResolveFilename (node:internal/modules/cjs/loader:1118:27)
#41 101.8     at defaultResolveImplForCJSLoading (node:internal/modules/cjs/loader:1142:10)
#41 101.8     at resolveForCJSWithHooks (node:internal/modules/cjs/loader:1169:12)
#41 101.8     at Module._load (node:internal/modules/cjs/loader:1341:5)
#41 101.8     at wrapModuleLoad (node:internal/modules/cjs/loader:261:19)
#41 101.8     at Module.require (node:internal/modules/cjs/loader:1674:12)
#41 101.8     at require (node:internal/modules/helpers:157:16)
#41 101.8     at Object.<anonymous> (/app/node_modules/lightningcss/node/index.js:20:12)
#41 101.8     at Module._compile (node:internal/modules/cjs/loader:1929:14)
#41 101.8     at aggregateBindingErrorsIntoJsError (file:///app/node_modules/rolldown/dist/shared/error-BuvQYXuZ.mjs:48:18)
#41 101.8     at unwrapBindingResult (file:///app/node_modules/rolldown/dist/shared/error-BuvQYXuZ.mjs:18:128)
#41 101.8     at #build (file:///app/node_modules/rolldown/dist/shared/rolldown-build-CrPk_lZe.mjs:3246:34)
#41 101.8     at async buildEnvironment (file:///app/node_modules/vite/dist/node/chunks/node.js:33253:64)
#41 101.8     at async Object.build (file:///app/node_modules/vite/dist/node/chunks/node.js:33675:19)
#41 101.8     at async Object.buildApp (file:///app/node_modules/vite/dist/node/chunks/node.js:33672:153)
#41 101.8     at async CAC.<anonymous> (file:///app/node_modules/vite/dist/node/cli.js:777:3) {
#41 101.8   errors: [Getter/Setter]
#41 101.8 }
#41 ERROR: process "/bin/sh -c npm run build" did not complete successfully: exit code: 1

Seems like an issue with lightningcss.linux-arm64-musl.node, which triggers my question: Do we care about arm right now?

@evrardj-roche

Copy link
Copy Markdown
Contributor Author

Rebuilding in case it was a flake (it used to work at the beginning of this initiative)

@evrardj-roche

Copy link
Copy Markdown
Contributor Author

@o1da even though this is not fully improving, it's better than without it. I will work on two other fixes: 1) resilience to 502 errors 2) improving test stack to produce less 502 errors (moving to multinode). This patch is still worth it, imo.

@evrardj-roche

Copy link
Copy Markdown
Contributor Author

I am still investigating something that might improve further, and had different settings (going multi node mode). But as I am doing this on the side, it might take time.

Hence why I am thinking this is worth it already.

@o1da
o1da merged commit 41b7e14 into Roche:main Aug 31, 2026
160 of 180 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI and local tests are flaky

2 participants