[Question] Docker: apk add: network error (check Internet connection and firewall)

[mk-pmb]

Describe the bug

apk in docker fails to connect to the outbound proxy on my LAN, and thus cannot install packages.

Steps to Reproduce

1. Clone the repo. (on branch master = 97c4165)
2. Run dockerized docker-compose with "up" command.
   • If that fails with syntax error, change the compose file version to "2.1".
   • If you get errors like "canceling authentication due to timeout" or "Invalid memory access (signal 11) at address 0x…", change the postgres version to 16.
3. See connection timeout from apk albeit https_proxy is set in the environment of docker-compose.
4. In docker-compose.yml, above the line INVIDIOUS_CONFIG: |, insert https_proxy: 'http://proxy-hostname.lan:8884/'
5. Retry, see apk now has DNS failure for resolving the LAN hostname. (So at least we know the proxy setting was effective.)
6. Modify the env var to use the LAN IP instead.
7. Retry, see apk being back to network error (check Internet connection and firewall)

Logs

Building invidious
Step 1/30 : FROM crystallang/crystal:1.8.2-alpine AS builder
 ---> 9bdcd2cc2656
Step 2/30 : RUN apk add --no-cache sqlite-static yaml-static
 ---> Running in 35763f4cdece
fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/main/x86_64/APKINDEX.tar.gz
WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.17/main: network error (check Internet connection and firewall)
fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/community/x86_64/APKINDEX.tar.gz
WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.17/community: network error (check Internet connection and firewall)
ERROR: unable to select packages:
  sqlite-static (no such package):
    required by: world[sqlite-static]
Service 'invidious' failed to build: The command '/bin/sh -c apk add --no-cache sqlite-static yaml-static' returned a non-zero code: 1

Screenshots

Additional context

• OS (if applicable): Ubuntu 20.04.6 LTS (focal)
• Docker version 20.10.22, build 3a2c30b

[unixfox]

Not an issue with invidious itself. Please read the official documentation for docker: https://docs.docker.com/network/proxy/

[unixfox]

But even if you manage to compile the image, invidious will not work because it doesn't support yet http proxies: #301

[mk-pmb]

Please read the official documentation for docker:

Which part were you referring to? All my other projects work fine, so it seems I already have the proper config in place. The fact that choosing the hostname causes DNS lookup error shows that apk does receive and use my proxy variable. Still, for unknown reason, it cannot connect, while other containers can:

$ docker run --rm --env https_proxy node:16 curl --silent --head https://github.com/ | grep html
content-type: text/html; charset=UTF-8

Edit: And it seems to not be apk's bug either:

$ docker run --rm --env https_proxy alpine:latest apk add --no-cache yaml-static
fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/community/x86_64/APKINDEX.tar.gz
(1/1) Installing yaml-static (0.2.5-r0)
OK: 6 MiB in 15 packages

Grasping for straws, and ignoring my own observation about the DNS error, I also tried adding ARG https_proxy in docker/Dockerfile above the RUN apk add line, but it didn't change behavior.

Update: I managed to docker build --build-arg https_proxy . successfully (in the docker subdir), but when I then run docker-compose up, it still tries and fails apk.

Update: Setting a literal value in the Dockerfile (ARG https_proxy='http:// … proxy's LAN IP … :8884/) makes it run up to Checking player dependencies, and an error in ../scripts/fetch-player-dependencies.cr, so I guess that is the issue you meant above.

Once we get proxies to work in crystal, we should try and see if we can make the Dockerfile and docker-compose work more smoothly.

[absidue]

so I guess that is the bug you meant above.

Features that haven't been implemented yet are not bugs. Supporting proxies requires extra code that hasn't been added yet, so it's not a bug. Bugs are problems in the existing code.

[mk-pmb]

You're right. I changed my wording from "bug" to "issue". I considered it a bug because it bugged me. However, the important part is that a reliable patch is already available.

Edit: On 2nd thought, it's more of a personal preference where to draw the line whether lack of code is a a bug, missing feature, or even desirable to not bloat the code base.
Some people expect cleanup tasks like freeing memory or closing file descriptors. Others may think it bloats the codebase because the process usually terminates quickly anyway.
Some people expect their grep to have a --line-buffered option. Others prefer a slightly smaller grep because they already have line buffer setup code in stdbuf.
Some people expect that any file download code supports proxy config via env vars. Others think it's bloat that can be easily shimmed using LD_PRELOAD.