Base image cannot verify package repo certs #322

benlangfeld · 2021-09-30T17:12:20Z

❯ docker run -it --rm phusion/passenger-full:1.0.19 apt-get update
Unable to find image 'phusion/passenger-full:1.0.19' locally
1.0.19: Pulling from phusion/passenger-full
345e3491a907: Pull complete
57671312ef6f: Pull complete
5e9250ddb7d0: Pull complete
412fc6a5183b: Pull complete
5972567e8811: Pull complete
2680f0dfbe5a: Pull complete
0ad992210c71: Pull complete
2465a49e78dd: Pull complete
Digest: sha256:f73634c743a094dc3b3e1a1b50434e6d80e50d7988cfa40c274bdfdcfe40b738
Status: Downloaded newer image for phusion/passenger-full:1.0.19
Ign:1 https://deb.nodesource.com/node_14.x focal InRelease
Err:2 https://deb.nodesource.com/node_14.x focal Release
  Certificate verification failed: The certificate is NOT trusted. The certificate chain uses expired certificate.  Could not handshake: Error in the certificate verification. [IP: 201.17.30.137 443]
Get:3 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]
Get:4 http://archive.ubuntu.com/ubuntu focal InRelease [265 kB]
Ign:5 https://oss-binaries.phusionpassenger.com/apt/passenger focal InRelease
Get:6 http://security.ubuntu.com/ubuntu focal-security/main Sources [223 kB]
Err:7 https://oss-binaries.phusionpassenger.com/apt/passenger focal Release
  Certificate verification failed: The certificate is NOT trusted. The certificate chain uses expired certificate.  Could not handshake: Error in the certificate verification. [IP: 109.107.35.58 443]
Get:8 http://security.ubuntu.com/ubuntu focal-security/restricted Sources [31.4 kB]
Get:9 http://security.ubuntu.com/ubuntu focal-security/multiverse Sources [8,644 B]
Get:10 http://security.ubuntu.com/ubuntu focal-security/universe Sources [87.7 kB]
Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [580 kB]
Get:12 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [30.1 kB]
Get:13 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [794 kB]
Get:14 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
Get:15 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,133 kB]
Get:16 http://archive.ubuntu.com/ubuntu focal-backports InRelease [101 kB]
Get:17 http://archive.ubuntu.com/ubuntu focal/universe Sources [12.3 MB]
Get:18 http://archive.ubuntu.com/ubuntu focal/main Sources [1,079 kB]
Get:19 http://archive.ubuntu.com/ubuntu focal/multiverse Sources [208 kB]
Get:20 http://archive.ubuntu.com/ubuntu focal/restricted Sources [7,198 B]
Get:21 http://archive.ubuntu.com/ubuntu focal/universe amd64 Packages [11.3 MB]
Get:22 http://archive.ubuntu.com/ubuntu focal/multiverse amd64 Packages [177 kB]
Get:23 http://archive.ubuntu.com/ubuntu focal/restricted amd64 Packages [33.4 kB]
Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages [1,275 kB]
Get:25 http://archive.ubuntu.com/ubuntu focal-updates/main Sources [551 kB]
Get:26 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Sources [20.7 kB]
Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Sources [31.4 kB]
Get:28 http://archive.ubuntu.com/ubuntu focal-updates/universe Sources [240 kB]
Get:29 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [630 kB]
Get:30 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,580 kB]
Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,082 kB]
Get:32 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [33.3 kB]
Get:33 http://archive.ubuntu.com/ubuntu focal-backports/universe Sources [3,509 B]
Get:34 http://archive.ubuntu.com/ubuntu focal-backports/main Sources [1,357 B]
Get:35 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [6,310 B]
Get:36 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [2,668 B]
Reading package lists... Done
E: The repository 'https://deb.nodesource.com/node_14.x focal Release' does not have a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
E: The repository 'https://oss-binaries.phusionpassenger.com/apt/passenger focal Release' does not have a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.

https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/

The text was updated successfully, but these errors were encountered:

lmancilla · 2021-09-30T18:10:46Z

Same thing here!

CamJN · 2021-09-30T18:15:05Z

We're working on a release with the ca-certificates bundle updated, in the meantime you can add this to your Dockerfile:

RUN mv /etc/apt/sources.list.d{,.bak}
RUN apt update && apt install -y ca-certificates
RUN mv /etc/apt/sources.list.d{.bak,}

kellyeryan · 2021-09-30T20:08:56Z

This syntax isn't working for me. Here is the error.

Step 2/97 : RUN mv /etc/apt/sources.list.d{,.bak}

 ---> Running in 2e6f6bbda13a

mv: missing destination file operand after '/etc/apt/sources.list.d{,.bak}'

Try 'mv --help' for more information.

The command '/bin/sh -c mv /etc/apt/sources.list.d{,.bak}' returned a non-zero code: 1

Command exited with non-zero status 1

I'll post if I find a solution.

CamJN · 2021-09-30T20:11:14Z

Ah sorry that's bash syntax, I usually switch my shell to bash, just use

Run mv /etc/apt/sources.list.d /etc/apt/sources.list.d.bak

And reverse the mv args when putting it back.

kellyeryan · 2021-09-30T20:12:57Z

Thanks so much and for responding so quickly!

kellyeryan · 2021-09-30T20:28:58Z

Still getting errors.

E: Failed to fetch https://oss-binaries.phusionpassenger.com/apt/passenger/dists/xenial/main/binary-amd64/Packages  server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none

Here's what I've put in the Dockerfile:

RUN mv /etc/apt/sources.list.d /etc/apt/sources.list.d.bak
RUN apt update && apt install -y ca-certificates
RUN mv /etc/apt/sources.list.d.bak /etc/apt/sources.list.d

aovertus · 2021-09-30T21:10:27Z

@kellyeryan its seem to work on our side, we added the snipped you shared

RUN mv /etc/apt/sources.list.d /etc/apt/sources.list.d.bak
RUN apt update && apt install -y ca-certificates
RUN mv /etc/apt/sources.list.d.bak /etc/apt/sources.list.d

CamJN · 2021-09-30T21:12:26Z

Hmm can you grep the /etc/apt/ directory for the passenger repo definition file? It's in /etc/apt/sources.list.d/passenger.list in the most recent passenger docker image, but you might be using an older one (xenial based). @kellyeryan

kellyeryan · 2021-09-30T21:17:28Z

Yes! It looks like it is xenial based.

RUN echo deb https://download:$PASSENGER_TOKEN@www.phusionpassenger.com/enterprise_apt xenial main > /etc/apt/sources.list.d/passenger.list

Is that what you need?

benlangfeld · 2021-09-30T21:20:17Z

Yes! It looks like it is xenial based.

RUN echo deb https://download:$PASSENGER_TOKEN@www.phusionpassenger.com/enterprise_apt xenial main > /etc/apt/sources.list.d/passenger.list

Is that what you need?

A 16.04 image has OpenSSL 1.0.2g. The new Let's Encrypt root certs require OpenSSL 1.1, which isn't available until 18.04. The latest version of the Passenger base image uses 20.04, which is why it works there.

CamJN · 2021-09-30T22:15:33Z

Hmm odd that it still tried to update the passenger repo while it was disabled. or was that after putting the /etc/apt/sources.list.d back? @kellyeryan

CamJN · 2021-10-01T02:38:32Z

@benlangfeld the xenial situation is fixable by removing the expired root (which i assume is what the ca-certificates package update does). I bumped several of our internal docker images today to fix our CI while working on the release so i know xenial CAN work.

prdanelli · 2021-10-01T09:06:30Z

I'm getting the following error after adding the suggested lines to our production Dockerfile.

E: Failed to fetch https://oss-binaries.phusionpassenger.com/apt/passenger/dists/focal/main/binary-amd64/Packages.gz  File has unexpected size (8692 != 7637). Mirror sync in progress? [IP: 109.107.35.58 443]
   Hashes of expected file:
    - Filesize:7637 [weak]
    - SHA512:75e76ea6c830822f1d0c999083573745f3cc0c574e139d0506390aed49f00279e7f38a9c89904e311eabd8b2657e952b4bc9faec4cee5e2adf2e64213447f17f
    - SHA256:a8e9ba32c5712648af09f66687179759c5bbd516f3e20eb75bd51ed3df642f52
    - SHA1:9ddc75fe982a40176ff798e3f3219c412f337932 [weak]
    - MD5Sum:d8d7d2a0ec172c41a522b11fddfa2796 [weak]
   Release file created at: Wed, 14 Jul 2021 07:29:51 +0000
E: Some index files failed to download. They have been ignored, or old ones used instead.
error building image: error building stage: failed to execute command: waiting for process to exit: exit status 100
ERROR: Job failed: command terminated with exit code 1

We're current on focal:

# cat /etc/apt/sources.list.d/passenger.list
deb https://oss-binaries.phusionpassenger.com/apt/passenger focal main

Gauravbtc · 2021-10-01T09:41:56Z

@prdanelli

I'm getting the following error after adding the suggested lines to our production Dockerfile.

E: Failed to fetch https://oss-binaries.phusionpassenger.com/apt/passenger/dists/focal/main/binary-amd64/Packages.gz  File has unexpected size (8692 != 7637). Mirror sync in progress? [IP: 109.107.35.58 443]
   Hashes of expected file:
    - Filesize:7637 [weak]
    - SHA512:75e76ea6c830822f1d0c999083573745f3cc0c574e139d0506390aed49f00279e7f38a9c89904e311eabd8b2657e952b4bc9faec4cee5e2adf2e64213447f17f
    - SHA256:a8e9ba32c5712648af09f66687179759c5bbd516f3e20eb75bd51ed3df642f52
    - SHA1:9ddc75fe982a40176ff798e3f3219c412f337932 [weak]
    - MD5Sum:d8d7d2a0ec172c41a522b11fddfa2796 [weak]
   Release file created at: Wed, 14 Jul 2021 07:29:51 +0000
E: Some index files failed to download. They have been ignored, or old ones used instead.
error building image: error building stage: failed to execute command: waiting for process to exit: exit status 100
ERROR: Job failed: command terminated with exit code 1

We're current on focal:

# cat /etc/apt/sources.list.d/passenger.list
deb https://oss-binaries.phusionpassenger.com/apt/passenger focal main

thanks for the solutions
I added this RUN echo "deb https://oss-binaries.phusionpassenger.com/apt/passenger focal main
command in production docker file but still it raising same error

prdanelli · 2021-10-01T12:24:09Z

@CamJN Sorry to bump this so soon, but have you managed to bundle the certificates for the focal image? We're not able to deploy fixes to production at the moment and its stressing management more than we'd like for a Friday :)

niltonvasques · 2021-10-01T13:45:30Z

After perform the proposed fix:

RUN mv /etc/apt/sources.list.d /etc/apt/sources.list.d.bak
RUN apt update && apt install -y ca-certificates
RUN mv /etc/apt/sources.list.d.bak /etc/apt/sources.list.d

We are still receiveing errors:

Err:8 https://oss-binaries.phusionpassenger.com/apt/passenger bionic/main amd64 Packages
  File has unexpected size (22378 != 21335). Mirror sync in progress? [IP: 109.107.35.58 443]
  Hashes of expected file:
   - Filesize:21335 [weak]
   - SHA512:4769d5f763843893dbd5a9001a28eb22cd6eb0fa377b18558832867e86742d9c4636a66759f83125d165b7ee9a87de083ada7cfaafa35ae250b3a6907cb9e262
   - SHA256:cfc513204b1b759d3e88afe699b75c8f1f148c3ea9cfc228c1212b9fdb4ffbc2
   - SHA1:53d40c29ebdb67664cd474b5f4023b445036adc4 [weak]
   - MD5Sum:1aef34387df9bb4fcaed56a2852d72b7 [weak]
  Release file created at: Wed, 14 Jul 2021 07:29:25 +0000
Fetched 9,034 B in 2s (4,243 B/s)
Reading package lists... Done
E: Failed to fetch https://oss-binaries.phusionpassenger.com/apt/passenger/dists/bionic/main/binary-amd64/Packages.gz  File has unexpected size (22378 != 21335). Mirror sync in progress? [IP: 109.107.35.58 443]
   Hashes of expected file:
    - Filesize:21335 [weak]
    - SHA512:4769d5f763843893dbd5a9001a28eb22cd6eb0fa377b18558832867e86742d9c4636a66759f83125d165b7ee9a87de083ada7cfaafa35ae250b3a6907cb9e262
    - SHA256:cfc513204b1b759d3e88afe699b75c8f1f148c3ea9cfc228c1212b9fdb4ffbc2
    - SHA1:53d40c29ebdb67664cd474b5f4023b445036adc4 [weak]
    - MD5Sum:1aef34387df9bb4fcaed56a2852d72b7 [weak]
   Release file created at: Wed, 14 Jul 2021 07:29:25 +0000
E: Some index files failed to download. They have been ignored, or old ones used instead

niltonvasques · 2021-10-01T14:06:54Z

For now, I was able to solve temporary by disabling the phusion sources list. Once this ticket get resolved we will reenable back.

RUN rm /etc/apt/sources.list.d/passenger.list
RUN apt-get update && apt-get install ca-certificates && apt-get update

illdelph · 2021-10-01T14:09:59Z

@niltonvasques This should only be temporary until a fix is deployed, but you should only revert back to the original source.list after any other package install has run.

RUN mv /etc/apt/sources.list.d /etc/apt/sources.list.d.bak
RUN apt update && apt install -y ca-certificates

.... other package installs .....

RUN mv /etc/apt/sources.list.d.bak /etc/apt/sources.list.d

Clearer example

RUN mv /etc/apt/sources.list.d /etc/apt/sources.list.d.bak
RUN apt update && apt install -y ca-certificates

RUN apt-get update && apt-get install -y -qq --no-install-recommends postgresql-client

RUN mv /etc/apt/sources.list.d.bak /etc/apt/sources.list.d

Again this is only a temporary fix but I hope it helps

lmancilla · 2021-10-01T14:21:55Z

I'm getting the following error after adding the suggested lines to our production Dockerfile.

E: Failed to fetch https://oss-binaries.phusionpassenger.com/apt/passenger/dists/focal/main/binary-amd64/Packages.gz  File has unexpected size (8692 != 7637). Mirror sync in progress? [IP: 109.107.35.58 443]
   Hashes of expected file:
    - Filesize:7637 [weak]
    - SHA512:75e76ea6c830822f1d0c999083573745f3cc0c574e139d0506390aed49f00279e7f38a9c89904e311eabd8b2657e952b4bc9faec4cee5e2adf2e64213447f17f
    - SHA256:a8e9ba32c5712648af09f66687179759c5bbd516f3e20eb75bd51ed3df642f52
    - SHA1:9ddc75fe982a40176ff798e3f3219c412f337932 [weak]
    - MD5Sum:d8d7d2a0ec172c41a522b11fddfa2796 [weak]
   Release file created at: Wed, 14 Jul 2021 07:29:51 +0000
E: Some index files failed to download. They have been ignored, or old ones used instead.
error building image: error building stage: failed to execute command: waiting for process to exit: exit status 100
ERROR: Job failed: command terminated with exit code 1

We're current on focal:

# cat /etc/apt/sources.list.d/passenger.list
deb https://oss-binaries.phusionpassenger.com/apt/passenger focal main

In the meanwhile, try replacing focal by groovy dist for Passenger:

RUN mv /etc/apt/sources.list.d /etc/apt/sources.list.d.bak
RUN apt update && apt install -y ca-certificates
RUN mv /etc/apt/sources.list.d.bak /etc/apt/sources.list.d
RUN sed -i 's/focal/groovy/g' /etc/apt/sources.list.d/passenger.list

Tested using phusion/passenger-ruby27

@niltonvasques @prdanelli

prdanelli · 2021-10-01T16:11:27Z

@illdelph this seems to have worked for me.

RUN mv /etc/apt/sources.list.d /etc/apt/sources.list.d.bak
RUN apt update && apt install -y ca-certificates

RUN apt-get update && apt-get install -y -qq --no-install-recommends postgresql-client

RUN mv /etc/apt/sources.list.d.bak /etc/apt/sources.list.d

Thank you.

CamJN · 2021-10-01T17:32:00Z

Ok new version (2.0.0) pushed to docker hub. This should be fixed now.

kwstannard · 2021-10-02T03:39:18Z

Can you tag latest ruby25 please?

teyamagu · 2021-10-04T01:50:48Z

passenger-ruby25 has been updated, but the content looks the same as 1.0.19

CamJN · 2021-10-04T06:46:33Z

@teyamagu @kwstannard the ruby 2.5 and 2.4 images were discontinued because those rubies are EOL, the tags got touched by the release scripts (this has been fixed since, so shouldn't happen again) but there are no changes.

teyamagu · 2021-10-04T07:20:50Z

@CamJN I got it. thanks!

CamJN · 2021-10-04T09:21:28Z

Since the new images are out i'm closing this.

CamJN mentioned this issue Sep 30, 2021

Passenger docker builds failing in CI. #324

Closed

codemonium mentioned this issue Oct 1, 2021

Update image to fix certificate issue powerhome/playbook#1625

Merged

6 tasks

CamJN mentioned this issue Oct 1, 2021

oss-binaries expired certificate causing issues phusion/passenger#2385

Closed

garytaylor added a commit to hmcts/et_full_system_gem that referenced this issue Oct 4, 2021

Bundled ca-certificates as workaround to phusion/passenger-docker#322

f828707

CamJN closed this as completed Oct 4, 2021

elsapassaro mentioned this issue Oct 7, 2021

Upgrade to Ubuntu 20.04 materialscloud-org/tools-barebone#17

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Base image cannot verify package repo certs #322

Base image cannot verify package repo certs #322

benlangfeld commented Sep 30, 2021

lmancilla commented Sep 30, 2021

CamJN commented Sep 30, 2021

kellyeryan commented Sep 30, 2021 •

edited

Loading

CamJN commented Sep 30, 2021

kellyeryan commented Sep 30, 2021

kellyeryan commented Sep 30, 2021

aovertus commented Sep 30, 2021

CamJN commented Sep 30, 2021

kellyeryan commented Sep 30, 2021 •

edited

Loading

benlangfeld commented Sep 30, 2021

CamJN commented Sep 30, 2021

CamJN commented Oct 1, 2021

prdanelli commented Oct 1, 2021 •

edited

Loading

Gauravbtc commented Oct 1, 2021 •

edited

Loading

prdanelli commented Oct 1, 2021

niltonvasques commented Oct 1, 2021

niltonvasques commented Oct 1, 2021 •

edited

Loading

illdelph commented Oct 1, 2021

lmancilla commented Oct 1, 2021

prdanelli commented Oct 1, 2021

CamJN commented Oct 1, 2021

kwstannard commented Oct 2, 2021

teyamagu commented Oct 4, 2021

CamJN commented Oct 4, 2021

teyamagu commented Oct 4, 2021

CamJN commented Oct 4, 2021

Base image cannot verify package repo certs #322

Base image cannot verify package repo certs #322

Comments

benlangfeld commented Sep 30, 2021

lmancilla commented Sep 30, 2021

CamJN commented Sep 30, 2021

kellyeryan commented Sep 30, 2021 • edited Loading

CamJN commented Sep 30, 2021

kellyeryan commented Sep 30, 2021

kellyeryan commented Sep 30, 2021

aovertus commented Sep 30, 2021

CamJN commented Sep 30, 2021

kellyeryan commented Sep 30, 2021 • edited Loading

benlangfeld commented Sep 30, 2021

CamJN commented Sep 30, 2021

CamJN commented Oct 1, 2021

prdanelli commented Oct 1, 2021 • edited Loading

Gauravbtc commented Oct 1, 2021 • edited Loading

prdanelli commented Oct 1, 2021

niltonvasques commented Oct 1, 2021

niltonvasques commented Oct 1, 2021 • edited Loading

illdelph commented Oct 1, 2021

lmancilla commented Oct 1, 2021

prdanelli commented Oct 1, 2021

CamJN commented Oct 1, 2021

kwstannard commented Oct 2, 2021

teyamagu commented Oct 4, 2021

CamJN commented Oct 4, 2021

teyamagu commented Oct 4, 2021

CamJN commented Oct 4, 2021

kellyeryan commented Sep 30, 2021 •

edited

Loading

kellyeryan commented Sep 30, 2021 •

edited

Loading

prdanelli commented Oct 1, 2021 •

edited

Loading

Gauravbtc commented Oct 1, 2021 •

edited

Loading

niltonvasques commented Oct 1, 2021 •

edited

Loading