Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Debian package repositories broken: Unknown date format Bad header data #1399

Closed
aahlenst opened this issue Jun 17, 2020 · 33 comments
Closed
Assignees
Milestone

Comments

@aahlenst
Copy link
Contributor

Since Wed, June 17, around noon (UTC):

rm -rf /var/lib/apt/lists/* && apt-get update
Get:1 http://archive.ubuntu.com/ubuntu focal InRelease [265 kB]
Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [107 kB]                                                
Get:3 http://security.ubuntu.com/ubuntu focal-security InRelease [107 kB]                                  
Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [98.3 kB]                                 
Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages [1275 kB]                                             
Ign:6 https://adoptopenjdk.jfrog.io/adoptopenjdk/deb focal InRelease                                                  
Get:7 http://archive.ubuntu.com/ubuntu focal/restricted amd64 Packages [33.4 kB]                      
Get:8 http://archive.ubuntu.com/ubuntu focal/universe amd64 Packages [11.3 MB]                                  
Err:9 https://adoptopenjdk.jfrog.io/adoptopenjdk/deb focal Release                                             
  Unknown date format Bad header data [IP: 35.231.52.82 443]
Get:10 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [42.3 kB]
Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [13.2 kB]
Get:12 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [1077 B]
Get:13 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [134 kB]
Get:14 http://archive.ubuntu.com/ubuntu focal/multiverse amd64 Packages [177 kB]             
Get:15 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [137 kB]      
Get:16 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [252 kB]        
Get:17 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [13.2 kB]  
Get:18 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [1077 B]
Get:19 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [2900 B]
Reading package lists... Done                   
E: The repository 'https://adoptopenjdk.jfrog.io/adoptopenjdk/deb focal Release' does not have a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.

Problem seems to be the HTTP headers:

$ curl -I https://adoptopenjdk.jfrog.io/adoptopenjdk/deb/dists/focal/Release
HTTP/1.1 200 OK
Date: Wed, 17 Jun 2020 13:19:39 GMT
Content-Type: application/octet-stream
Content-Length: 5283
Connection: keep-alive
X-Artifactory-Id: adoptopenjdk-artifactory-primary-0-adoptopenjdk
X-Artifactory-Node-Id: adoptopenjdk-primary
Last-Modified: Wed, 17 Jun 2020 14:56:24 CEST
ETag: dba4ba63e15993dfac53b7f89a541a1a37224c74
X-Checksum-Sha1: dba4ba63e15993dfac53b7f89a541a1a37224c74
X-Checksum-Sha256: a9b67e576a3de6d35097144d6ed73e4ae2dbf6f0a722d2362a113e2d528e2b4e
X-Checksum-Md5: b60071b2b99200098204a6fa76c24387
Accept-Ranges: bytes
X-Artifactory-Filename: Release
Content-Disposition: attachment; filename="Release"; filename*=UTF-8''Release
Cache-Control: no-store
Strict-Transport-Security: max-age=15724800; includeSubDomains
X-Request-ID: 0064f103ef6d81f8fd3765b85c9d5910

Problem seems to be this line: Last-Modified: Wed, 17 Jun 2020 14:56:24 CEST. This should be UTC or GMT, not CEST.

This seems to be a known problem in Artifactory: https://www.jfrog.com/jira/browse/RTFACT-21640. But it should already have been fixed, so I have no idea why it pops up now.

Regenerating the package index hasn't helped.

I'm going to raise this with JFrog.

@aahlenst aahlenst self-assigned this Jun 17, 2020
@karianna karianna added this to TODO in infrastructure via automation Jun 17, 2020
@karianna karianna moved this from TODO to In Progress in infrastructure Jun 17, 2020
@fvlankvelt
Copy link

Running into this as well - am not very familiar with the Debian formats, but a quick look through the specs seems to indicate it's the 'Date' field in the 'Release' file:
Date: Wed, 17 Jun 2020 12:56:24 UTC

@detro
Copy link

detro commented Jun 17, 2020

Yep, the Release file DOES contain UTC. An example one:

Origin: Artifactory
Label: Artifactory
Suite: jessie
Codename: jessie
Date: Wed, 17 Jun 2020 12:56:30 UTC
Acquire-By-Hash: yes
Components: main
Architectures: amd64 arm64 armhf i386 ppc64el s390x
MD5Sum:
 f5d500a3fba5d6412b1faa372ff5fdac           124112 main/binary-amd64/Packages
 b3b5b043b29650043c624b97c0aa1b6e             9591 main/binary-amd64/Packages.bz2
 2cc9db1090ede6ca6ad6427a174c90e2            11277 main/binary-amd64/Packages.gz
 9b8bb25f51d5e9fb30c04ded2ec5510b            39582 main/binary-arm64/Packages
... snip ...

File downloaded from: https://adoptopenjdk.jfrog.io/adoptopenjdk/deb/dists/jessie/

@karianna karianna added this to the June 2020 milestone Jun 17, 2020
@detro
Copy link

detro commented Jun 17, 2020

For reference, this happens when then running an apt update:

root@9db94be72395:/# apt update
Hit:1 http://security.debian.org/debian-security stretch/updates InRelease
Ign:2 http://deb.debian.org/debian stretch InRelease
Hit:3 http://deb.debian.org/debian stretch-updates InRelease
Hit:4 http://deb.debian.org/debian stretch Release
Ign:6 https://adoptopenjdk.jfrog.io/adoptopenjdk/deb stretch InRelease
Err:7 https://adoptopenjdk.jfrog.io/adoptopenjdk/deb stretch Release
  Unknown date format
Reading package lists... Done
E: The repository 'https://adoptopenjdk.jfrog.io/adoptopenjdk/deb stretch Release' does not have a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.

@tader
Copy link

tader commented Jun 17, 2020

Debian's own Release also contains a UTC Date, so that is probably fine...

curl -s http://ftp.debian.org/debian/dists/buster/Release |head:

Origin: Debian
Label: Debian
Suite: stable
Version: 10.4
Codename: buster
Changelogs: http://metadata.ftp-master.debian.org/changelogs/@CHANGEPATH@_changelog
Date: Sat, 09 May 2020 09:51:02 UTC
Acquire-By-Hash: yes
Architectures: amd64 arm64 armel armhf i386 mips mips64el mipsel ppc64el s390x
Components: main contrib non-free

@mkey-cdx
Copy link

We're running into this issue has well and also noticed that the Date field seems to comply with Date format specifications https://wiki.debian.org/DebianRepository/Format#Date.2C_Valid-Until

I cannot tell what is going wrong.

@darkwizard242
Copy link

darkwizard242 commented Jun 17, 2020

@mkey-cdx Following header seems to be the cause:

Last-Modified: Wed, 17 Jun 2020 08:56:25 EDT

basically, the Last-Modified header.

@darkwizard242
Copy link

@aahlenst - Sorry to be a bother buddy, just wondering if there was any update from JFrog's side in terms of the resolution to this issue?

@aahlenst
Copy link
Contributor Author

No, no answer yet. I'll update as soon as we hear anything. Unfortunately, we neither have control over the machine nor the domain, so we cannot provide an interim fix.

@darkwizard242
Copy link

darkwizard242 commented Jun 17, 2020

@aahlenst - No worries. Thanks very much for the response and keeping us posted.

As a temporary workaround in the meantime, if someone would like to setup AdoptOpenJDK using the tar files ... you can do something like the following

## Using wget to download AdoptOpenJDK 11 tar file
sudo wget https://github.com/AdoptOpenJDK/openjdk11-binaries/releases/download/jdk-11.0.7%2B10/OpenJDK11U-jdk_x64_linux_hotspot_11.0.7_10.tar.gz -O jdk.tar.gz

## Extract jdk.tar.gz
sudo tar -xzf jdk.tar.gz

## Create a storage path and move it there:
sudo mv jdk-11.0.7+10 /usr/lib/jvm/jdk

## Add a file to set JAVA_HOME environment variable and /usr/lib/jvm/jdk/bin to PATH globally:
sudo bash -c 'cat <<EOF >/etc/profile.d/java.sh
export JAVA_HOME=/usr/lib/jvm/jdk
export PATH=$PATH:/usr/lib/jvm/jdk/bin
EOF'

## Source it:
sudo bash -c 'source /etc/profile.d/java.sh'

## Try:
java -version

FYI - only tested it on Ubuntu 16.04 LTS & Ubuntu 18.04 LTS .. Shouldn't be any different for a debian family OS.

@LeLobster
Copy link

Yesterday evening I was able to add the repo and install an adoptopenjdk package, but running update this morning results in the above mentioned error. Maybe that helps narrowing down where things went wrong?
it has to be a change introduced sometime between then I imagine

@instantlinux
Copy link

Related to this release - 7.5.8?
https://bintray.com/package/release/jfrog/artifactory-pro-debs/jfrog-artifactory-pro-deb

Perhaps some of y'all could start tweeting up a storm about this, it's been persisting longer than I would have expected.

@keefertaylor
Copy link

Is there a dependent issue with JFrog? Is the best way to reach them really on Twitter?

@aahlenst
Copy link
Contributor Author

@keefertaylor As I wrote in the first message, we have an open support request with JFrog.

@schowave
Copy link

Is there another repository than the one from jfrog?

@Weltraumschaf
Copy link

What's going on here? We can't rollout our infrastructure due to this error. Is someone working on this?

@rubentrf
Copy link

(Disclaimer: we only use this package on our CI for running tests) For those of you using Debian Buster and need OpenJDK 8, you can switch to Debian's Stretch package openjdk-8-jdk-headless adding it as an APT repo:

apt-add-repository 'deb http://security.debian.org/debian-security stretch/updates main'
apt-get update && apt-get install openjdk-8-jdk-headless

it only installs a few dependencies being the headless variant, and for us it did the trick.

@aahlenst
Copy link
Contributor Author

The packages can be manually pulled from https://adoptopenjdk.jfrog.io/adoptopenjdk/deb/pool/main/a/. Because no further dependencies have to be downloaded from our server, you can work around the problem with wget+apt:

$ wget https://adoptopenjdk.jfrog.io/adoptopenjdk/deb/pool/main/a/adoptopenjdk-14-hotspot/adoptopenjdk-14-hotspot_14.0.0+36-2_amd64.deb
$ sudo apt install ./adoptopenjdk-14-hotspot_14.0.0+36-2_amd64.deb

There's currently no other server to download the packages from via apt. I do not want to rush the introduction of a replacement, as inconvenient as the current situation is.

We're still waiting on a response from JFrog.

deadalnix pushed a commit to Bitcoin-ABC/bitcoin-abc that referenced this issue Jun 18, 2020
Summary:
This is a workaround for the adoptopenjdk8 issue:
adoptium/infrastructure#1399

It should help keeping the CI running until this is fixed upstream.
The real solution would be to update to latest Teamcity, which will cut
the dependency to the old Java 8 version.

Test Plan: Run any CI build.

Reviewers: #bitcoin_abc, jasonbcox

Reviewed By: #bitcoin_abc, jasonbcox

Differential Revision: https://reviews.bitcoinabc.org/D6635
@abhijeetnarvekar
Copy link

Facing the same issue:
Reading package lists... E: The repository 'https://adoptopenjdk.jfrog.io/adoptopenjdk/deb bionic Release' does not have a Release file. The command '/bin/sh -c ( wget -qO - https://adoptopenjdk.jfrog.io/adoptopenjdk/api/gpg/key/public | sudo apt-key add - ) && sudo apt-add-repository --yes https://adoptopenjdk.jfrog.io/adoptopenjdk/deb/ && sudo apt-get update -y --allow-releaseinfo-change && sudo apt-get install -y adoptopenjdk-8-hotspot' returned a non-zero code: 100

eak12913 added a commit to gruntwork-io/gruntwork-io.github.io that referenced this issue Jun 18, 2020
@duckulaj
Copy link

Still geting the same problem

Ign:17 https://adoptopenjdk.jfrog.io/adoptopenjdk/deb bionic InRelease
Err:19 https://adoptopenjdk.jfrog.io/adoptopenjdk/deb bionic Release
Unknown date format Bad header data [IP: 35.231.52.82 443]
Reading package lists... Done
E: The repository 'https://adoptopenjdk.jfrog.io/adoptopenjdk/deb bionic Release' does not have a Release file.

@darkwizard242
Copy link

@abhijeetnarvekar , @duckulaj - As per @aahlenst, the issue has been raised with JFrog and team is likely waiting on a response from them.

The packages can be manually pulled from https://adoptopenjdk.jfrog.io/adoptopenjdk/deb/pool/main/a/. Because no further dependencies have to be downloaded from our server, you can work around the problem with wget+apt:

$ wget https://adoptopenjdk.jfrog.io/adoptopenjdk/deb/pool/main/a/adoptopenjdk-14-hotspot/adoptopenjdk-14-hotspot_14.0.0+36-2_amd64.deb
$ sudo apt install ./adoptopenjdk-14-hotspot_14.0.0+36-2_amd64.deb

There's currently no other server to download the packages from via apt. I do not want to rush the introduction of a replacement, as inconvenient as the current situation is.

We're still waiting on a response from JFrog.

As you may notice, @aahlenst also shared alternative method to install the debian package for the time being, if you are interested.

@tschutte
Copy link

Looks like it is related to this issue, caused by a bug in the underlying Tomcat engine. https://www.jfrog.com/jira/browse/RTFACT-21640

@karianna
Copy link
Contributor

karianna commented Jun 18, 2020

Their support is on the case

@karianna karianna moved this from To do to In progress in Top Priorities Jun 18, 2020
@RangerRick
Copy link

Hopefully it sticks, but it just worked for me:

ranger@ghosts:~$ sudo apt-get update
[sudo] password for ranger:
Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Hit:2 http://us.archive.ubuntu.com/ubuntu bionic InRelease
Hit:4 https://download.virtualbox.org/virtualbox/debian bionic InRelease
Get:5 http://us.archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]
Hit:6 http://apt.postgresql.org/pub/repos/apt xenial-pgdg InRelease
Hit:7 https://downloads.plex.tv/repo/deb public InRelease
Ign:8 https://adoptopenjdk.jfrog.io/adoptopenjdk/deb bionic InRelease
Get:9 http://us.archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]
Get:10 https://adoptopenjdk.jfrog.io/adoptopenjdk/deb bionic Release [5,285 B]
Hit:11 http://ppa.launchpad.net/certbot/certbot/ubuntu bionic InRelease
Ign:3 https://debian.opennms.org stable InRelease
Hit:12 http://archive.canonical.com/ubuntu bionic InRelease
Get:14 https://adoptopenjdk.jfrog.io/adoptopenjdk/deb bionic Release.gpg [821 B]
Hit:13 https://debian.opennms.org stable Release
Hit:15 http://ppa.launchpad.net/hizo/mkv-extractor-gui/ubuntu xenial InRelease
Hit:16 http://ppa.launchpad.net/mkusb/ppa/ubuntu xenial InRelease
Hit:17 http://ppa.launchpad.net/ondrej/php/ubuntu xenial InRelease
Fetched 258 kB in 2s (146 kB/s)
Reading package lists... Done

@darkwizard242
Copy link

darkwizard242 commented Jun 18, 2020

Yup, seems like it has been fixed... Last-Modified header TZ is GMT now.

$ curl -IL https://adoptopenjdk.jfrog.io/adoptopenjdk/deb/dists/xenial/Release
HTTP/1.1 200 OK
Date: Thu, 18 Jun 2020 18:46:17 GMT
Content-Type: application/octet-stream
Content-Length: 5285
Connection: keep-alive
X-Artifactory-Id: adoptopenjdk-artifactory-primary-0-adoptopenjdk
X-Artifactory-Node-Id: adoptopenjdk-primary
Last-Modified: Wed, 17 Jun 2020 12:56:25 GMT
ETag: 3b1e63e4fdf360bc8e04f5b688adf9184ceab02d
X-Checksum-Sha1: 3b1e63e4fdf360bc8e04f5b688adf9184ceab02d
X-Checksum-Sha256: a34410d12fff57b039269433835a674f3fb94b4dee624d9cf63f294255a2bd93
X-Checksum-Md5: 21cce52419c5920a0d39786e17bc32dd
Accept-Ranges: bytes
X-Artifactory-Filename: Release
Content-Disposition: attachment; filename="Release"; filename*=UTF-8''Release
Cache-Control: no-store
Strict-Transport-Security: max-age=15724800; includeSubDomains
X-Request-ID: 3b9188cc78466f83dcbccab40e9959ba

Tested installation of debian packages from the repo.. Looks all good 👍

@devcovato
Copy link

devcovato commented Jun 19, 2020

Hi, it seems JFrog fixed the issue.

I have an automation script and it works again, no manual steps as suggested above.

@mkey-cdx
Copy link

This issue seems to have been solved.

curl -I https://adoptopenjdk.jfrog.io/adoptopenjdk/deb/dists/xenial/Release

HTTP/1.1 200 OK
Date: Fri, 19 Jun 2020 06:49:43 GMT
Content-Type: application/octet-stream
Content-Length: 5285
Connection: keep-alive
X-Artifactory-Id: adoptopenjdk-artifactory-primary-0-adoptopenjdk
X-Artifactory-Node-Id: adoptopenjdk-primary
Last-Modified: Wed, 17 Jun 2020 12:56:25 GMT
ETag: 3b1e63e4fdf360bc8e04f5b688adf9184ceab02d
X-Checksum-Sha1: 3b1e63e4fdf360bc8e04f5b688adf9184ceab02d
X-Checksum-Sha256: a34410d12fff57b039269433835a674f3fb94b4dee624d9cf63f294255a2bd93
X-Checksum-Md5: 21cce52419c5920a0d39786e17bc32dd
Accept-Ranges: bytes
X-Artifactory-Filename: Release
Content-Disposition: attachment; filename="Release"; filename*=UTF-8''Release
Cache-Control: no-store
Strict-Transport-Security: max-age=15724800; includeSubDomains
X-Request-ID: dad535c29c92f759399941668df94248

apt update

Ign:2 https://adoptopenjdk.jfrog.io/adoptopenjdk/deb xenial InRelease                                                                                            
Hit:3 https://adoptopenjdk.jfrog.io/adoptopenjdk/deb xenial Release

Can you confirm action taken?

@buchmui
Copy link

buchmui commented Jun 19, 2020

Our Jenkins Jobs were finished successfully today. I can confirm, that this problem is solved for us. Thank you everyone.

@aahlenst
Copy link
Contributor Author

This problem should be fixed. If anyone experiences any problems, please get in touch.

I'm very sorry for the extended outage of our Debian package repositories and that it took more than a day to sort it out. This is not an acceptable level of service for any AdoptOpenJDK offering.

To the best of my knowledge, this is what happened (all times in UTC):

On Wed, June 17, around noon, our hosted Artifactory instance has been upgraded to v6.18.1. This included a version of Apache Tomcat that could return HTTP date headers in other time zones than GMT which is forbidden by the relevant RFCs. Apt, the package management tool of all Debian flavours, relies on those headers and as a consequence stopped working. We investigated the issue, but quickly discovered that we cannot fix it on our end. At 13:48, we contacted JFrog support by email and asked for a rollback to the previous version or an upgrade to a patched version. JFrog support got back to us on Thu, Jun 18, 16:45, and acknowledged the problem and promised an upgrade to an unaffected version of Artifactory. At 22:59, we were informed that Artifactory has been upgraded to v6.20.0 and that the problem should be resolved.

This incident highlighted multiple issues with our current setup and arrangements with our suppliers. We'll carefully review everything and institute the necessary changes.

@abhijeetnarvekar
Copy link

Works again! Great! Thank you everyone!

infrastructure automation moved this from In Progress to Done Jun 19, 2020
Top Priorities automation moved this from In progress to Done Jun 19, 2020
@mkey-cdx
Copy link

Thanks for the detailed information @aahlenst

@draganbjedov
Copy link

Works again!

fatmcgav pushed a commit to fatmcgav/puppet-elasticsearch that referenced this issue Jun 24, 2020
@gdams gdams removed this from Done in Top Priorities May 17, 2021
@Splarkszter
Copy link

E: The repository 'https://adoptopenjdk.jfrog.io/adoptopenjdk/deb jammy Release' does not have a Release file.

Jan/8/2023

@Splarkszter
Copy link

Tried to manually get the termurin-8-jdk but when trying to execute it, it displays: "Error: Dependency not satisfiable: adoptium-ca-certificates" i do have ca-certificates-java installed though.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
No open projects
Development

No branches or pull requests