Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Action does not download package - fails with "gzip: stdin: not in gzip format" error #89

Closed
GalOzRlz opened this issue Aug 29, 2023 · 27 comments
Closed

Action does not download package - fails with "gzip: stdin: not in gzip format" error #89

GalOzRlz opened this issue Aug 29, 2023 · 27 comments

Comments

@GalOzRlz
Copy link

GalOzRlz commented Aug 29, 2023 •
edited

For more than 24 hours our action is failing from time to time with this error - it looks like it won't download the package. Is this a CDN issue on the servers?

Run tailscale/github-action@v1
  with:
    authkey: ***
    version: 1.30.0
  env:
    CODEARTIFACT_URL:****
    pythonLocation: /opt/hostedtoolcache/Python/3.8.17/x64
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.8.17/x64/lib
    AWS_DEFAULT_REGION: us-east-[2]
    AWS_REGION: us-east-2
    AWS_ACCESS_KEY_ID: ***
    AWS_SECRET_ACCESS_KEY: ***
    AWS_SESSION_TOKEN: ***
    POSTGRES_URL: ***
Run MINOR=$(echo $VERSION | awk -F '.' {'print $2'})
  MINOR=$(echo $VERSION | awk -F '.' {'print $2'})
  if [ $((MINOR % 2)) -eq 0 ]; then
    URL="https://pkgs.tailscale.com/stable/tailscale_${VERSION}_amd64.tgz"
  else
    URL="https://pkgs.tailscale.com/unstable/tailscale_${VERSION}_amd64.tgz"
  fi
  curl -H user-agent:tailscale-github-action-v1 -L "$URL" -o tailscale.tgz
  tar -C ${HOME} -xzf tailscale.tgz
  rm tailscale.tgz
  TSPATH=${HOME}/tailscale_${VERSION}_amd64
  sudo mv "${TSPATH}/tailscale" "${TSPATH}/tailscaled" /usr/bin
  shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
  env:
    CODEARTIFACT_URL: ****
    pythonLocation: /opt/hostedtoolcache/Python/[3](https://github.com/SeemplicityDev/engine/actions/runs/6010592451/job/16302345813#step:17:3).8.17/x6[4](https://github.com/SeemplicityDev/engine/actions/runs/6010592451/job/16302345813#step:17:4)
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.8.17/x64/lib
    AWS_DEFAULT_REGION: us-east-2
    AWS_REGION: us-east-2
    AWS_ACCESS_KEY_ID: ***
    AWS_SECRET_ACCESS_KEY: ***
    AWS_SESSION_TOKEN: ***
    POSTGRES_URL: ****(https://github.com/SeemplicityDev/engine/actions/runs/6010592451/job/16302345813#step:17:5)j.us-east-2.rds.amazonaws.com/postgres
    VERSION: 1.30.0
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100    81  100    81    0     0    [6](https://github.com/SeemplicityDev/engine/actions/runs/6010592451/job/16302345813#step:17:6)99      0 --:--:-- --:--:-- --:--:--   [7](https://github.com/SeemplicityDev/engine/actions/runs/6010592451/job/16302345813#step:17:7)04

100    1[9](https://github.com/SeemplicityDev/engine/actions/runs/6010592451/job/16302345813#step:17:9)  [10](https://github.com/SeemplicityDev/engine/actions/runs/6010592451/job/16302345813#step:17:10)0    [19](https://github.com/SeemplicityDev/engine/actions/runs/6010592451/job/16302345813#step:17:20)    0     0     [37](https://github.com/SeemplicityDev/engine/actions/runs/6010592451/job/16302345813#step:17:38)      0 --:--:-- --:--:-- --:--:--    37
100    19  100    19    0     0     37      0 --:--:-- --:--:-- --:--:--     0

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
Error: Process completed with exit code 2.
@zonorti
Copy link

zonorti commented Aug 29, 2023

I want to fix it with using github cache #87, so after one run it should work.
But I am going to touch only v2

@GalOzRlz
Copy link
Author

no problem - we will update to version 2 if needed.

@zonorti zonorti mentioned this issue Aug 29, 2023
Closed
@zonorti
Copy link

zonorti commented Aug 29, 2023

@GalOzRlz please have a look at my PR. it works for me, but I am new to GH Actions, so YMMV

@creachadair
Copy link
Member

I suspect this may have been related to #88, as the error responses would not have been gzipped.

@GalOzRlz
Copy link
Author

GalOzRlz commented Oct 3, 2023

This is happening again today!

@beaugunderson
Copy link

yes, I'm also seeing this hundreds of times today as we use tailscale in our deployment workflow... we upgraded from v1 to v2 and still have the same issue:

69540

@beaugunderson beaugunderson mentioned this issue Oct 4, 2023
Closed
@DentonGentry
Copy link
Contributor

DentonGentry commented Oct 4, 2023 •
edited

@beaugunderson @GalOzRlz
What tailscale version is it trying to download? The v2 action defaults to 1.42.0, is that it or do you set version ?

@GalOzRlz
Copy link
Author

GalOzRlz commented Oct 4, 2023 •
edited

@beaugunderson @GalOzRlz What tailscale version is it trying to download? The v2 action defaults to 1.42.0, is that it or do you set version ?

We fix the version to version: 1.30.0 --

  - name: Tailscale
    uses: tailscale/github-action@v1
    with:
      authkey: ${{ secrets.TAILSCALE_AUTHKEY }}
      version: 1.30.0

@beaugunderson
Copy link

beaugunderson commented Oct 4, 2023 via email

@DentonGentry
Copy link
Contributor

I assume we could upgrade to something more recent without major issues if it would help

At this point I don't think a version change would help. We're focussing on the pkgs server and the CDN in front of it, which is used for all Tailscale versions.

@avi-city
Copy link

avi-city commented Oct 4, 2023

Thanks @DentonGentry , much appreciated

@DentonGentry
Copy link
Contributor

It isn't failing everywhere:

$ curl -H user-agent:tailscale-github-action -L https://pkgs.tailscale.com/stable/tailscale_1.24.2_amd64.tgz --output /tmp/tailscale_1.24.2_amd64.tgz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    81  100    81    0     0    194      0 --:--:-- --:--:-- --:--:--   195
100 18.2M  100 18.2M    0     0  16.6M      0  0:00:01  0:00:01 --:--:-- 41.9M
$ curl -H user-agent:tailscale-github-action -L https://pkgs.tailscale.com/stable/tailscale_1.24.2_amd64.tgz.sha256
a25bba595af9a67fa2b3ef7df5f92b343179470b14115c76fd2f6e92cc3bc9ac
$ echo "a25bba595af9a67fa2b3ef7df5f92b343179470b14115c76fd2f6e92cc3bc9ac /tmp/tailscale_1.24.2_amd64.tgz" | sha256sum -c
/tmp/tailscale_1.24.2_amd64.tgz: OK

$ curl -H user-agent:tailscale-github-action -L https://pkgs.tailscale.com/stable/tailscale_1.30.0_amd64.tgz --output /tmp/tailscale_1.30.0_amd64.tgz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    81  100    81    0     0    188      0 --:--:-- --:--:-- --:--:--   188
100 19.5M  100 19.5M    0     0  17.0M      0  0:00:01  0:00:01 --:--:-- 17.0M
$ curl -H user-agent:tailscale-github-action -L https://pkgs.tailscale.com/stable/tailscale_1.30.0_amd64.tgz.sha256
8660387a79d539838f032e0cbe620cb241ee3f0636fd8942275d37ca38f456fd
$ echo "8660387a79d539838f032e0cbe620cb241ee3f0636fd8942275d37ca38f456fd /tmp/tailscale_1.30.0_amd64.tgz" | sha256sum -c
/tmp/tailscale_1.30.0_amd64.tgz: OK

A workflow using this GitHub Action passed here a few minutes ago: https://github.com/tailscale/github-action/actions/runs/6403096635/job/17380921729?pr=91

I suppose a problem in the CDN serving pkgs.tailscale.com might only manifest in GitHub datacenters other than the one we used for that run. However it looks like all GitHub Actions still run in the US, and the CDN doesn't have a huge number of different regional nodes.

For anyone experiencing trouble today: are you self-hosting runners?

@beaugunderson
Copy link

beaugunderson commented Oct 4, 2023 via email

@GalOzRlz
Copy link
Author

GalOzRlz commented Oct 4, 2023

No self running hosts in these workflows

@beaugunderson
Copy link

any news here? we're still seeing lots of failures

@beaugunderson
Copy link

fwiw 1.48.2 is installing for us with no failures yet!

DentonGentry added a commit that referenced this issue Oct 5, 2023
@DentonGentry
action.yml: output expected and actual checksum
3ffb1f1 
Updates #89

Signed-off-by: Denton Gentry <dgentry@tailscale.com>
@DentonGentry
Copy link
Contributor

  • We updated the CI/CD in this repo to allow a manual trigger and have run it a number of times today with no failures.
  • We see no concerning metrics from pkgs.tailscale.com nor the CDN in front of it.
  • Right now I don't see a sign that the pkg server is involved in the problem. The described symptoms are what would happen were the GitHub runner to give up on the download.

@beaugunderson 1.24.2 gets a checksum error and 1.48.2 does not?

@DentonGentry DentonGentry mentioned this issue Oct 5, 2023
Merged
@beaugunderson
Copy link

@DentonGentry zero failures out of ~60 after switching to 1.48.2 👍

DentonGentry added a commit that referenced this issue Oct 5, 2023
@DentonGentry
action.yml: output expected and actual checksum (#92)
a5ed86c 
Updates #89

Signed-off-by: Denton Gentry <dgentry@tailscale.com>
@DentonGentry
Copy link
Contributor

If someone who can reproduce the issue is willing to run the action @a5ed86cd4900e943e2aa3f760fd8472f04e46469 instead of v2, it will output the expected and actual checksums to get an idea what is happening.

@GalOzRlz
Copy link
Author

GalOzRlz commented Oct 5, 2023

@a5ed86cd4900e943e2aa3f760fd8472f04e46469

I'm going to try and make this happen and get back to you

@GalOzRlz
Copy link
Author

GalOzRlz commented Oct 5, 2023 •
edited

@DentonGentry

We got one!
image

still using version: 1.30.0 btw

@DentonGentry
Copy link
Contributor

It got the redirect to the CDN, then downloaded 19 bytes and stopped.
b16e15764b8bc06c5c3f9f19bc8b99fa48e7894aa5a6ccdad65da49bbf564793 is the SHA256 of "404 page not found"

@DentonGentry
Copy link
Contributor

@GalOzRlz

  • Could I ask the exact content of the "version" string passed as an input to the GitHub action? Is it "1.30.0" and nothing more?
  • Could I ask what continent you are located on? I guess your jobs might run in a different GitHub datacenter, which in turn might be directed to a different CDN node.

@GalOzRlz
Copy link
Author

GalOzRlz commented Oct 5, 2023

@GalOzRlz

  • Could I ask the exact content of the "version" string passed as an input to the GitHub action? Is it "1.30.0" and nothing more?

Yep just 1.30.0

  • name: Tailscale
    uses: tailscale/github-action@v1
    with:
    authkey: ${{ secrets.TAILSCALE_AUTHKEY }}
    version: 1.30.0
  • Could I ask what continent you are located on? I guess your jobs might run in a different GitHub datacenter, which in turn might be directed to a different CDN node.

We're in Israel.

@XciD
Copy link

XciD commented Oct 5, 2023

Getting the same.

Version: 1.36.0
us-east-1 in AWS

@DentonGentry
Copy link
Contributor

I think we've worked out the series of issues starting from a data consistency problem in a provider's tools, which led to not deploying updates to two of the nodes in our infrastructure, which led to a 404 for those unfortunate enough to hit one of those nodes.

We think this is resolved now. If you see checksum errors occurring after 5:30pm Pacific time on October 5, please update this issue.

@DentonGentry
Copy link
Contributor

We also added metrics for this specific issue, and have seen no occurrences since October 5. I'm going to close this as resolved.

For people reading this in the future, if you see "gzip: stdin: not in gzip format" or a checksum error please do not comment on this issue or reopen it. This one is fixed. If you encounter a new problem, even with the same symptom, please open a new bug.

@rasdani rasdani mentioned this issue Apr 11, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@creachadair @beaugunderson @zonorti @DentonGentry @XciD @avi-city @GalOzRlz