[Packaging][Other] Downloading arrrow dependencies fails with 403 error on debian

[Infinical]

Describe the bug, including details regarding any error messages, version, and platform.

While installing arrow dependencies in a docker container it fails with 403 error when downloading the dependencies

#8 222.2 Err:7 https://apache.jfrog.io/artifactory/arrow/ubuntu jammy/main amd64 libarrow-glib1100 amd64 11.0.0-1
#8 222.2   403  Forbidden [IP: 52.92.243.129 443]
#8 222.2 Err:15 https://apache.jfrog.io/artifactory/arrow/ubuntu jammy/main amd64 libparquet1100 amd64 11.0.0-1
#8 222.2   403  Forbidden [IP: 52.92.243.129 443]
#8 222.2 Err:26 https://apache.jfrog.io/artifactory/arrow/ubuntu jammy/main amd64 libarrow-dev amd64 11.0.0-1
#8 222.2   403  Forbidden [IP: 52.92.243.129 443]
#8 222.2 Err:43 https://apache.jfrog.io/artifactory/arrow/ubuntu jammy/main amd64 libparquet-dev amd64 11.0.0-1
#8 222.2   403  Forbidden [IP: 52.92.243.129 443]
#8 222.4 Fetched 126 MB in 3min 25s (613 kB/s)

Component(s)

Packaging, Other

[kou]

Could you share a Dockerfile that reproduces this problem?

[Infinical]

@kou

FROM ubuntu:22.04


RUN apt-get update && apt install -y -V  wget lsb-release


RUN wget https://apache.jfrog.io/artifactory/arrow/$(lsb_release --id --short | tr 'A-Z' 'a-z')/apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb \
  && apt install -y -V ./apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb > /dev/null 2>&1

RUN apt-get -y update \
    && apt-get -y install --no-install-recommends libarrow-glib-dev libparquet-glib-dev libarrow-dev libparquet-dev libgirepository1.0-dev libglib2.0 libglib2.0-dev

[raulcd]

Hi @Infinical ! I was having some issues yesterday morning with artifactory returning 403 for some requests. It seems to be solved now. Could you give it a try?
I think it was apache.jfrog.io having a temporary issue.

[Thingus]

Hi @raulcd
Letting you know that this has been an intermediate issue for me today and last week on a similar situation, with the following (snipped) dockerfile:

FROM postgres:12.11@sha256:8bff0179347982a6c083e1ed69cf4b65f12b8ae76e61f63b238c8c1ba4714c86
...snip...
RUN apt-get update \
     && apt install -y -V ca-certificates lsb-release wget
RUN  wget https://apache.jfrog.io/artifactory/arrow/$(lsb_release --id --short | tr 'A-Z' 'a-z')/apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb \
         && apt install -y -V ./apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb \
         && apt update \
         && apt install -y -V libarrow-dev libparquet-dev \

The second RUN failed with 403 forbidden, so I tried running it from bash in the cached container; apt install libarrow-dev failed with the same error the first time, but happily ran on a retry.

[agoncharuk]

Hi, I am consistently getting this error when installing Arrow in an ubuntu:22.04 container (the IP in the error is 52.218.235.67). Are there any known workarounds?
Should we file an infra ticket with Apache?

[raulcd]

Thanks. I haven't been able to reproduce on my end. I've opened the following ticket on INFRA:
https://issues.apache.org/jira/browse/INFRA-24569
Please let them know any more details there! Thanks!

[ronocod]

We were seeing this a lot, and I think it's because the signed URLs used to download files from S3 are only valid for 30 seconds. The failures we saw were all on occasions that the download part of apt install took over 30 seconds. We updated our code to rerun apt install if it fails which mitigates the problem for now.

[kou]

https://issues.apache.org/jira/browse/INFRA-24569?focusedCommentId=17728543&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17728543

The timeout was updated on Monday. Have you had any further reports? If we don't hear back in a couple days I'll close this. If reports continue to come in let us know and we can have JFrog increase the timeout further.

I close this.

If this problem is still happen, please report it here.

[joemccall86]

We are still seeing this issue exactly as reported. Here is our relevant script that we use to set up our CI:

  sudo apt update
  sudo apt install -y -V ca-certificates lsb-release wget
  if [ $(lsb_release --codename --short) = "stretch" ]; then
    sudo tee /etc/apt/sources.list.d/backports.list <<APT_LINE
deb http://deb.debian.org/debian $(lsb_release --codename --short)-backports main
APT_LINE
  fi
  pushd /tmp
  wget https://apache.jfrog.io/artifactory/arrow/$(lsb_release --id --short | tr 'A-Z' 'a-z')/apache-arrow-archive-keyring-latest-$(lsb_release --codename --short).deb
  sudo apt install -y -V ./apache-arrow-archive-keyring-latest-$(lsb_release --codename --short).deb
  popd

This doesn't happen every time.

[kou]

I reported it to INFRA but no response yet: https://issues.apache.org/jira/browse/INFRA-24569?focusedCommentId=17731160&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17731160

[raulcd]

A minor note, even though I clicked no to keep the issue opened it was closed when merging the related PR. This was due to the Closes XXX on the PR:

Would you like to update the associated issue? (y/n): n
Traceback (most recent call last):
  File "/home/raulcd/code/arrow/dev/merge_arrow_pr.py", line 781, in <module>
    cli()
  File "/home/raulcd/code/arrow/dev/merge_arrow_pr.py", line 768, in cli
    cmd.continue_maybe("Would you like to update the associated issue?")
  File "/home/raulcd/code/arrow/dev/merge_arrow_pr.py", line 476, in continue_maybe
    self.fail("Okay, exiting")
  File "/home/raulcd/code/arrow/dev/merge_arrow_pr.py", line 462, in fail
    raise Exception(msg)
Exception: Okay, exiting

[agoncharuk]

The issue indeed seems to be related to a timeout associated with a token for S3 downloads. As a workaround, we explicitly install some of the largest dependencies of the Arrow package before installing the Arrow itself.

[kou]

@raulcd Ah, sorry. You're right.

@agoncharuk Thanks for sharing your opinion. Could you comment information you have to https://issues.apache.org/jira/browse/INFRA-24569 instead of here? I think that we need to work together with INFRA to solve this.

[kou]

I close this.
We need to do #37350 to fix this.

[SumitAgrawal03071989]

I see this problem still happening

the command I am using
wget https://apache.jfrog.io/artifactory/arrow/$(lsb_release --id --short | tr 'A-Z' 'a-z')/apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb

the error message

E: Failed to fetch https://jfrog-prod-usw2-shared-oregon-main.s3.amazonaws.com/ --- blah blah --- 403 Forbidden [IP: 52.92.212.233 443]