403 Forbidden during debian install from lf repository #14837
Comments
|
With regards to the issue, we have reported downloads failing with 403 Forbidden, we have parsed through your instance logs and I found the below errors.
This issue occurs when the bucket URL ( https:///jfrog-prod-usw2-shared-oregon-main.s3.amazonaws.com) is not whitelisted in your network or the S3-signed URL expired and failed to download the package. As a next step, could you please whitelist the S3 bucket URL in your network ( https:///jfrog-prod-usw2-shared-oregon-main.s3.amazonaws.com ) Also, please try to download the package from the same client using the below REST call and share the output. #curl -u: https://jfrog-prod-usw2-shared-oregon-main.s3.amazonaws.com/aol-linuxfoundation/filestore/e0/e0fdd83f07b6741f70067e46ceb9cf90010841c7?X-Artifactory-username=anonymous&X-Artifactory-repositoryKey=magma-packages-prod&X-Artifactory-artifactPath=focal-1.8.0/libfolly-dev_2018.02.26.00-6ubuntu20.04_amd64.deb -output libfolly-dev_2018.02.26.00-6ubuntu20.04_amd64.deb -vvv -L Looking forward to hearing from you. |
|
@fabiopalumbo, The second curl call is damaged and returns nothing reasonable for me. Even with quotation marks (needed for using URL parameters) around the URL I get errors. The following curl is downloading the package as expected But from our analysis we expect downloading a single package to be working. The issue is happening when a couple of packages are installed in a single |
|
Thank you for your patience. If the above is correct, we might try increasing the timeout value (i.e signedUrlExpirySeconds) to 240 seconds depending on the time it is taking, by default signedUrlExpirySeconds value is 60 seconds. For making this change we will need to restart the server for the change to take effect. |
|
Hi @fabiopalumbo,
We expect this to be the case - based on the analysis above. In my opinion it is reasonable to try this and have a server restart. A maintenance window would be nice so we can interpret expected CI failures and communicate this to end-users. I will get feedback from the TSC. |
Change and restart scheduled for Feb 15th 7am PT. Note: 06:45am the issue was reproduced locally by the script above (with a less restricted bandwidth) - check will be done again after the config change |
|
Update from Slack: https://magmacore.slack.com/archives/C01Q1T14YJ2/p1676887747835589?thread_ts=1675887162.599359&cid=C01Q1T14YJ2 I still can reproduce the issue. When I do -> i.e., install magma and at the end summary If I click on the link, I see This is: |
|
One observation: I'm still getting this error when I have Spain's IP address but when I switch to the other IP like Germany it doesn't give the error. Moreover, When I'm in Spain IP address the packages that fail due to this error are not similar everytime. |
Your Environment
Describe the Issue
When setting up components where the installation of debian packages from the LF reposirory is needed,
403 Forbiddencan occur and the setup fails. Has been observed in CI when VMs are setup or docker containers are build, locally and when setting up production environments. It looks like this only happens if a lot of packages are installed by oneapt installcall. See example logging below:[1]
To Reproduce
It looks like this is happening when doing
apt installin an environment with a low bandwidth. On a system with a sufficient bandwidth it can be reproduced using:Expected behavior
Installing magma packages without issues.
Additional context
Hypothesis: it looks like the tokens in the generated URL have a low lifetime. On a system with a low bandwidth this lifetime can be exceeded.
Workarounds
RUNis started from scratch for each call).apt install package_1 package_2 ... package_nintoapt install package_1 && apt install package_2 && ... && apt install package_nThe text was updated successfully, but these errors were encountered: