Automating downloads from launchpad with CI #28

aepfli · 2021-06-23T06:08:57Z

I am not sure if this is even the right place to ask, if not, please let me know, where i should instead ask :)

I want to download the SAP Commerce artifacts via CI so i can easier distribute them with my co workers. We are using a Package Registry tool for storing most of our artifacts. To reduce the overhead, we thought it is a good ides to use our CI infrastructure to download via CURL and store it in the Package registry.

Sadly this seems to be not working, i assume it is related to our infrastructure within GCP. But if i use the same docker image locally for downloading, i have no issues at all downloading. But if i use it with the CI system, i am always redirected to a page via AkamaiGHost and my first request is a Temporarly Moved request.

I thought i ask here based on https://github.com/SAP/commerce-gradle-plugin/blob/master/docs/FAQ.md#faq - as this was my main orientation for the tooling. Maybe somebody else already had this problem, and found a solution.

Thank you

The text was updated successfully, but these errors were encountered:

aepfli · 2021-06-23T06:14:33Z

BASIC_AUTH="$(echo -n "${SAP_USER_ID}:${SAP_PASSWORD}" | base64)"

curl -L -v -b cookies.txt  \
	-H "Authorization: Basic $BASIC_AUTH" \
	-o "download.file" \
	"https://softwaredownloads.sap.com/file/${SAP_FILE_ID}"

this is the magic i am using

mpern · 2021-06-24T16:59:07Z

Hm, that's strange.

I just ran a little test on my end using exactly the script you provided and I can download 2011.8 without a hitch.

Some food for thought:

Have you tested the particular S-User in the browser too?
Missing shell escaping?
Try using -u "$SAP_USER_ID:$SAP_PASSWORD" instead of building the Auth header manually. Maybe it helps?

A couple of redirects until the download actually starts are expected though.

SAP_USER_ID='S0123456789'
SAP_PASSWORD='<redacted>'
SAP_FILE_ID='0020000000820402021'

curl -L -v -b cookies.txt  \
    -u "$SAP_USER_ID:$SAP_PASSWORD" \
    -o "download.file" \
    "https://softwaredownloads.sap.com/file/${SAP_FILE_ID}"

aepfli · 2021-06-25T09:53:25Z

here is the strange thing, the basic auth header, the url everything is the same when i am inspecting the log closely. So i do not assume it has something to do with the Script or the data.

This is the log from locally:

> GET /file/0020000000820402021 HTTP/1.1
> Host: softwaredownloads.sap.com
> User-Agent: curl/7.61.1
> Accept: */*
> Authorization: Basic <stripped>

< HTTP/1.1 302 Found
< Date: Mon, 21 Jun 2021 13:46:35 GMT
* Added cookie JSESSIONID=<stripped> for domain softwaredownloads.sap.com, path /, expire 0
< Set-Cookie: JSESSIONID=<stripped> Path=/; Secure; HttpOnly
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< Strict-Transport-Security: max-age=31536000; includeSubDomains
< Location: https://softwaredownloads.sap.com/?file=0020000000820402021&downloadId=05672df4-7d4d-4599-b9d1-fc9f35a9ada7
< DownloadId: 05672df4-7d4d-4599-b9d1-fc9f35a9ada7
< Content-Type: text/html;charset=utf-8
< Transfer-Encoding: chunked
< Server: SAP Software Download

compared to CI system:

> GET /file/0020000000820402021 HTTP/1.1
> Host: softwaredownloads.sap.com
> User-Agent: curl/7.61.1
> Accept: */*
> Authorization: Basic <stripped>

< HTTP/1.1 302 Moved Temporarily
< Server: AkamaiGHost
< Content-Length: 0
< Location: https://origin.softwaredownloads.sap.com/tokengen/?file=0020000000820402021
< Expires: Wed, 23 Jun 2021 08:48:43 GMT
< Cache-Control: max-age=0, no-cache, no-store
< Pragma: no-cache
< Date: Wed, 23 Jun 2021 08:48:43 GMT
< Connection: keep-alive
< Strict-Transport-Security: max-age=31536000

Might this be related to the machines hosted in GCP and a special treatment on Akamai?
After the temporarly moved, i am getting forwarded to the token page (but without my auth header) and i get a form which points me to the SSO page.

<?xml version="1.0"?>
<!DOCTYPE html>
<html>
    <head>
        <meta charset="utf-8"/>
            </head>
    <body onload="document.forms[0].submit()">
        <noscript>
            <p>
                <strong>Note:</strong> Since your browser does not support JavaScript,
                you must press the Continue button once to proceed.
            </p>
        </noscript>
        
        <form action="https://accounts.sap.com/saml2/idp/sso" method="post">
            <div>
                <input type="hidden" name="RelayState" value="https://origin.softwaredownloads.sap.com/tokengen/?file=0020000000820402021"/>                
                <input type="hidden" name="SAMLRequest" value="<stripped>"/>                
                                
            </div>
            <noscript>
                <div>
                    <input type="submit" value="Continue"/>
                </div>
            </noscript>
        </form>
            </body>
</html>

i even tried to do the login via curl and mimic the redirect functionality, but that is not as easy as i hoped it will be. (after providing the j_username i get an error that it was not successful, and i should clean everything and start over)

Worst case i thought about tunneling my request through another machine, eg. mine to see if this works, to verify my GCP IP theory - but that is some effort, i am not sure i currently have time for :)

mpern · 2021-06-25T10:32:40Z

At that stage, your guess is as good as mine.

But maybe there is some IP filtering going on, I honestly don't know.

For me at takes 2 redirects until I end up at the file. Do you cache any cookies by chance?

aepfli · 2021-06-25T10:46:08Z

I hope that you maybe can tell me a fancy header to bypass this ;) I will see if I find a workaround. And I will also add here my progress, if I make some (hopefully) Thank you for your input.

…

On Fri, Jun 25, 2021, 12:32 Markus Perndorfer ***@***.***> wrote: At that stage, your guess is as good as mine. But maybe there is some IP filtering going on, I honestly don't know. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#28 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACMGKQR34JSCBL4MM2ELSZ3TURLNFANCNFSM47FCVNIA> .

aepfli · 2021-06-28T07:20:39Z

it seems like i managed to bypass this by first calling the token endpoint manually.

BASIC_AUTH="$(echo -n "${SAP_USER_ID}:${SAP_PASSWORD}" | base64)"

curl -L -v -b cookies.txt  -c cookie.txt \
	-H "Authorization: Basic $BASIC_AUTH" \
	-o "download.file" \
	"https://origin.softwaredownloads.sap.com/tokengen/?file=${SAP_FILE_ID}"

curl -L -v -b cookies.txt  -c cookie.txt \
	-H "Authorization: Basic $BASIC_AUTH" \
	-o "download.file" \
	"https://softwaredownloads.sap.com/file/${SAP_FILE_ID}"

aepfli · 2021-06-28T12:33:04Z

as i found i workarround i close this ticket - thank you @mpern for your support

mpern · 2021-06-29T06:12:12Z

Thank you for taking the time to find a workaround.

One other thing came to mind:

Try using --location-trusted (send Authorization header on every redirect) instead of -L and see if it changes the behaviour.

Background:

aepfli closed this as completed Jun 28, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Automating downloads from launchpad with CI #28

Automating downloads from launchpad with CI #28

aepfli commented Jun 23, 2021

aepfli commented Jun 23, 2021 •

edited

mpern commented Jun 24, 2021

aepfli commented Jun 25, 2021

mpern commented Jun 25, 2021 •

edited

aepfli commented Jun 25, 2021 via email

aepfli commented Jun 28, 2021

aepfli commented Jun 28, 2021

mpern commented Jun 29, 2021 •

edited

Automating downloads from launchpad with CI #28

Automating downloads from launchpad with CI #28

Comments

aepfli commented Jun 23, 2021

aepfli commented Jun 23, 2021 • edited

mpern commented Jun 24, 2021

aepfli commented Jun 25, 2021

mpern commented Jun 25, 2021 • edited

aepfli commented Jun 25, 2021 via email

aepfli commented Jun 28, 2021

aepfli commented Jun 28, 2021

mpern commented Jun 29, 2021 • edited

aepfli commented Jun 23, 2021 •

edited

mpern commented Jun 25, 2021 •

edited

mpern commented Jun 29, 2021 •

edited