Cannot update AppImages hosted on GitLab CI #94

probonopd · 2018-06-10T11:04:18Z

# Get "old" AppImage
linux@host:~> wget -c "https://gitlab.com/probono/QtQuickApp/-/jobs/73879740/artifacts/raw/QtQuickApp-x86_64.AppImage"

# Check whether it has a correct zsync URL
linux@host:~> Downloads/appimageupdatetool-329-7260f32-x86_64.AppImage -d QtQuickApp-x86_64.AppImage 
Parsing file: QtQuickApp-x86_64.AppImage
AppImage type: 2
Raw update information: zsync|https://gitlab.com/probono/QtQuickApp/-/jobs/artifacts/master/raw/QtQuickApp-x86_64.AppImage.zsync?job=trusty
Update information type: Generic ZSync URL
Assembled ZSync URL: https://gitlab.com/probono/QtQuickApp/-/jobs/artifacts/master/raw/QtQuickApp-x86_64.AppImage.zsync?job=trusty

# Check
linux@host:~> Downloads/appimageupdatetool-329-7260f32-x86_64.AppImage -j QtQuickApp-x86_64.AppImage 
zsync2: Bad status code 200 while trying to download .zsync file!
zsync2: Reading and/or parsing .zsync file failed!

# Try to update
linux@host:~> Downloads/appimageupdatetool-329-7260f32-x86_64.AppImage QtQuickApp-x86_64.AppImage 
Checking for updates...
zsync2: Bad status code 200 while trying to download .zsync file!
zsync2: Reading and/or parsing .zsync file failed!
Update check failed, exiting!

I suspect it has to do with the fact that there is a HTTP argument (?=...) involved?

The text was updated successfully, but these errors were encountered:

probonopd · 2018-06-10T11:06:25Z

How can one turn on curl debugging output in appimageupdatetool?

Edit: CURLOPT_VERBOSE=1 Downloads/appimageupdatetool-329-7260f32-x86_64.AppImage QtQuickApp-x86_64.AppImage

probonopd · 2018-06-10T11:13:11Z

In the case of GitLab,

https://gitlab.com/probono/QtQuickApp/-/jobs/artifacts/master/raw/QtQuickApp-x86_64.AppImage.zsync?job=trusty

redirects to the latest build (the number may have increased since I posted this) at

https://gitlab.com/probono/QtQuickApp/-/jobs/73880276/artifacts/raw/QtQuickApp-x86_64.AppImage.zsync

The real AppImage is available in the same un-redirected location:

https://gitlab.com/probono/QtQuickApp/-/jobs/artifacts/master/raw/QtQuickApp-x86_64.AppImage?job=trusty

which also gets redirected to the same directory (the number may have increased since I posted this)

https://gitlab.com/probono/QtQuickApp/-/jobs/73880276/artifacts/raw/QtQuickApp-x86_64.AppImage

So it looks good to me, I suspect there must be a bug in zsync2 and/or AppImageUpdate that gets confused.

probonopd · 2018-06-10T11:21:36Z

Similar issue: antony-jr/AppImageUpdater#11

probonopd · 2018-07-02T06:37:23Z

Ping @TheAssassin. Would be great if you could look into it. Thanks!

TheAssassin · 2018-07-02T22:19:23Z

This is caused by https://github.com/AppImage/zsync2/blob/293d65d2683cca8d579efd97a0e31634b21956ca/src/zsclient.cpp#L321-L323. I think it should be save to store the original URL as "referer", and resolve relative links from there.

TheAssassin · 2018-07-02T22:27:50Z

Commenting out the line doesn't seem to make a difference, but actually I can update the QtQuick AppImage from GitLab just fine as is. I stepped into the relevant functions, and the referer is set to "https://gitlab.com/probono/QtQuickApp/-/jobs/artifacts/master/raw/QtQuickApp-x86_64.AppImage.zsync?job=trusty", which appears to be correct.

probonopd · 2018-07-03T05:43:28Z

So it is working for you using the continuous build? I'll need to test again then...

TheAssassin · 2018-07-03T12:45:23Z

@probonopd I tested a locally built version. I will test with the released AppImage later.

probonopd · 2018-07-05T05:05:23Z

Definitely not working for me with today's latest continous build. Tested on Xubuntu 18.04:

2018-07-05 07:03:57 (4.62 MB/s) - ‘QtQuickApp-x86_64.AppImage’ saved [21309368/21309368]

me@host:~$ Downloads/appimageupdatetool-330-6f8f515-x86_64.AppImage QtQuickApp-x86_64.AppImage 
Checking for updates...
zsync2: Bad status code 200 while trying to download .zsync file!
zsync2: Reading and/or parsing .zsync file failed!
Update check failed, exiting!
me@host:~$ Downloads/appimageupdatetool-330-6f8f515-x86_64.AppImage QtQuickApp-x86_64.AppImage -d

Parsing file: QtQuickApp-x86_64.AppImage
AppImage type: 2
Raw update information: zsync|https://gitlab.com/probono/QtQuickApp/-/jobs/artifacts/master/raw/QtQuickApp-x86_64.AppImage.zsync?job=trusty
Update information type: Generic ZSync URL
Assembled ZSync URL: https://gitlab.com/probono/QtQuickApp/-/jobs/artifacts/master/raw/QtQuickApp-x86_64.AppImage.zsync?job=trusty
me@host:~$ Downloads/appimageupdatetool-330-6f8f515-x86_64.AppImage QtQuickApp-x86_64.AppImage -j
zsync2: Bad status code 200 while trying to download .zsync file!
zsync2: Reading and/or parsing .zsync file failed!

TheAssassin · 2018-07-05T12:39:28Z

At a first glance, it looks like GitLab don't support partial responses... I will have to test it a bit myself with curl, I guess.

TheAssassin · 2018-07-05T12:46:33Z

Check this out:

curl -v "https://gitlab.com/probono/QtQuickApp/-/jobs/73890732/artifacts/raw/QtQuickApp-x86_64.AppImage.zsync"
# redirects with "302 Found" to below URL
# trying to get a partial response
curl -v -X GET -H "range: bytes=-100" "https://gitlab.com/probono/QtQuickApp/-/jobs/73890732/artifacts/raw/QtQuickApp-x86_64.AppImage.zsync"
# yields a "200 OK" response, not a "206 Partial Content" one, and provides the entire body of the file

AppImageUpdate tries to fetch only the first few bytes of the zsync file when checking for updates to not have to download it entirely, as it only needs the header for this. Therefore and also for the later update process, it requires Partial Content support from the webservers. GitLab doesn't support this at all, so we can't use it with AppImageUpdate. This is out of scope of AppImageUpdate.

probonopd · 2018-07-05T15:09:20Z

requires Partial Content support from the webservers. GitLab doesn't support this at all

Uh. Thanks for finding this out. I will open a feature request with them.

probonopd · 2018-07-05T15:20:10Z

Blocked by https://gitlab.com/gitlab-com/support-forum/issues/3655

TheAssassin · 2018-07-16T01:26:19Z

So, as we found out today, there's two issues here. One is that the .zsync file contains a URL lacking the mandatory ?job= query string parameter (in this case ?job=trusty). Therefore, when the relative URL is calculated, that URL also lacks the parameter, and therefore GitLab returns status code 404.

Now, this problem can be fixed easily, and so we did. The issue now is however that GitLab lacks support for partial responses, making binary delta updates impossible (or at least very much less inefficient, the client could read until the chunk it wants is received, and terminate the connection afterwards, but that's far from ideal). So, not really an option.

To fix the first part, we should make sure that when the .zsync file is generated, the URL within it contains the ?job=trusty part. Another option is to check for this suffix in the update information, and add it to the composed URL, but that'd be just a cheap workaround (and we'd have to make sure to parse the actual job name out of the query string). Ideally, GitLab would put the job name into the actual URL path.

The second part however is completely out of our scope, and we can't fix it, we could only implement some dirty hacks, accepting some major inefficiency. Therefore, we should start to accept 200 responses but instead rather have GitLab implement partial response support. See https://gitlab.com/gitlab-com/support-forum/issues/3655.

probonopd · 2019-01-15T18:51:51Z

As a workaround, one might want to upload to GitLab Pages as that seems to support the required HTTP Range Requests:

curl -v -X GET -H "range: bytes=0-10" https://pages.gitlab.io/vuepress/

TheAssassin · 2019-02-12T22:33:34Z

Bintray also broke partial requests, see AppImageCommunity/zsync2#5 (comment). This is so annoying...

probonopd · 2019-06-10T14:38:27Z

As another workaround, one might want to use https://raw.githack.com/ which uses https://www.cloudflare.com/. Works!

me@host:~$ curl -v -X GET -H "range: bytes=8-10" https://glcdn.githack.com/scribus/scribus/-/jobs/227157068/artifacts/raw/Scribus-2f0ab15-x86_64.AppImage | xxd
Note: Unnecessary use of -X or --request, GET is already inferred.
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 2606:4700:30::681b:b7fc...
* TCP_NODELAY set
* Connected to glcdn.githack.com (2606:4700:30::681b:b7fc) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [215 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [100 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [3632 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [115 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [37 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-ECDSA-CHACHA20-POLY1305
* ALPN, server accepted to use h2
* Server certificate:
*  subject: OU=Domain Control Validated; OU=PositiveSSL Multi-Domain; CN=sni177193.cloudflaressl.com
*  start date: Apr  5 00:00:00 2019 GMT
*  expire date: Oct 12 23:59:59 2019 GMT
*  subjectAltName: host "glcdn.githack.com" matched cert's "*.githack.com"
*  issuer: C=GB; ST=Greater Manchester; L=Salford; O=COMODO CA Limited; CN=COMODO ECC Domain Validation Secure Server CA 2
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x55cf2dc9bbb0)
} [5 bytes data]
> GET /scribus/scribus/-/jobs/227157068/artifacts/raw/Scribus-2f0ab15-x86_64.AppImage HTTP/2
> Host: glcdn.githack.com
> User-Agent: curl/7.58.0
> Accept: */*
> range: bytes=8-10
> 
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
} [5 bytes data]
< HTTP/2 206 
< date: Mon, 10 Jun 2019 14:37:54 GMT
< content-type: application/octet-stream
< content-length: 3
< set-cookie: __cfduid=d8b3230ff46d36bacfeb1569ec2416a141560177474; expires=Tue, 09-Jun-20 14:37:54 GMT; path=/; domain=.githack.com; HttpOnly
< x-content-type-options: nosniff
< x-request-id: oadW70qqp92
< x-runtime: 0.042011
< x-ua-compatible: IE=edge
< strict-transport-security: max-age=31536000
< referrer-policy: strict-origin-when-cross-origin
< expires: Thu, 31 Dec 2037 23:55:55 GMT
< cache-control: max-age=315360000
< x-robots-tag: none
< cache-control: public, immutable
< access-control-allow-origin: *
< x-githack-cache-status: MISS
< cf-cache-status: HIT
< content-range: bytes 8-10/121565160
< expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
< server: cloudflare
< cf-ray: 4e4c1601898c63fb-FRA
< 
{ [3 bytes data]
100     3  100     3    0     0     38      0 --:--:-- --:--:-- --:--:--    38
* Connection #0 to host glcdn.githack.com left intact
00000000: 4149 02                                  AI.

TheAssassin · 2019-06-10T14:39:23Z

Isn't that something people should just put in as update information then?

probonopd · 2019-06-10T14:43:13Z

Yes, I'm just thinking about how. In my Scribus example, 227157068 is the GitLab CI build number. How would we deal with that? Wouldn't we need a GitLab specific Transport mechanism that would trigger the updater to resolve to "latest"? Or did we already find a better solution?

probonopd · 2019-06-10T14:43:56Z

Prominent AppImage users like Inkscape and Scribus are on GitLab CI, so getting this to work would be really beneficial.

TheAssassin · 2019-06-10T14:47:10Z

You need a special transport type I guess, so magic strings are . But that shouldn't be called "GitLab" simply then. This is clearly a baaaaaaad hack around their inability to provide a feature that will save them money.
I think at first we should apply more pressure on GitLab. I don't see Inkscape/Scribus here, nor in that bug report. Why not raise awareness and wait before implementing bad hacks? Those two rather large projects can apply some pressure.

probonopd · 2019-06-10T14:53:30Z

Actually there are two separate issues:

GitLab makes it way too hard to have a permalink to the latest build artefacts. Inkscape was forced to publish these lengthy instructions | https://gitlab.com/gitlab-com/support-forum/issues/4570
GitLab CI artifacts hosting does not support Range Requests without the use of external CDNs such as https://raw.githack.com/ | https://gitlab.com/gitlab-com/support-forum/issues/3655

I think if we would implement a Transport mechanism to work around 1, then nothing special would be needed to also support 2 if implemented correctly, the user would just put a githack.com URL instead of a gitlab.com URL into the update information.

I have opened a ticket on 1, https://gitlab.com/gitlab-com/support-forum/issues/4570.

probonopd · 2019-06-10T16:06:50Z

1 is actually solved, https://gitlab.com/gitlab-com/support-forum/issues/4570

TheAssassin · 2019-06-10T16:21:58Z

Then maybe raise awareness for 2 with those projects you mentioned and try to get it solved that way. Untrusted third party services are really really annoying. They should not be used in general, and especially not without the consent of the the owner of the project.

probonopd · 2019-06-10T16:56:20Z

Semi-related, also opened

[Feature Request] Please support HTTP Redirects for GitLab CI artifacts downloads

https://gitlab.com/gitlab-com/support-forum/issues/4572

probonopd assigned TheAssassin Jun 10, 2018

probonopd added the bug label Jun 10, 2018

This was referenced Jun 10, 2018

Cannot update from GitLab CI antony-jr/AppImageUpdater#11

Closed

Figure out AppImage download link given a GitLab.com project URL AppImage/appimage.github.io#605

Open

probonopd added the high priority label Jul 2, 2018

TheAssassin added invalid and removed bug high priority labels Jul 5, 2018

TheAssassin added blocked and removed invalid labels Jul 5, 2018

probonopd mentioned this issue May 4, 2019

AppImages for Linux KiCad/kicad-website#370

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cannot update AppImages hosted on GitLab CI #94

Cannot update AppImages hosted on GitLab CI #94

probonopd commented Jun 10, 2018 •

edited

Loading

probonopd commented Jun 10, 2018 •

edited

Loading

probonopd commented Jun 10, 2018 •

edited

Loading

probonopd commented Jun 10, 2018

probonopd commented Jul 2, 2018

TheAssassin commented Jul 2, 2018

TheAssassin commented Jul 2, 2018

probonopd commented Jul 3, 2018

TheAssassin commented Jul 3, 2018

probonopd commented Jul 5, 2018

TheAssassin commented Jul 5, 2018

TheAssassin commented Jul 5, 2018

probonopd commented Jul 5, 2018

probonopd commented Jul 5, 2018

TheAssassin commented Jul 16, 2018

probonopd commented Jan 15, 2019 •

edited

Loading

TheAssassin commented Feb 12, 2019

probonopd commented Jun 10, 2019

TheAssassin commented Jun 10, 2019

probonopd commented Jun 10, 2019

probonopd commented Jun 10, 2019

TheAssassin commented Jun 10, 2019 •

edited

Loading

probonopd commented Jun 10, 2019 •

edited

Loading

probonopd commented Jun 10, 2019

TheAssassin commented Jun 10, 2019 •

edited

Loading

probonopd commented Jun 10, 2019

Cannot update AppImages hosted on GitLab CI #94

Cannot update AppImages hosted on GitLab CI #94

Comments

probonopd commented Jun 10, 2018 • edited Loading

probonopd commented Jun 10, 2018 • edited Loading

probonopd commented Jun 10, 2018 • edited Loading

probonopd commented Jun 10, 2018

probonopd commented Jul 2, 2018

TheAssassin commented Jul 2, 2018

TheAssassin commented Jul 2, 2018

probonopd commented Jul 3, 2018

TheAssassin commented Jul 3, 2018

probonopd commented Jul 5, 2018

TheAssassin commented Jul 5, 2018

TheAssassin commented Jul 5, 2018

probonopd commented Jul 5, 2018

probonopd commented Jul 5, 2018

TheAssassin commented Jul 16, 2018

probonopd commented Jan 15, 2019 • edited Loading

TheAssassin commented Feb 12, 2019

probonopd commented Jun 10, 2019

TheAssassin commented Jun 10, 2019

probonopd commented Jun 10, 2019

probonopd commented Jun 10, 2019

TheAssassin commented Jun 10, 2019 • edited Loading

probonopd commented Jun 10, 2019 • edited Loading

probonopd commented Jun 10, 2019

TheAssassin commented Jun 10, 2019 • edited Loading

probonopd commented Jun 10, 2019

probonopd commented Jun 10, 2018 •

edited

Loading

probonopd commented Jun 10, 2018 •

edited

Loading

probonopd commented Jun 10, 2018 •

edited

Loading

probonopd commented Jan 15, 2019 •

edited

Loading

TheAssassin commented Jun 10, 2019 •

edited

Loading

probonopd commented Jun 10, 2019 •

edited

Loading

TheAssassin commented Jun 10, 2019 •

edited

Loading