Failed to extract the current "stop releases"

[ghost]

Hey,
Is this warning dangerous for the upgrade operation? What could I do about it?

# sfos-upgrade 3.4.0.24
[D] unknown:0 - Got http_proxy from environment, will not talk to connman
Warning: Failed to extract the current "stop releases" from https://jolla.zendesk.com/hc/en-us/articles/201836347#4.1
Hence using an internal, potentially outdated list of stop releases instead.
Do you really want to continue? (Y/N)

Also, why those information is needed?
Any help would be appreciated.

[Olf0]

Is this warning dangerous for the upgrade operation?

No, because currently sfos-upgrade's internal list of stop releases is up-to-date (plus 2.0.5.6 and 1.0.0.5).
Background: Additionally downloading and extracting the current stop releases from https://jolla.zendesk.com/hc/en-us/articles/201836347#4.1 is just a safety measure, as the internal list may become outdated.

What could I do about it?

The error message looks as if a system-wide HTTP-proxy is set on your device, which prevents the page https://jolla.zendesk.com/hc/en-us/articles/201836347#4.1 from being downloaded.

1. Does that web-page load in the native Browser on this device?
2. Does a curl -LO https://jolla.zendesk.com/hc/en-us/articles/201836347 (at the command line) work on this device?

Also, why those information is needed?

See https://jolla.zendesk.com/hc/en-us/articles/201836347#4.1
TL;DR: "Jumping over" a stop release will likely break your SailfishOS installation.

[ghost]

Yes, I am using a HTTP proxy and have relevant environment variables set up.

Does that web-page load in the native Browser on this device?

Yes.

But I noticed a request with cURL returns 403 from that URL:

$ curl -I https://jolla.zendesk.com/hc/en-us/articles/201836347
HTTP/1.0 200 Connection established
Proxy-agent: tinyproxy/1.10.0

HTTP/1.1 403 Forbidden
Date: Wed, 14 Oct 2020 17:39:52 GMT
Content-Type: text/html; charset=UTF-8
Connection: close
...other headers...

Ignoring proxy with flag --noproxy "*" gives the same result.

[Olf0]

Well, then something is strange about your HTTP-proxy setup.
Interestingly, the cURL call before that one seems to succeed with your setup. You may check that at the command line per
curl -LO https://coderus.openrepos.net/whitesoft/sailversion
If my assumption is correct, you have a positive and a negative test to experiment with your proxy settings.

But as the only relevant difference between those two cURL calls is the DNS-domain visited (AFAICS), can you assure that no DNS-based filtering is involved (be it "on device", in your LAN, by your internet provider etc.)?

[carmenfdezb]

Hi @Olf0! I obtained the same warning when I updated my XA2 and my Jolla Tablet and I don't use proxy

[ghost]

It seems like a blank page with an error "Please turn JavaScript on and reload the page." is returned from jolla.zendesk.com when User-Agent is curl/<curl_version>.

[Olf0]

Oh, now I see. Thank you @wickedsp1d3r for insisting on this and @carmenfdezb for confirming this issue in a proxy-less environment.
I should have looked inside the file, when I tested downloading it manually. Yes, it does not have the expected content, which is displayed in the browser. This was definitely working from when I wrote this functionality in 2019 to at least spring 2020 (when SFOS 3.3.0 was released).

I am not sure, how this is related to @wickedsp1d3r's original proxy warning, but I will experiment with user agent settings etc. to let sfos-upgrade download and extract the recent list of stop releases from there, again.
Do you have any specific suggestions how to implement this?

[Olf0]

After some research and experimenting, I have the impression, that one has to enable JavaScript for Cloudflare (the content delivering company) to be able to see any content at jolla.zendesk.com

Edit: No, it loads in Firefox with JavaScript disabled by NoScript, but setting the user agent string does not seem to make a difference, as curl -A 'Mozilla/5.0 (X11; Linux x86_64) Gecko/20100101 Firefox/47' https://jolla.zendesk.com/hc/en-us/articles/201836347 still downloads a "No JavaScript"-warning page.

As I am not a web expert and have no idea how to resolve this (or even if my quick analysis is correct), I think I will likely delete or disable this safety measure.
This is unfortunate, because when sfos-upgrade cannot automatically determine the recent stop releases anymore, this increases maintenance burden and responsibility. :-(

[Olf0]

Well, so I released sfos-upgrade 3.7.8, in which downloading the list of current stop releases from jolla.zendesk.com is disabled.

I still leave this issue open, in hope someone has an idea how to circumvent the hurdles Zendesk / Jolla now impose for accessing this list in an automated manner.

[carmenfdezb]

Thank you so much!

[Olf0]

Filing a request at jolla.zendesk.com may help, which kindly asks to offer a current list of "SailfishOS stop releases" suitable for automated ("scripted") download (e.g., per curl or libcurl) either at its usual place (i.e., by switching the measure(s) off again, which now prevent downloading it) or another (then, also maybe in a better machine-readable / structured / formatted form).

Edit: And / or alternatively, depict this issue (still along the approach denoted above) as a "+ New Topic" at FSO in the "Platform Development" category (seems to fit best for this).

[Olf0]

Because none of the categories the bug reporting process at jolla.zendesk.com offers is well suited for this, I decided to report this at FSO in the "Platform Development" category: https://forum.sailfishos.org/t/scripted-download-of-https-jolla-zendesk-com-hc-en-us-articles-201836347-fails/7935

[Olf0]

The replies provided by sailor (i.e., a Jolla employee) Jovirkku in the original thread and a different one are not really addressing this issue, unfortunately.

[Olf0]

Accidentally I noticed that Jolla has silently provided a machine-downloadable list of "stop releases" they maintain, see FSO issue 7935 "Scripted download of https://jolla.zendesk.com/hc/en-us/articles/201836347 fails!".

So finally I was able to fix this by commit 5ed9547.

[Olf0]

After some research and experimenting, I have the impression, that one has to enable JavaScript for Cloudflare (the content delivering company) to be able to see any content at jolla.zendesk.com

Edit: No, it loads in Firefox with JavaScript disabled by NoScript, but setting the user agent string does not seem to make a difference, as curl -A 'Mozilla/5.0 (X11; Linux x86_64) Gecko/20100101 Firefox/47' https://jolla.zendesk.com/hc/en-us/articles/201836347 still downloads a "No JavaScript"-warning page.

As I am not a web expert and have no idea how to resolve this (or even if my quick analysis is correct), […]. :-(

For some technical background on the bot-detection and -prevention measures the two big CDNs (Content Delivery Networks) Coudflare and Akamai emply, see Forum.Sailfishos.Org: Scripted download of https://jolla.zendesk.com/hc/en-us/articles/201836347 fails! - Reply 16.
From there:
<sub>@flyping provided a pointer to the corresponding "bots"-related measures the CDNs (Content Delivery Networks) Cloudflare and Akamai employ.
The documentation by Cloudflare describes very well, what I experienced for almost 3 years. As expected these measures are designed to be non-circumventable.

Side note: Both, Akamai and Cloudflare emphasise, that the bot-related measures are highly customer-configurable and can be enabled per web-page (at least for Akamai).
Nevertheless, moving Jolla's official list of stop releases to the SailfishOS-Wiki and open it up for collaboration at GitHub is a vastly better solution.
Thanks!</sub>