Plugin doesn't follow HTTPS redirections when downloading package tools

[wmarkow]

Describe the bug
I have prepared my custom package_wmarkow_index.json - I just needed to build with some latest GCC compiler. It contains two custom avr-1.8.2 platforms which uses two different compiler tools:

• avr-gcc-8.3.0-x64-mingw hosted as https://blog.zakkemble.net/download/avr-gcc-8.3.0-x64-mingw.zip
• avr-gcc-9.1.0-x64-mingw hosted as https://blog.zakkemble.net/download/avr-gcc-9.1.0-x64-mingw.zip

When I want to use one of these platforms in Sloeber, then compiler is not downloaded correctly, because the downloader doesn't follow the HTTP redirections to reach the destination.

To Reproduce
Take a look at the GIF:

Expected behavior
The tools should be downloaded.
As a workaround I downloaded them manually and copied into arduinoPlugin\downloads directory.

Screenshots
Take a look at the attached GIF above.

Desktop (please complete the following information):

• OS: Windows 7
• Browser Firefox 71.0
• Sloeber V4.3.1_win64.2018-10-10_08-21-58

[wimjongman]

are you able to create a patch for this issue?

[jantje]

I'm pretty sure redirections have been implemented for something. Not sure if it was only for json or for all downloads.
Not sure why you want to use redirections for the zip files though ....

[wmarkow]

@wimjongman theoretcally and practically I'm able to create a patch for this issue, but I'm not giving any promisses. Please wait a bit.

@jantje I'm not the owner of this webpage. I just found that the guy hosts there the avr-gcc dostributions so I just used them in my own package json. This webpage uses Wordpress under the hood and redirects links to the download folder.

[wimjongman]

@wmarkow cool. Thanks!

[wmarkow]

I had some time to take a look at this. Just put a breakpoint in
io.sloeber.core.managers.InternalPackageManager.downloadAndInstall(String, String, IPath, boolean, IProgressMonitor)
and execute the test scenario. The breakpoint will hit twice but continue until it wants to download
https://blog.zakkemble.net/download/avr-gcc-8.3.0-x64-mingw.zip
Now debug what is going on.

@jantje indeed the method io.sloeber.core.api.PackageManager.myCopy(URL, File, boolean) handles the redirections - but only once. In my case I had a multi-redirection (like a chain), like below:

https://blog.zakkemble.net/download/avr-gcc-8.3.0-x64-mingw.zip redirects with 302 code to:
http://blog.zakkemble.net/wp-content/uploads/downloads/2019/02/avr-gcc-8.3.0-x64-mingw.zip redirects with 301 to:
https://blog.zakkemble.net/wp-content/uploads/downloads/2019/02/avr-gcc-8.3.0-x64-mingw.zip which can be finally downloaded with code 200

Proposed fix: when HTTP status is 3xx then call io.sloeber.core.api.PackageManager.myCopy(URL, File, boolean) recursively with a redirect URL.

Possible side effect: what if url A redirects to url B and url B redirects to url A? We will have an endless loop and probably Sloeber will crash with out of memory error. Is that even possible?

[jantje]

Possible side effect: what if url A redirects to url B and url B redirects to url A? We will have an endless loop and probably Sloeber will crash with out of memory error. Is that even possible?

that will happen for sure

I would prefer to point to https://blog.zakkemble.net/wp-content/uploads/downloads/2019/02/avr-gcc-8.3.0-x64-mingw.zip in the json file instead of https://blog.zakkemble.net/download/avr-gcc-8.3.0-x64-mingw.zip
No fix needed and will survive one new redirect

[erdemuncuoglu]

Possible side effect: what if url A redirects to url B and url B redirects to url A? We will have an endless loop and probably Sloeber will crash with out of memory error. Is that even possible?

Web browsers overcome this by limiting loop count (by 10 redirects for example).

[jantje]

If all goes well the nightly should contain this fix tomorrow.
Thanks