Download a file with Redirection corrupts the file data

[gtnardy]

If I download a file from a link (e.g: https://github.com/gtnardy/nanos-world-halloween/releases/download/v1.0.0/halloween.zip) which redirects me with set_follow_location enabled, it may end up adding the 'between' received data into my file:

Causing the top part of the file being the content before the file start to be downloaded itself:

I could identify with progress callback that the maximum size changed as well during the download:

Downloading 487/621.
Downloading 621/621.
Downloading 1378/1688665.
Downloading 1616/1688665.
Downloading 2994/1688665.
Downloading 4372/1688665.
...

Would that be a bug? Or should I try to handle that on my side?

[yhirose]

@gtnardy, could you try the same with curl with auto redirect option? If curl handles it as you expect, the current behavior in cpp-httplib is a bug. Please let me know your result. Thanks!

[gtnardy]

I've tested with curl on linux and Invoke-WebRequest on windows/powershell, both with redirection enabled (-L and -MaximumRedirection 10 respectively), and they downloaded the file correctly and not corrupted.

I could do a workaround for this problem using ResponseHandler and getting the status 200 from it, once I get it, I move my file pointer to the beginning and it starts downloading the file itself. The status 200 is returned just after the redirection and is the only status/response I receive from the ResponseHandler.

[yhirose]

@gtnardy, thanks for the excellent investigation. I think it's a bug. Could you do the workaround until I fix the problem? Hope I find time to work on this sometime this month.

[gtnardy]

sure thing! thank you for this amazing lib!

[yhirose]

@gtnardy, could you try the latest httplib.h? If it works on your machine, I'll release it.

[gtnardy]

Tested! It does not writes to the file anymore =D! Thank you!

Some things I noticed:

It still "downloads" the first redirectional page, but does not trigger ContentReceiver which is great! But it still triggers ResponseHandler 200 when it redirects and also triggers Progress during downloading the redirectional page. I don't know if you think it is fine or not, but my progress bar goes from 0 to 100 (while redirecting), then starts going from 0 to 100 again to download the file ;)

The ResponseHandler only triggers once when it redirects, and do not triggers when it finishes, was it supposed to happen? (it does not bother me currently).

ResponseHandler = Got Response
Progress = Downloading
ContentReceiver = Writing to file...

[yhirose]

@gtnardy, good find! I updated my change. Could you try again?

[gtnardy]

No more ProgressHandler before redirection 😊! (still ResponseHandler 200 just after redirection and no ResponseHandler at the end of download, also I stopped receiving ResponseHandler 302 at the starting of download - are those a problem?)

[yhirose]

@gtnardy, thanks for testing. That's what I expected. On the auto redirection mode, all the redirections before the final request will be completely ignored. I think the issue is now fixed. Thanks for your help!