Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"%7E" in Location: header is unescaped to raw tilde instead of kept as-is #11

Closed
OrIdow6 opened this issue May 9, 2021 · 2 comments
Closed
Labels
bug Something isn't working

Comments

@OrIdow6
Copy link
Collaborator

OrIdow6 commented May 9, 2021

When wget-lua follows a redirect with a "%7E" in the Location header, it unescapes the %7E into a tilde before sending it. This can be demonstrated as follows (somewhat convoluted, but it's clear what's happening):

  • Set up one terminal with nc -lp 5555
  • Set up another with nc -lp 5557
  • In a third, do wget http://127.0.0.1:5555/page.html
  • Paste the following into the first one (if you're on a Unix system, this isn't proper HTTP because of the newlines, but wget doesn't care):
HTTP/1.1 301 Redirect
Location: http://127.0.0.1:5557/what%2Ewhat.html
Content-Length: 3

eee
  • Wget should follow the redirect, and on your other nc instance you will see something like
GET /what%7Ewhat.html HTTP/1.1
User-Agent: Wget/1.21
Accept: */*
Accept-Encoding: identity
Host: 127.0.0.1:5557
Connection: Keep-Alive

on vanilla wget (1.21), but

GET /what~what.html HTTP/1.1
User-Agent: Wget/1.20.3-at.20210504.01 (linux-gnu)
Accept: */*
Accept-Encoding: identity
Host: 127.0.0.1:5557
Connection: Keep-Alive

on wget-lua/wget-at (1.20.3-at.20210504.01).

This caused problems on Bintray, where it unescaped this in the AWS CDN URL that https://dl.bintray.com/lamyj/generic/apt/pool/main/o/odil/python3-odil_0.12.1-1~stretch_amd64.deb (will hopefully be in the Wayback Machine soon) redirected to, which caused the download to fail. Rewby discovered that --no-iri makes wget-at behave properly (i.e. like upstream wget), and this was indeed adopted on the Bintray project, but this fix can't be use everywhere.

@OrIdow6
Copy link
Collaborator Author

OrIdow6 commented May 9, 2021

JAA apparently got this same issue, also with AWS, on aiohttp

@Arkiver2 Arkiver2 added the bug Something isn't working label Mar 23, 2022
@Arkiver2
Copy link
Member

The bug report notes

HTTP/1.1 301 Redirect
Location: http://127.0.0.1:5557/what%2Ewhat.html
Content-Length: 3

eee

should be run, however later examples seem to have used %7E instead of %2E.

Assuming %7E is what is meant, this issue does not happen anymore in version 1.21.3-at.20230621.01.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants