Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http: Default Client / HTTP mangles requests with specific long urls causing remote servers to throw 400s on valid requests #9543

Closed
nemosupremo opened this issue Jan 9, 2015 · 3 comments

Comments

Projects
None yet
4 participants
@nemosupremo
Copy link

commented Jan 9, 2015

1.) Go Version: go1.3 darwin/amd64 (Also affects go version go1.3 linux/amd64)
2.) OSX 10.9.5 / Ubuntu 12.04.2 LTS (GNU/Linux 3.2.0-24-generic x86_64)
3.)

When making a request with this specific a long url (it seems when the query_string is 266 characters), the go http client seems to mangle the request.

Here is a sample hits that hits the google homepage, making the query longer and longer until the error occurs: https://gist.github.com/nemothekid/501fb7e520999342560a

4.) For comparison, the python requests library doesn't have a problem with this:

>>> import requests
>>> r = requests.get("https://www.google.com?part=id&id=ZKfj0-ygXbg%2C8SINiaqgNIQ%2CxeQhXEJaHRo%2CU2VHAMOP5DY%2CGMr1JhKms5M%2CyKjYj5HMaNU%2CJgFAaxNc-tk%2CoMnDCxiZVMw%2Ck4VG725R4X0%2CU-s10nNV_N0%2CkN-p51dFYrk%2CmmIFr0NgRi8%2C4uuQ-qxjqbU%2CuRaePS8fjQE%2CJt_G6a4g9lw%2C5GaFxfjdfX8%2CoKoDg2aG-z4%2CrArN_38Xil8%2C -RuZqxSiFIw%2CNBvTI73iNDo%2CDaRvvkud9MQ%2CZm4y67kxpBk%2CG9MQIHE4pd0%2C2YwKlqgCdQw%2C6pc67X22kFo%2CYvxGDTawvAI%2CgpdFKiMGrDA%2CVEqtKfXYX6A%2ClqQCJ-xCEBE%2CmW8WwnqTmDA%2CZ0W_4kN8isA%2C3i8icj_wJGM%2C0i6E8FPecGc%2Cw8ISxSwSz-0%2CGx2ahhyKqEU%2CrFsJdFKsJDU%2C9ET_qEJYEew%2CVdXBiVM6hyI%2CefL0fPMZGCo%2CHH1OMx7dhhk%2ClvCd9zjXeZE%2CIJ0uQjwTc8o%2CMqoReZ6WJjU%2C0UkWtQGzdXc%2CIZjgPfWX8Jc%2CGvnULl2TRlw%2C6UNXAoOt-zk%2CQDaDKoGV-YA%2CX253vC7vvb8%2Cc-MWDL4C8PY&maxResults=50&")
>>> r.text
u'<!doctype html><html itemscope="" itemtype="http://schema.org/WebPage" lang="en"><head><meta content="Search the world\'s information, including webpages, images, videos and more. Google has many special features to help you find exactly what you\'re looking for." name="description"><meta content="noodp" name="robots"><meta content="/images/google_favicon_128.png" itemprop="image"><title>Google</title><script>(function(){window.google={kEI:\'tzGvVITFIomDoQTBrYGgCA...

5.) If you run the code above, Google (and other servers) you will get a 400 error. (I'm not sure what Google's 400 looks like, but I'm unsure even if the response completes).

<!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 400 (Bad Request)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/errors/logo_sm_2.png) no-repeat}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/errors/logo_sm_2_hr.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/errors/logo_sm_2_hr.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/errors/logo_sm_2_hr.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:55px;width:150px}
  </style>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>400.</b> <ins>That’s an error.</ins>
  <p>Your client has issued a malformed or illegal request.  <ins>That’s all we know.</ins>

This doesn't effect every long url, for example in the gist provided if you set u to:

u := h + "?part=id&id=LxtN--HMNsA%2CgiIT5ALtjPY%2C5fmzALLsofw%2CahBMSnKmMHs%2C2lduGacyTTo%2CQTXrc-XrHc4%2ChleMcTD8qrk%2CLVa9IMK6-bg%2CF9SyVu74gWg%2Cz5ITVelLBYY%2Ccx-lqXEU_aI%2Cvcr4MzfQQvQ%2CCNBVPuHmn70%2C0VmHS9CQcZI%2CS7IDGVC1KN4%2CsuUOAusvDEc%2CMpg41_O5XOY%2CHa1a_UEIsiE%2CRLkWr_ybDqc%2C_YFGr_9vBwg%2CeIo9iuZ6REA%2CePCalOWbEn8%2CwYt_PSUtp-A%2CJzt6JODbssE%2ClfYHdaUfSjY%2CEtcGmHs3Qss%2CyaqNkw0HW7M%2CqxCfvgeeT58%2Cc2MSgxccv_0%2Cm8UnfePn2fg%2C4h0TiFKao2w%2CVo8yWORV7TI%2C2JxxLHafwMg%2CL_YTkZtfuAA%2CODV566R-4wc%2CPlZ4MQR0de4%2CzsbpRQDNQKQ%2CAGWNVxcF4P0%2CtgSAufCbf6M%2C1jXC6L4WREE%2C7GnhVg61l5w%2C2YEccc4Vp44%2CDg0E7pzAm8o%2CRKzWK1eyPAQ%2Ct-dPaxAuDkY%2Cjh3hRvNy1C0%2CtitAChBO0jc%2CBinjgB0bWjE%2Cx7rHmM-cUjs%2CttT0GmCLeVs&maxResults=50"

you can successfully execute that entire request with no problems. (I also don't know what the difference between the two urls, both are essentially random strings).

@minux minux closed this Jan 9, 2015

@minux

This comment has been minimized.

Copy link
Member

commented Jan 9, 2015

There is a space at u[285], so when i = 287, the URL looks like this:
http://...../Xil8%2C -

And the GET request looks like this:
GET /?part=id&id=ZKfj0-....Xil8%2C - HTTP/1.1\r\n
...

Obviously an invalid GET request.

This is working as intended. You've passed an invalid URL.

@nemosupremo

This comment has been minimized.

Copy link
Author

commented Jan 9, 2015

Crap, sorry. I was looking at this so long I didn't see the obvious. Apologies for the unnecessary issue, and thanks for looking into this. (Although I still don't get why it worked in Python and in my browser)

@jayschwa

This comment has been minimized.

Copy link
Contributor

commented Jan 9, 2015

Python and your browser might URL encode the space.

@golang golang locked and limited conversation to collaborators Jun 25, 2016

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.