No response for Bad URI requests

[romovs]

Previously bmp used to provide responses for requests with malformed URIs:

{ 
  "request" : { "bodySize" : 0,
    "comment" : "",
    "cookies" : [  ],
    "headers" : [  ],
    "headersSize" : 0,
    "httpVersion" : "HTTP/1.1",
    "method" : "GET",
    "queryString" : [  ],
    "url" : "http://:/"
  },
  "response" : { "bodySize" : 0,
    "comment" : "",
    "content" : { "comment" : "",
        "mimeType" : "",
        "size" : 0
      },
    "cookies" : [  ],
    "headers" : [  ],
    "headersSize" : 0,
    "httpVersion" : "HTTP/1.1",
    "redirectURL" : "",
    "status" : -998,
    "statusText" : "Bad URI"
  },
  ...
}

This doesn't seem to be the case any more with LittleProxy.

[jekh]

This bad request behavior is not actually covered in the HAR 1.2 spec. Do you know how chrome, firefox, or other tools record bad requests in the HARs they generate?

[romovs]

Looks like the problem here is twofold; certain URLs not being proxied as expected and responses not being generated for failed requests.

Main issue here seems to be that browsers generally don't care much about URLs to be RFC2396 compliant, while BMP does. So in browser generated HARs we would see an actual server response no matter how invalid the URL is (well it least when it comes to the query part. authority with invalid characters is treated slightly differently, but let's ignore that for now).

Take these two requests for example:

http://host/%;
http://google.com/%%

Browser would make actual requests and return real server responses - 404, 400, or something similar depending on server configuration.

However when proxied through BMP both of these will fail.
As far as I remember the issue stems simply from java.net.URI failing on everything that is not strictly RFC2396 compliant. This leads to all sort of problems when URLs which work in browsers stop working when proxied with BMP.
It's arguable whether browsers should even support such URLs but it is what it is, and unfortunately a lot of websites abuse this: %uHHHH style of unicode encoding in the query string example link, or just forbidden unescaped characters which happens a lot with miscellaneous badly written AJAX requests example link

I am not sure if the underlying reason could be easily fixed or even needs to be fixed? But the Bad URI response workaround was rather useful as it gave at least some indication about what's going on.

As far as I can see, completely omitting responses leads to two problems

1. It breaks most HAR viewers. http://www.softwareishard.com/har/viewer/ won't even display the page if one of the entries doesn't have a response.
2. Essential error related information is being discarded thus making things very hard to debug.
   We just ran a few browser automation tests, and I see a lot of requests with missing responses in the HAR. From past tests I remember that one of those had to do with Bad URI, but what about the rest? Do they fail due to domain resolution, internal proxy problem, something else? (actually except of one Bad URI request those tests are not supposed to have any other issues, so the high number of requests with missing response is rather suspicious...)

[jekh]

Thanks for the detailed analysis. I just tested www.google.com/% and www.google.com/%u20AC using the LittleProxy implementation, and both worked as expected and the HAR was correct (Google returns a 404). The legacy implementation has many limitations, and you ran into one of them.

With HAR capture disabled, the two example links you provided do work with BMP+LittleProxy, but the maccabi4u.co.il link fails when HAR capture is turned on. It looks like the netty QueryStringDecoder is throwing an exception:

java.lang.IllegalArgumentException: invalid escape sequence `%u0' at index 0 of: %u05DE
    at io.netty.handler.codec.http.QueryStringDecoder.decodeComponent(QueryStringDecoder.java:345) ~[netty-all-4.0.23.Final.jar:4.0.23.Final]

As you note, netty is "correct" in that it is not a valid URL. Nevertheless, the HAR (and BMP in general) shouldn't care whether the URL is valid. The HAR does need to decode the query parameters to capture them, but if it cannot decode them, it should handle the error gracefully.

In this specific case, perhaps there is some sort of relaxed query parameter capture would fix the issue and provide sufficient information to the HAR. At the very least, if the query param decode fails, it should still capture the URL as-is along with the rest of the request.

In general, if the HAR capture filter throws an exception for some reason (indicating a code defect), it should not abort the entire request.

Regarding the second part of your issue, when there is an error condition and HAR capture is turned on, there should be some defined behavior. Unfortunately the HAR spec provides no guidance here. A good example would be when an upstream host is unreachable. I'm open to suggestions, but I'd like BMP to follow an established pattern if one exists. Do you have any error examples from Chrome, Firefox, or other HAR capture utilities?

[romovs]

Do you have any error examples from Chrome, Firefox, or other HAR capture utilities?

• Firefox (with Firebug) - discards failed requests.
• webpagetest.org - discards failed requests.
• Internet Explorer - exports a non compliant HAR... in an XML format no less!
• Chrome - saves failed requests. Uses custom fields (names starting with underscore) in the response object for error data.
• Charles Proxy - same as Chrome except error description is saved in the comment field.

I haven't tested other tools but my guess would be that those tools which do record failed requests probably also use custom fields.
Custom fields sound like the safest approach imo - HAR viewers/parsers would simply ignore them according to the spec, yet the error details still can be extracted if necessary.

Here are some snippets (cleaned up everything but request/response) from Chrome and Charles Proxy.

Chrome - network timeout:

{
    "request": {
      "method": "GET",
      "url": "http://192.168.0.200/cb-multiplex.js",
      "httpVersion": "unknown",
      "headers": [
        {
          "name": "Accept",
          "value": "*/*"
        },
        {
          "name": "Referer",
          "value": "http://web.cloudbeat/monitor/8?location=5"
        },
        {
          "name": "User-Agent",
          "value": "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
        }
      ],
      "queryString": [],
      "cookies": [],
      "headersSize": -1,
      "bodySize": 0
    },
    "response": {
      "status": 0,
      "statusText": "",
      "httpVersion": "unknown",
      "headers": [],
      "cookies": [],
      "content": {
        "size": 0,
        "mimeType": "x-unknown"
      },
      "redirectURL": "",
      "headersSize": -1,
      "bodySize": -1,
      "_transferSize": 0,
      "_error": "net::ERR_CONNECTION_TIMED_OUT"
    },
    "timings": {
      "blocked": -1,
      "dns": -1,
      "connect": -1,
      "send": 0,
      "wait": 0,
      "receive": 0,
      "ssl": -1
   }
},

Chrome - DNS failure

      {
        "request": {
          "method": "GET",
          "url": "http://portalstage.hot.net.il/UploadedImages//01_2013/existing_client_banner.png",
          "httpVersion": "unknown",
          "headers": [
            {
              "name": "Accept",
              "value": "image/webp,*/*;q=0.8"
            },
            {
              "name": "Referer",
              "value": "http://www.hot.net.il/Static/1037_1484740438.css"
            },
            {
              "name": "User-Agent",
              "value": "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
            }
          ],
          "queryString": [],
          "cookies": [],
          "headersSize": -1,
          "bodySize": 0
        },
        "response": {
          "status": 0,
          "statusText": "",
          "httpVersion": "unknown",
          "headers": [],
          "cookies": [],
          "content": {
            "size": 0,
            "mimeType": "x-unknown"
          },
          "redirectURL": "",
          "headersSize": -1,
          "bodySize": -1,
          "_transferSize": 0,
          "_error": "net::ERR_NAME_NOT_RESOLVED"
        }
      }

Charles Proxy - network timeout

{ 
  "request" : { 
      "bodySize" : 0,
      "cookies" : [  ],
      "headers" : [ { 
            "name" : "Host",
            "value" : "192.168.0.200"
          },
          { "name" : "User-Agent",
            "value" : "Mozilla/5.0 (Windows NT 6.1; rv:37.0) Gecko/20100101 Firefox/37.0"
          },
          { "name" : "Accept",
            "value" : "*/*"
          },
          { "name" : "Accept-Language",
            "value" : "en-US,en;q=0.5"
          },
          { "name" : "Accept-Encoding",
            "value" : "gzip, deflate"
          },
          { "name" : "Referer",
            "value" : "http://web.cloudbeat/monitor/8?location=5"
          },
          { "name" : "Connection",
            "value" : "keep-alive"
          }
        ],
      "headersSize" : 304,
      "httpVersion" : "HTTP/1.1",
      "method" : "GET",
      "queryString" : [  ],
      "url" : "http://192.168.0.200/cb-multiplex.js"
    },
  "response" : { 
      "_charlesStatus" : "Failed",
      "bodySize" : 0,
      "comment" : "Connect: Connection timed out: connect",
      "content" : { 
          "mimeType" : "",
          "size" : 0
        },
      "cookies" : [  ],
      "headers" : [  ],
      "headersSize" : 0,
      "httpVersion" : "",
      "redirectURL" : "",
      "status" : 0,
      "statusText" : ""
    }
}

Charles Proxy - DNS failure:

{ 
  "request" : { 
      "bodySize" : 0,
      "cookies" : [ ],
      "headers" : [ { "name" : "Host",
            "value" : "portalstage.hot.net.il"
          },
          { "name" : "User-Agent",
            "value" : "Mozilla/5.0 (Windows NT 6.1; rv:37.0) Gecko/20100101 Firefox/37.0"
          },
          { "name" : "Accept",
            "value" : "image/png,image/*;q=0.8,*/*;q=0.5"
          },
          { "name" : "Accept-Language",
            "value" : "en-US,en;q=0.5"
          },
          { "name" : "Accept-Encoding",
            "value" : "gzip, deflate"
          },
          { "name" : "Referer",
            "value" : "http://www.hot.net.il/Static/1037_1484740438.css"
          },
          { "name" : "Cookie",
            "value" : "__utma=262333715.1009848784.1420231726.1420231726.1430422952.2; __utmz=262333715.1420231726.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __utmv=262333715.|1=unique_users=1420231725131=1; __CT_Data=gpv=5&apv_9890_www08=5&cpv_9890_www08=4; WRUID=584958560.745084182; __utmb=262333715.5.9.1430422954329; __utmc=262333715; WRIgnore=true"
          },
          { "name" : "Connection",
            "value" : "keep-alive"
          },
          { "name" : "Pragma",
            "value" : "no-cache"
          },
          { "name" : "Cache-Control",
            "value" : "no-cache"
          }
        ],
      "headersSize" : 788,
      "httpVersion" : "HTTP/1.1",
      "method" : "GET",
      "queryString" : [  ],
      "url" : "http://portalstage.hot.net.il/UploadedImages//01_2013/existing_client_banner.png"
    },
  "response" : { 
      "_charlesStatus" : "Failed",
      "bodySize" : 0,
      "comment" : "UnknownHost: portalstage.hot.net.il",
      "content" : { 
          "mimeType" : "",
          "size" : 0
        },
      "cookies" : [  ],
      "headers" : [  ],
      "headersSize" : 0,
      "httpVersion" : "",
      "redirectURL" : "",
      "status" : 0,
      "statusText" : ""
    },
}

[jekh]

This is fantastic, thank you. I think the approach you suggest, using custom fields, is an excellent one. My preference would be to mirror Chrome's behavior. Given the other issues, it's unlikely I'll be able to add this feature before the beta-1 release to Central, but if anybody else in the community would like to take it on, that would be fantastic!

[jekh]

Hi @romovs -- this is finally fixed! The code in master has been updated to include requests/responses that error out in the HAR. The _error field will be set with the relevant error message.

Would you mind testing this with your use case? If it works for you, and you see the right error information, I'm going to go ahead and release beta-2.

[romovs]

Hi,
Just tested with a few sites and it seems to work perfectly fine so far. Errors are present where expected and HAR is viewable once again!

Although I think I might have stumbled upon that never loading page issue again.
will elaborate about it in another topic...

[jekh]

Thanks @romovs -- very glad this issue seems to be fixed, but sad that the old issue is back again. I'll close this one and reopen the other.