HAR

[dufferzafar]

This adds a script that can dump flows as HAR but does not depend on harparser.

Tasks:

• Tests based on comparison of entire HAR files?
• Unit tests for formatting functions
• .zhar Compression
• SSL and Connect Timings
• Logging via ctx.log
• Basic Test
• Proper Cookies Format
• Remove harparser dependency in setup.py
• Append to file on every request, rather than writing at end (forgot that this is json!)
• Read har file? (Separate PR if at all)

New Fields:

• text in response content
• postData for POST, PUT, PATCH
• serverIPAddress
• connection (we have no unique ids for connections)
• pages (mitmproxy has no concept of pages)

Fixes #1320

[mkagenius]

The post Data field is missing in case of POST method (was missing in original har_extractor too). Would it possible to add that? Something like this, maybe:

    if flow.request.method == "POST":
        entry["request"]["postData"] = format_post_data(flow.request)

and

def format_params(obj):
    if obj:
        return [{"name": k, "value": unicode(v, errors='ignore')} for k, v in obj.items()]
    return ""

def format_post_data(obj):
    content_type = obj.headers.get("content-type", "")
    post_data = {"mimeType" : content_type}
    is_valid_content_type = "application/x-www-form-urlencoded" in content_type.lower()
    if is_valid_content_type:
        post_data["params"] = format_params(obj.urlencoded_form)

    post_data["text"] = unicode(obj.content, errors='ignore')
    return post_data

[mkagenius]

One more thing, regarding cookies (in response), they are coming like (same behavior in har_extractor too):

  "cookies": [
            {
              "name": "__cfduid",
              "value": "SetCookie(value='d4554e2d8e2c1406a252163f6f693300a1470146048', attrs=CookieAttrs[('expires', 'Wed, 02-Aug-17 13:54:08 GMT'), ('path', '/'), ('domain', '.mydomain.co'), ('HttpOnly', None)])"
            }
          ],

While each one should be a separate field as in the specification:

"cookies": [
    {
        "name": "TestCookie",
        "value": "Cookie Value",
        "path": "/",
        "domain": "www.janodvarko.cz",
        "expires": "2009-07-24T19:20:30.123+02:00",
        "httpOnly": false,
        "secure": false,
        "comment": ""
    }
]

[mhils]

This looks great. So much clearer than what we had before 👍

[dufferzafar]

How can I unit test these functions? examples directory has no __init__.py so har_dump can't directly be imported!

[mhils]

Based on the stuff in test_examples:

m, sc = tscript("har_dump.py", six.moves.shlex_quote(path))
format_datetime = sc.ns["format_datetime"]
# work with format_datetime

Not really ideal, but I feel that's still better than making the examples directory a module.

[dufferzafar]

@mkagenius Should the postData field be included in PUT, PATCH requests too?

@mhils This is ready for review!

[mkagenius]

@dufferzafar Good question. I am also not sure as the specification is not too clear about that. But I checked what chrome does when we do copy ALL as HAR from inspect element window and found that in case of PUT / PATCH too it creates the post body.

[dufferzafar]

@mkagenius If chrome does this, then I think we should too. It seems like the correct thing to do anyways.

[Kriechi]

Do we have a policy about cookie attribute capitalisation?
Sometimes we use only-lowercase, sometimes mixed...

[dufferzafar]

There's no policy, i guess because the _kconv function converts these keys to lowercase.

IMO, we should keep using mixed cases just to show that they don't matter.