New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

First class support for HAR #1477

Open
dufferzafar opened this Issue Aug 11, 2016 · 4 comments

Comments

Projects
None yet
3 participants
@dufferzafar
Copy link
Contributor

dufferzafar commented Aug 11, 2016

Once we get HAR dumping support (#1464) I think we should make it an integral part of mitm rather than keeping it as a external example:

  • Allow user to export flows as HAR selectively. Just like we do for saving flows (w key.)
  • Allow reading of HAR files. This will complete a bridge to/from other tools like browsers and other proxies.

@dufferzafar dufferzafar changed the title HAR as a first class citizen First class support for HAR Aug 11, 2016

@mhils

This comment has been minimized.

Copy link
Member

mhils commented Aug 11, 2016

Being able to read from multiple places would be awesome, that'd also be highly beneficial for pcap2mitm. The way to implement this is probably similar to what we do for contentviews and addons.

@mhils mhils added the kind/feature label Aug 11, 2016

@dufferzafar

This comment has been minimized.

Copy link
Contributor

dufferzafar commented Aug 17, 2016

@mhils: Can you please explain the "contentviews and addons" part a bit more, I’m not exactly sure how this relates to them?

@mhils mhils added the area/core label Aug 24, 2016

@dufferzafar

This comment has been minimized.

Copy link
Contributor

dufferzafar commented Sep 29, 2016

Paraphrasing some discussion I had on this on Slack with @mhils:

The best thing would be to have HAR support in FlowReader:

FlowReader/FlowWriter should be ABCs that define the reading/writing API and then MitmFlowReader/HarFlowReader as implementations.

Then we could have a make_reader(filename) or something along those lines that returns the proper thing depending on the file extension.

For now we just want to focus on HAR & not PCAP (though that is something nice too) but pcap2mitmproxy is GPLed so we can't include it right now.

@cortesi

This comment has been minimized.

Copy link
Member

cortesi commented Sep 29, 2016

I think we should keep our core FlowReader and FlowWriter concepts separate from exporters and importers.

Exporters are the more complicated case, and there are many semantic ways in which they don't fit with our generalised FlowReader/Writer scheme. HAR, for example, has to be terminated after the last flow has been written to make it valid JSON. I don't think we should build the termination concept into our core, because there streaming flows one by one independently is the key attribute. I can also imagine a range of exporters that do things like, for instance, calculate summaries over multiple flows (in fact, I'm working on beasts like this right now) that only write data after all flows are consumed. A further issue is that many exporters will only support exporting a single flow - i.e. curl command export. This again doesn't quite fit with flowreader/writer.

What I'd really like is a way to register exporters and importers a la contentviews, as Max described. Addons should easily be able to add custom exporters/importers, and, in fact, it might be a useful discipline to choose to implement the built-in exporters using the addons mechanism. For exporters, we should have an API that allows the exporter to indicate if it supports multiple-flow export, or only single-flow export.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment