mitmproxy's libmproxy as a replacement for our proxy

[andresriancho]

Analyze the possibility of using mitmproxy's libmproxy as a replacement for our proxy. Ours is not well tested is repeatedly been subject to awful bugs.

Things to do:

• Check how good the code of mitmproxy is
• Does the code have unittests?
• Test it, try to browse different sites using it.
• Does it support websockets? Ours doesn't, so support for that would be awesome.

https://github.com/mitmproxy/mitmproxy
http://mitmproxy.org/doc/scripting/libmproxy.html

[andresriancho]

Their test suite is very impressing, covering 97% of the code.

[andresriancho]

Sent email to author to see if we can use the libmproxy, and if he can make mitmproxy and libmproxy two independent projects.

[andresriancho]

@mitmproxy answered, the key points were:

• We can split mitmproxy in two projects, one for the library (which w3af will use) and another for the rest
• We can use it, license is compatible and he feels good about it
• He recommends, and I agree, that the best thing is to add libmproxy as an external dependency of w3af; not as a fork that I maintain.

[andresriancho]

While working on this, create a new SSL certificate for w3af.org (does that actually make sense?) instead of sf.net:

The certificate is not trusted because it is self-signed. 
The certificate is only valid for w3af.sourceforge.net

Also, maybe we should have the CA cert available so users can add it to their browsers? How is ZAP handling this?

[andresriancho]

In the future I'll want to have JS support, which will require a browser that uses a local proxy to reach the target. In that case I'll need a very fast and stable proxy!

Also, I'll need a way to configure SSL CA certificate in the automated browser to be able to reach SSL sites.

[andresriancho]

From the mailing list:

Hi! Is there a tutorial somewhere I can follow on how to setup the SSL Certificate Authority (CA) for the spiderman plugin? When I attempt to manually browse my site via the spiderman proxy, I'm presented with the "This connection is untrusted" dialog in Firefox and I'm not permitted to the SSL pages. For Metasploit, I used a certificate that it provided for me and that worked beautifully for their framework. Just curious if there's a certificate I can install for w3af located somewhere that I can install for spiderman or if I can get instructions on how to approach this problem with w3af.

Actions to be performed:

• Distribute the CA file with w3af
• Write a RST document (that will be included in the docs) on how to use the spiderman proxy with the latest libmitmproxy

[m0sth8]

What do you think about libmproxy performance? Libmproxy concurrency is based on threading as well as w3af proxy implementation.

[andresriancho]

The performance shouldnt be an issue. Asked the main developer and he told me they used the proxy to route thousands of requests per second, something like a LAN proxy for a medium sized office.

[andresriancho]

Take a look at the code in daemons: localproxy and proxy. With the new implementation those would be gone, and all the uses replaced.

[m0sth8]

I've add PoC in commit: 7895ce7. It depends on the libmproxy master.

So, there is a couple of positive and negative things about libmproxy:

• All communications with HTTP handler pass through messages, it's good for future distributed proxies.
• Libmproxy has very good primitives for HTTPRequest, HTTPResponse, which support serialization and states.

• They are in the middle of libmproxy rewriting. Current master and the last version have very different API. Also, there is no backward compatibility in API. We should talk with them to ask about their plans.
• Sometimes it's difficult to change their implementations without monkey patching.
• Current Master implementation depends on global variable for shutdown proxy server, but it's easy to fix.

The next step might be

• Create handler interface with stable api depends only on w3af (BaseHTTPRequestHandler or something easier and more convenient)
• Create libmproxy implementation

[andresriancho]

The code looks excellent 👍 , some comments:

• Asked the circleci guys to add libffi-dev, they did. You should be able to run the test suite now. Please note that last week I've worked a lot in improving the test suite and many tests which were failing are now passing at develop branch. You might want to merge develop into your feature branch at some point, that will give you more stable builds.
• If 7895ce7 provides all the same features we provided at https://github.com/andresriancho/w3af/blob/7895ce7a671e972908b2a4f9c0c8326c6004cb05/w3af/core/controllers/daemons/proxy.py , then I'll be very happy and impressed with libmproxy :)
• Sent a couple of comments to the code, nothing important.
• If possible lets send the pull-request to libmproxy so that we/they don't depend on that global variable.

[andresriancho]

Work to be completed at: Complete libmproxy implementation #10429