Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Added mitmXSS addon #1907
mitmXSS is a new addon for mitmproxy that automatically scans every visited page for XSS vulnerabilities. It outputs any found XSS vulnerabilities to the mitmproxy log (viewable by pressing
I just added another commit so that it automatically includes cookies in all of its requests. This is done using the flow object and pulling the information out of
Apologies for taking so long to get back to this issue - I've been nightmarishly busy.
If we resolve the structural issues and convince the other maintainers to re-add lxml, I'm in favor of pulling this in, on the understanding that we'll refactor once sessions are in tree.
Thanks for all the feedback!
On the topic of lxml, I totally get the hesitation towards reintroducing it as a dependency. If you are already building a new feature for addons, would it potentially make sense to build dependency management into that as a feature? That way one doesn't need lxml in order to run mitmproxy but there can still be a nice UI for running this addon.
On the topic of the license, yup that makes sense— just changed that.
As far as moving tests to the top level tests directory, I'm happy to do that but I'm not sure that it makes sense to do so. If the ultimate goal is to have a lot of addons for mitmproxy, I'm not sure it makes sense to include tests for each addon in the tests for the project as a whole. But if you are sure that it is the right decision for the time being, let me know and I'll move them over.
So, lxml is a difficult topic. It caused a lot of headaches for me by being the only dependency which was not working on Windows with Python 3 while we were already on track to being Python 3 only (see #1444). I initially tried to solve this the nice way and spent a significant amount of time and effort building Windows wheels for lxml (mhils/libxml2-win-binaries and a couple of PRs to lxml), which really wasn't fun. Unfortunately, after doing all of this, there were no visible signs that a lxml release with these changes would eventually appear within a reasonable timeframe. We had to come up with alternatives (#1657) and I finally replaced the lxml contentview implementation in #1831. For the record, I do think the lxml maintainer is well-intentioned (and he clearly has no obligations to us), but he seems to have very little time at hand and even less interest in supporting downstream. He did manage to push out a release with Windows wheels by now, but I still think that depending on lxml is risky and likely to put us in a similar situation again. Sorry for being negative on this, but I've been burnt.
Let's get a bit more positive and productive:
Right now the major selling point of addons is that they allow us to structure and separate our internal logic. External addons that come with their own dependencies are the next logical step, but if I read @cortesi's comment correctly he thinks that this feature is important enough to be right in the core so we can skip that problem for now :). If we can do that without lxml, I would tend to agree.
Can we use https://docs.python.org/3.6/library/xml.etree.elementtree.html instead of lxml? IIRC we discarded that option for the contentviews as it auto-fixes HTML when prettyprinting and failed for some malformed documents, but those limitations shouldn't be dealbreakers for this feature. This would also allow us to mitigate some DoS attack vectors if we use defusedxml, an option we don't have with lxml.
That sounds reasonable, I should have some free time this weekend so I'll try switching it over to see if that works. Depending on the extent to which it auto-fixes the HTML, it may or may not end up being a problem (I don't think it will be problematic though).
We need to think about this carefully, and there's no rush. A few quick comments:
I just finished switching it over to html.parser and it seems to work fine with that. Even though it isn't as robust as lxml or beautifulsoup, it should be fine for this given that 100% reliability for it is not essential. I think we can leave it with this and if at any point lxml is pulled in as a dependency, then we can switch back to that.
I'll move the tests over to /tests later once I get the chance to figure out how to get imports working for /tests.
Could we please also remove the images from the patch. They're very nice, but they're very soon going to be out of date, and will be a pain to maintain. I propose that we add this sort of thing to the documentation at the point where this addon becomes a built-in (which will be sooner rather than later).
We just discussed this on Slack and decided to move this into examples/ for now. This needs a few changes beforehand, see comments here. For the structure:
- Move mitmproxy/addons/xss.py to examples/complex/, feel free to keep the name or rename it to xss_checker or whatever you think is best
- Move test/mitmproxy/addons/test_xss.py to test/mitmproxy/examples/xss.py (examples folder needs to be created)
Thanks again, @ddworken!
This is getting into a good shape!
A few minor things are left - but overall a big improvement! Thanks for taking the time and addressing our comments!
Feb 27, 2017
added a commit
this pull request
Mar 3, 2017
Not much reason behind those values, just needed a unique string that can be used to pick out the place where our input is reflected into the page. If you look at those strings while looking at a standard US qwerty keyboard there is some amount of a pattern but that is just a factor of how I mashed the keyboard.