New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add transparent mode for own outgoing traffic #1261

Open
tlbdk opened this Issue Jun 15, 2016 · 19 comments

Comments

Projects
None yet
6 participants
@tlbdk

tlbdk commented Jun 15, 2016

I did a bit of research on how to get PF to also handle own outgoing traffic, it works a bit like this:

Browser -> (outgoing interface) -> (PF filter rule) -> (lo interface) -> (PF translation rule) -> mitmproxy

There are some limits, the "Browser" needs to run as a different user than mitmproxy as the uid is used as a filter rule to make sure the mitmproxy traffic is also not sent back to it self.

I have a more in-depth explanation of how to do this:
http://tlbdk.github.io/mac/proxy/mitmproxy/fiddler/2016/04/14/redirect-outgoing-traffic-for-user-on-mac.html

All this could be turned into a switch for mitmproxy so it was a bit less work to setup.

@mhils

This comment has been minimized.

Member

mhils commented Jun 16, 2016

Thanks for the fantastic write-up! Easily intercepting traffic for our own machine is certainly on the agenda for mitmweb.

refs #630

@mhils mhils added the enhancement label Jun 23, 2016

@mhils mhils added the os/osx label Aug 24, 2016

@tomlabaude

This comment has been minimized.

tomlabaude commented Nov 14, 2016

Thanks tlbdk for your detailed post, I was stuck testing transparent mode and it helped me understand my errors.

(btw link is not good anymore, found your post here: https://github.com/tlbdk/tlbdk.github.io/blob/master/_posts/2016-04-19-redirect-outgoing-traffic-for-user-on-mac.md)

Now I question myself about seeing the whole traffic of macOS, for all users & system traffic, not only the user running the browser.
I tried a negation of the proxy user like: keep state {not or !} user { <user id running the proxy> } without success.

A solution could be to write a line per other user, but with the possibility to miss one, or to automate by first listing candidate system users and then creating the list:

keep state user root
keep state user macports
keep state user _hidd
keep state user _ctkd
keep state user _softwareupdate
keep state user _mdnsresponder
...

Also not sure if it's technically feasible, but this hard work with pf is done in 1 line with iptables, would compiling iptables work on macOS?

@tlbdk

This comment has been minimized.

tlbdk commented Nov 14, 2016

If PF supports user negation and you want to transparently proxy all traffic on the machine, you would only need to run mitmproxy under it's own user/uid.

iptables is Linux only and uses the Linux kernel firewall to do it's magic, PF is the macOS equivalent and does not seem to support filtering based on pid's, so I think the easiest options is to change random uid and add rules for that.

@srenatus

This comment has been minimized.

srenatus commented Mar 3, 2017

Just stumbled upon this -- I haven't tried it, but I think keep state and user <user> are separate things, and that you could probably do user { != nobody }. I'm not sure how it might interact with "unknown" user traffic. Details here.

 user <user>
      ...
      User and group refer to the effective (as opposed to the real) IDs, in case the socket is created
      by a setuid/setgid process.  User and group IDs are stored when a socket is created; when a
      process creates a listening socket as root (for instance, by binding to a privileged port) and
      subsequently changes to another user ID (to drop privileges), the credentials will remain root.

      User and group IDs can be specified as either numbers or names.  The syntax is similar to the one
      for ports.  The value unknown matches packets of forwarded connections.  unknown can only be used
      with the operators = and !=.  Other constructs like user >= unknown are invalid.  Forwarded pack-ets packets
      ets with unknown user and group ID match only rules that explicitly compare against unknown with
      the operators = or !=.  For instance user >= 0 does not match forwarded packets. 
      ...
@tomlabaude

This comment has been minimized.

tomlabaude commented Mar 3, 2017

Thanks, that's interesting indeed, will make some tests.
(And now I feel ashamed of my tests to negate the user, before the user keyword...)

@tomlabaude

This comment has been minimized.

tomlabaude commented Apr 20, 2017

user != nobody works perfectly to see the traffic of all users of your Mac

@mhils

This comment has been minimized.

@mhils

This comment has been minimized.

Member

mhils commented Feb 24, 2018

We can also already do this on Windows, where one can execute the platform/windows.py script manually with the right options.

@mhils mhils changed the title from Add transparent mode for MacOS X with own outgoing traffic to Add transparent mode for own outgoing traffic Feb 24, 2018

@mhils

This comment has been minimized.

Member

mhils commented Feb 24, 2018

I hear this quite often and we maybe should prioritize this - having something like mitmproxy --mode os would be really interesting.

@aniketpanjwani

This comment has been minimized.

Contributor

aniketpanjwani commented Feb 24, 2018

Hey Max - I'm not sure we need to remove the OSX tags, since this is really only tricky on OSX. In Linux, it's pretty easy to configure transparent mode for your own outgoing traffic - you just have to replace PREROUTING in step 4 here with OUTPUT. I implement this in Chomper for Linux. If you think it's worthwhile, I can make a PR to have a note in the docs to use the OUTPUT chain if you're using transparent mode for your own outgoing traffic on Linux.

I'm working on OSX compatibility for Chomper, which really just means figuring out how to solve #1261 for OSX. We've been discussing this a bit in the developer chat, and the most promising stackoverflow link I've found is this one. I can't try it out, since I don't have access to a Mac yet, but @miheerdew is trying to implement the instructions in the stackoverflow link.

@mhils

This comment has been minimized.

Member

mhils commented Feb 24, 2018

I think the main part here is putting that into a mitmproxy switch, and that's not really OS specific. There are roughly three steps that need to be done:

  1. Define an API like platform.start_os_proxy/platform.stop_os_proxy and implement it for at least one OS. Maybe that even includes certificate installation? Make sure that it tears down properly, even if mitmproxy exited ungracefully before.
  2. Add a mode to mitmproxy so that we can say mitmproxy --mode os/self/os-transparent/.... Not entirely straightforward as we AFAIK depend on the mode at some places in the proxy logic.
  3. Implement the API defined in 1 for all other OSes.
@aniketpanjwani

This comment has been minimized.

Contributor

aniketpanjwani commented Feb 24, 2018

I see. It should be pretty easy to implement in Linux. Figuring out how to automate certificate installation is something I'm working on. The mitmproxy docs suggest doing this for Linux, but that won't work for firefox/chrome, both of whom do not use system-wide certificates. Automating iptables configuration would require running mitmproxy as root - would that requirement be okay? If so, I should be able to work on this for Linux within the next week or two.

However, it seems like from the above discussion that no one ever figured out how to get it working with pf, so if @miheerdew figures something out, I'll write about it here.

@mhils

This comment has been minimized.

Member

mhils commented Feb 24, 2018

Any pointers are definitely welcome. For a first version we shouldn't install certificates. That can come later, this is already very tricky to do right cross-platform.

@tomlabaude

This comment has been minimized.

tomlabaude commented Feb 25, 2018

If it's of any help, for the macOS transparent mode, I've digged a lot inside pf to implement it in Debookee and came to the same solution as @tlbdk

  • Running mitmproxy with user nobody as pf can't redirect only part of a traffic of a macOS user (so it can't make the distinction between mitmproxy outgoing traffic, and users' or system traffic running under the same OS user)
  • Few lines of redirect in /etc/pf.conf

It's in production and I'm fully satisfied: mitmproxy intercepts all my machine traffic (root, system, user), except the traffic of user nobody...

@miheerdew

This comment has been minimized.

Contributor

miheerdew commented Mar 4, 2018

Hi @tomlabaude, have you tested this on MacOS High Sierra? Strangely, I can still access the internet if I block/redirect en0 (or en1). I was looking at tcpdump, and all the traffic seems to passing automatically through lo0 to the internet.

@tomlabaude

This comment has been minimized.

tomlabaude commented Mar 4, 2018

Not sure about blocking, but to redirect own traffic to mitmproxy, you need pfctl nat (redirection of lo0 to mitmproxy port) and filter (redirection of en0 traffic to lo0 except for mitmproxy user) rules to make it work. Look at those screenshots, or better, run Debookee in free trial and check pftctl status & pf.conf when SSL module is running.

Here's a screenshot of both nat and filter rules, mitmdump runs on port 6970 with user nobody, 4294967294 is user nobody
screenshot

screenshot

@miheerdew

This comment has been minimized.

Contributor

miheerdew commented Mar 6, 2018

Thanks. Debookee seems to be working fine, but when I manually load the same (nat and filter) rules into the (now empty) "debookee" anchor, my traffic isn't redirected anymore. I am definitely missing something here. Do you know what it might be?

@tomlabaude

This comment has been minimized.

tomlabaude commented Mar 7, 2018

Not sure this thread is the best place to follow such troubleshooting, as the thread is not OS related anymore, maybe @mhils can tell.
Also I could spend some time to troubleshoot & help you make it work if there's a direct enhancement/code of mitmproxy, not another project.

@miheerdew

This comment has been minimized.

Contributor

miheerdew commented Mar 7, 2018

I tried the instructions in OP's link but those don't redirect my traffic to mitmproxy, so I am trying to reproduce @tomlabaude's results on my computer.

It would actually be great if the docs are updated with the latest information on how to redirect machines own traffic to mitmproxy transparently for macOS. Once those instructions work, they can be implemented into --mode os-transparent argument.

@mhils Should I start a new issue for transparent proxy setup on macOS?

@tomlabaude: created a forum post

miheerdew added a commit to miheerdew/mitmproxy that referenced this issue Mar 11, 2018

Workaround for MacOS transparent proxy
This workaround (adopted from mitmproxy#1261) allows MacOS users to redirect their machine's outgoing traffic to mitmproxy transparently.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment