Inbuilt Map Remote Functionality #1454

Open
dufferzafar opened this Issue Jul 31, 2016 · 12 comments

Comments

Projects
None yet
6 participants
@dufferzafar
Contributor

dufferzafar commented Jul 31, 2016

Charles Proxy has a Map Remote feature which maps one URL to another.

This can be achieved in mitm by using a script similar to:

def request(flow):
    if "google.com" in flow.url:
        flow.request.url = flow.request.url.replace("google.com", "google.jp")

I think we should have this feature inbuilt (just like content replacements) rather than expecting users to write custom scripts.

@mhils mhils added the ready label Aug 1, 2016

@dufferzafar

This comment has been minimized.

Show comment
Hide comment
@dufferzafar

dufferzafar Aug 10, 2016

Contributor

I just saw that Charles has a neat UI for Map remotes & Map locals. I think mitm should have a UI for it too.

Contributor

dufferzafar commented Aug 10, 2016

I just saw that Charles has a neat UI for Map remotes & Map locals. I think mitm should have a UI for it too.

@dufferzafar

This comment has been minimized.

Show comment
Hide comment
@dufferzafar

dufferzafar Aug 10, 2016

Contributor

An Issue related to Map local: #1471.

User experience related to map local: https://discourse.mitmproxy.org/t/emulate-charles-proxys-map-local/116

Contributor

dufferzafar commented Aug 10, 2016

An Issue related to Map local: #1471.

User experience related to map local: https://discourse.mitmproxy.org/t/emulate-charles-proxys-map-local/116

@dufferzafar dufferzafar removed the ready label Aug 17, 2016

@mhils mhils added area/core area/addons and removed area/core labels Aug 24, 2016

@daveyarwood

This comment has been minimized.

Show comment
Hide comment
@daveyarwood

daveyarwood Oct 5, 2016

I've been using Charles' Map Remote extensively for work, and would love to see this feature in mitmproxy.

Given some time to familiarize myself with the codebase, I might be able to submit a PR for this. Any direction you could give me would be super helpful!

I've been using Charles' Map Remote extensively for work, and would love to see this feature in mitmproxy.

Given some time to familiarize myself with the codebase, I might be able to submit a PR for this. Any direction you could give me would be super helpful!

@dufferzafar

This comment has been minimized.

Show comment
Hide comment
@dufferzafar

dufferzafar Oct 5, 2016

Contributor

@daveyarwood Try setting up the dev environment first. See if all tests pass to ensure you have stuff working. Then come discuss this feature at our Slack channel.

I haven't used Charles much, but I remember using it's Map Remote once, and really liked it. Having this in mitm would be great! So it's nice you're picking this up. 👍

Contributor

dufferzafar commented Oct 5, 2016

@daveyarwood Try setting up the dev environment first. See if all tests pass to ensure you have stuff working. Then come discuss this feature at our Slack channel.

I haven't used Charles much, but I remember using it's Map Remote once, and really liked it. Having this in mitm would be great! So it's nice you're picking this up. 👍

@daveyarwood

This comment has been minimized.

Show comment
Hide comment
@daveyarwood

daveyarwood Oct 5, 2016

Cool -- I'm a bit busy at the moment, but hope to give this a try soon!

Cool -- I'm a bit busy at the moment, but hope to give this a try soon!

@daveyarwood

This comment has been minimized.

Show comment
Hide comment
@daveyarwood

daveyarwood Oct 6, 2016

After playing around with this a bit, I was able to create a simple script that replicates an example of one thing I've been using Map Remote for with Charles:

def request(flow):
    if flow.request.pretty_host == "mynetwork.adzerk.com":
        flow.request.host = "127.0.0.1"
        flow.request.port = 8080

mynetwork.adzerk.com is my login to the (hypothetical) "mynetwork" account in the Adzerk UI. When running the Adzerk UI locally on port 8080, I can go to any URL in my browser where the URL is yarwood.adzerk.com, and mitmproxy successfully redirects all requests from mynetwork.adzerk.com/any/route/can/go/here.html to 127.0.0.1:8080/the/same/route.html. In testing the UI, this appears to have the exact same behavior as Map Remote in Charles! Minor success! 😄

Before I start hacking on this further, I could use some input about what this experience should be like for mitmproxy.

We could do a simple reproduction of the Charles Map Remote feature, which would be something like:

  • For each entry, for both "Map From" and "Map To," you can specify the following things:
    • protocol
    • host
    • port
    • path
    • query (? I'm not sure what this actually does in Charles... I'm guessing something about the query string)
  • Any request matching all of the "Map From" criteria gets each field replaced with the strings in the entry's "Map To" settings.

A simpler, possibly cleaner alternative is to do this in a more general way, using regex to search and replace over the entire URL. The replacement process would also parse the replaced URL and put each of the URL components where it needs to go. So for example, I might map ^(https?)://mynetwork.adzerk.com to \\1://127.0.0.1:8080, and that would replace any URL starting with http:// or https:// followed by mynetwork.adzerk.com with the same protocol followed by 127.0.01:8080 and the rest of the URL. The replaced string would be parsed as a URL and the correct replacement protocol, host, and port would be set in the request.

I think the first way (let's call it "the Charles Way") makes sense for Charles because it's a GUI application, and it's nice to work with fields in this way in a GUI. But for a more low-level CLI tool like mitmproxy, I think having a way to do regex replacement in the URL might be the more useful thing, and for convenience, we could always build more specific Map Remote-like behavior on top of that.

By the way, it seems like --replace is almost what I'm describing, but in testing it, it seems to do the replacement in the entire request and/or response, not just the request URL components.

Thoughts on all this?

daveyarwood commented Oct 6, 2016

After playing around with this a bit, I was able to create a simple script that replicates an example of one thing I've been using Map Remote for with Charles:

def request(flow):
    if flow.request.pretty_host == "mynetwork.adzerk.com":
        flow.request.host = "127.0.0.1"
        flow.request.port = 8080

mynetwork.adzerk.com is my login to the (hypothetical) "mynetwork" account in the Adzerk UI. When running the Adzerk UI locally on port 8080, I can go to any URL in my browser where the URL is yarwood.adzerk.com, and mitmproxy successfully redirects all requests from mynetwork.adzerk.com/any/route/can/go/here.html to 127.0.0.1:8080/the/same/route.html. In testing the UI, this appears to have the exact same behavior as Map Remote in Charles! Minor success! 😄

Before I start hacking on this further, I could use some input about what this experience should be like for mitmproxy.

We could do a simple reproduction of the Charles Map Remote feature, which would be something like:

  • For each entry, for both "Map From" and "Map To," you can specify the following things:
    • protocol
    • host
    • port
    • path
    • query (? I'm not sure what this actually does in Charles... I'm guessing something about the query string)
  • Any request matching all of the "Map From" criteria gets each field replaced with the strings in the entry's "Map To" settings.

A simpler, possibly cleaner alternative is to do this in a more general way, using regex to search and replace over the entire URL. The replacement process would also parse the replaced URL and put each of the URL components where it needs to go. So for example, I might map ^(https?)://mynetwork.adzerk.com to \\1://127.0.0.1:8080, and that would replace any URL starting with http:// or https:// followed by mynetwork.adzerk.com with the same protocol followed by 127.0.01:8080 and the rest of the URL. The replaced string would be parsed as a URL and the correct replacement protocol, host, and port would be set in the request.

I think the first way (let's call it "the Charles Way") makes sense for Charles because it's a GUI application, and it's nice to work with fields in this way in a GUI. But for a more low-level CLI tool like mitmproxy, I think having a way to do regex replacement in the URL might be the more useful thing, and for convenience, we could always build more specific Map Remote-like behavior on top of that.

By the way, it seems like --replace is almost what I'm describing, but in testing it, it seems to do the replacement in the entire request and/or response, not just the request URL components.

Thoughts on all this?

@mhils

This comment has been minimized.

Show comment
Hide comment
@mhils

mhils Oct 14, 2016

Member

@daveyarwood: Thanks for writing that up (and sorry for the slow-ish response)! 😃 map remote (and map local) functionality would a be fantastic addition to mitmproxy. I'm not sure what a good UI would be either, but here's a quick braindump from my end:

There's a certain feature overlap with --replace and this is somewhat related to #1469. We could change the replacement editor to have three boolean flags (url, header, body) to indicate where replacements should be done. The replacement editor then would look like this:

Apply to flows that match Regex Replacement URL Header Body
~q example.com example.org X
~s foo bar X

(One could also use an enum instead of three boolean flags)

I like this approach for being super flexible, but the UI is getting too complicated. It's also a bit of an open question if/how we can integrate map to local and file replacements with this. We could implement a special case that when a URL starts with file:// after replacement, we treat this as map to local. However, that would conflict with the proposed integration of file replacements (#1458), so we'd have to come up with different prefixes, or ... ? It's certainly a powerful tool, but likely too overwhelming.

Another approach would be to simply have a dedicated Map Remote/Local editor:

URL to match Replacement
https://example.com http://example.org
http://example.xyz/ file://./data

(if these were regexes you could still enter a plain domain name and it'd work)

That is partially duplicating the replace feature (with the possible addition of map to local), but it would make a considerably simpler UI.

@mitmproxy/devs, this feature request came up a couple of times now and I think it'd be super useful. Any opinions?

Member

mhils commented Oct 14, 2016

@daveyarwood: Thanks for writing that up (and sorry for the slow-ish response)! 😃 map remote (and map local) functionality would a be fantastic addition to mitmproxy. I'm not sure what a good UI would be either, but here's a quick braindump from my end:

There's a certain feature overlap with --replace and this is somewhat related to #1469. We could change the replacement editor to have three boolean flags (url, header, body) to indicate where replacements should be done. The replacement editor then would look like this:

Apply to flows that match Regex Replacement URL Header Body
~q example.com example.org X
~s foo bar X

(One could also use an enum instead of three boolean flags)

I like this approach for being super flexible, but the UI is getting too complicated. It's also a bit of an open question if/how we can integrate map to local and file replacements with this. We could implement a special case that when a URL starts with file:// after replacement, we treat this as map to local. However, that would conflict with the proposed integration of file replacements (#1458), so we'd have to come up with different prefixes, or ... ? It's certainly a powerful tool, but likely too overwhelming.

Another approach would be to simply have a dedicated Map Remote/Local editor:

URL to match Replacement
https://example.com http://example.org
http://example.xyz/ file://./data

(if these were regexes you could still enter a plain domain name and it'd work)

That is partially duplicating the replace feature (with the possible addition of map to local), but it would make a considerably simpler UI.

@mitmproxy/devs, this feature request came up a couple of times now and I think it'd be super useful. Any opinions?

@mhils mhils added the help wanted label Oct 14, 2016

@ujjwal96

This comment has been minimized.

Show comment
Hide comment
@ujjwal96

ujjwal96 Mar 19, 2017

Member

@mhils The file replacement feature has been added now and URLs can be replaced using the current replacements option. So is there something more which needs to be done for this?
Do we need a dedicated Map Remote/Local editor?

Member

ujjwal96 commented Mar 19, 2017

@mhils The file replacement feature has been added now and URLs can be replaced using the current replacements option. So is there something more which needs to be done for this?
Do we need a dedicated Map Remote/Local editor?

@mhils

This comment has been minimized.

Show comment
Hide comment
@mhils

mhils Mar 20, 2017

Member

@ujjwal96: I think there are still lots of improvements to be made regarding the UX. Consider the following user story:

I want to show the contents of ~/foo.css when I access example.com/style.css.

I think this is something that's not viable with replacements right now. I think it would be very useful to have a dedicated "map remote" functionality that can be used to map URLs to other URLs or local files. It would be interesting to see how other tools are solving this (@dufferzafar mentions Charles Proxy), I haven't looked into that yet. To make the distinction a bit more clear, we may want to have the replacements addon operate on the request path only, not the full URL, and handle the URL in the map remote editor.

Member

mhils commented Mar 20, 2017

@ujjwal96: I think there are still lots of improvements to be made regarding the UX. Consider the following user story:

I want to show the contents of ~/foo.css when I access example.com/style.css.

I think this is something that's not viable with replacements right now. I think it would be very useful to have a dedicated "map remote" functionality that can be used to map URLs to other URLs or local files. It would be interesting to see how other tools are solving this (@dufferzafar mentions Charles Proxy), I haven't looked into that yet. To make the distinction a bit more clear, we may want to have the replacements addon operate on the request path only, not the full URL, and handle the URL in the map remote editor.

@eu81273

This comment has been minimized.

Show comment
Hide comment
@eu81273

eu81273 Jul 10, 2018

@mhils
I think there is not enough documation about this feature.

I tried like below,

$ mitmproxy --replacements @~u@http://target.com/js/aaa.js@http://localhost:3000/js/aaa.js

but I got error response like this.

/usr/local/bin/mitmproxy: Invalid filter pattern: ~u

From the documentation, I couldn't find any related information.

eu81273 commented Jul 10, 2018

@mhils
I think there is not enough documation about this feature.

I tried like below,

$ mitmproxy --replacements @~u@http://target.com/js/aaa.js@http://localhost:3000/js/aaa.js

but I got error response like this.

/usr/local/bin/mitmproxy: Invalid filter pattern: ~u

From the documentation, I couldn't find any related information.

@kajojify

This comment has been minimized.

Show comment
Hide comment
@kajojify

kajojify Jul 10, 2018

Contributor

@eu81273 Hello! Yes, you are right, unfortunately, our documentation doesn't describe some moments very clearly. I hope we will change it in the nearest future. But nevertheless there is something in the documentation you may be interested in:
Filter expressions -> https://docs.mitmproxy.org/stable/concepts-filters/
Replacements -> https://docs.mitmproxy.org/stable/overview-features/#replacements

Shortly speaking, here @~u@http://target.com/js/aaa.js@http://localhost:3000/js/aaa.js,
~u - mitmproxy filter expression that defines which flows a replacement applies to;
http://target.com/js/aaa.js - a valid Python regular expression that defines what gets replaced;
http://localhost:3000/js/aaa.js - a string literal that is substituted in.

The thing is ~u isn't correct. To get the flows you want relying on url, you need to use ~u regex or just regex (which is equivalent to ~u regex). For example, you can write:
@"~u google\.com"@google@yahoo
@google\.com@google@yahoo
This replacement takes all flows, which have google.com into their urls and replace google with yahoo. You can also write @google@yahoo. It means, that the pattern set to the default .*.

Contributor

kajojify commented Jul 10, 2018

@eu81273 Hello! Yes, you are right, unfortunately, our documentation doesn't describe some moments very clearly. I hope we will change it in the nearest future. But nevertheless there is something in the documentation you may be interested in:
Filter expressions -> https://docs.mitmproxy.org/stable/concepts-filters/
Replacements -> https://docs.mitmproxy.org/stable/overview-features/#replacements

Shortly speaking, here @~u@http://target.com/js/aaa.js@http://localhost:3000/js/aaa.js,
~u - mitmproxy filter expression that defines which flows a replacement applies to;
http://target.com/js/aaa.js - a valid Python regular expression that defines what gets replaced;
http://localhost:3000/js/aaa.js - a string literal that is substituted in.

The thing is ~u isn't correct. To get the flows you want relying on url, you need to use ~u regex or just regex (which is equivalent to ~u regex). For example, you can write:
@"~u google\.com"@google@yahoo
@google\.com@google@yahoo
This replacement takes all flows, which have google.com into their urls and replace google with yahoo. You can also write @google@yahoo. It means, that the pattern set to the default .*.

@eu81273

This comment has been minimized.

Show comment
Hide comment
@eu81273

eu81273 Jul 11, 2018

@kajojify
Thank you!!! You save my life..
Can I ask you one more question?
When I tried like this, port number(3000) occurs 502(HTTP Status code) error.

$ mitmproxy --replacements @"~u .*targetdomain\.com\/js.*"@targetdomain.com@localhost:3000

Is there any way to replace url with port number with command line?

eu81273 commented Jul 11, 2018

@kajojify
Thank you!!! You save my life..
Can I ask you one more question?
When I tried like this, port number(3000) occurs 502(HTTP Status code) error.

$ mitmproxy --replacements @"~u .*targetdomain\.com\/js.*"@targetdomain.com@localhost:3000

Is there any way to replace url with port number with command line?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment