Virtual Hosts Plugin

[DUOLabs333]

Is your feature request related to a problem? Please describe.
I want to run a bunch of local services on different ports, and have a proxy that can connect to each of them

Describe the solution you'd like
For example, the proxy is on port 80, and I have my personal DNS that resolves *.test to 127.0.0.1. When I go to example.test, the proxy will then go to 127.0.0.1:5000 for example.

[abhinavsingh]

I think RedirectToCustomServerPlugin and CustomDnsResolverPlugin can both do this for you.

• https://github.com/abhinavsingh/proxy.py#redirecttocustomserverplugin
• https://github.com/abhinavsingh/proxy.py#customdnsresolverplugin

Let me know how it goes.

[DUOLabs333]

So, to modify proxy.plugin.RedirectToCustomServerPlugin, I'll need to clone the source code, right?

[DUOLabs333]

custom_dns_resolver.py looks interesting, but it only maps domain to ip address, not port too. Also, it may interfere with my actual DNS.

[abhinavsingh]

So, to modify proxy.plugin.RedirectToCustomServerPlugin, I'll need to clone the source code, right?

No you don't need to clone the source code.

1. Simply copy/paste RedirectToCustomServerPlugin into your own directory
2. Start proxy with your new plugin e.g. proxy --plugin /path/to/my/directory/my_file.RedirectToCustomServerPlugin and it should work fine.

[abhinavsingh]

custom_dns_resolver.py looks interesting, but it only maps domain to ip address, not port too. Also, it may interfere with my actual DNS.

Yep feel free to modify it per your needs. RedirectToCustomServerPlugin is just a placeholder example on-top of which folks can build their own solutions. Lemme know.

[DUOLabs333]

Ok, in the example, it has import ..http.proxy, but obviously that won't work. So, how do I fix that? Also, what about HTTPS? Can I just do the steps under TLS interception and remove the HTTPS check?

[DUOLabs333]

This fails: PYTHONPATH=$PWD /home/system/.local/bin/proxy --plugins test.RedirectToCustomServerPlugin

[abhinavsingh]

This fails: PYTHONPATH=$PWD /home/system/.local/bin/proxy --plugins test.RedirectToCustomServerPlugin

Not sure what's in there. Below is a working example I just tried on local machine:

1. Contents of /tmp/plug folder

╰─ ls -1 /tmp/plug                                                                                                                       ─╯
__pycache__
my_plugin.py

2. My custom plugin class

╰─ cat /tmp/plug/my_plugin.py                                                                                                            ─╯
from proxy.http.proxy import HttpProxyBasePlugin


class MyPlugin(HttpProxyBasePlugin):
  pass

3. Load it

╰─ PYTHONPATH=/tmp/plug python -m proxy --plugin my_plugin.MyPlugin                                                                      ─╯
2021-11-17 18:45:59,656 - pid:26374 [I] plugins.load:82 - Loaded plugin proxy.http.proxy.HttpProxyPlugin
2021-11-17 18:45:59,657 - pid:26374 [I] plugins.load:82 - Loaded plugin my_plugin.MyPlugin
2021-11-17 18:45:59,657 - pid:26374 [I] listener.setup:84 - Listening on ::1:8899
2021-11-17 18:45:59,715 - pid:26374 [I] executors.setup:158 - Started 16 threadless workers
2021-11-17 18:45:59,830 - pid:26374 [I] pool.setup:91 - Started 16 acceptors

[abhinavsingh]

You can use both --plugin and --plugins, both are same.

[abhinavsingh]

Also, what about HTTPS? Can I just do the steps under TLS interception and remove the HTTPS check?

Can you expand? Remove HTTPS checks at which stage of TLS interception

[DUOLabs333]

In the example plugin, you skip the plugin if it has the https header. So, if I implement TLS interception, can I remove that. Also, got it to work, but there's an error: TypeError: Can't instantiate abstract class RedirectToCustomServerPlugin with abstract methods handle_client_request, handle_upstream_chunk, on_upstream_connection_close

[abhinavsingh]

Also, got it to work, but there's an error: TypeError: Can't instantiate abstract class RedirectToCustomServerPlugin with abstract methods handle_client_request, handle_upstream_chunk, on_upstream_connection_close

:) Yes you need to implement the plugin abstract methods. I just gave you an example on how to load a plugin.

Also looks like you are using stable release from v2.3.1 from pip. If you grab the latest from develop, all those methods are now optional and no longer "required". You can continue using v2.3.1, simply implement those methods. HttpProxyBasePlugin is an abstract class.

[abhinavsingh]

you skip the plugin if it has the https header. So, if I implement TLS interception, can I remove that.

Did you try TLS Interception or are you speculating. IIRC, if you enable TLS Interception, all plugins will continue to work as is. Within the context of proxy, requests are now HTTP, not HTTPS.

[DUOLabs333]

Oh, that's perfect. Ok, going to give this another go.

[DUOLabs333]

Yeah, so if I try to run it with the git version, it just hangs. I'm guessing the methods are still required, they just won't give an error?

[abhinavsingh]

Yeah, so if I try to run it with the git version, it just hangs.

I won't recommend using git version in production. But we'll make a release soon, so you will soon have the latest code on pip too.

I'm guessing the methods are still required, they just won't give an error?

develop branch is almost always kept stable. I use proxy.py all the time out of develop. I am posting this comment and browsing via proxy.py right now (all out of develop). In rare scenarios, develop may break due to missing tests or other reasons.

So nothing should break or hang. Locally I am able to run it just fine. Made an http and https request, still using the same empty plugin.

I am on develop branch which is the default branch.

╰─ PYTHONPATH=/tmp/plug python -m proxy --plugins my_plugin.MyPlugin                 ─╯
2021-11-17 19:05:42,695 - pid:28677 [I] plugins.load:82 - Loaded plugin proxy.http.proxy.HttpProxyPlugin
2021-11-17 19:05:42,696 - pid:28677 [I] plugins.load:82 - Loaded plugin my_plugin.MyPlugin
2021-11-17 19:05:42,696 - pid:28677 [I] listener.setup:84 - Listening on ::1:8899
2021-11-17 19:05:42,751 - pid:28677 [I] executors.setup:158 - Started 16 threadless workers
2021-11-17 19:05:42,881 - pid:28677 [I] pool.setup:91 - Started 16 acceptors

2021-11-17 19:05:55,212 - pid:28705 [I] server.access_log:430 - ::1:49195 - GET httpbin.org:80/get - 200 OK - 484 bytes - 645.35ms
2021-11-17 19:06:04,957 - pid:28699 [I] server.access_log:430 - ::1:49205 - CONNECT httpbin.org:443 - 5900 bytes - 1151.83ms

[DUOLabs333]

Hmmm, even the empty one isn't working.

[abhinavsingh]

Does proxy work fine for you without any external plugin?

[DUOLabs333]

Oh, forgot to enable --enable-web-server

[DUOLabs333]

It loads, but not to the address (the redirect to custom server plugin).

[abhinavsingh]

It loads, but not to the address

Your address might be unreachable. Always verify using curl first, whether your upstream actually works.

But overall looks like you have got what we were after.

Feel free to leave more comments for discussion, but I think good to close this for now.

[DUOLabs333]

Ok, it works with curl, but not in a browser.

[DUOLabs333]

Aka curl -v -x localhost:8899 http://google.com works, but not going to localhost:8899 in the browser.

[abhinavsingh]

Yes, because browsers are not dumb like curl, that's why :D. I am pretty sure, google has HSTS enabled which will prohibit browsers from even making an http request to google.

Anyways, does it go to https://google.com, or what is the error you see in browsers dev console?

[abhinavsingh]

Also note that default response from in-built web server is 404 Not Found, so browser will not show anything even for success case. So take a deep look into actual request/response.

$ proxy --plugin proxy.plugin.RedirectToCustomServerPlugin --enable-web-server

╰─ curl -v -x localhost:8899 http://httpbin.org/get                                                            ─╯
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8899 (#0)
> GET http://httpbin.org/get HTTP/1.1
> Host: httpbin.org
> User-Agent: curl/7.64.1
> Accept: */*
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 404 NOT FOUND
< Server: proxy.py vv2.3.2-dev147.g35b643c.d20211114
< Content-Length: 0
< Connection: close
< 
* Closing connection 0

[DUOLabs333]

Hmmm, seems like more work is needed to figure this out. Thank you though.

[abhinavsingh]

No worries, lemme know, feel free to open another ticket for questions if necessary.