Implementing proxy.py in existing code

[RedByte1337]

I am trying to create some sort of a python proxy that has the following functionality:

• Intercepting TLS to be able to read all encrypted data (This part I got working already with proxy.py)
• When a certain HTTP (GET or POST) request is intercepted by the client, this should be let through, but in the background, a new arbitrary HTTP request has to be quickly sent to the server. (To a different URL with a different body)
• A websocket connection should be monitored for certain responses from the server, and a new arbitrary HTTP request might need to be sent to the server depending on what was received in the websocket.
• Some specific data from HTTP responses or websocket data from the server will need to be logged into a file.

As you can see these are a lot of different parts. I'm not very experienced with python and only use it from time to time for simple actions, but I'm doing this little project just to learn from it (so any tips are welcome).

I've looked at some of the default plugins, as well as at some of the library examples, however, I'm having a hard time getting started.

I'm fairly sure most of the functionality can be implemented as a custom plugin, but I don't think that would be the best approach in my scenario? Doing everything in one or multiple plugins would make it harder for the different pieces to communicate with each other as far as I understand. For instance, the arbitrary HTTP requests that would need to be sent out in certain scenarios would be one of about 20 ~ 50 predefined API calls that I would need to implement in python as functions as well. (e.g. get_something_api_1(arg1, arg2), set_something_api_1(arg1, arg2, arg3, arg4), ...)

Therefore I thought I might need more of a library approach such as in these examples instead of using plugins.

I started this project a while back by implementing some sort of communicator class using python http.client and I already have multiple functions implemented in that class to send specific API calls. So if I want to use that functionality, I just need to create an instance of the communicator class and call the function with the specific arguments.

Now I realize that I might benefit from being able to send those API calls when specific requests from the browser are sent, requiring a proxy to monitor the traffic. This is how I came to this awesome project. But I'm having a hard time trying to integrate it into my existing code.

I also realize that this will require implementing some multithreading or multiprocessing in order to not block the browser while doing some actions. I'm fairly new to that as well.

[abhinavsingh]

@KeanuNys Welcome to proxy.py :). Here are some pointers that came to me as I read your comment:

I am not sure why will 2 request need to communicate, pass data or synchronize with each other for your use case. IIUC, you want to perform certain checks by calling pre-written API methods, before processing an incoming request.

FWIW, this can be achieved using a simple plugin. Within your plugin's handle_client_request implementation, you can choose to perform checks based upon incoming request attributes (headers/cookies). If checks passes, allow the request to go through. Otherwise drop the request with an error.

See handle_client_request doc and packaged plugin for example usage.

proxy.py/proxy/http/proxy/plugin.py

Lines 108 to 124 in 8922310

	def handle_client_request(
	self, request: HttpParser,
	) -> Optional[HttpParser]:
	"""Handler called before dispatching client request to upstream.
	Note: For pipelined (keep-alive) connections, this handler can be
	called multiple times, for each request sent to upstream.
	Note: If TLS interception is enabled, this handler can
	be called multiple times if client exchanges multiple
	requests over same SSL session.
	Return optionally modified request object to dispatch to upstream.
	Return None to drop the request data, e.g. in case a response has already been queued.
	Raise HttpRequestRejected or HttpProtocolException directly to
	tear down the connection with client.
	"""

I'll not necessarily recommend using core library directly, until we run into a use-case that cannot be solved by proxy.py plugins. Remember that, your plugins will work as-is, iff, tomorrow, we decide to switch to a core based codebase. Because plugins is a core functionality :)

Let me know if this helps and how it went.

Best

[RedByte1337]

Hi abhinavsingh,

Thanks for your answer.

You're almost correct, but I don't want to block the incoming request from the client based on the results of an API call. The goal is to allow the request to go through, but for some requests, an additional API call too the server will be made from python afterwards to get some data or to do some changes. Example:

Client ---a---> Proxy ---a---> Server
Client ---a---> Proxy ---a---> Server
Client ---a---> Proxy ---a---> Server
Client ---b---> Proxy ---b---> Server
                Proxy ---c---> Server
Client ---a---> Proxy ---a---> Server
Client ---a---> Proxy ---a---> Server
Client ---b---> Proxy ---b---> Server
                Proxy ---c---> Server
Client ---a---> Proxy ---a---> Server
Client ---a---> Proxy ---a---> Server
Client ---x---> Proxy ---x---> Server
                Proxy ---x---> Log_to_local_file
Client ---a---> Proxy ---a---> Server
Server --ws1--> Proxy --ws1--> Client
Client ---a---> Proxy ---a---> Server
Server --ws2--> Proxy --ws2--> Client
	        Proxy ---d---> Server
Client ---a---> Proxy ---a---> Server
Client ---a---> Proxy ---a---> Server ---y---> Proxy ---y---> Client
                Proxy ---z---> Server
Client ---a---> Proxy ---a---> Server

So for most requests, we will just take no action and let them pass through to the server. These are requests (a).
Although if request b is made by the client, we forward it directly to the server, but after that, we want to send a different request (c) from the proxy to the server directly to do a different action (so calling one of the API functions I implemented previously).

In other cases, when we see request x for instance, we just log some data about request x to a file after the request was made.

There is also a websocket between the client and the server where the server can send useful data at random intervals to the client. For some websocket responses from the server, no action will need to be taken (e.g. ws1), while for other responses, we will send an API call to the server to do some action (API call d).

Another use case will be that the client sends request a, which we don't take action on, unless a specific response (response y) is received, at which case some action is performed (e.g. API call z to the server)

In most cases, we don't really want to delay the requests made by the client to the server. We just want to passively sit between them and take some actions in certain scenarios, but never modify the actual request to or response from the server.

I hope that clarifies it a bit :)
I'm afraid this will become too complex to implement completely in a plugin? Or am I wrong here and is this doable?

Thanks again!

[abhinavsingh]

Just wanted to update, I'll get back to this and reply with my thoughts over the weekend. Got kind of tied up with something.