Enabling shared state across processes

[toumorokoshi]

There are a couple use cases for being able to share objects across processes:

1. sharing counters / gauges to ensure that the value accurately reflects the correct value for all processes (helpful when using a multi-process web server such as gunicorn)

2. enable handing of trace information (may not be needed as oftentimes this is done via fork(), which will just create a copy-on-write copy).

We have some prior art with how prometheus does it: https://github.com/prometheus/client_python

[toumorokoshi]

I'm going to close this issue, as we can discuss it as we get closer to a tangible need or problem.

[ocervell]

Hi @toumorokoshi , I think this use case is very valid and matches real-world situations (gunicorn > Flask / Django apps). Shared counters would be highly appreciated. Have we moved the discussion forward at all on this ?

[toumorokoshi]

@ocervell unfortunately not: as you can imagine, metrics aggregating across worker processes is just as hair as it was a year ago :)

feel free to re-open (or create a new ticket) if you're interested in driving the discussion, or have a specific use case we can design toward.

As another thought, the other design pattern we can leverage instead of some shared object across Python processes is just use a sync that knows how to aggregate them all, such as the opentelemetry-collector.

I'm a bit hesitate toward any approach that does shared state in Python, primarily because there's significant performance implications for the wrong design, and processing overhead and complexity with a design that can handle all edge cases.

[hacst]

@toumorokoshi Isn't the use-case is pretty much all of the standard production python web-service deployments (e.g. #365)? They mostly use processes (through gunicorn, uwsgi, ...) unless specifically designed around async. And even those tend to additionally use processes in production as far as I have seen (e.g. https://www.uvicorn.org/#running-with-gunicorn). The GIL makes it extremely hard to scale reliably using threads like you would in other languages.

W.r.t. opentelemetry-collector: Is it actually able to perform this task (and for all metric types/aggregations)? I'm a bit unclear how this would be configured (e.g. to get a histogram). Would this require something like https://github.com/open-telemetry/opentelemetry-collector/issues/1422 ?

[toumorokoshi]

@toumorokoshi Isn't the use-case is pretty much all of the standard production python web-service deployments (e.g. #365)?

The use case does apply to anything that needs to aggregate metrics across multiple processes, and I agree in Python it's fairly common.

In the context of #365 specifically, a lot of those metrics could be collected in the top-level process, since it does interact with every request.

Regarding how we would accomplish this in in the open-telemetry collector: I admit I'm not 100% sure either, I just know theoretically that's where we landed.

Sorry I think to be clear: I am in no way saying that this isn't valuable. But as with most open source projects, we need someone to contribute in this area to make it viable.

The reason I closed this issue originally was because it wasn't focused on an example we wanted to enable (from which an implementation would be clearer), and was focused on a broad concept.

If you have a strong need, I think we would love a contribution (even a prototype) to help shape this discussion.