Exposing prometheus metrics as a separate fastapi app and on a different port

[manjiler]

I am currently using the sub application mount to have both the business service application and a separate prometheus endpoint.

I really like to run it on a separate port have a different worker pool for it.

When using gunicorn with uvicorn.workers.UvicornWorker class we can set the max_request to a number greater than 0 so that worker process is restarted after that number. This help limit any memory leak.

Since I am using sub application mount, they run as apart of the same worker(s) process and share the same port. Everytime prometheus scraps for metrics, the request get considered against the max_request.
For example, if the max_request is set to 100 and prometheus scrapper is set a period of 10 seconds. Then in 1000 seconds the worker process is restarted.
If prometheus sub application can run on a different port then I can set this max_request=0 so it won't restart the worker that is serving the prometheus app.

Has only in the community tried something like this?

[jgould22]

Why are you concerned about worker restarts?

[manjiler]

I am not concerned about the worker restarts.. I want the worker to restart only it served a actual API and prometheus metric endpoint...
At this point, it just an unnecessary restart.

uwsgi had an option to restart the worker if RSS reached a particular value.

[manjiler]

A separation of these would be better as we can handle them accordingly.

Hypothetically say, there was to deploy 2 fastapi on to a single gunicorn instance have them bounded to different ports and we have different gunicorn configuration for it.
Also this makes sure that is a sets of workers dedicated for handling the main business API and another sets of workers (smaller in number) to handle the monitoring aspects..

[manjiler]

@jgould22 Is there way I can 2 fastapi application running on different ports in a single gunicorn instance?

[jgould22]

I do not believe that Gunicorn has that functionality.

UWSGI likely does as you mention above. If it is better for you use case then you should use UWSGI.

That said I would not recommend introducing such complication to your application just to avoid gunicorn cycling its workers.

[manjiler]

Over the few weeks, I tried various ways do this. Nothing seems simple and straight forward...

[rafsaf]

Memory leak after 100 requests?? That is very low limit.

Is this good idea to separate metrics from app: it depends, with higher traffic I would definitely go for that, then if something is wrong with app, metrics are still alive and vice-versa. And be aware its very bad idea to reset processes every 100 reqs using prometheus-client, note every created process writes to new file and you shortly end up with 1000000 files with metrics in mutli proc dir and it takes 10s for one scrape :P, I hardly recommend following discussion in their issues - starting prometheus/client_python#275 and there are many more in past years.

Much better idea is to kill processes when they reach memory limit. In Samsung (SGG) where I work in one of our projects we use in-memory SQLites and some other YDD stuff that increase memory infinitely and it is problem after some time, of course it's all to be rewritten some day, but for now in uwsgi https://uwsgi-docs-additions.readthedocs.io/en/latest/Options.html#process-management for example you have both reload-on-rss (kindly ask for reload on certain memory usage) and evil-reload-on-rss (unkindly reload on certain memory usage). Use equivalent settings whatever server you use, kill it on say 500mb and forget max_request (or use much more value if for some reason that's your choice), but don't get me wrong, this is bad when you have memory leaks like that and need to reload processes, first you should investigate if this is real problem for you, only then try to solve it.

[manjiler]

@rafsaf I agree on the RSS part. I have used USGI for my previous projects and reload-on-rss was a good thing to have rather than max_request.

With ASGI, its either gunicorn or uvicorn. They support only max_request.

For now, I have increased the max_request to 10000 for now.

[rafsaf]

Hmm it appears you're right, moreover Nginx Unit which is a very nice alternative, has also only max requests limit, nothing for memory.

Hypercon and Daphne not seem to support even that much. Unfortunately I'm not aware of anything more, but if someone does please share a knowledge with us :D

Maybe we should start just digging in 4 mentioned webservers issues (I would not use daphne for fastapi app) and someone definitely could have this kind of a problem, not documented solutions may exist.

[manjiler]

gunicorn is the preferred way for fastapi production deployment with uvicorn workers.
I will be good to do some digging in gunicorn to see if they can support reload-on-rss.