Memory increase with V2

[samuelcolvin]

Discussed in #6550

^{Originally posted by accountForIssues July 10, 2023}
I use FastAPI (with Pydantic) and uvicorn (with 8 workers) in a docker container with a 2 GB memory limit. Currently in process of migrating/testing the latest versions, I observed that a lot of the workers weren't starting with no errors.

When removing the memory limit, the workers ran fine but the memory usage increased significantly.

The production instance (pydantic 1.10.8 and FastAPI 0.96) uses about 1 GB with 8 workers. Whereas with just 4 workers, the dev instance (with Pydantic v2 and FastAPI 0.100) uses > 2GB.

I ran the "uvicorn run ..." command line with memray to profile the memory usage and this was the summary result.

I have also attached the flamegraph html (as txt) when running with 1 worker. Note that this is on start-up with no incoming requests.

memray-flamegraph-uvicorn.1.html.txt

Can anyone see what could be the cause ? And whether this is pydantic or FastAPI related ? I am also doing a lot of refactoring so it's possible that it's something that I broke in my code.

also related to tiangolo/fastapi#9857

• @accountForIssues sees > 100% increase in memory
• @MattiasOlla sees 35% increase in memory

Selected Assignee: @Kludex

[MattiasOlla]

Actually, I can reproduce the effect even in a very minimal example (essentially just a hello world FastAPI application) https://gist.github.com/MattiasOlla/966f2dc5e1c505eb1e04eb8759eaee5c In my test, the memory usage is about 40MiB using version 1 and 65MiB with version 2, so 60% greater. Of course, with such a small application, it's hard to tell how it scales, but the effect seems to be there.

[accountForIssues]

Thanks for starting an issue.

In the flamegraph, there are some references to my code. So, I've been trying to see what could be causing the bloat in there but can't seem to find anything common between them.

There are no large default values in the models. I might try commenting out each route and see if it has any drastic effect.

I also have a smaller project (3 workers) that uses about ~175 MB with Pydantic v1 and the same with v2 uses ~200 MB, so about 10-15% increase there as well. This project however only has about 5 pydantic models compared to 50+ in the other one.

[samuelcolvin]

We would expect some increase in memory in V2 because some python values are copied into rust objects and store in every validator and serialised.

But 500mb for ~50 models is a lot.

Btw, @dmontagu noted that he thinks including routes twice deep copies the validator, this was free or very cheap in V1, but not in V2. Might be fixable in fastapi.

Also @Kludex mentioned that multiple workers are not supposed to be used in production with uvicorn - something to do with restarts.

[Kludex]

Also @Kludex mentioned that multiple workers are not supposed to be used in production with uvicorn - something to do with restarts.

https://www.uvicorn.org/deployment/#using-a-process-manager

[accountForIssues]

Is there something I can look at in my code ? It's entirely possible that the huge increase for me is partly due to how the models are being used and/or the way coroutines are defined. Maybe it had no effect in v1 but now it's coming out.

On a first glance, I don't really see anything. Most of the models are not nested and rarely have any default values.

As for using uvicorn, I had some trouble logging with gunicorn and hence uvicorn was preferred. I'm aware it has issues. I will definitely take a look once this memory issue is solved. :)

[accountForIssues]

@samuelcolvin Tested the new Pydantic 2.0.3 release and with 2 workers, the memory use is now down to ~440 MB from ~1.2 GB with Pydantic 2.0.2 with the code/environment being the same.

It is a great improvement over 2.0.2 and is manageable but still a large increase over ~125 MB / worker in Pydantic v1.

[DeoLeung]

I'm still experiencing memory issue on 2.0.3, it doubles the usage v1 used, any information I can provide or actions I can take?

[hramezani]

@DeoLeung we released Pydantic 2.1.1. Please test it first.

[DeoLeung]

upgrading to 2.1.1 is much better now, while it's still around 20% more

[accountForIssues]

Upgrading to 2.1.1 is another great improvement but for me it's still slightly less than 2x increase. With 1 worker, it's about 240 MB. Hopefully, it goes down further with more updates but it's manageable in the current state.

[StasEvseev]

Hello folks! Just sharing some more info regarding the issue around memory usage for Pydantic v2.
We has been using v2 on prod for couple of days. Our setup is basically gunicorn with 4 workers using gevent hubs.
We noticed huge bumps in memory usage and tried to overcome it with max_requests configuration, but it didn't help much because main process is also growing in virtual memory (VSZ on ps command), and respawing processes doesn't help then.

Let us know what the update? And is there any way to make it work?

Thank you very much!

UPD: memory on main process didn't increase over time my mistake. So, main process actually imported pydantic and then it forked to children, I would hope that CoW would safe us here, but for some reason memory usage only increase over time and process respawning doesn't help for some reason.

UPD2: After couple of more days of observation, noticed that our child workers got 139 code and killed and even thought it killed memory is not released fully and therefore lead to leaks. Happens quite often, might be related to the fact that we are using gevent?

Since we are running on the k8s with PID namespaces it is hard to backtrace the error logs on the host machine with an actual container, but I found these logs just before the 139 error.

Those are different segfaults we were facing.

segfault at 0 ip 000078b1d0ff6f1c sp 00007ffe971366c0 error 6 in libpython3.11.so.1.0[78b1d0eed000+1bb000]

segfault at 100 ip 00007ddfb64fe349 sp 00007ffc351395b0 error 4 in _pydantic_core.cpython-311-x86_64-linux-gnu.so

segfault at e4 ip 00000000000000e4 sp 00007ffc30d59a58 error 14

Hope it helps!

@samuelcolvin

[StasEvseev]

@samuelcolvin do you think, those segfaults we were facing were result of using MiMalloc allocator?

[healfy]

hello, we have faced huge memory increase with pydantic 2.1.1 and gunicorn too.
Should we expect some fixes or just it will be as it is?

[dmontagu]

In general memory usage will go up because we store more information about how to validate types in memory (a major part of getting the large factor increase in validation speed over v1). There were some issues with mimalloc in earlier versions of pydantic v2 reserving a large amount of unused memory, but on the latest versions of pydantic v2 the memory hit may be harder to address.

If you are using Fastapi, one of the reasons for the large amount of memory use is that, as far as I am aware, the typeadapters are not cached/reused between endpoints, so you end up having memory usage that scales with the number of endpoints rather than just the number of types you are using. I don't know how hard that would be to address but I suspect is a more realistic short term target for improvement than the amount of memory used to represent the pydantic core objects themselves.

(I'll note that I would not be surprised if there was still low-hanging fruit for memory usage optimization in pydantic core, but I also wouldn't be surprised if there wasn't.)

[healfy]

Thank you for the detailed explanation. It is very interesting, but with Fastapi we haven't faced some memory leaks and everything works smooth as it should. We have encountered issues with aiohttp + gunicorn. I didn't get a chance to dive in it, because it quite challenging to install memray in our environment. Hope I will find way to investigate it

[dmontagu]

I’ll note that various memory leaks have been fixed since Pydantic 2 was released. If you suspect a memory leak issue it may be worth trying a more recent minor version than 2.1, I’m just not sure off the top of my head when those fixes were added.