Performance type issues since upgrading to 2026.1.31

[ChrisNolan]

Good morning fellow shippers,

Since I upgraded to 2026.1.31 (from 2024.5 ish era) I've had two periods of downtime due to the server seemingly 'running away'. Has anyone else had these problems too? Where do I start? Do I need to throw more hardware at it, or are there outstanding PRs or something related to the issue I can apply perhaps?

The first incident presented as the db itself refusing new connections -- it was maxed out (default seems to be to allow 100). After a restart I was 'watching' the connections (watch -n2 'sudo docker compose exec -T db psql -U postgres -d db -c "select count(*) from pg_stat_activity;"') and they were holding 'steady' at 12 or 14 (of the 100).

This morning, the full server ended up crashing due to filling the swap file and the available ram...

This is a 'self hosted' instance. I did build the image myself so it could include PRs that I needed that aren' tin 2026.1.31 yet so perhaps my method of rebuilding the images themselves is related... perahps there are some config or environment variables I could tweak? (i.e. it seems like it might be in 'debug' mode which likely takes more resources?).

[ChrisNolan]

This seems to be related to

fix(smartkargo): fix trace persistence and shipment cancellation

From 2026.1.26 -- from e7c19e9

I think I'll look at wrapping the ORM stuff in tracing/utils so that old connections can get cleared up ... or enqueue it to Huey instead? idk. Thoughts?

[ChrisNolan]

Ok, so my agent made #1120 as a 'phase 1' fix with minimal blast radius that allows for the connections to be cleaned up. The fix mentioned above allowed for many more successful writes which made it more obvious they were accumulating.

It also put in place a document at PRDs/TRACING_PERSISTENCE_PHASE2_HUEY_REFACTOR.md which will need review and consideration as a 'proper' long term fix that is more consistent with the way the rest of the code base handles this type of db update.