Terminates during load testing

[lukepighetti]

When running on a Digital Ocean 512MB VPC, if I load test it with hey -m GET -c 5000 -n 10000 <host> (5000 concurrent, 10000 connections), pocketbase will terminate. my expectation is that it would be resilient enough to stay running

What's the recommended way to make this more durable? One person could take down my server quite easily

[thangngoc89]

@lukepighetti Looks like it was terminated by the OS? Need more information on which limit has been reached that lead to termination.

You could test again with GOMEMLIMT. Set it to something like 450MiB.

GOMEMLIMT=450MiB pocketbase ....

[ganigeorgiev]

Without clear reproducible steps, how the environment was configured and what are those TLS errors, there is nothing much I can help with.

If the process was terminated it was most likely due to OOM but again without details what is even the above screenshot supposed to show I can't help.

For benchmarks you can check https://github.com/pocketbase/benchmarks.

If you need further assistance, please provide more details in a Q&A discussion as the threaded format is more suitable for this.

[lukepighetti]

Is there any way to convert this into a discussion? That would have been a better place to post it. The overarching theme is making a pocketbase installation durable. I have no concerns about the actual speed during the load test.

[ganigeorgiev]

I've converted the issue into a Q&A discussion, but again without details about the running environment and what is your setup there is no much I can help with.

In general there are 2 main reasons why your PocketBase process may get terminated:

1. OOM - if you are in memory-constrained server environment (eg. under 1GB RAM) then as mentioned above you can set the GOMEMLIMIT env variable to instructs the GC to run more often (it is a soft-limit).
2. Not enough file descriptors (see "Increase the open file descriptor limit").

You can try to find the actual reason why the PocketBase process was terminated by checking the stderr logs and/or if you are running a systemd service the recent journalctl logs.

In both cases there is nothing really much we can do at the moment. If you want to apply a rate limit or other more advanced network configurations you'll have to setup a reverse proxy (nginx, apache, caddy, etc.) but even with a rate limit in place there is no guarantee that someone DoS-ing your server will not take it down since the requests will have reached the server anyway.

[lukepighetti]

What's the ideal way to automatically restart a pocketbase process if it is terminated by the OS?

[ganigeorgiev]

I prefer and recommend using systemd. You can find an example in Step 4 of the "Minimal setup" guide.

[lukepighetti]

I don't appear to be getting termination errors now that it's running in systemd, any idea why that might be the case?

[ganigeorgiev]

I have no idea but systemd will not prevent the OS killing your app process, it will just attempt to restart it if that happens.

Since your VPS has ~512MB my guess is that your issue is caused by OOM (especially if you are running hey from the same server). There will be some performance and allocations improvements with the refactoring but even with that if you are planning processing for example large amount of data or generating thumbs for large images (see #3794 (comment)) it will require bumping your RAM (or at least enable swap if not already).

[lukepighetti]

hey was being run from an external computer

process seems to be totally durable running under systemd. Appreciate the assistance. Really grateful to have access to this tech (pocketbase)