High resource usage and server lockup after video upload on MediaCMS 8.x (Docker)

[epinox]

Hi everyone,

I'm running the Docker version of MediaCMS 8.3.0 on a small vServer (Debian Trixie, 4 vCPUs, 4 GB RAM). Everything worked smoothly with version 7.2, especially video uploads. Since upgrading to the 8.x series, however, I'm facing a serious issue:
After uploading a video, the server almost always gets completely overloaded. CPU and memory usage go to 100%, the whole system becomes unresponsive (even SSH access stops responding), and the only way to recover is a full vServer restart.
Additionally, even when MediaCMS is completely idle, the Celery workers seem to consume nearly all available RAM.

Has anyone else experienced similar problems with MediaCMS 8.x in a Docker environment on limited hardware? Are there any known configuration options, Docker environment variables, Celery settings, or best practices to significantly reduce the resource footprint (especially for Celery workers and during video processing)?

Any tips, configuration examples, or pointers to relevant documentation would be greatly appreciated!
Thanks in advance for your help — really loving the project and would like to keep using it on smaller servers.

Frank

[krotname]

This looks very likely to be the stock Docker Celery sizing being too large for a 4 GB VPS.

In the current Docker config, prestart.sh copies the Celery supervisor configs when ENABLE_CELERY_SHORT / ENABLE_CELERY_LONG are enabled. The defaults are quite aggressive for small hosts:

• supervisord-celery_short.conf starts short1 short2 ... -c10 -Q short_tasks, so that is up to 20 short-task worker processes.
• supervisord-celery_long.conf has no explicit -c, so Celery prefork defaults to the CPU count. On a 4 vCPU VPS that can run several long tasks at once.
• upload processing goes through chunkize_media on short_tasks, then schedules encode_media on long_tasks; those long tasks run ffmpeg. There are also long-queue tasks for sprites/HLS/Whisper-related work.

For a small Docker install I would try this first:

# deploy/docker/local_settings.py
DO_NOT_TRANSCODE_VIDEO = True

The admin docs describe this as the low-capability-server option: it disables transcoding and sprite creation, and only the original file is shown.

If you still need transcoding, reduce the worker concurrency instead of using the defaults:

# deploy/docker/supervisord/supervisord-celery_long.conf
- ... -Ofair --prefetch-multiplier=1 -Q long_tasks
+ ... -Ofair --prefetch-multiplier=1 -c1 -Q long_tasks

# deploy/docker/supervisord/supervisord-celery_short.conf
- celery multi start short1 short2 ... --soft-time-limit=300 -c10 -Q short_tasks
+ celery multi start short1 ... --soft-time-limit=300 -c2 -Q short_tasks

Then recreate the worker container so prestart.sh copies the changed supervisor files:

docker compose up -d --force-recreate celery_worker

Also check how many encode profiles are active; every active profile means more ffmpeg work per upload:

docker compose exec web python manage.py shell -c "from files.models import EncodeProfile; print(list(EncodeProfile.objects.filter(active=True).values('id','name','resolution','codec')))"

On 4 GB RAM I would start with DO_NOT_TRANSCODE_VIDEO = True or long_tasks -c1; otherwise one upload can legitimately saturate both CPU and memory.

[epinox]

Hi krotname,

I just wanted to say a huge THANK YOU!

Your solution works perfectly. After applying the changes in the deploy/docker/supervisord/ folder, the memory usage has been almost halved. The server no longer freezes or becomes unresponsive after video uploads - even during transcoding.

During transcoding the CPU still goes to 100% (which is expected with ffmpeg), but the big difference is that there is now more than 1 GB of free RAM left, so the system stays stable and responsive. Exactly what I needed!

I kept Transcoding enabled because I only use the standard encodings and upload mostly 720p videos. With just those supervisord adjustments everything runs smoothly now.

I'm really happy with the result and super grateful that you took the time to help me.
Thank you again - you saved me a lot of headaches! 😊

Frank

[krotname]

Yes, exposing the worker concurrency as settings/env overrides makes sense to me, with the current values left as defaults.

The two knobs I would start with are:

• long queue concurrency, e.g. default unset/current behavior, override to -c1 for small VPS installs
• short queue concurrency / number of short workers, so small installs do not get the short1 short2 ... -c10 shape

I would keep DO_NOT_TRANSCODE_VIDEO as the blunt low-capability option, but make "transcode, just slowly" easier than editing supervisor files under deploy/docker/supervisord.

I can't promise a PR immediately, but if someone picks it up I think the clean shape is "settings/env -> rendered supervisor command" rather than asking Docker users to patch the shipped supervisor confs.

[mgogoulos]

Hi all,

@krotname good catch regarding the celery number of workers. I also have been running the default settings in a number of small capabilities VMs (eg https://demo.mediacms.io/ runs in a 2cpu/2GB VM) without issues, but maybe it makes sense to expose these settings in a more easily configurable way, like somehow pass them from settings.py. I'd still keep the ones that are currently set as the default ones but give a way to override more easily. If you're interested to having a look and submit a PR that would be great.