You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Despite Kernel Culling parameters being configured, Voila Python process holds open a large number of File Descriptors, which leads to instability such as high load average, or ulimit reached causing the process and container to exit.
As seen in the screen recording, the amount of PID (as seen in the bottom docker stats window) kept increasing despite kernels being culled.
The amount of file descriptors open (as seen in middle-right window) climbed rapidly, and did not recover after requests have stopped.
The Load Average of this container also increased over time, and it did not recover after requests stopped.
Depending on where this container was run (e.g., local vs AWS ECS Fargate), it could hit ulimit very quickly and terminate, or becomes so sluggish to new requests and requires restarting.
Expected behavior
Expected behaviour is Voila increase in CPU and memory usage.
The amount of PIDs and file descriptors (fd) will also increase initially, and then stabilise.
When there are no new connections made to Voila, PID count and fd count will reduce back down to nominal value.
Context
voila version 0.2.10
Operating System and version: Docker on macOS 11.5.2, container image jupyter/base-notebook:python-3.8.8
Browser and version: apachebench
The text was updated successfully, but these errors were encountered:
Description
Despite Kernel Culling parameters being configured, Voila Python process holds open a large number of File Descriptors, which leads to instability such as high load average, or ulimit reached causing the process and container to exit.
Reproduce
App bundle is attached: simple-app.zip
1. Launch container
2. Watch Voila thread counts in container
3. Watch voila file descriptor count in container
docker exec -it --privileged --user root voila-simple-app ./simple-app-lsof.sh
4. Simulate traffic with ApacheBench
Observed behaviour
As seen in the screen recording, the amount of PID (as seen in the bottom
docker stats
window) kept increasing despite kernels being culled.The amount of file descriptors open (as seen in middle-right window) climbed rapidly, and did not recover after requests have stopped.
The Load Average of this container also increased over time, and it did not recover after requests stopped.
Depending on where this container was run (e.g., local vs AWS ECS Fargate), it could hit ulimit very quickly and terminate, or becomes so sluggish to new requests and requires restarting.
Expected behavior
Expected behaviour is Voila increase in CPU and memory usage.
The amount of PIDs and file descriptors (fd) will also increase initially, and then stabilise.
When there are no new connections made to Voila, PID count and fd count will reduce back down to nominal value.
Context
jupyter/base-notebook:python-3.8.8
The text was updated successfully, but these errors were encountered: