Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Increase file descriptor limits on ECS/Fargate workers (#174)
* This change resolves the issue of the maximum number of open file descriptors being exceeded at runtime. This is documented in the Dask Distributed FAQs (https://distributed.dask.org/en/latest/faq.html). The issue occurs on FargateCluster nodes under some types of heavy workloads. * The `nofile` hard and soft ulimits are set to 65535 (from default 1024) in both the scheduler and worker nodes via the ECS task definitions for each. * Also see dask/distributed#1858
- Loading branch information