DVC failing with "too many open files" error on large dataset

[aayostem]

I'm getting a OSError: [Errno 24] Too many open files when running dvc add on a dataset with 50,000+ files. The error occurs consistently around the 10,000 file mark:

$ dvc add data/raw_images/
Adding...
Error: Failed to add 'data/raw_images' - OSError: [Errno 24] Too many open files: 'data/raw_images/image_9847.jpg'

My environment:

• Ubuntu 20.04
• DVC 3.0.0
• Python 3.8
• Default ulimit: 1024

What I've tried:

• Increasing ulimit to 4096 (ulimit -n 4096)
• Using dvc add --file .dvc to process in smaller batches
• Restarting the process multiple times

The dataset structure:

data/raw_images/
├── category_a/    # ~15,000 files
├── category_b/    # ~20,000 files  
├── category_c/    # ~15,000 files
└── metadata/      # ~500 files

How can I resolve this file descriptor limit issue and successfully add large file collections to DVC?

[skshetry]

DVC 3.0.0 is no longer supported. Please update to the latest version and try.

Also, DVC only supports Python >=3.9, so please update Python version as well.
If not, DVC provides binary package (.deb) that you can install from following instructions from https://dvc.org/doc/install/linux.

[aayostem]

@skshetry what i did was to increase system file descriptor limits. I tried doing a temporary increase using the current session $ ulimit -n 65536 and it worked. Then i did a permanent system wide increase before restarting.