During batch feed, OSError: [Errno 24] Too many open files

[neo-anderson]

Hi 👋
When I try to ingest data into Vespa Cloud, I get this error - OSError: [Errno 24] Too many open files.

When I select only the first few documents in my dataset, the feed works. If I use the whole dataset, I get that error. I dont see a way to reset the connections/close files. So, pyvespa wont let me upload anymore data unless I quit the python session and do it all over again. Synchronous batch feed works, but it is too slow for my usecase.
Code:

# works
app.feed_batch(schema="myschema", batch=batch_data[:1000], batch_size=1000, total_timeout=200, asynchronous=True)
# fails
app.feed_batch(schema="myschema", batch=batch_data, batch_size=1000, total_timeout=200, asynchronous=True)

[thigm85]

This seems to be an OS limitation. I was once running a script on an AWS EC2 instance and I had to increase the open file soft limit when using app.feed_batch.

I increased the limit with ulimit -n 10000 and checked with ulimit -Sn. It worked after that.

If the above does not work, try to reduce the number of async connections via the connections parameter:

app.feed_batch(..., connections, ...)

[neo-anderson]

I didn't change ulimit, but tried reducing the number of connections like you suggested and it worked! The default number of connections (100) was fine for a local docker deployment. Had to reduce it to ingest the same data into Vespa Cloud. Thanks for your help.

[thigm85]

Glad it worked. Did you check what the highest value that worked was? I might consider changing the default connections value.

[neo-anderson]

Hi @thigm85 , connections set to 100 and 50 didn't work for me, but setting it to 20 worked. I dont know if it's the max value that would work though. Cheers!

[thigm85]

Thanks @neo-anderson

[neo-anderson]

I tried ulimit -n 10000. It fixed the issue completely. Ingesting with 100 connections without an issue 👍. Thanks!

[bratseth]

@lesters put this in the doc somewhere and/or change default?

[thigm85]

@bratseth @lesters I can create a troubleshooting section and include this. It was on my to-do list anyway.

[bratseth]

Yes, great!