OSError: [Errno 24] Too many open files

[lsabi]

Checklist

• I searched the HTTPX documentation but couldn't find what I'm looking for.
• I looked through similar issues on GitHub, but didn't find anything.
• I looked up "How to do ... in HTTPX" on a search engine and didn't find any information.
• I asked the community chat for help.

Question

I'm using fastAPI which pushes data to Apache Solr via httpx. When I want to import some data, it posts a lot of sequential requests (i.e. one after the other) and I get the following error

  File "./app/utils/search.py", line 79, in update

  File "/home/user/.local/lib/python3.8/site-packages/httpx/_client.py", line 1009, in __init__
  File "/home/user/.local/lib/python3.8/site-packages/httpx/_client.py", line 1046, in init_transport
  File "/home/user/.local/lib/python3.8/site-packages/httpx/_config.py", line 62, in __init__
  File "/home/user/.local/lib/python3.8/site-packages/httpx/_config.py", line 85, in load_ssl_context
  File "/home/user/.local/lib/python3.8/site-packages/httpx/_config.py", line 142, in load_ssl_context_verify

OSError: [Errno 24] Too many open files


During handling of the above exception, another exception occurred:
...

Which is strange because I'm not using HTTPS connection, since I'm within a private network and just testing for now. The provided URLs are without the S, so no ssl should be loaded. Could it be that httpx tries to open the certificate and not closing it?

My code looks as below (indentation is correct in the file, Markdown is just messing it up)

try:
        async with httpx.AsyncClient() as client:
            res = await client.post(url, json=data, params=options)

        if res.status_code == 200:
            return True
        else:
            raise ValueError("Wrong status code", res.status_code, res.json())
    except Exception as e:
        print(e)
        logger.exception("Cannot update document in SOLR")

QUESTION
Is it me or is it a bug (see between the error and the code)? I haven't found anything that could relate to the problem except for #556, but the problem was due to the library it was used.

Thanks.

[florimondmanca]

Hey,

“Too many open files” is a pretty generic Linux error, and I’m not sure it’s caused by the SSL code in particular, but...

You can either increase the limit on your machine, or alternatively (better in any case) reuse the client instance across your requests, instead of instantiating one for each request. That way the SSL config will only be loaded once, instead of once per request (which by the way can lead to pretty big memory usage).

Going to close this for now, if these two action items don’t help feel free to reach out on the Gitter chat. Thanks!

PS: by default SSL is enabled on clients so the client loads up a context on init, but the context may or may not be used in requests depending on http vs https usage.