You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I'm using fastAPI which pushes data to Apache Solr via httpx. When I want to import some data, it posts a lot of sequential requests (i.e. one after the other) and I get the following error
File "./app/utils/search.py", line 79, in update
File "/home/user/.local/lib/python3.8/site-packages/httpx/_client.py", line 1009, in __init__
File "/home/user/.local/lib/python3.8/site-packages/httpx/_client.py", line 1046, in init_transport
File "/home/user/.local/lib/python3.8/site-packages/httpx/_config.py", line 62, in __init__
File "/home/user/.local/lib/python3.8/site-packages/httpx/_config.py", line 85, in load_ssl_context
File "/home/user/.local/lib/python3.8/site-packages/httpx/_config.py", line 142, in load_ssl_context_verify
OSError: [Errno 24] Too many open files
During handling of the above exception, another exception occurred:
...
Which is strange because I'm not using HTTPS connection, since I'm within a private network and just testing for now. The provided URLs are without the S, so no ssl should be loaded. Could it be that httpx tries to open the certificate and not closing it?
My code looks as below (indentation is correct in the file, Markdown is just messing it up)
try:
async with httpx.AsyncClient() as client:
res = await client.post(url, json=data, params=options)
if res.status_code == 200:
return True
else:
raise ValueError("Wrong status code", res.status_code, res.json())
except Exception as e:
print(e)
logger.exception("Cannot update document in SOLR")
QUESTION
Is it me or is it a bug (see between the error and the code)? I haven't found anything that could relate to the problem except for #556, but the problem was due to the library it was used.
Thanks.
The text was updated successfully, but these errors were encountered:
“Too many open files” is a pretty generic Linux error, and I’m not sure it’s caused by the SSL code in particular, but...
You can either increase the limit on your machine, or alternatively (better in any case) reuse the client instance across your requests, instead of instantiating one for each request. That way the SSL config will only be loaded once, instead of once per request (which by the way can lead to pretty big memory usage).
Going to close this for now, if these two action items don’t help feel free to reach out on the Gitter chat. Thanks!
PS: by default SSL is enabled on clients so the client loads up a context on init, but the context may or may not be used in requests depending on http vs https usage.
Checklist
Question
I'm using fastAPI which pushes data to Apache Solr via httpx. When I want to import some data, it posts a lot of sequential requests (i.e. one after the other) and I get the following error
Which is strange because I'm not using HTTPS connection, since I'm within a private network and just testing for now. The provided URLs are without the S, so no ssl should be loaded. Could it be that httpx tries to open the certificate and not closing it?
My code looks as below (indentation is correct in the file, Markdown is just messing it up)
QUESTION
Is it me or is it a bug (see between the error and the code)? I haven't found anything that could relate to the problem except for #556, but the problem was due to the library it was used.
Thanks.
The text was updated successfully, but these errors were encountered: