SERVER_DOWN exception raised during batch operations
#516
Comments
|
We certainly need more details. Generally though, I suggest asking on the OpenLDAP mailing list - openldap-technical@openldap.org - as it's probably the configuration issue. |
|
I am not certain if the bug I observe is due to a misconfiguration of OpenLDAP or a misusage of python-ldap, this is the point of the issue. I will try to share an example project that reproduces the issue soon. |
|
Why are you using async? Also, performance wise, it is best to use a single connection and do the modifications serially. You will slow the system down by trying to do many modifications concurrently. I.e., your code is far from optimal as described. |
|
Sorry to have taken so long to answer.
Naively I thought that would help performances.
Out of curiosity, why is that? I can still reproduce the error with async methods. Here is a MRE for the bug I describe in OP. I use python 3.11.3 and python-ldap 3.4.3. You will need python-slapd to run the snippet (or adapt it to another LDAP server): >>> import ldap
>>> import slapd
>>> NB_ITEMS = 5000
>>> server = slapd.Slapd(
>>> schemas=[
>>> "core.ldif",
>>> "cosine.ldif",
>>> "inetorgperson.ldif",
>>> ],
>>> )
>>> server.start()
>>> server.init_tree()
>>> conn = ldap.initialize(server.ldap_uri)
>>> conn.simple_bind_s(server.root_dn, server.root_pw)
>>> for i in range(NB_ITEMS):
... conn.add(
... f"cn=cn{i},{server.suffix}",
... [
... ("objectClass", [b"inetOrgPerson"]),
... ("cn", [f"cn{i}".encode()]),
... ("sn", [b"foo"]),
... ],
... )
Traceback (most recent call last):
File ".../test.py", line 16, in <module>
conn.add(
File "/.../lib/python3.11/site-packages/ldap/ldapobject.py", line 233, in add
return self.add_ext(dn,modlist,None,None)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ".../lib/python3.11/site-packages/ldap/ldapobject.py", line 218, in add_ext
return self._ldap_call(self._l.add_ext,dn,modlist,RequestControlTuples(serverctrls),RequestControlTuples(clientctrls))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ".../lib/python3.11/site-packages/ldap/ldapobject.py", line 128, in _ldap_call
result = func(*args,**kwargs)
^^^^^^^^^^^^^^^^^^^^
ldap.SERVER_DOWN: {'result': -1, 'desc': "Can't contact LDAP server", 'errno': 32, 'ctrls': [], 'info': 'Broken pipe'}With 5000 items I get the error at all times, but decreasing the number of items to ~1000 won't crash. |
|
I suspect the server is getting frustrated because you're not reading the response side of the stream: you should call Check your server logs, especially if the above doesn't help. |
|
For the record, here is the last logs from the server: I understand by your answers that this might be a server configuration issue, but also that there is a misusage on my side. If async is not optimal and sync can only perform one operation at a time, how would you recommend to perform batch operations? |
Looks like the connection got lost somehow. Since you're using OpenLDAP, back-mdb will serialise your writes anyway, if you're concerned that network latency slows down your operations, limit your batches to a small amount, only issuing new operations as the old ones have been confirmed (using one of the |
Operating system: Archlinux
Python version: 3.10.9
python-ldap version: 3.4.3
ldap server: slapd 2.6.4
When I try to perform a large number (several thousands) of async operations (groups of search, add, modify), there is a point where a
SERVER_DOWNexception is raised.The exception is randomly raised by
addorresult, but if the list is, say 5000 items long, it never gets to the end. It feels like I open too much connections to the ldap server and at a moment I reach some kind of limit.Is it due to:
Thank you for your help
The text was updated successfully, but these errors were encountered: