SocketException(Socket closed) when adding a very big entry

[fanf]

Hello,

When I try to add a very big entry (big == (nunber of values for an attribute x size of values) goes above some threshold), I get the following exception:

	at com.unboundid.ldap.sdk.LDAPConnectionInternals.sendMessage(LDAPConnectionInternals.java:611)
	at com.unboundid.ldap.sdk.LDAPConnection.sendMessage(LDAPConnection.java:4406)
	at com.unboundid.ldap.sdk.AddRequest.processAsync(AddRequest.java:1121)
	at com.unboundid.ldap.sdk.AddRequest.process(AddRequest.java:1025)
	at com.unboundid.ldap.sdk.LDAPConnection.add(LDAPConnection.java:2078)
       ......
Caused by: java.net.SocketException: Socket closed
	at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:118)
	at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
	at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
	at com.unboundid.util.ByteStringBuffer.write(ByteStringBuffer.java:1662)
	at com.unboundid.asn1.ASN1Buffer.writeTo(ASN1Buffer.java:1012)
	at com.unboundid.ldap.sdk.LDAPConnectionInternals.sendMessage(LDAPConnectionInternals.java:562)
	... 12 more

So it seems that somewhere inside unboundid, there is something that is limited by Integer.maxvalue.

You can reproduce it with the following code (it's scala, but should really be understandable anyhonw: add a sufficient number of sufficiently big values => you reach the point where the error happens).

  val provider = new RWPooledSimpleAuthConnectionProvider(
      host = "localhost",
      port = 1389,
      authDn = "cn=Manager,cn=rudder-configuration",
      authPw = "secret",
      poolSize = 1
  )
  def str(size: Int) = new String( Array.fill[Char](size)('x'))

  // run
  def main(args: Array[String]): Unit = {
    val dn = new DN("cn=Nodes Configuration,ou=Rudder,cn=rudder-configuration")
    // here, if I use 454 bytes for the last value, I get the exception. Below that, OK
    val values = for(i <- 0 until 167) yield i + "-"+str(100000) ++ str(454)


  val addRequest = new AddRequest(
        dn
      , new Attribute("objectClass", "top", "nodeConfigurations")
      , new Attribute("cn", "Nodes Configuration")
      , new Attribute("nodeConfig", values:_*) // here, we add 168 big value and one small
    )

    val x = for {
      ldap <- provider
      _ =  println("**** got connection")
      _ <- ldap.delete(dn) // that's just to clean up when testing multiple time
      _ =  println("**** deleted")
      _ =  ldap.backed.add(addRequest) // KABOUM
      _ =  println("**** added")
    } yield {
      "done"
    }
    println(x)
  }

I tried to understand both the code and the debug output, but failed for each one.

I understand that the use case is a little bit extreme (even if encoutered in Rudder https://www.rudder-project.org/redmine/issues/10646#note-24), but I'm wondering if there is a workaround ?

Thanks

[fanf]

Some more information: if I add the entry with a sub-set of values, then add more and more values with modification change requests, I can go beyond the threshold. And I can read back the entry with unboundid SDK.
So really, the problem seems to lie in the encoding of one attribute with a lots of values in one go.

[dirmgr]

This is almost certainly a case of the directory server closing the connection rather than a problem with the LDAP SDK itself. Most directory servers impose a limit on the size of a request that the client can send, although that limit is usually configurable.

Imposing a limit on the maximum size of an LDAP request is an important safety feature because of the way that LDAP works at the protocol level. LDAP requests are encoded using the ASN.1 Basic Encoding Rules (BER), as described at https://ldap.com/ldapv3-wire-protocol-reference-asn1-ber/. A BER element contains three components:

• A byte (or multiple bytes in rare cases that you’re not likely to encounter in LDAP) that provides information about the type of data held in that element.
• One or more bytes that specify the number of bytes in the element’s value.
• An encoded representation of the element value.

Each LDAP message is encoded as a BER sequence, which is a kind of container that basically holds an array of other elements (the message ID, the body of the request or response, and an optional set of controls). But ultimately, each LDAP request and response is encapsulated in a BER element that has all of the other elements inside it.

When the server receives a request from the client, it first reads the the type and length of the BER element that holds the LDAP message, and then it needs to allocate enough memory to hold the value of that element. In this case, the client is sending a big request, and the server is refusing to allocate that much memory. It’s probably a relatively small amount of memory (maybe just a couple of megabytes), but it’s a safety mechanism nonetheless because it prevents a malicious client from establishing a connection and sending the start of a BER message that says the element value is something big, like a gigabyte, and causing the server to allocate that much memory. If the client establishes a lot of connections that all do that, it can cause the server to run out of memory and crash or start swapping or exhibit some other kind of erratic behavior.

The directory server’s only real safeguard against this type of attack is to terminate any connection on which a client indicates that it wants to send a big request. It could send a notice of disconnection unsolicited notification before it does that to explain why it’s closing the connection, but not all servers do that, and you would have also needed to register an unsolicited notification handler in the LDAP SDK to be able to get that notification anyway.

But the good news is that the maximum request size limit is probably configurable. Most servers have a relatively low limit (I think it’s usually in the 1–5 megabyte range) since LDAP requests are typically pretty small, and about the only time you really run into this issue is when you’re trying to add a really big entry. But you can probably update the configuration to raise the limit. In the Ping Identity Directory Server, there’s a max-request-size property in the LDAP connection handler configuration. In directory servers derived from the Netscape Directory Server codebase (including the Fedora 389 Directory Server, Red Hat Directory Server, and Oracle DSEE), I think that the configuration attribute is called nsslapd-maxbersize. I’m not as familiar with the configuration for other types of servers like OpenLDAP or Active Directory, but there’s probably some way to raise the limit. I’d recommend checking the documentation or using the support channels for the server that you’re using to see what the options are for the server that you’re using.

[fanf]

Many, many thanks for your answer. I'm using openldap, and I was almost sure that there was such a parameter to change. I had already tested that the request was arriving to openldap and didn't saw anything, so I though that the problem was before that.
In fact, there is a little message "ber_get_next on fd 10 failed errno=34 (Numerical result out of range)" which only appears in high debug and was lost in the middle of everything.
With that in hand, it was easier to find that the correct parameter is 'sockbuf_max_incoming_auth':

sockbuf_max_incoming_auth <integer>
    Specify the maximum incoming LDAP PDU size for authenticated sessions. The default is 4194303.

Inceasing that parameter in slapd.conf made the deal.

Many thanks to have forced me to double (decuple, actually) check!