LdapException: Size Limit Exceeded (4) #139

sphaero · 2023-02-07T15:16:33Z

It seems this plugin is querying the full ldap database and tries to compare a username to the returned list of users? Correct?

jellyfin-plugin-ldapauth/LDAP-Auth/LDAPAuthenticationProviderPlugin.cs

Lines 382 to 396 in 3a5b07b

    
           while (ldapUsers.HasMore() && foundUser == false) 
        
           { 
        
               var currentUser = ldapUsers.Next(); 
        
               foreach (var attr in LdapUsernameAttributes) 
        
               { 
        
                   var toCheck = GetAttribute(currentUser, attr); 
        
                   if (toCheck?.StringValueArray != null) 
        
                   { 
        
                       foreach (var name in toCheck.StringValueArray) 
        
                       { 
        
                           if (string.Equals(username, name, usernameComparison)) 
        
                           { 
        
                               ldapUser = currentUser; 
        
                               foundUser = true; 
        
                               break;

If that's the case I don't think this how this is supposed to be done. One should do an ldap search for the username attribute matching the username. If that returns a hit one can continue.

Am I missing something here? An ldap administrator wouldn't be happy with these queries I guess?

I would help with the code but I'm no C# dev so I would need help setting up a test env?

sphaero · 2023-02-07T15:17:25Z

Related to #39 ???

joshuaboniface · 2023-02-07T15:38:34Z

As far as I can recall, this was done because the LDAP library we were using (the only one I'm aware of, or was at the time, for .NET/C#) didn't allow per-user queries.

It could probably be chunked if required, or if it can do searches for the single user that's definitely ideal.

Test env shouldn't be too complicated: you just need the dotnet-sdk package and then run dotnet build in the root of the repo to get a plugin file out.

sphaero · 2023-02-07T20:20:44Z

Is that this library? https://github.com/dsbenghe/Novell.Directory.Ldap.NETStandard

I'm no LDAP ninja but I think there are multiple ways to verify users.

First is to just use a ldap bind call with the user credentials. I think this example does that: https://github.com/dsbenghe/Novell.Directory.Ldap.NETStandard/blob/master/original_samples/Samples/Bind.cs
the other is to either do an anonymous bind or some user bind and search for the user credentials to compare them. I think this example does that: https://github.com/dsbenghe/Novell.Directory.Ldap.NETStandard/blob/master/original_samples/Samples/VerifyPassword.cs

ParhamGhafouri · 2023-02-28T09:41:06Z

Greetings,

I have this problem on my organization with more than 1000 users to query from LDAP server and still facing "Base Search: Size Limit Exceeded" and "Failure: User Filter: Size Limit Exceeded". was wonder if there is any workaround to fix this problem.

as side note I did change Page Size limit on my LDAP which is active directory but still no luck .

sphaero · 2023-02-28T09:51:54Z

I posted links to the LDAP library of how to do the ldap calls correctly. This needs to be done in the plugin code. If you're familiar with C# a PR is very welcomed I think!

ParhamGhafouri · 2023-02-28T10:13:41Z

Greetings Sphaero,

Thanks for your respond , unfortunately I don't have that knowledge and therefor can't implement those change into source code, would be nice if you could do us a favor and fix this problem as I've been using this plugin lately and due to increase users that access our media servers we are facing this problem right now.

With kind regards, Parham

h3ge · 2023-02-28T10:14:46Z

It seems this plugin is querying the full ldap database and tries to compare a username to the returned list of users? Correct?

That's not directly true. It depends on the search filter. In the default Configuration the SearchFilter is "(memberOf=CN=JellyfinUsers,DC=contoso,DC=com)"

If you want to work with huge amount of users permitted to access jellyfin, it would be better to use the username in the SearchFilter.

(&(memberOf:1.2.840.113556.1.4.1941:=CN=MyUserGroup,DC=Groups,DC=dummy,DC=local)(objectClass=user)(sAMAccountName=%ESCAPEDUSERNAME%))

A way could be, that the SearchFilter is dynamically build depending on the login attributes like that

SearchFilter = "(&" + SearchFilter + "(objectClass=user)(" + UsernameAttr + "=" + username + "))"

Or maybe

            ILdapSearchResults ldapUsers;
            try
            {

                if (SearchFilter.Contains("##USERNAME##")) {
                	SearchFilter = string.Replace("##USERNAME##",username);
                }
                ldapUsers = ldapClient.Search(
                    LdapPlugin.Instance.Configuration.LdapBaseDn,
                    LdapConnection.ScopeSub,
                    SearchFilter,
                    LdapUsernameAttributes,
                    false);
            }

so that users are be able to use advanced ldap filtering techniques?

sphaero · 2023-02-28T10:58:30Z

That might work as well. Are you saying this is already implemented or that this could be easily implemented?

ParhamGhafouri · 2023-02-28T14:46:40Z

Greetings,

This not implemented as far as I know , I tried to modify LDAP query but still no luck with it.

joshuaboniface · 2023-02-28T14:51:09Z

Is that this library? dsbenghe/Novell.Directory.Ldap.NETStandard

Yes, that is the correct library.

I'm no LDAP ninja but I think there are multiple ways to verify users.

* First is to just use a ldap bind call with the user credentials. I think this example does that: [dsbenghe/Novell.Directory.Ldap.NETStandard@`master`/original_samples/Samples/Bind.cs](https://github.com/dsbenghe/Novell.Directory.Ldap.NETStandard/blob/master/original_samples/Samples/Bind.cs?rgh-link-date=2023-02-07T20%3A20%3A44Z)

* the other is to either do an anonymous bind or some user bind and search for the user credentials to compare them. I think this example does that: [dsbenghe/Novell.Directory.Ldap.NETStandard@`master`/original_samples/Samples/VerifyPassword.cs](https://github.com/dsbenghe/Novell.Directory.Ldap.NETStandard/blob/master/original_samples/Samples/VerifyPassword.cs?rgh-link-date=2023-02-07T20%3A20%3A44Z)

The problem with both examples, that we ran into, is that they seem to require knowing the dn in advance. The library, at least as far as I recall from when we made this in early 2019, did not have an easy way to search for a particular attribute. Perhaps that has changed, but this is why we do the full dump and then search.

It's probably worth revisiting, I'll see if I can spend some time on this over the next few days.

h3ge · 2023-02-28T15:01:39Z

I tried to modify LDAP query but still no luck with it.

What query did you use?
I use LDAP very heavy in small to very large environments, maybe I can help with this.

ParhamGhafouri · 2023-02-28T15:10:53Z

I would like to allow users that are member of ALL-Media group able to login into media server , this group is about 5000 members , this is what I'm using right now :

(memberOf=CN=Org-AllMedia,OU=Departments,OU=Security,OU=Domain Groups,DC=local,DC=com)

h3ge · 2023-02-28T15:36:53Z

Ah ok, I thought you already tried to adjust the plugin code accordingly as I described in one of my previous posts. - Since the LDAP filter currently doesn't have any info about which user is logging in, no further restriction can be made here.

joshuaboniface · 2023-02-28T16:36:14Z

This actually ended up being fairly easy. It just wasn't obvious how best to use the Search() method.

I have an improved search system incoming.

joshuaboniface · 2023-02-28T17:37:16Z

And, it's not, because I was working from a much older version of the plugin that didn't try to do stuff like case-insensitive name searching. That will never work with a filter-based approach and will need to be removed.

The previous implementation would search for all users that potentially matched the (mandatory) configured search filter, then loop through the results to find a user matching one of the attributes. This method worked fine for small instances, but became unwieldy for larger LDAP instances, and could even result in a Size Limit Exceeded error for extremely large databases. This commit introduces a better method by integrating the user search component into the initial LDAP search by way of the search filter. This will ideally result in only a single LDAP search result for a given user rather than potentially dozens or hundreds, reducing both the runtime complexity of the plugin as well as the load on the LDAP server. This new method has two potential ways of generating the limited search filter: First, the existing search filter option may now accept a "{username}" variable within it, which will be interpolated at runtime based on what the user entered. This method allows for very complex queries to be crafted at will, providing better administrator flexibility and options. Second, if no "{username}" variable is found within the search filter, the filter will be modified at runtime to include all of the search attributes combined with the username to generate a search query which will return any matches between the username and those queries. For example, given attributes "uid" and "mail", a (base) search filter of "(objectclass=mailUser)", and the entered username "joshua", the following "real" search query would be produced: (&(objectclass=mailUser)(|(uid=joshua)(mail=joshua))) These two methods are, functionally, mutually exclusive: if the search filter contains the "{username}" variable, the search attributes will be ignored and may be blank/empty; on the other side, with valid search attributes, it is no longer necessary to specify a search filter at all, since there will still be a filter on the attributes and username. Thus, the plugin now accepts blank input for both fields in the configuration, though leaving both blank would not work properly. In both cases with the new method, searches are case-insensitive, so the option for case-insensitive username searches has been removed. The new configuration is also backwards-compatible: if no changes are made to the search filter, the second method above will be used and this should continue to function exactly as expected. Some debug messages have also been updated to provide a clearer picture of what the plugin is doing at various steps and aid in troubleshooting. Closes jellyfin#139 jellyfin#34 Obsoletes PR jellyfin#71

h3ge · 2023-02-28T19:23:14Z

And, it's not, because I was working from a much older version of the plugin that didn't try to do stuff like case-insensitive name searching. That will never work with a filter-based approach and will need to be removed.

Why? Ldap search is case insensitive in most cases. (as long it's not a IA5String), usually you need to work with matching rules if you want it case sensitive.

The previous implementation would search for all users that potentially matched the (mandatory) configured search filter, then loop through the results to find a user matching one of the attributes. This method worked fine for small instances, but became unwieldy for larger LDAP instances, and could even result in a Size Limit Exceeded error for extremely large databases. This commit introduces a better method by integrating the user search component into the initial LDAP search by way of the search filter. This will ideally result in only a single LDAP search result for a given user rather than potentially dozens or hundreds, reducing both the runtime complexity of the plugin as well as the load on the LDAP server. This new method has two potential ways of generating the limited search filter: First, the existing search filter option may now accept a "{username}" variable within it, which will be interpolated at runtime based on what the user entered. This method allows for very complex queries to be crafted at will, providing better administrator flexibility and options. Second, if no "{username}" variable is found within the search filter, the filter will be modified at runtime to include all of the search attributes combined with the username to generate a search query which will return any matches between the username and those queries. For example, given attributes "uid" and "mail", a (base) search filter of "(objectclass=mailUser)", and the entered username "joshua", the following "real" search query would be produced: (&(objectclass=mailUser)(|(uid=joshua)(mail=joshua))) These two methods are, functionally, mutually exclusive: if the search filter contains the "{username}" variable, the search attributes will be ignored and may be blank/empty; on the other side, with valid search attributes, it is no longer necessary to specify a search filter at all, since there will still be a filter on the attributes and username. Thus, the plugin now accepts blank input for both fields in the configuration, though leaving both blank would not work properly. In both cases with the new method, searches are case-insensitive, so the option for case-insensitive username searches has been removed. The new configuration is also backwards-compatible: if no changes are made to the search filter, the second method above will be used and this should continue to function exactly as expected. Some debug messages have also been updated to provide a clearer picture of what the plugin is doing at various steps and aid in troubleshooting. Closes jellyfin#139 jellyfin#34 Obsoletes PR jellyfin#71

The previous implementation would search for all users that potentially matched the (mandatory) configured search filter, then loop through the results to find a user matching one of the attributes. This method worked fine for small instances, but became unwieldy for larger LDAP instances, and could even result in a Size Limit Exceeded error for extremely large databases. This commit introduces a better method by integrating the user search component into the initial LDAP search by way of the search filter. This will ideally result in only a single LDAP search result for a given user rather than potentially dozens or hundreds, reducing both the runtime complexity of the plugin as well as the load on the LDAP server. This new method has two potential ways of generating the limited search filter: First, the existing search filter option may now accept a "{username}" variable within it, which will be interpolated at runtime based on what the user entered. This method allows for very complex queries to be crafted at will, providing better administrator flexibility and options. Second, if no "{username}" variable is found within the search filter, the filter will be modified at runtime to include all of the search attributes combined with the username to generate a search query which will return any matches between the username and those queries. For example, given attributes "uid" and "mail", a (base) search filter of "(objectclass=mailUser)", and the entered username "joshua", the following "real" search query would be produced: (&(objectclass=mailUser)(|(uid=joshua)(mail=joshua))) These two methods are, functionally, mutually exclusive: if the search filter contains the "{username}" variable, the search attributes will be ignored and may be blank/empty; on the other side, with valid search attributes, it is no longer necessary to specify a search filter at all, since there will still be a filter on the attributes and username. Thus, the plugin now accepts blank input for both fields in the configuration, though leaving both blank would not work properly. In both cases with the new method, searches are case-insensitive, so the option for case-insensitive username searches has been removed. The new configuration is also backwards-compatible: if no changes are made to the search filter, the second method above will be used and this should continue to function exactly as expected. Some debug messages have also been updated to provide a clearer picture of what the plugin is doing at various steps and aid in troubleshooting. An explanation of the two methods above is included in the plugin configuration page, along with some rearranging of the options, to assist users in configuring the two fields the way they want. Closes jellyfin#139 jellyfin#34 Obsoletes PR jellyfin#71

The previous implementation would search for all users that potentially matched the (mandatory) configured search filter, then loop through the results to find a user matching one of the attributes. This method worked fine for small instances, but became unwieldy for larger LDAP instances, and could even result in a Size Limit Exceeded error for extremely large databases. This commit introduces a better method by integrating the user search component into the initial LDAP search by way of the search filter. This will ideally result in only a single LDAP search result for a given user rather than potentially dozens or hundreds, reducing both the runtime complexity of the plugin as well as the load on the LDAP server. This new method has two potential ways of generating the limited search filter: First, the existing search filter option may now accept a "{username}" variable within it, which will be interpolated at runtime based on what the user entered. This method allows for very complex queries to be crafted at will, providing better administrator flexibility and options. Second, if no "{username}" variable is found within the search filter, the filter will be modified at runtime to include all of the search attributes combined with the username to generate a search query which will return any matches between the username and those attributes. For example, given attributes "uid" and "mail", a (base) search filter of "(objectclass=mailUser)", and the entered username "joshua", the following "real" search query would be produced: (&(objectclass=mailUser)(|(uid=joshua)(mail=joshua))) These two methods are, functionally, mutually exclusive: if the search filter contains the "{username}" variable, the search attributes will be ignored and may be blank/empty; on the other side, with valid search attributes, it is no longer necessary to specify a search filter at all, since there will still be a filter on the attributes and username. Thus, the plugin now accepts blank input for both fields in the configuration, though leaving both blank would not work properly. In both cases with the new method, searches are case-insensitive, so the option for case-insensitive username searches has been removed. The new configuration is also backwards-compatible: if no changes are made to the search filter, the second method above will be used and this should continue to function exactly as expected. Some debug messages have also been updated to provide a clearer picture of what the plugin is doing at various steps and aid in troubleshooting. An explanation of the two methods above is included in the plugin configuration page, along with some rearranging of the options, to assist users in configuring the two fields the way they want. Closes jellyfin#139 jellyfin#34 Obsoletes PR jellyfin#71

The previous implementation would search for all users that potentially matched the (mandatory) configured search filter, then loop through the results to find a user matching one of the attributes. This method worked fine for small instances, but became unwieldy for larger LDAP instances, and could even result in a Size Limit Exceeded error for extremely large databases. This commit introduces a better method by integrating the user search component into the initial LDAP search by way of the search filter. This will ideally result in only a single LDAP search result for a given user rather than potentially dozens or hundreds, reducing both the runtime complexity of the plugin as well as the load on the LDAP server. This new method has two potential ways of generating the limited search filter: First, the existing search filter option may now accept a "{username}" variable within it, which will be interpolated at runtime based on what the user entered. This method allows for very complex queries to be crafted at will, providing better administrator flexibility and options. Second, if no "{username}" variable is found within the search filter, the filter will be modified at runtime to include all of the search attributes combined with the username to generate a search query which will return any matches between the username and those attributes. For example, given attributes "uid" and "mail", a (base) search filter of "(objectclass=mailUser)", and the entered username "joshua", the following "real" search query would be produced: (&(objectclass=mailUser)(|(uid=joshua)(mail=joshua))) These two methods are, functionally, mutually exclusive: if the search filter contains the "{username}" variable, the search attributes will be ignored and may be blank/empty; on the other side, with valid search attributes, it is no longer necessary to specify a search filter at all, since there will still be a filter on the attributes and username. Thus, the plugin now accepts blank input for both fields in the configuration, though leaving both blank would not work properly. In both cases with this change, searches are case-insensitive due to the case-insensitivity of LDAP search queries,, so the option for case-insensitive username searches has been removed as obsolete. The new configuration is also backwards-compatible: if no changes are made to the search filter, the second method above will be used and this should continue to function exactly as expected. Some debug messages have also been updated to provide a clearer picture of what the plugin is doing at various steps and aid in troubleshooting. An explanation of the two methods above is included in the plugin configuration page, along with some rearranging of the options, to assist users in configuring the two fields the way they want. Closes jellyfin#139 jellyfin#34 Obsoletes PR jellyfin#71

joshuaboniface · 2023-02-28T20:06:30Z

@h3ge Yep, I realized that - have a fairly robust implementation incoming (just ignore those various force-pushes above lol).

The previous implementation would search for all users that potentially matched the (mandatory) configured search filter, then loop through the results to find a user matching one of the attributes. This method worked fine for small instances, but became unwieldy for larger LDAP instances, and could even result in a Size Limit Exceeded error for extremely large databases. This commit introduces a better method by integrating the user search component into the initial LDAP search by way of the search filter. This will ideally result in only a single LDAP search result for a given user rather than potentially dozens or hundreds, reducing both the runtime complexity of the plugin as well as the load on the LDAP server. This new method has two potential ways of generating the limited search filter: First, the existing search filter option may now accept a "{username}" variable within it, which will be interpolated at runtime based on what the user entered. This method allows for very complex queries to be crafted at will, providing better administrator flexibility and options. Second, if no "{username}" variable is found within the search filter, the filter will be modified at runtime to include all of the search attributes combined with the username to generate a search query which will return any matches between the username and those attributes. For example, given attributes "uid" and "mail", a (base) search filter of "(objectclass=mailUser)", and the entered username "joshua", the following "real" search query would be produced: (&(objectclass=mailUser)(|(uid=joshua)(mail=joshua))) These two methods are, functionally, mutually exclusive: if the search filter contains the "{username}" variable, the search attributes will be ignored and may be blank/empty; on the other side, with valid search attributes, it is no longer necessary to specify a search filter at all, since there will still be a filter on the attributes and username. Thus, the plugin now accepts blank input for both fields in the configuration, though leaving both blank would not work properly. In both cases with this change, searches are case-insensitive due to the case-insensitivity of LDAP search queries,, so the option for case-insensitive username searches has been removed as obsolete. The new configuration is also backwards-compatible: if no changes are made to the search filter, the second method above will be used and this should continue to function exactly as expected. Some debug messages have also been updated to provide a clearer picture of what the plugin is doing at various steps and aid in troubleshooting. An explanation of the two methods above is included in the plugin configuration page, along with some rearranging of the options, to assist users in configuring the two fields the way they want. Closes jellyfin#139 jellyfin#34 Obsoletes PR jellyfin#71

ParhamGhafouri · 2023-03-01T04:34:16Z

@h3ge Yep, I realized that - have a fairly robust implementation incoming (just ignore those various force-pushes above lol).

Greetings Joshua,

Glad that you are working to solve this problem. was wonder if you have any ETA on new release.

Regards

sphaero mentioned this issue Feb 28, 2023

LDAP not working up 1.000 users #39

Closed

joshuaboniface mentioned this issue Feb 28, 2023

Rework search flow to minimize LDAP search results #140

Merged

joshuaboniface closed this as completed in #140 Mar 2, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

LdapException: Size Limit Exceeded (4) #139

LdapException: Size Limit Exceeded (4) #139

sphaero commented Feb 7, 2023

sphaero commented Feb 7, 2023

joshuaboniface commented Feb 7, 2023

sphaero commented Feb 7, 2023

ParhamGhafouri commented Feb 28, 2023

sphaero commented Feb 28, 2023

ParhamGhafouri commented Feb 28, 2023

h3ge commented Feb 28, 2023 •

edited

sphaero commented Feb 28, 2023

ParhamGhafouri commented Feb 28, 2023

joshuaboniface commented Feb 28, 2023 •

edited

h3ge commented Feb 28, 2023

ParhamGhafouri commented Feb 28, 2023

h3ge commented Feb 28, 2023

joshuaboniface commented Feb 28, 2023

joshuaboniface commented Feb 28, 2023

h3ge commented Feb 28, 2023

joshuaboniface commented Feb 28, 2023

ParhamGhafouri commented Mar 1, 2023 •

edited

LdapException: Size Limit Exceeded (4) #139

LdapException: Size Limit Exceeded (4) #139

Comments

sphaero commented Feb 7, 2023

sphaero commented Feb 7, 2023

joshuaboniface commented Feb 7, 2023

sphaero commented Feb 7, 2023

ParhamGhafouri commented Feb 28, 2023

sphaero commented Feb 28, 2023

ParhamGhafouri commented Feb 28, 2023

h3ge commented Feb 28, 2023 • edited

sphaero commented Feb 28, 2023

ParhamGhafouri commented Feb 28, 2023

joshuaboniface commented Feb 28, 2023 • edited

h3ge commented Feb 28, 2023

ParhamGhafouri commented Feb 28, 2023

h3ge commented Feb 28, 2023

joshuaboniface commented Feb 28, 2023

joshuaboniface commented Feb 28, 2023

h3ge commented Feb 28, 2023

joshuaboniface commented Feb 28, 2023

ParhamGhafouri commented Mar 1, 2023 • edited

h3ge commented Feb 28, 2023 •

edited

joshuaboniface commented Feb 28, 2023 •

edited

ParhamGhafouri commented Mar 1, 2023 •

edited