LDAP Sync: final batch not processed (without pagination)

[sscholl]

Description:

I use LDAP without pagination (by setting LDAPP_Search_Page_Size to 0), because my LDAP Server is not able to provide pragination. When I use the LDAP Sync Feature (by hand or by cronjob), 500 users are imported at maximum. I guess this is because of the missing pagination feature, but my LDAP sends all ~800 users.

Can you increase the default value or make it configurable?

Thank you!!!

Server Setup Information:

• Version of Rocket.Chat Server: 0.60.1
• Database Migration: 105
• Operating System: linux 3.10.0-693.el7.x86_64 x64
• Deployment Method(snap/docker/tar/etc): docker
• Number of Running Instances: 1
• Node Version: v8.9.3

Steps to Reproduce:

1. Configure LDAP with more than 500 users
2. set LDAPP_Search_Page_Size to 0
3. set Search_Size_Limit to 1000
4. Execute LDAP_Sync_Now

Expected behavior:

All LDAP Users shoud be imported (more than 500).

Actual behavior:

Only 500 users are imported.
After log message LDAPSync ➔ info Import running. Users imported until now: 500 import stops.

Relevant logs:

20171229-11:58:44.846(0) rocketchat_logger rocketchat_logger.js:278 LDAPSync ➔ info Import running. Users imported until now: 500

[rodrigok]

@sscholl There is no hard limit in our code.

Was it working before version 0.60? Could it be a hard search limit in your LDAP server configuration?

[rodrigok]

We tested imports with more then 200.000 users with and without pagination.

[sscholl]

@rodrigok Thanks for your answer! I can see, that my LDAP is returning ~800 users, so my guess here is Rocket.Chat. I see the value 500 for example here in the source code:

Rocket.Chat/packages/rocketchat-ldap/server/ldap.js

Line 389 in e2c0fc9

const internalPageSize = options.paged && options.paged.pageSize > 0 ? options.paged.pageSize * 2 : 500;

Or is it possible, that I have a problem with asynchronous jobs?

EDIT: I did not test it before 0.60. Thats my first time.
EDIT2: I was able to import the users by using setting LDAP_Search_Page_Size to 10000 and Search_Size_Limit 100000. But this gave me the missing pagination error again. However initially the users are present now. Further automatic Full-Syncs still not possible - of course, I disabled pagination again.

PagedError: missing paged control
  at SearchPager._onEnd (/app/bundle/programs/server/npm/node_modules/ldapjs/lib/client/search_pager.js:91:15)

[rodrigok]

@sscholl That's the internal batch control, is not related with LDAP pagination.

But I found the error, it's not processing the final results of the final batch, I'll fix ASAP, release a new version and let you know.

Thanks

[sscholl]

Very nice!!! Thank you :)