Paging not working as documented

[driverjb]

I am working with a large directory at my company (40000+ accounts). I have a group that has a very large number of members (10000+). I'm following the documented procedure (http://ldapjs.org/client.html) for paging.

I've wrapped the search function in a Promise for ease of use. Here's an excerpt from my function. If you don't see it defined, assume it's provided by the function:

let params = { filter: `(sAMAccountName=${samAccountName})`, scope 'sub', paged: true, sizeLimit: 500 }
return new Promise<SearchResult[]>((resolve, reject) => {
  this.client.search(searchBase, params, (error, response) => {
    if (error) return reject(error);
    response.on('page', () => {
      console.log('page end');
    });
    response.on('searchEntry', (entry: SearchResult) => {
      // definitions are not yet available for 3.*
      //@ts-ignore
      results.push(entry.pojo);
    });
    response.once('end', () => {
      return resolve(results);
    });
  });
});

When I execute the search for my group that has a memberOf field containing 10000+ members, I see the "page end" text print once and then I get a result containing a field called member;range=0-1499. First off, I set the pageSize to 500, not 1500. Second, I know for a fact that that there are more than 1500 group members. In the past, I used the manual method of adjusting the ldap calls to use that attribute, parse it, and walk it out until it came back with a '*' in the range, i.e. member;range=1500-2999, member;range=3000-*. This works but I had to write a custom recursive function to do it, and it's even more complicated when you have nested groups that you need to flatten into members who are users. Paging doesn't appear to do anything except tell you that there is a page.

What am I missing?

[jsumners]

Please see #886. It looks to me like the paged option is working correctly.

sizeLimit is specifically tied to the sizeLimit option of the search request. It does not set the maximum number of results per search page. If you wish to manage how many results are returned per page, see

node-ldapjs/lib/client/client.js

Lines 618 to 629 in a37daf1

	if (options.paged) {
	// Perform automated search paging
	const pageOpts = typeof (options.paged) === 'object' ? options.paged : {}
	let size = 100 // Default page size
	if (pageOpts.pageSize > 0) {
	size = pageOpts.pageSize
	} else if (options.sizeLimit > 1) {
	// According to the RFC, servers should ignore the paging control if
	// pageSize >= sizelimit. Some might still send results, but it's safer
	// to stay under that figure when assigning a default value.
	size = options.sizeLimit - 1
	}

[driverjb]

Okay I see that. However, that still doesn't really answer how to get all the search results using paging.

[jsumners]

The test shows how to collect the results. If you want to reduce memory, then you'd use the page event to know when the end of a page has been encountered and manage your in-memory storage accordingly.

node-ldapjs/lib/client/search_pager.js

Lines 16 to 22 in a37daf1

	* 1. page - Emitted whenever the end of a result page is encountered.
	* If this is the last page, 'end' will also be emitted.
	* The event passes two arguments:
	* 1. The result object (similar to 'end')
	* 2. A callback function optionally used to continue the search
	* operation if the pagePause option was specified during
	* initialization.

[driverjb]

Thank you for that! I was writing the code correctly, but I wasn't understanding my LDAP search properly. Rather than expect it to automatically page the large "memberOf" attribute of the single group that is returned, I need to run two searches. One to get the group data, and a second search to get the entire "memberOf" list for that group. In that case, the paging worked as expected.

Thanks again!