eg apps list and eg users list return incomplete results or empty

[Tiberriver256]

It seems that the SCAN in the findAll method for both the application dao and the user dao only does a single call instead of a full iteration which works when testing but not when your database grows outside of the default query range for SCAN.

The SCAN should be called again and again until the return cursor is 0 to guarantee the search is complete (called a 'full iteration'). From the docs:

As you can see the SCAN return value is an array of two values: the first value is the new cursor to use in the next call, the second value is an array of elements.

Since in the second call the returned cursor is 0, the server signaled to the caller that the iteration finished, and the collection was completely explored. Starting an iteration with a cursor value of 0, and calling SCAN until the returned cursor is 0 again is called a full iteration.

I will put in a PR for what is working as a fix in our environment.

[XVincentX]

Hey @Tiberriver256,

We do not return the whole set from the Admin API on purpose. In case of a huge list of users or applications, this would blow up the gateway. Given the Admin API and the gateway itself live in the same process, we do want to minimize it's impact as much as possible.

That's why we leverage the Redis cursors to paginate through the result.

Basically, for you, this means:

• Make your first call to the Admin API:
  curl http://localhost:9876/users

• Grab the nextKey value from the response:

{"users":[],"nextKey":0}

• Use it in the following user call:

curl http://localhost:9876/users?start=nextKeyValue

That should fulfill your use case.
I think we do not have the feature properly documented in our website. I'm going to check and open an issue there if it's the case.

In case it's not or you need more clarification, feel free to reopen the issue or jump in our Gitter channel.

Thanks!

[Tiberriver256]

Ah, so maybe this is a bug in the cli consuming the API then?

[XVincentX]

This might be. I'll reopen the issue!

[Tiberriver256]

Ah, I think I found it. This should be checking for a nextKey then and calling again if there is one correct?

express-gateway/admin/users.js

Lines 35 to 42 in 76d32c1

	list (params) {
	return client
	.get(baseUrl)
	.query(params)
	.then(res => {
	return res.body;
	});
	},

[XVincentX]

Correct!

[XVincentX]

@Tiberriver256 I proposed a solution for this problem in #762 pull request. If you have some time I'd appreciate your input there. Thanks!

[Tiberriver256]

That works for me. Thank you sir.

[altsang]

@XVincentX has this been reviewed? been so long

[XVincentX]

The pull request has been reviewed some time ago. I addressed all the comments and pinging Kevin for another round of review — but no answer yet.