Directory_ExpiredPageToken when requesting next page with $filter request option

[jansohn]

I'm getting a Directory_ExpiredPageToken error message when using a $filter option for assignedLicenses and requesting the next page results. If I remove the $filter option it does not throw the error.
As a work-around it seems to work if one just tries often enough but this creates unnecessary traffic load and slows down the whole process.

This has also been reported on StackOverflow already without any answers so far:
https://stackoverflow.com/questions/60047153/microsoft-graph-api-users-endpoint-with-filter-provides-nextlink-token-expired
Same on Microsoft UserVoice:
https://microsoftgraph.uservoice.com/forums/920506-microsoft-graph-feature-requests/suggestions/15813055-allow-complex-query-for-assignedlicenses

Expected behavior

Requesting the next page results returns the next page results.

Actual behavior

Requesting the next page results throws a Directory_ExpiredPageToken error.

Debug log:

2020-05-06 12:32:02,185 DEBUG [main] com.microsoft.graph: Created DefaultExecutors
2020-05-06 12:32:02,225 DEBUG [main] com.microsoft.graph: Created DefaultSerializer
2020-05-06 12:32:02,225 DEBUG [main] com.microsoft.graph: Created CoreHttpProvider
2020-05-06 12:32:02,252 DEBUG [main] com.microsoft.graph: Starting to send request, URL https://graph.microsoft.com/v1.0/users?$filter=assignedLicenses/any(x:x/skuId+eq+53818b1b-4a27-454b-8896-0dba576410e6)&$select=userPrincipalName,mail,onPremisesSamAccountName,displayName,givenName,accountEnabled,assignedLicenses&$top=999
2020-05-06 12:32:02,297 DEBUG [main] com.microsoft.graph: Request Method GET
2020-05-06 12:32:06,284 DEBUG [main] com.microsoft.graph: Response code 200, OK
2020-05-06 12:32:06,285 DEBUG [main] com.microsoft.graph: Response json
2020-05-06 12:32:06,672 DEBUG [main] com.microsoft.graph: Deserializing type UserCollectionResponse
2020-05-06 12:32:06,714 DEBUG [main] com.microsoft.graph: Starting to send request, URL https://graph.microsoft.com/v1.0/users?$filter=assignedLicenses%2fany(x%3ax%2fskuId+eq+53818b1b-4a27-454b-8896-0dba576410e6)&$select=userPrincipalName%2cmail%2conPremisesSamAccountName%2cdisplayName%2cgivenName%2caccountEnabled%2cassignedLicenses&$top=999&$skiptoken=X%27xxx%27
2020-05-06 12:32:06,715 DEBUG [main] com.microsoft.graph: Request Method GET
2020-05-06 12:32:06,918 DEBUG [main] com.microsoft.graph: Response code 400, Bad Request
2020-05-06 12:32:06,920 DEBUG [main] com.microsoft.graph: Handling error response
2020-05-06 12:32:06,927 DEBUG [main] com.microsoft.graph: Deserializing type GraphErrorResponse
2020-05-06 12:32:06,929 ERROR [main] com.microsoft.graph: Graph service exception Error code: Directory_ExpiredPageToken
Error message: The specified page token value has expired and can no longer be included in your request.

GET https://graph.microsoft.com/v1.0/users?$filter=assignedLicenses%2fany(x%3ax%2fskuId+eq+53818b1b-4a27-454b-8896-0dba576410e6)&$select=userPrincipalName%2cmail%2conPremisesSamAccountName%2cdisplayName%2cgivenName%2caccountEnabled%2cassignedLicenses&$top=999&$skiptoken=X%27xxx%27
SdkVersion : graph-java/v1.7.1
Authorization : Bearer xxx


400 : Bad Request
Strict-Transport-Security : max-age=31536000
Cache-Control : private
x-ms-ags-diagnostic : {"ServerInfo":{"DataCenter":"North Central US","Slice":"SliceC","Ring":"3","ScaleUnit":"003","RoleInstance":"AGSFE_IN_72"}}
client-request-id : 6e3f0a33-0f12-4439-9d5c-fcdfd1f5482b
request-id : 438b6243-fc06-48fc-8fbf-bca3bd52b9ce
Content-Length : 301
Date : Wed, 06 May 2020 10:32:06 GMT
Content-Type : application/json
{
  "error": {
    "code": "Directory_ExpiredPageToken",
    "message": "The specified page token value has expired and can no longer be included in your request.",
    "innerError": {
      "request-id": "438b6243-fc06-48fc-8fbf-bca3bd52b9ce",
      "date": "2020-05-06T10:32:06"
    }
  }
}

com.microsoft.graph.http.GraphServiceException: Error code: Directory_ExpiredPageToken
Error message: The specified page token value has expired and can no longer be included in your request.

GET https://graph.microsoft.com/v1.0/users?$filter=assignedLicenses%2fany(x%3ax%2fskuId+eq+53818b1b-4a27-454b-8896-0dba576410e6)&$select=userPrincipalName%2cmail%2conPremisesSamAccountName%2cdisplayName%2cgivenName%2caccountEnabled%2cassignedLicenses&$top=999&$skiptoken=X%27xxx%27
SdkVersion : graph-java/v1.7.1
Authorization : Bearer xxx


400 : Bad Request
Strict-Transport-Security : max-age=31536000
Cache-Control : private
x-ms-ags-diagnostic : {"ServerInfo":{"DataCenter":"North Central US","Slice":"SliceC","Ring":"3","ScaleUnit":"003","RoleInstance":"AGSFE_IN_72"}}
client-request-id : 6e3f0a33-0f12-4439-9d5c-fcdfd1f5482b
request-id : 438b6243-fc06-48fc-8fbf-bca3bd52b9ce
Content-Length : 301
Date : Wed, 06 May 2020 10:32:06 GMT
Content-Type : application/json
{
  "error": {
    "code": "Directory_ExpiredPageToken",
    "message": "The specified page token value has expired and can no longer be included in your request.",
    "innerError": {
      "request-id": "438b6243-fc06-48fc-8fbf-bca3bd52b9ce",
      "date": "2020-05-06T10:32:06"
    }
  }
}

	at com.microsoft.graph.http.GraphServiceException.createFromConnection(GraphServiceException.java:446) ~[microsoft-graph-1.7.1.jar:?]
	at com.microsoft.graph.http.CoreHttpProvider.handleErrorResponse(CoreHttpProvider.java:485) ~[microsoft-graph-1.7.1.jar:?]
	at com.microsoft.graph.http.CoreHttpProvider.sendRequestInternal(CoreHttpProvider.java:375) [microsoft-graph-1.7.1.jar:?]
	at com.microsoft.graph.http.CoreHttpProvider.send(CoreHttpProvider.java:204) [microsoft-graph-1.7.1.jar:?]
	at com.microsoft.graph.http.CoreHttpProvider.send(CoreHttpProvider.java:184) [microsoft-graph-1.7.1.jar:?]
	at com.microsoft.graph.http.BaseCollectionRequest.send(BaseCollectionRequest.java:89) [microsoft-graph-1.7.1.jar:?]
	at com.microsoft.graph.requests.extensions.UserCollectionRequest.get(UserCollectionRequest.java:52) [microsoft-graph-1.7.1.jar:?]
	at com.test.MicrosoftGraphIT.testGraphCall(MicrosoftGraphIT.java:119) [test-classes/:?]
	at com.test.MicrosoftGraphIT$Proxy$_$$_WeldClientProxy.testGraphCall(Unknown Source) [test-classes/:?]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_181]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_181]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_181]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_181]
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) [junit-4.12.jar:4.12]
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) [junit-4.12.jar:4.12]
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) [junit-4.12.jar:4.12]
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) [junit-4.12.jar:4.12]
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12]
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) [junit-4.12.jar:4.12]
	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) [junit-4.12.jar:4.12]
	at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.12.jar:4.12]
	at org.jglue.cdiunit.CdiRunner$2.evaluate(CdiRunner.java:200) [cdi-unit-4.1.0.jar:?]
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) [junit-4.12.jar:4.12]
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) [junit-4.12.jar:4.12]
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) [junit-4.12.jar:4.12]
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) [junit-4.12.jar:4.12]
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) [junit-4.12.jar:4.12]
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) [junit-4.12.jar:4.12]
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) [junit-4.12.jar:4.12]
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) [junit-4.12.jar:4.12]
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363) [junit-4.12.jar:4.12]
	at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:89) [.cp/:?]
	at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:41) [.cp/:?]
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:542) [.cp/:?]
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:770) [.cp/:?]
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:464) [.cp/:?]
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:210) [.cp/:?]

Steps to reproduce the behavior

Eclipse 2019-12, microsoft-graph 1.7.1, msal4j 1.4.0, Oracle JDK 8u181

Example code:

List<Option> requestOptions = Collections.singletonList(new QueryOption("$filter",
        "assignedLicenses/any(x:x/skuId eq 53818b1b-4a27-454b-8896-0dba576410e6)"));

IUserCollectionPage users = graphClient.users()
        .buildRequest(requestOptions)
        .select("userPrincipalName,mail,onPremisesSamAccountName,displayName,givenName,accountEnabled,assignedLicenses")
        .top(999)
        .get();

int totalUsers = 0;
while (users.getCurrentPage().size() > 0) {
    totalUsers += users.getCurrentPage().size();
    users = users.getNextPage().buildRequest().get();
}
log.info("User count: {}", totalUsers);

[baywet]

Thank you for your question and your patience on the matter.
This was a temporary service issue which has been fixed since.
A few notes about your code:

IUserCollectionPage users = graphClient.users()
        .buildRequest()
        .filter("assignedLicenses/any(x:x/skuId eq 53818b1b-4a27-454b-8896-0dba576410e6)") //version 2.2 of the SDK will allow you to do this
        .select("userPrincipalName,mail,onPremisesSamAccountName,displayName,givenName,accountEnabled,assignedLicenses")
        .top(999)
        .get();

int totalUsers = 0;
while (users.getCurrentPage().size() > 0) {
    totalUsers += users.getCurrentPage().size();
   final IUserCollectionRequestBuilder nextPage = users.getNextPage();
   if(nextPage != null) { //nextPage will be null when you get to the end, so you should check for that.
	users = nextPage.buildRequest().get();
   } else {
	break;
   }
}
log.info("User count: {}", totalUsers);

Closing as it's a fairly old question. Feel free to comment/ask to re-open if required.

[jansohn]

@baywet please re-open, we are still seeing this issue with microsoft-graph 2.4.1 and msal4j 1.8.0. This has not been fixed...

Edit: I've changed to your proposed .filter() method without any changes in the behavior.

[baywet]

@jansohn What happens if you remove the top statement? (it'll result in more requests, but those should return faster)
How many object (users, groups, applications...) are present in this tenant?
Have you contacted support for that issue? It looks like it might be a data related issue with a large number of objects?
The filter change it simply to make the code easier to write and maintain, no behavior change is expected.

[jansohn]

• no change when removing the top statement
• 164113 users, 138705 groups (I'm not allowed to count the number of applications...)
• I haven't contacted support so far except by opening this GitHub issue
• Search result size is about 5600 users, nothing special in my opinion

[baywet]

Thanks for providing the numbers. This qualifies as a large tenant and probably explains why the service is not behaving like it should. FYI this is behaving properly on a "regular developer tenant".
Follow up questions:

• is the sequence always failing, only sometimes, only rarely?
• if the sequence completes on some instances, how long does it take to complete?

I'm going to transfer that issue into our service issues repository for visibility. Can you contact support please?

[jansohn]

• it's always failing to retrieve the results
• we currently retry the calls until they finally finish which takes about 3 minutes

Where/How do I contact the official support?

[baywet]

thanks for the additional information, see the two boxes at the bottom of this page. It's important that you contact support from the tenant that's experiencing the issue, so the support team can access the logs of that tenant.

[jansohn]

Seems like I can only open a ticket in the Azure Admin Center concerning billing and subscription management which both do not seem to fit...

[baywet]

A tenant this large should have a premier support contract and you (or a tenant admin) should be able to open a ticket through the Office 365 admin portal.

[jansohn]

Yes, I'm "just" a developer so I do not have access to the admin portal. I'll see what I can do but would appreciate it if this can be looked at either way.