MalformedContinuationTokenException starting with SDK v3.7 #1364

codecadwallader · 2020-04-10T15:27:38Z

We are continuously addressing and improving the SDK, if possible, make sure the problem persist in the latest SDK version.

Describe the bug
We are seeing consistent MalformedContinuationTokenException exceptions starting with SDK v3.7 that did not occur with the same query in SDK v3.6. The same issue happens with SDK v3.8 as well.

To Reproduce
This is a cross-partition query over Gateway mode and as we continue to drain more results from the FeedIterator the exception will be thrown on a call to await feed.ReadNextAsync(). It looks like it may work for the first set of results from every physical partition then throws as it goes back to get a second set of results from a physical partition (my guess). Looking at Fiddler there is no error coming back over the wire, it appears to be purely inside the SDK.

Expected behavior
The query to continue to return results as it did with SDK v3.6

Actual behavior
A MalformedContinuationTokenException is thrown with a reason "CompositeContinuationToken is missing field".

Environment summary
SDK Version: v3.7 and v3.8
OS Version: Windows

Additional context
Activity ID examples:
b0e3b405-f7e3-4c22-bf11-0a86fcd2782c
eed47b7d-f177-475b-849b-d0a5a67f4e77

Error message:

Response status code does not indicate success: BadRequest (400); Substatus: 0; ActivityId: ; Reason: (Response status code does not indicate success: BadRequest (400); Substatus: 0; ActivityId: ; Reason: (Response status code does not indicate success: BadRequest (400); Substatus: 0; ActivityId: ; Reason: (Response status code does not indicate success: BadRequest (400); Substatus: 0; ActivityId: ; Reason: (Response status code does not indicate success: BadRequest (400); Substatus: 0; ActivityId: ; Reason: (Response status code does not indicate success: BadRequest (400); Substatus: 0; ActivityId: ; Reason: (Response status code does not indicate success: BadRequest (400); Substatus: 0; ActivityId: ; Reason: (Response status code does not indicate success: BadRequest (400); Substatus: 0; ActivityId: ; Reason: (CompositeContinuationToken is missing field: 'token': {"range":{"min":"05C189CD6732","max":"05C18F5D153C"}});););););););); CompositeContinuationToken is missing field: 'token': {"range":{"min":"05C189CD6732","max":"05C18F5D153C"}}

Call stack:

   at Microsoft.Azure.Cosmos.Query.Core.Monads.TryCatch`1.Try[T](Func`2 onSuccess)
   at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.PipelinedDocumentQueryExecutionContext.TryCreateAsync(ExecutionEnvironment executionEnvironment, CosmosQueryContext queryContext, CrossPartitionInitParams initParams, CosmosElement requestContinuationToken, CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.PipelinedDocumentQueryExecutionContext.TryCreateAsync(ExecutionEnvironment executionEnvironment, CosmosQueryContext queryContext, CrossPartitionInitParams initParams, CosmosElement requestContinuationToken, CancellationToken cancellationToken)
   at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.CosmosQueryExecutionContextFactory.TryCreateSpecializedDocumentQueryExecutionContextAsync(CosmosQueryContext cosmosQueryContext, InputParameters inputParameters, PartitionedQueryExecutionInfo partitionedQueryExecutionInfo, List`1 targetRanges, String collectionRid, CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.CosmosQueryExecutionContextFactory.TryCreateSpecializedDocumentQueryExecutionContextAsync(CosmosQueryContext cosmosQueryContext, InputParameters inputParameters, PartitionedQueryExecutionInfo partitionedQueryExecutionInfo, List`1 targetRanges, String collectionRid, CancellationToken cancellationToken)
   at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.CosmosQueryExecutionContextFactory.TryCreateFromPartitionedQuerExecutionInfoAsync(PartitionedQueryExecutionInfo partitionedQueryExecutionInfo, ContainerQueryProperties containerQueryProperties, CosmosQueryContext cosmosQueryContext, InputParameters inputParameters, CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.CosmosQueryExecutionContextFactory.TryCreateFromPartitionedQuerExecutionInfoAsync(PartitionedQueryExecutionInfo partitionedQueryExecutionInfo, ContainerQueryProperties containerQueryProperties, CosmosQueryContext cosmosQueryContext, InputParameters inputParameters, CancellationToken cancellationToken)
   at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.CosmosQueryExecutionContextFactory.TryCreateCoreContextAsync(CosmosQueryContext cosmosQueryContext, InputParameters inputParameters, CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.CosmosQueryExecutionContextFactory.TryCreateCoreContextAsync(CosmosQueryContext cosmosQueryContext, InputParameters inputParameters, CancellationToken cancellationToken)
   at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.CosmosQueryExecutionContextFactory.<>c__DisplayClass2_0.<Create>b__1(CancellationToken innerCancellationToken)
   at Microsoft.Azure.Cosmos.Query.Core.AsyncLazy`1.GetValueAsync(CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Microsoft.Azure.Cosmos.Query.Core.AsyncLazy`1.GetValueAsync(CancellationToken cancellationToken)
   at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.LazyCosmosQueryExecutionContext.ExecuteNextAsync(CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.LazyCosmosQueryExecutionContext.ExecuteNextAsync(CancellationToken cancellationToken)
   at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.CosmosQueryExecutionContextWithNameCacheStaleRetry.ExecuteNextAsync(CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.CosmosQueryExecutionContextWithNameCacheStaleRetry.ExecuteNextAsync(CancellationToken cancellationToken)
   at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.CatchAllCosmosQueryExecutionContext.ExecuteNextAsync(CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.CatchAllCosmosQueryExecutionContext.ExecuteNextAsync(CancellationToken cancellationToken)
   at Microsoft.Azure.Cosmos.Query.QueryIterator.ReadNextAsync(CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Microsoft.Azure.Cosmos.Query.QueryIterator.ReadNextAsync(CancellationToken cancellationToken)
   at Microsoft.Azure.Cosmos.FeedIteratorCore`1.ReadNextAsync(CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Microsoft.Azure.Cosmos.FeedIteratorCore`1.ReadNextAsync(CancellationToken cancellationToken)
   at Microsoft.Azure.Cosmos.FeedIteratorInlineCore`1.<>c__DisplayClass5_0.<ReadNextAsync>b__0()
   at Microsoft.Azure.Cosmos.TaskHelper.RunInlineIfNeededAsync[TResult](Func`1 task)
   at Microsoft.Azure.Cosmos.FeedIteratorInlineCore`1.ReadNextAsync(CancellationToken cancellationToken)
   at ViralLaunch.Core.Services.AzureCosmosDBService.PerformFeedRetrieve[TOut](Func`3 feedBuilder, String queryText, Nullable`1 maxItemCount, String continuationTokenIn, Boolean trackMetrics)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.<>c.<OutputWaitEtwEvents>b__12_0(Action innerContinuation, Task innerTask)
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(Action action, Boolean allowInlining)
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
   at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetResult(TResult result)
   at Microsoft.Azure.Cosmos.FeedIteratorCore`1.ReadNextAsync(CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.<>c.<OutputWaitEtwEvents>b__12_0(Action innerContinuation, Task innerTask)
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(Action action, Boolean allowInlining)
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
   at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetResult(TResult result)
   at Microsoft.Azure.Cosmos.Query.QueryIterator.ReadNextAsync(CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.<>c.<OutputWaitEtwEvents>b__12_0(Action innerContinuation, Task innerTask)
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(Action action, Boolean allowInlining)
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
   at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(TResult result)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetResult(TResult result)
   at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.CatchAllCosmosQueryExecutionContext.ExecuteNextAsync(CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.<>c.<OutputWaitEtwEvents>b__12_0(Action innerContinuation, Task innerTask)
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(Action action, Boolean allowInlining)
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
   at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(TResult result)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetResult(TResult result)
   at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.CosmosQueryExecutionContextWithNameCacheStaleRetry.ExecuteNextAsync(CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.<>c.<OutputWaitEtwEvents>b__12_0(Action innerContinuation, Task innerTask)
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(Action action, Boolean allowInlining)
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
   at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(TResult result)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetResult(TResult result)
   at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.LazyCosmosQueryExecutionContext.ExecuteNextAsync(CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.<>c.<OutputWaitEtwEvents>b__12_0(Action innerContinuation, Task innerTask)
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(Action action, Boolean allowInlining)
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
   at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(TResult result)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetResult(TResult result)
   at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.PipelinedDocumentQueryExecutionContext.ExecuteNextAsync(CancellationToken token)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.<>c.<OutputWaitEtwEvents>b__12_0(Action innerContinuation, Task innerTask)
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(Action action, Boolean allowInlining)
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
   at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(TResult result)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetResult(TResult result)
   at Microsoft.Azure.Cosmos.Query.Core.ExecutionComponent.SkipTake.TakeDocumentQueryExecutionComponent.ClientTakeDocumentQueryExecutionComponent.DrainAsync(Int32 maxElements, CancellationToken token)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.<>c.<OutputWaitEtwEvents>b__12_0(Action innerContinuation, Task innerTask)
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(Action action, Boolean allowInlining)
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
   at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
   at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.Parallel.CosmosParallelItemQueryExecutionContext.DrainAsync(Int32 maxElements, CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.<>c.<OutputWaitEtwEvents>b__12_0(Action innerContinuation, Task innerTask)
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(Action action, Boolean allowInlining)
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
   at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(TResult result)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetResult(TResult result)
   at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.ItemProducers.ItemProducerTree.TryMoveNextPageAsync(CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.<>c.<OutputWaitEtwEvents>b__12_0(Action innerContinuation, Task innerTask)
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(Action action, Boolean allowInlining)
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
   at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
   at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.ItemProducers.ItemProducerTree.ExecuteWithSplitProofingAsync(Func`2 function, Boolean functionNeedsBeReexecuted, CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.<>c.<OutputWaitEtwEvents>b__12_0(Action innerContinuation, Task innerTask)
   at System.Threading.Tasks.AwaitTaskContinuation.System.Threading.IThreadPoolWorkItem.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()

The text was updated successfully, but these errors were encountered:

bchong95 · 2020-04-10T17:17:37Z

Can you provide a reproduce for the error. If not we can start with a sample for how you are draining your query and passing the continuation token.

As for the error message :

CompositeContinuationToken is missing field: 'token': {"range":{"min":"05C189CD6732","max":"05C18F5D153C"}

It looks like your continuation token is missing a field. It should be something like:

{
       "token": "asdfsadfasdf",
       "range":{"min":"05C189CD6732","max":"05C18F5D153C"
}

codecadwallader · 2020-04-13T14:32:03Z

Thanks for the quick response. I took a look and the continuation token coming back from FeedIterator<TOut>.ReadNextAsync() appears to be the same in v3.6 as it is in v3.7:

v3.6

{
    "top": 978,
    "sourceToken": [
        {
            "range": {
                "min": "05C1653DB928",
                "max": "05C1753DB928"
            }
        }
    ]
}

v3.7

{
    "top": 978,
    "sourceToken": [
        {
            "range": {
                "min": "05C1653DB928",
                "max": "05C1753DB928"
            }
        }
    ]
}

The query we are trying to run is a SELECT TOP 1000 ... with a MaxItemCount of 100. The idea is we want to return the first 100 results, and allow for paging up to the next 900 results but not allow the user to keep going indefinitely.

Because we want to get back exactly 100 results (no more, no less) to support paging the way we drain results is something we worked out together with you last summer if you may recall over Azure/azure-cosmos-dotnet-v2#692 when the SDK changed partition behavior from v2.2.2 to v2.2.3

Pseudo-real-code:

string continuationToken = null;
var results = new List<TOut>();

while (true)
{
    var feed = Container.GetItemQueryIterator<TOut>(continuationToken, maxItemCount: 100 - results.Count);
    while (feed.HasMoreResults)
    {
        var response = await feed.ReadNextAsync();
        if (results.Count + response.Count > 100)
        {
            break; // Page has too many results, re-issue query with adjusted max item count
        }

        results.AddRange(response);
        continuationToken = feed.HasMoreResults ? response.ContinuationToken : null;
    }

    if (continuationToken == null || results.Count == 100)
    {
        break;
    }
}

Thanks for your help!

ealsur · 2020-04-13T15:38:06Z

@codecadwallader quick question, is this happening when working with the Emulator (if so, which version?) or the live service?

codecadwallader · 2020-04-13T16:57:40Z

@ealsur The live service

bchong95 · 2020-04-13T18:32:51Z

I wasn't able to reproduce this error. Can I get a console app that reproduces this issue? Thanks.

codecadwallader · 2020-04-14T20:55:47Z

Sure, I've been working on that. I'm also not able to reproduce it building up from a vanilla app so I'll need to spend some more time paring down our app and see what setting/aspect is making the difference.

codecadwallader · 2020-04-15T19:18:50Z

Ok, I've isolated the issue down to Newtonsoft.Json.NullValueHandling.Ignore. We've had this property set with our JSON serialization for years since it helps reduce the payload sizes stored in Cosmos. The token property is indeed coming back as null and being dropped out of the ContinuationToken. I suspect this behavior has always been present, but perhaps SDK v3.7 is no longer handling the absence of token when sending it back in the same way it was in SDK v3.6?

If you would still like a minimal reproducible solution I can provide it.. but it's probably easier for you to just feed this one line into your own test bed:

            JsonConvert.DefaultSettings = () => new JsonSerializerSettings { NullValueHandling = NullValueHandling.Ignore };

Thanks for your help!

Addendum: This does not happen with every attempted query. It requires a query that will attempt to utilize the continuation token to get more results.

codecadwallader · 2020-04-29T14:37:41Z

@bchong95 Just sanity checking to see if you were notified of my second update re: NullValueHandling. If you'd still like the minimal reproducible solution I can provide it.

vsadams · 2020-07-10T16:06:51Z

@bchong95 any update on this?

vsadams · 2020-07-11T00:13:26Z

@bchong95. @j82w I have found the cause of this issue and a hacky workaround.

So the reason this is happening is due to the code in FeedRangeCompositeContinuation.

The issue is the classes use of JsonConvert from the Newtonsoft library. JsonConvert will use the default func handle for creating a serializer. So for instance if the consumer of the cosmos libs does something like...

JsonConvert.DefaultSettings = () =>
{
    var settings = new JsonSerializerSettings();
    settings.NullValueHandling = NullValueHandling.Ignore;
    return settings;
};

Then the code in that class is going to attempt to deserialize\serialize the CompositeContinuationToken using ignore for the null value handling. The result is you will get a continuation token string without a token. then CompositeContinuationToken.TryCreateFromCosmosElement is going to end up throwing because the string no longer has a "token" value for any continuation token that returned a null token.

If you have code that is trying to drain the feed then fetching more, i.e draining the feed in a while loop then using the continuation token to get more results it will end up potentially throwing out.

I would like to make a suggestion that we utilize a static serializer in the FeedRangeCompositeContinuation class that ensures nulls are not ignored. EIther that, or making the TryCreateFromCosmosElement method skip the valiation checks for the properties.

I am more than happy to make a PR if you guys think thats best or I can leave it to you guys. I know as a whole, Newtonsoft is being pulled out of the libs so maybe just using the System.Text.Json serializer is best.

Here is my hacky workaround for anyone else that encounters this and needs a patch for the time being...

 public static void SetDefaults()
{
    JsonConvert.DefaultSettings = () =>
    {
        //apply internal defaults
        var settings = SetDefaults(new JsonSerializerSettings());

        //temp fix for cosmos
        settings.Converters.Add(new CosmosConverter());
        return settings;
    };
}

public class CosmosConverter : JsonConverter
{
    private const string CompositeContinuationTokenClass = "Microsoft.Azure.Cosmos.Query.Core.ContinuationTokens.CompositeContinuationToken";

    public CosmosConverter()
    {
    }

    public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
    {
        //utilize the newtonsoft default serializer settings
        var tempSerializer = new JsonSerializer();
        tempSerializer.Serialize(writer, value);
    }

    public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
        => throw new NotImplementedException();

    public override bool CanRead => false;

    public override bool CanConvert(Type objectType) => objectType?.FullName == CompositeContinuationTokenClass;
}

Thanks @codecadwallader for the tip.

ealsur · 2020-07-22T16:27:01Z

@vsadams This is great feedback, but query does not use FeedRangeCompositeContinuation. Query seems to go through https://github.com/Azure/azure-cosmos-dotnet-v3/blob/master/Microsoft.Azure.Cosmos/src/Query/v3Query/QueryIterator.cs#L80
which is the place where the Malformed error message is generated.

ealsur · 2020-07-22T16:41:58Z

Just to be sure, I also checked FeedRangeCompositeContinuation serialization/deserialization with your suggested global settings, with this test:

JsonConvert.DefaultSettings = () =>
            {
                JsonSerializerSettings settings = new JsonSerializerSettings();
                settings.NullValueHandling = NullValueHandling.Ignore;
                return settings;
            };


            const string containerRid = "containerRid";
            List<Documents.Routing.Range<string>> keyRanges = new List<Documents.Routing.Range<string>>()
            {
                new Documents.Routing.Range<string>("A", "B", true, false),
                new Documents.Routing.Range<string>("D", "E", true, false),
            };
            FeedRangeInternal feedRangeInternal = new FeedRangeEPK(new Documents.Routing.Range<string>("A", "E", true, false));
            FeedRangeCompositeContinuation token = new FeedRangeCompositeContinuation(containerRid, feedRangeInternal, keyRanges);
            // {"V":0,"Rid":"containerRid","Continuation":[{"range":{"min":"A","max":"B"}},{"range":{"min":"D","max":"E"}}],"Range":{"min":"A","max":"E"}}
            string serialized = token.ToString();
            Assert.IsTrue(FeedRangeContinuation.TryParse(serialized, out FeedRangeContinuation feedRangeContinuation));
            FeedRangeCompositeContinuation feedRangeCompositeContinuation = feedRangeContinuation as FeedRangeCompositeContinuation;

The serialization indeed does not have the token attribute within the Continuation attribute, but the deserialization works as expected and the resulting Token is null as expected.

FeedRangeCompositeContinuation does have a custom converter https://github.com/Azure/azure-cosmos-dotnet-v3/blob/master/Microsoft.Azure.Cosmos/src/Serializer/FeedRange/FeedRangeCompositeContinuationConverter.cs

The issue seems to be with the mechanism used on queries to serializer/deserialize its continuation (which is not FeedRangeCompositeContinuation).

vsadams · 2020-07-22T21:29:07Z

@ealsur it is definitely possible that I am leading this astray. I have not pulled the source and debugged against it. I am purely going from reading the code in GitHub.

However, the class FeedRangeCompositeContinuation does have a converter but CompositeContinuationToken does not. So when FeedRangeCompositeContinuation does something like

azure-cosmos-dotnet-v3/Microsoft.Azure.Cosmos/src/FeedRange/Continuations/FeedRangeCompositeContinuation.cs

Line 271 in 725931c

this.CurrentToken.Token = JsonConvert.SerializeObject(continuationAsComposite);

where is serializes the CompositeContinuationToken into a string the property will be missing.

Again, very possible, I am wrong on this. I can pull code to double check but that was my inclination on why my work around worked.

ealsur · 2020-07-22T21:46:45Z

We agree on the behavior of FeedRangeCompositeContinuation, the string that gets output does not have the "token" property if you alter the global Json settings. But what I'm saying is that it doesn't affect when FeedRangeCompositveContinuation calls TryParse, it works as expected (the Token property is null) so no errors happen there.

Now, going back to the original issue (the Malformed message), that is not related to FeedRangeCompositeContinuation, as queries do not use this class nor it is involved in the path that returns this error message.

vsadams · 2020-07-22T21:51:15Z

Gotcha, that makes sense!

vsadams · 2020-07-23T00:15:57Z

@ealsur I pulled the cosmos code and debugged it.
This is what I am seeing. This line here

azure-cosmos-dotnet-v3/Microsoft.Azure.Cosmos/src/Query/Core/ExecutionContext/Parallel/CosmosParallelItemQueryExecutionContext.ContinuationToken.cs

Line 44 in f72a33d

    
           continuationToken = JsonConvert.SerializeObject(compositeContinuationTokens, DefaultJsonSerializationSettings.Value);

is attempting to use a default serializer settings. However, due to the func handle attached to JsonConvert.DefaultSettings, the default settings are overriden.

In order to do a simple test to make sure this theory is correct, I removed my work around in my code base and tested it. I did get a malformed token originating there and eventually got the exception. I then modified the code and used a static serializer like


        private static JsonSerializer Serializer = new JsonSerializer();

        protected override string ContinuationToken
        {
            get
            {
                IEnumerable<ItemProducer> activeItemProducers = this.GetActiveItemProducers();
                string continuationToken;
                if (activeItemProducers.Any())
                {
                    IEnumerable<CompositeContinuationToken> compositeContinuationTokens = activeItemProducers.Select((documentProducer) => new CompositeContinuationToken
                    {
                        Token = documentProducer.CurrentContinuationToken,
                        Range = documentProducer.PartitionKeyRange.ToRange()
                    });
                    StringBuilder sb = new StringBuilder();
                    using StringWriter sringWriter = new StringWriter(sb);
                    using JsonWriter jsonTextWriter = new JsonTextWriter(sringWriter);
                    Serializer.Serialize(jsonTextWriter, compositeContinuationTokens);
                    continuationToken = sb.ToString();
                    //continuationToken = JsonConvert.SerializeObject(compositeContinuationTokens);
                }
                else
                {
                    continuationToken = null;
                }

                return continuationToken;
            }
        }

(Not pretty it was just a test)..but this worked.

Hope this is helpful!

ealsur · 2020-07-23T16:39:45Z

@bchong95 Can you verify what @vsadams is mentioning? Is this a possible fix?

vsadams · 2020-10-07T20:03:44Z

@ealsur any update on whether or not this is getting in?

ealsur · 2020-10-07T20:07:10Z

@bchong95 can you help verifying the scenario @vsadams is providing? Would his fix work? If so, I'm guessing you can send a PR @vsadams ?

vsadams · 2020-11-11T00:03:12Z

@ealsur or @bchong95 any update?

kirankumarkolli · 2021-01-08T14:16:30Z

@sboshra please share the update.

bchong95 · 2021-02-05T19:42:11Z

We no longer use Newtonsoft to deserialize tokens, so any changes to the default serializer won't impact the query code in SDK:

https://github.com/Azure/azure-cosmos-dotnet-v3/blob/master/Microsoft.Azure.Cosmos/src/Query/Core/Pipeline/CrossPartition/Parallel/ParallelContinuationToken.cs#L38

vsadams mentioned this issue Jul 10, 2020

Unexpected continuation token? Azure/azure-cosmos-dotnet-v2#692

Open

ealsur added the QUERY label Oct 7, 2020

ealsur added the bug Something isn't working label Nov 11, 2020

j82w assigned sboshra Dec 16, 2020

j82w mentioned this issue Jan 5, 2021

Query: Fixes ORDER BY undefined (and mixed type primitives) continuation token support #2103

Merged

bchong95 closed this as completed Feb 5, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MalformedContinuationTokenException starting with SDK v3.7 #1364

MalformedContinuationTokenException starting with SDK v3.7 #1364

codecadwallader commented Apr 10, 2020

bchong95 commented Apr 10, 2020

codecadwallader commented Apr 13, 2020 •

edited

Loading

ealsur commented Apr 13, 2020

codecadwallader commented Apr 13, 2020

bchong95 commented Apr 13, 2020

codecadwallader commented Apr 14, 2020

codecadwallader commented Apr 15, 2020 •

edited

Loading

codecadwallader commented Apr 29, 2020

vsadams commented Jul 10, 2020

vsadams commented Jul 11, 2020 •

edited

Loading

ealsur commented Jul 22, 2020

ealsur commented Jul 22, 2020

vsadams commented Jul 22, 2020

ealsur commented Jul 22, 2020

vsadams commented Jul 22, 2020

vsadams commented Jul 23, 2020 •

edited

Loading

ealsur commented Jul 23, 2020

vsadams commented Oct 7, 2020

ealsur commented Oct 7, 2020

vsadams commented Nov 11, 2020

kirankumarkolli commented Jan 8, 2021

bchong95 commented Feb 5, 2021

MalformedContinuationTokenException starting with SDK v3.7 #1364

MalformedContinuationTokenException starting with SDK v3.7 #1364

Comments

codecadwallader commented Apr 10, 2020

bchong95 commented Apr 10, 2020

codecadwallader commented Apr 13, 2020 • edited Loading

ealsur commented Apr 13, 2020

codecadwallader commented Apr 13, 2020

bchong95 commented Apr 13, 2020

codecadwallader commented Apr 14, 2020

codecadwallader commented Apr 15, 2020 • edited Loading

codecadwallader commented Apr 29, 2020

vsadams commented Jul 10, 2020

vsadams commented Jul 11, 2020 • edited Loading

ealsur commented Jul 22, 2020

ealsur commented Jul 22, 2020

vsadams commented Jul 22, 2020

ealsur commented Jul 22, 2020

vsadams commented Jul 22, 2020

vsadams commented Jul 23, 2020 • edited Loading

ealsur commented Jul 23, 2020

vsadams commented Oct 7, 2020

ealsur commented Oct 7, 2020

vsadams commented Nov 11, 2020

kirankumarkolli commented Jan 8, 2021

bchong95 commented Feb 5, 2021

codecadwallader commented Apr 13, 2020 •

edited

Loading

codecadwallader commented Apr 15, 2020 •

edited

Loading

vsadams commented Jul 11, 2020 •

edited

Loading

vsadams commented Jul 23, 2020 •

edited

Loading