How to connect to a provider that only answers one GetDirectory request per message?

[andreashuber-lawo]

Some providers seem to have issues with answering the requests sent by the Ember+ Sharp Library. The library typically sends messages containing many GetDirectory requests, but said providers only ever answer the first one. As a result, Consumer.CreateAsync fails with a TimeoutException.

[andreashuber-lawo]

To work around this issue, call a Consumer.CreateAsync overload that accepts a ChildrenRetrievalPolicy parameter and pass ChildrenRetrievalPolicy.DirectOnly. As a result, only the direct children of the root node are requested from the provider and the method returns as soon as they have been received. Client code can now inspect the root children and set INode.ChildrenRetrievalPolicy of the first node to ChildrenRetrievalPolicy.DirectOnly and call Consumer.SendAsync. As soon as that method has completed, the process can be repeated for the remaining nodes and recursively for any children of interest.

This approach ensures that each consumer message contains at most one GetDirectory request and will thus lead to many more messages being sent by the consumer. Consequentially, the process of retrieving all the elements of interest will take much longer than with a provider that implements the specification correctly.

[fredrikbergholtz-sr]

It seems as if I am having difficulties wrapping my head around these asynchronous thingies. I tried to implement the suggestion like this:

...
using (Consumer<MyRoot> consumer = await Consumer<MyRoot>.CreateAsync(client, 1000, ChildrenRetrievalPolicy.DirectOnly))
{
    await consumer.SendAsync();
    INode root = consumer.Root;
    await RetrieveChildren(root, consumer);
    ...

where RetrieveChildren looks like this:

protected static async Task RetrieveChildren (INode root, Consumer<MyRoot> consumer)
{
    if (!root.IsRoot)
    {
        root.ChildrenRetrievalPolicy = ChildrenRetrievalPolicy.DirectOnly;
        await consumer.SendAsync();
    }
    foreach (IElement child in root.Children)
    {
        if (child is INode)
        {
            INode node = (INode)child;
            await RetrieveChildren(node, consumer);
        }
    }
}

This, however gives me System.Threading.Tasks.TaskCanceledException in the AsyncPump. Single stepping through the code sometimes lets it traverse the tree further, so I am thinking that I am doing something stupid regarding my async/await bits.

I would appreciate an example of how to actually implement the work-around.

[fredrikbergholtz-sr]

I removed the recursion like this:

        protected static async Task RetrieveChildren(INode root, Consumer<MyRoot> consumer)
        {
            int i = 0;
            List<INode> nodes = new List<INode> {root};
            while (nodes.Count != 0)
            {
                INode node = nodes[0];
                nodes.RemoveAt(0);

                Console.WriteLine((i++) + ": " + node.Identifier + " (" + node.Number + ")");
                node.ChildrenRetrievalPolicy = ChildrenRetrievalPolicy.DirectOnly;
                await consumer.SendAsync();

                foreach (IElement child in node.Children)
                {
                    if (child is INode)
                    {
                        nodes.Add((INode)child);
                    }
                }
            }
        }

Then I get a "System.Threading.Tasks.TaskCancelledException" with the following stack trace:

   vid System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   vid System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   vid System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   vid Lawo.EmberPlusSharp.Model.Consumer`1.<SendAsync>d__13.MoveNext() i C:\Storage\Development\Libraries\EmberPlus\ember-plus-sharp\Lawo.EmberPlusSharp\Model\Consumer`1.cs:rad 183
--- Slut på stackspårningen från föregående plats där ett undantag utlöstes ---
   vid System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   vid System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   vid System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   vid System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   vid SimpleTest.Program.<RetrieveChildren>d__1.MoveNext() i C:\SVN\metodutv\NG\Lawo-EmberPlusSharp-Test\SimpleTest\SimpleTest\Program.cs:rad 56
--- Slut på stackspårningen från föregående plats där ett undantag utlöstes ---
   vid System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   vid System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   vid System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   vid System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   vid SimpleTest.Program.<>c.<<Main>b__0_0>d.MoveNext() i C:\SVN\metodutv\NG\Lawo-EmberPlusSharp-Test\SimpleTest\SimpleTest\Program.cs:rad 28

I apologise for the swedish text in the stack trace, but I hope that you can decipher it anyway.

Curiously enough, this happens on the same node as the earlier crash even though I do the traversal in the other direction, meaning that a lot more nodes are traversed before the crash, this time.

[andreashuber-lawo]

Have you tried to increase the timeout to say 10000? 1000 seems a little low.

[fredrikbergholtz-sr]

Wouldn't I get timeout exceptions if that was the case? I still get TaskCancelledExceptions, even with much longer time-outs. In addition, my code as it is written now, sometimes simply freezes. There must be something wrong with it. I just can't see what.

EDIT: I tested with a time-out of 5 and, yes, I get a timeout exception as I expected. So the TaskCancelled is because of something else. I have a feeling that it has something to do with the scope that creates threads expiring before the threads are completed somehow. But I am not sure.

[andreashuber-lawo]

Wouldn't I get timeout exceptions if that was the case?

You should, but I had the suspicion that there might be a bug that would prevent you from getting a proper TimeoutException, see below.

In addition, my code as it is written now, sometimes simply freezes. There must be something wrong with it. I just can't see what.

While I can't reproduce the TaskCancelledException, I do get the freezes. It appears that this is a bug (#40), I'm sorry for the inconvenience. I'll keep you posted.