Never recieving messages to stream (half-open connection).

[lavarith]

I'm using Curl Unity to connect to the HTTP/2 Alexa Voice Service directives endpoint. I've added the CurlMultiUpdater to a game object and I'm using the CurlMulti to create a downchannel connection like below (see AVS link for implementation details).

Unfortunately, no matter what I do, the OnPerformCallback (which creates a log entry and prints the inText output) is never called despite knowing that directives should be being sent to the application. It leads me to believe something must be wrong with the stream or connection that's created, or am I just missing something in the connection creation?

--- Part of function that creates Curl Connection ---
// Create Curl Easy Object
CurlEasy easy = new CurlEasy
{
    method = request.Method,
    debug = true,
    uri = new Uri(request.Url),
    contentType = request.ContentType,
    timeout = request.Timeout,
    lowSpeedTimeout = request.LowSpeedLimit,
    lowSpeedLimit = request.LowSpeedLimit,
    maxRetryCount = 0,
    outData = request.OutData,
    performCallback = OnPerformCallback
};

// Add Access Token
easy.SetHeader("Authorization", "Bearer " + authenticator.GetAccessToken());

if (request.AdditionalHeaders != null)
{
    foreach (KeyValuePair<string, string> entry in request.AdditionalHeaders)
    {
        easy.SetHeader(entry.Key, entry.Value);
    }
}

// Perform CurlMulti Action
Debug.Log("Connecting to " + request.Url);
easy.MultiPerform(curlMulti.DefaultMulti);

----

---- Downchannel callback ----
Debug.Log("In downchannel callback.");
// We're in downchannel callback, meaning we've either received a directive or a disconnect
if (result == CURLE.OK)
{
    // A directive?
	Debug.Log(easy.inText);
}
else
{
    // Server disconnect. Make new downchannel
}
easy.Abort();

[Jayatubi]

The OnPerformCallback will be called when the request is finished which also means the Lib.curl_multi_info_read returns DONE underhood.

If the link you request never ends, such as a living stream, then the OnPerformCallback might not have a chance to be called. If so you could consider to use ProgressCallback instead.

Beside, when the CurlEasy reqeust was going to callback it would put it into a TaskScheduler which belongs to the thread where the Perform was called. That means if you didn't perform this request on the Unity main thread the callback might not be called. If so I will mark this issue as a bug and find a way to handle it.

BTW, if you could provide a simple project to reproduce this issue would be much great helpful.

[lavarith]

Thanks for the quick reply!

Carving out a 'simple' project is a bit hard since I need LoginWithAmazon and even a bunch more stood up to support creating a DownChannel with AVS.

Using the ProgressCallback makes sense. When I added that to my Easy declaration, the function is basically called continuously, so I created this function:

public void DownChannelProgressCallback(long dltotal, long dlnow, long ultotal, long ulnow, CurlEasy easy)
    {
        // We're in downchannel callback, meaning we've either received a directive or a disconnect
        // A directive?
        if(easy.inText != null || easy.message != null || easy.outText != null)
        {
            Debug.Log("=========== In downchannel progress callback. ===========");
            Debug.Log(easy.inData);
            Debug.Log(easy.outData);
            Debug.Log(easy.inText);
            Debug.Log(easy.outText);
            Debug.Log(easy.contentType);
            Debug.Log(easy.message);
            Debug.Log($"Download - Total: {dltotal}; Now: {dlnow};");
            Debug.Log($"Upload - Total: {ultotal}; Now: {ulnow};");

            easy.Abort();
        }
    }

However, this never prints anything. Any thoughts there? Am I looking in the right place? Am I handling the response correctly?

[Jayatubi]

The inText will be available after the request is finised so does the message. The outText was the data you put into the request and in the code you provided it seems to be empty.

If you just want the response headers, without any response body, you may try to set the method to HEAD such as:

var easy = new CurlEasy();
easy.method = "HEAD";

Then this request will auto abort after all the headers retrieved.

If the server doesn't accept the HEAD method then you could try to use the default GET instead and set the option CURLOPT.NOBODY to true:

easy.SetOpt(CURLOPT.NOBODY, true);

[lavarith]

The directives endpoint has no body, so it makes sense that it's empty.

But I want the response body as well. Let me try the CURLOPT setting.

[Jayatubi]

If you want the response body you should wait the request to finish.

If you really want to access incoming data before it finishes you could take a look at the private member responseBodyStream of CurlEasy which holds the unfinished incoming data. Just be careful this stream is being writing on other thread than the Unity main thread.

[lavarith]

That makes sense, but the directives endpoint is designed to never close (unless you close it). I'll look at the responseBodyStream.

[lavarith]

That did it!

I changed responseBodyStream to public and then do this in my ProgressCallback:

public void DownChannelProgressCallback(long dltotal, long dlnow, long ultotal, long ulnow, CurlEasy easy)
    {
        // We're in downchannel callback, meaning we've either received a directive or a disconnect
        // A directive?
        var ms = easy.responseBodyStream as MemoryStream;
        // Parse data in the stream
        byte[] inData = ms.ToArray();
        if (inData.Length > 0)
        {
            Debug.Log("========== In the DownChannel! ==========");
            Debug.Log(Encoding.UTF8.GetString(inData));

            // Clear out the stream
            easy.responseBodyStream = new MemoryStream();
        }
    }

Any idea if there's a better way to implement this? I'm worried a bit about obliterating a message when I clear responseBodyStream. I'm also wondering if I should use a Coroutine or task to periodically query the responseBodyStream rather than the progressCallback, which seems to fire continuously.

[Jayatubi]

Yes you could access the stream in a coroutine if you make it thread safe, such as by adding some necessary locks.
In my opinion the most handy way is to make your own duplex steam which support reading and writing at the same time. And since you are reading an endless stream it is not wise to use a memory stream because it may cause an out of memory issue.
You should consider to use a file stream or a ring buffer.