client.purge_instance_history_by returns aiohttp.ContentTypeError

[hergott]

I’m deleting orchestration histories, and this demo code wasn’t working for me in multiple ways. I reprogrammed that example, but it was failing with this error:

System.Private.CoreLib: Exception while executing function: Functions.regime_regression. System.Private.CoreLib: Result: Failure
Exception: ContentTypeError: 0, message='Attempt to decode JSON with unexpected mimetype: ', url=URL('http://127.0.0.1:17071/durabletask/instances/?createdTimeFrom=1938-07-03T22:59:05.650856Z&createdTimeTo=2020-08-21T22:59:05.650856Z&runtimeStatus=Completed,Terminated,Failed') 
Stack:   File "C:\...\npm\node_modules\azure-functions-core-tools\bin\workers\python\3.8/WINDOWS/X64\azure_functions_worker\dispatcher.py", line 338, in _handle__invocation_request
    call_result = await fi.func(**args)
  File "C:\...\__app__\regime_regression\__init__.py", line 87, in main
    await purge_history
  File "C:\...\__app__\.venv\lib\site-packages\azure\durable_functions\models\DurableOrchestrationClient.py", line 361, in purge_instance_history_by
    response = await self._delete_async_request(request_url)
  File "C:\...\__app__\.venv\lib\site-packages\azure\durable_functions\models\utils\http_utils.py", 
line 68, in delete_async_request
    data = await response.json()
  File "C:\...\__app__\.venv\lib\site-packages\aiohttp\client_reqrep.py", line 1026, in json        
    raise ContentTypeError(
.

For me, this error occurs when either (a) there are no past instances to delete, or (b) the Function App is run locally (regardless of whether there are instances eligible to be purged).

On my setup, client.purge_instance_history_by works as expected if there are instances that meet the delete criteria and I’m calling a Durable Function deployed to Azure.

This is what I wrote to handle the error; I don’t know whether people view this as long-term solution:

import azure.functions as func          
import azure.durable_functions as df    
from azure.durable_functions.models.OrchestrationRuntimeStatus import OrchestrationRuntimeStatus 

from datetime import datetime, timedelta
from aiohttp import ContentTypeError

...

    purge_past_instances = info.get('purge_past_instances', False)

    if purge_past_instances:
        keep_days = info.get('keep_days', 3)

        client = df.DurableOrchestrationClient(starter)

        created_time_from = datetime.utcnow() + timedelta(days=-30000)
        created_time_to = datetime.utcnow() + timedelta(days=-keep_days)

        runtime_statuses = [OrchestrationRuntimeStatus.Completed,
                            OrchestrationRuntimeStatus.Terminated, OrchestrationRuntimeStatus.Failed]

        try:
            purge_coroutine = client.purge_instance_history_by(
                created_time_from, created_time_to, runtime_statuses)

            purge_history = await purge_coroutine

        except ContentTypeError as e:
            purge_err = f'\nFailed to delete past instances: ContentTypeError.\n{repr(e)}\n'

            return func.HttpResponse(purge_err, status_code=500, mimetype='text/plain')

        else:
            purge_msg = f'Deleted past instances, without analyzing any submitted data. Purged {purge_history.instances_deleted} past instances that were created from {created_time_from} to {created_time_to}.'

            return func.HttpResponse(purge_msg, status_code=200, mimetype='text/plain')

[davidmrdavid]

Hi @hergott ,

Thanks for reaching out. Hmm, I remember this feature appeared a little brittle upon first implementation, so this could very well be a bug on our end. I'll review this in the coming days and get back to you on an ETA. This will most likely not be a part of our most immediate next release, but we can certainly integrate the fix as part of the next.

I'll get back to you!

⚡ ⚡

[hergott]

Hi @davidmrdavid ,

Thanks for your response. I'm not sure there's a big rush on this.

IMO, the most important thing for now is that client.purge_instance_history_by seems to work as intended under the conditions I described in my post.

Matt

[davidmrdavid]

Ah yes, those docs need some re-doing. It would appear that those are missing await statements.

As for the error itself, I think I know what would cause it, there's a missing flag in our HTTP response processor. However, I can't seem to reproduce the error you list in this issue, I get a different one:

8/28/2020 3:34:23 AM] Executed 'Functions.DurableFunctionsHttpStart2' (Failed, Id=bfe94583-bebb-4b73-94cd-9309896d81ce, Duration=87ms)
[8/28/2020 3:34:23 AM] System.Private.CoreLib: Exception while executing function: Functions.DurableFunctionsHttpStart2. System.Private.CoreLib: Result: Failure
Exception: TypeError: unable to encode outgoing TypedData: unsupported type "<class 'azure.functions.http.HttpResponseConverter'>" for Python type "coroutine"
Stack:   File "C:\Users\david\AppData\Roaming\npm\node_modules\azure-functions-core-tools\bin\workers\python\3.8\WINDOWS\X64\azure_functions_worker\dispatcher.py", line 372, in _handle__invocation_request
    return_value = bindings.to_outgoing_proto(
  File "C:\Users\david\AppData\Roaming\npm\node_modules\azure-functions-core-tools\bin\workers\python\3.8\WINDOWS\X64\azure_functions_worker\bindings\meta.py", line 93, in to_outgoing_proto
    raise TypeError(

Do you mind attaching a zip'ed reproducer, and your version of the azure-functions-durable python library? Might also be good to make sure you're in the latest version of the functions core tools, which you can update as such:

npm i -g azure-functions-core-tools@3 --unsafe-perm true

If you still get the same error with the latest version of the core tools, then I would greatly appreciate it if you could attach a reproducer. Thanks!!

[hergott]

Hi @davidmrdavid ,

I just looked at the error you posted, and I think you are going off the demo code from the Azure website.

That example tries to return a Python coroutine when the function binding is expecting an HTTP response.

You should use the code I posted above to initiate/test a purge command; the error you describe reflects one of several changes I had to make to the demo.

Matt

[davidmrdavid]

Hi @hergott

You're right, I was accidentally running the wrong sample. I managed to get your error, and find the fix. It's in the PR associated with this issue! Should be fixed soon :)

⚡ ⚡

[hergott]

Fantastic!

[davidmrdavid]

Hi @hergott ,

This should have been fixed in the latest version of Durable Python (v1.0.0b8), please let us know if that works on your end. In the meantime, I'll be closing this ticket. Thank you so much for bringing this to our attention!

⚡ ⚡

[hergott]

Hi @davidmrdavid ,

I changed the requirements.txt file to azure-functions-durable>=1.0.0b8, and then I tested this new version both locally and through a deployed Azure Function App.

It sure looks like it works!

The .instances_deleted return value gives the correct number of purged instances, and I can see that the instance histories are deleted on Storage Explorer. If there are no orchestration histories to delete, .instances_deleted correctly returns a 0 value.

At first, my local testing was failing to delete orchestration histories. But I figured out that there is a 6-hour mismatch between my computer's time and the UTC time that Azure uses. Using the code from my original posting, I got the local testing working by changing these lines:

        created_time_from = datetime.today() + timedelta(days=-30000)
        created_time_to = datetime.today() + timedelta(days=-keep_days)

to

        created_time_from = datetime.utcnow() + timedelta(days=-30000)
        created_time_to = datetime.utcnow() + timedelta(days=-keep_days)

Great job!

Matt