Invoking a non-HTTP triggered function locally using the Azure Functions Core Tools Admin API (cardinality: one)

[amotl]

Is your question related to a specific version? If so, please specify:

Found Python version 3.8.5 (python3).
Azure Functions Core Tools (3.0.2931 Commit hash: d552c6741a37422684f0efab41d541ebad2b2bd2)
Function Runtime Version: 3.0.14492.0

What binding does your question apply to, if any? (e.g. Blob Trigger, Event Hub Binding, etc)

Event Hub Trigger Binding (cardinality: one)

Question

Hi there,

we are trying to follow the documentation about how to pass test data to a non-HTTP triggered function to invoke a Python function locally when running within the Azure Functions Core Tools host.

For all kinds of functions other than HTTP triggers and webhooks and Event Grid triggers, you can test your functions locally by calling an administration endpoint. Calling this endpoint with an HTTP POST request on the local server triggers the function.

We are using the EventHub Trigger binding, so the function signature looks like that:

def main(events: List[func.EventHubEvent]):
    # Process all events.
    for event in events:
        pass

Now, we are trying to figure out how exactly to formulate what to put into <trigger_input> in our scenario.

The message body is required to have the following JSON format:

{
    "input": "<trigger_input>"
}

The <trigger_input> value contains data in a format expected by the function.

When submitting an obviously wrong request like

http http://0.0.0.0:7071/admin/functions/eventHubTrigger input=foobar

the host says

[2020-10-14T15:03:03.439] Executed 'Functions.EventHubTrigger' (Failed, Id=57244492-47f9-4801-a99f-431f64a2cd6e, Duration=86ms)
[2020-10-14T15:03:03.439] System.Private.CoreLib: Exception while executing function: Functions.EventHubTrigger. Microsoft.Azure.WebJobs.Host: Exception binding parameter 'events'. Microsoft.Azure.WebJobs.Host: Binding parameters to complex objects (such as 'Object') uses Json.NET serialization.
[2020-10-14T15:03:03.439] 1. Bind the parameter type as 'string' instead of 'Object' to get the raw values and avoid JSON deserialization, or
[2020-10-14T15:03:03.439] 2. Change the queue payload to be valid json. The JSON parser failed: Error parsing boolean value. Path '', line 1, position 1.

When naively sending real JSON like {"input": [{"device-id": "1"}]}, the host responds with <Response [400]> (Bad Request).

After these observations, we started digging through the source code of the host and the python function worker in order to figure out how to build an appropriate payload reflecting the "real" thing but quickly got lost.

While it is obvious that there will be more efforts required to send a complex object instead of a simple scalar string as outlined within the example

curl --request POST -H "Content-Type:application/json" --data '{"input":"sample queue data"}' http://localhost:7071/admin/functions/QueueTrigger

we are now wondering if that would be possible at all. Maybe you can help us out?

Thank you very much in advance and with kind regards,
Andreas.

[amotl]

Hi again,

I believe I should share some more background where we are aiming at. Originally, I've misplaced that within Azure/azure-functions-host#6603 (comment) and Azure/azure-functions-host#6603 (comment).

We are currently looking into running full integration tests with pytest through a host instance provided by azure-functions-core-tools.

Our idea was to spin up a function/host instance from a pytest fixture, submit a single or more messages to it by e.g. invoking

http http://0.0.0.0:7071/admin/functions/eventHubTrigger input=foobar

and then check the outcome of what has been done when processing the message(s) (e.g. writing to a database).

Now, after sublimating myself more thoroughly into the code base, I found azure_functions_worker.testutils.start_webhost as well as tests.endtoend.test_eventhub_functions.test_eventhub_trigger, which probably perfectly reflect our goal. Kudos to @1st1, @Hazhzeng, @elprans, @vrdmr, @ankitkumarr and @maiqbal11.

From reading that code, I learned that there is probably no way to invoke an EventHub Trigger function directly. Instead, the idea seems to be to use a HTTP Trigger in order to indirectly submit an event to EventHub, right?

azure-functions-python-worker/tests/endtoend/test_eventhub_functions.py

Lines 30 to 34 in c465de4

	# Invoke eventhub_output HttpTrigger to generate an Eventhub Event.
	r = self.webhost.request('POST', 'eventhub_output',
	data=json.dumps(doc))
	self.assertEqual(r.status_code, 200)
	self.assertEqual(r.text, 'OK')

With kind regards,
Andreas.

cc @faymarie, @quodt, @kamejosh

[amotl]

Hi again,

despite learning that the endtoend integration tests within this repository indirectly invoke an Event Hubs Trigger using a HTTP Trigger, I would like to share some resources where others have also been trying to do it directly by submitting a HTTP POST request to the Admin API.

• Non-HTTP triggered functions do not support being executed via HTTP POST MicrosoftDocs/azure-docs#35469
• https://stackoverflow.com/questions/47002514/azure-function-is-not-triggered-locally
• https://stackoverflow.com/questions/47739232/azure-functions-developing-locally-cannot-register-eventhub-triggered-function

It is all revolving around the aspect how to phrase the request body and whether it would work using a JSON body at all.

With kind regards,
Andreas.

[kepalas]

@amotl If I understood your problem correctly, I think that I found the "magic" format that needs to be submitted through the POST call.

This is how I was able to use the Admin API with Event Hub trigger function:

curl --request POST 'http://localhost:7071/admin/functions/MyFunctionName' --header 'Content-Type: application/json' --data-raw '{"input": "{\"SystemProperties\":{},\"myteststuff\":{\"testid\":\"810d9efd-458c-49f2-a9c0-822c897a02e3\",\"testtimestamp\":\"2020-10-28 23:24:16.315294\"},\"testdata\":\"Ping from Az CLI IoT Extension #1\"}"}'

First thing, the JSON in the POST request needs to follow this pattern: {"input":"<escaped JSON string>"}.
In addition, the escaped JSON string must include "SystemProperties":{} object.

If the "SystemProperties":{} object is missing, the Admin API still returns a 202 response, but in the host logs you can find the following error:

System.Private.CoreLib: Exception while executing function: Functions.MyFunctionName. System.Private.CoreLib: Result: Failure
Exception: TypeError: unable to decode incoming TypedData: unsupported combination of TypedData field 'string' and expected binding type <class 'azure.functions.eventhub.EventHubTriggerConverter'>

[amotl]

Dear Tadas,

I think that I found the "magic" format that needs to be submitted through the POST call.
The escaped JSON string must include a "SystemProperties":{} object.

Ah, that is awesome! Thank you so much for applying your wizardry to the problem we have been facing here. I just wanted to leave you a quick appreciation and will follow up on this next week.

With kind regards,
Andreas.

cc @faymarie, @quodt, @kamejosh

[amotl]

Dear Tadas,

we translated the curl request to an invocation using HTTPie and it works like a treat (this saves us from having to escape the JSON payload on the first hand):

http http://0.0.0.0:7071/admin/functions/eventHubTrigger \
  input='{"SystemProperties":{},"foo":"Ping from Az CLI IoT Extension #1","bar":{"id":"810d9efd-458c-49f2-a9c0-822c897a02e3","timestamp":"2020-10-28 23:24:16.315294"}}' \
  --print hHbB

Thanks again!

Now, we are actually aiming at submitting a list of events, reflected by using a binding attribute "cardinality": "many" within function.json. We've deliberately created another issue for that (see #773) in order to be able to close this one successfully.

With kind regards,
Andreas.

[amotl]

Dear @stefanushinardi, @anirudhgarg, @anthonychu and @Hazhzeng,

may I humbly suggest to add the outcome of this research by @kepalas to the aforementioned documentation at
Passing test data to a function > Non-HTTP triggered functions?

The gist is:

• When supplying a complex real-world JSON object as <trigger_input> value, it must be properly escaped.
• The escaped JSON string must include a "SystemProperties":{} object.

With kind regards,
Andreas.

[feesma]

Also facing issues trying to figure out what to sent as <trigger_input> for blob triggers and other events