Problem with using a CustomInputSchema

[romankiss]

This document states the following:
"If your event data doesn't match the expected Event Grid schema, you can still use Event Grid to route event to subscribers."

That's the great feature of the custom/domain event topic to integrate basically any REST POST driven publisher to the AEG eventing Pub/Sub model directly without any integrator (for example: the Azure Function).

But, the current release of the CustomInputSchema didn't allow to do it without using an additional component for integrating to the AEG because it is required to use a hardcoded array of the events doesn't matter how many events have been published either one or multiple.

The document shows an example of the Original Event Schema, such as

[
   {
    "myEventTypeField":"Created",
    "resource":"Users/example/Messages/1000",
    "resourceData":{"someDataField1":"SomeDataFieldValue"}
  }
]

Note, that the above example is an array of the custom input schema, which practical is in the most cases useless, because we don't have a possibility to change any webhook (3rd party, Azure, Google, AWS, etc.).

In other word, it looks like that the implementation of the CustomInputSchema has a design bug and it should be handled a single event as a JObject in the case of the JArray with a one event item, so the following event should be accepted:

  {
    "myEventTypeField":"Created",
    "resource":"Users/example/Messages/1000",
    "resourceData":{"someDataField1":"SomeDataFieldValue"}
  }  

without the error like the following:

{
  "error": {
    "code": "BadRequest",
    "message": "Unexpected token when reading JSON. Expected: StartArray, actual: StartObject. Report '3aa1b8e7-dc93-480e-81fe-0947e4b7ae19:1:11/14/2019 5:18:00 AM (UTC)' to our forums for assistance or raise a support ticket.",
    "details": [
      {
        "code": "InputJsonInvalid",
        "message": "Unexpected token when reading JSON. Expected: StartArray, actual: StartObject. Report '3aa1b8e7-dc93-480e-81fe-0947e4b7ae19:1:11/14/2019 5:18:00 AM (UTC)' to our forums for assistance or raise a support ticket."
      }
    ],
  }
}

Thanks
Roman

---

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

• ID: ccb247e6-02af-3828-4818-3c3327818009
• Version Independent ID: f8ab9dff-4aed-de20-e6e8-23287a06092a
• Content: Map custom field to Azure Event Grid schema
• Content Source: articles/event-grid/input-mappings.md
• Service: event-grid
• GitHub Login: @spelluru
• Microsoft Alias: spelluru

[DixitArora-MSFT]

@romankiss Thanks for your feedback. We will review and update as appropriate.

[KetanChawda-MSFT]

@spelluru / @banisadr Any thoughts here from product side.

[romankiss]

@banisadr How this "design issue" has been solved in the version 2020-06-01?
Can we have for input single event the same like we have it at the CloudEvent, such as no array wrapper?

Thanks
Roman

[carlreid]

@spelluru Was there any progress on this?

We've just ran into the same issue where we would like to map a custom input schema to EventGrid but we also only receive a JObject. We can't modify the source to send an array so we're a bit stuck. Is there an option to change to accepting without the array?

[spelluru]

@jfggdl - Could you please help with this issue? Thanks

#reassign:jfggdl

[jfggdl]

@romankiss and @carlreid, thank you for reporting this issue. I am the Product Manager for Event Grid. Unfortunately, single event publishing is not available for custom and event grid schemas. Given our support for open standards, our roadmap includes more features based on open standards and as a consequence, any new investment will be done on Cloud Events schema specification and not on the other schemas. Cloud Events 1.0 does support publishing single events. Please let me know if you need more information on that. I understand that this involves some refactoring on your side, but by embracing the new Cloud Events 1.0 schema you will have more extensibility and use cases supported now and in the future. Thank you for your understanding.

[carlreid]

I understand the direction and completely agree with it. But in the cases where we can't control the source that sends the event in terms of the structure then it doesn't work.

I think the solution would be to have a custom mapper of some sort to the cloud event schema.

Otherwise the solution we just implemented was to have a function that takes an event, maps it and then posts to EventGrid. Just a little annoying to have some extra step of our own running. Unless you have a better suggestion?

[romankiss]

@jfggdl thanks for info about the focusing on the Cloud Events schema.

Currently, we can use a 3rd party single event (JObject) only for the AEG Custom topic created for a CloudEvent input schema (the AEG Domain topic has still an issue with the source attribute, see #43468 (comment)) when the producer has a capability to setup the following required headers:
Content-Type: application/json
ce-id
ce-source
ce-type
ce-specversion

and optional other headers such as:
ce-subject
ce-topic
ce-*
where * must consist of lower-case letters ('a' to 'z') or digits ('0' to '9').

Note, that the saskey can be added either to the header 'aeg-sas-key' or as a url query parameter.

In many real cases (included also the AEG subscriber webhook) the producer didn't allow to add additional headers, so in this case we have only way to use a url query parameters.

It will be very useful to have a capability for passing the missing ce-* attributes the same way like we have today for an aeg-sas-key, such as via the query parameters.
Based on the cloudevent spec, the source + id must be unique, so in the case of missing the ce-id should be generated by guid or mapping from the payload, for instance, ce-id=@myid (something similar like is used for binding expression in the Azure Function)

Thanks
Roman

[spelluru]

@jfggdl - Please track this in a work item internally. Thanks.

#please-close

[jfggdl]

Thank you @spelluru. We are tracking now this item with internal work item 9489937.