Implement custom event pipeline

[c-w]

Test EventHub to play with this.

Resolves #30

[erikschlegel]

Thanks! please see my comment below re: consolidating the EH related environment so it's inline with graphql.

[erikschlegel]

Thanks! please see my comment below re: consolidating the EH related environment so it's inline with graphql.

[erikschlegel]

from an environment standpoint Im keeping things consistent with GQL, and setting only three vars

--from-literal=PUBLISH_EVENTS_EVENTHUB_CONNECTION_STRING="${eh_conn_str}" --from-literal=PUBLISH_EVENTS_EVENTHUB_PATH="${eh_path}" --from-literal=PUBLISH_EVENTS_EVENTHUB_PARTITION="${eh_consumer_group}"

[c-w]

Unfortunately the Spark connector for EventHubs doesn't support the connection string. I'll update the GQL library instead.

[c-w]

Oh wow, the nodejs EventHubs library only supports the connection-string. Looks like we'll have to provide both unfortunately.

[erikschlegel]

Spark can just parse the connection string which is just delimited ;

[erikschlegel]

Azure only gives me the connection string so it's going to need to get parsed one way or another 😞

[c-w]

Can't you fetch the shared access policy and get the details from there? It's all exposed in the Azure portal (deep link).

[c-w]

You should be able to get the auth keys via the EventHubs API: https://docs.microsoft.com/en-us/rest/api/eventhub/eventhubs#EventHubs_RegenerateKeys

[erikschlegel]

this is going to mean that @Smarker will have to add a record into the streams table upon a called to CreateOrReplaceSite, as long as the eventhub environment is properly setup.

[kevinhartman]

LGTM

[kevinhartman]

LGTM