Implement custom event pipeline #36
Conversation
| ConnectorConfig( | ||
| "CustomEvents", | ||
| Map ( | ||
| "policyName" -> envOrElse("CUSTOMEVENTS_EH_POLICY_NAME", "project-fortis-spark"), |
There was a problem hiding this comment.
from an environment standpoint Im keeping things consistent with GQL, and setting only three vars
--from-literal=PUBLISH_EVENTS_EVENTHUB_CONNECTION_STRING="${eh_conn_str}" --from-literal=PUBLISH_EVENTS_EVENTHUB_PATH="${eh_path}" --from-literal=PUBLISH_EVENTS_EVENTHUB_PARTITION="${eh_consumer_group}"
There was a problem hiding this comment.
Unfortunately the Spark connector for EventHubs doesn't support the connection string. I'll update the GQL library instead.
There was a problem hiding this comment.
Oh wow, the nodejs EventHubs library only supports the connection-string. Looks like we'll have to provide both unfortunately.
There was a problem hiding this comment.
Spark can just parse the connection string which is just delimited ;
There was a problem hiding this comment.
Azure only gives me the connection string so it's going to need to get parsed one way or another 😞
There was a problem hiding this comment.
Can't you fetch the shared access policy and get the details from there? It's all exposed in the Azure portal (deep link).
There was a problem hiding this comment.
You should be able to get the auth keys via the EventHubs API: https://docs.microsoft.com/en-us/rest/api/eventhub/eventhubs#EventHubs_RegenerateKeys
| @@ -42,6 +43,12 @@ object StreamProviderFactory { | |||
| new File(settings.progressDir, Constants.EventHubProgressDir).getPath) | |||
| ) | |||
| ) | |||
| .withFactories( | |||
| List( | |||
There was a problem hiding this comment.
this is going to mean that @Smarker will have to add a record into the streams table upon a called to CreateOrReplaceSite, as long as the eventhub environment is properly setup.
c-w
dismissed
erikschlegel’s stale review
Parsing the connection string here would be wrong.
Test EventHub to play with this.
Resolves #30