-
Notifications
You must be signed in to change notification settings - Fork 4
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks! please see my comment below re: consolidating the EH related environment so it's inline with graphql.
ConnectorConfig( | ||
"CustomEvents", | ||
Map ( | ||
"policyName" -> envOrElse("CUSTOMEVENTS_EH_POLICY_NAME", "project-fortis-spark"), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
from an environment standpoint Im keeping things consistent with GQL, and setting only three vars
--from-literal=PUBLISH_EVENTS_EVENTHUB_CONNECTION_STRING="${eh_conn_str}" --from-literal=PUBLISH_EVENTS_EVENTHUB_PATH="${eh_path}" --from-literal=PUBLISH_EVENTS_EVENTHUB_PARTITION="${eh_consumer_group}"
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Unfortunately the Spark connector for EventHubs doesn't support the connection string. I'll update the GQL library instead.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh wow, the nodejs EventHubs library only supports the connection-string. Looks like we'll have to provide both unfortunately.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Spark can just parse the connection string which is just delimited ;
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Azure only gives me the connection string so it's going to need to get parsed one way or another 😞
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can't you fetch the shared access policy and get the details from there? It's all exposed in the Azure portal (deep link).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You should be able to get the auth keys via the EventHubs API: https://docs.microsoft.com/en-us/rest/api/eventhub/eventhubs#EventHubs_RegenerateKeys
@@ -42,6 +43,12 @@ object StreamProviderFactory { | |||
new File(settings.progressDir, Constants.EventHubProgressDir).getPath) | |||
) | |||
) | |||
.withFactories( | |||
List( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this is going to mean that @Smarker will have to add a record into the streams
table upon a called to CreateOrReplaceSite
, as long as the eventhub environment is properly setup.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Parsing the connection string here would be wrong.
Test EventHub to play with this.
Resolves #30