Skip to content
This repository has been archived by the owner. It is now read-only.

Implement custom event pipeline #36

Merged
merged 2 commits into from Jun 28, 2017
Merged

Implement custom event pipeline #36

merged 2 commits into from Jun 28, 2017

Conversation

@c-w
Copy link
Contributor

@c-w c-w commented Jun 27, 2017

Test EventHub to play with this.

Resolves #30

@c-w c-w requested a review from kevinhartman Jun 28, 2017
@c-w c-w force-pushed the customevents branch from 810064e to b9798ab Jun 28, 2017
Copy link
Contributor

@erikschlegel erikschlegel left a comment

Thanks! please see my comment below re: consolidating the EH related environment so it's inline with graphql.

ConnectorConfig(
"CustomEvents",
Map (
"policyName" -> envOrElse("CUSTOMEVENTS_EH_POLICY_NAME", "project-fortis-spark"),
Copy link
Contributor

@erikschlegel erikschlegel Jun 28, 2017

from an environment standpoint Im keeping things consistent with GQL, and setting only three vars

--from-literal=PUBLISH_EVENTS_EVENTHUB_CONNECTION_STRING="${eh_conn_str}" --from-literal=PUBLISH_EVENTS_EVENTHUB_PATH="${eh_path}" --from-literal=PUBLISH_EVENTS_EVENTHUB_PARTITION="${eh_consumer_group}"

Copy link
Contributor Author

@c-w c-w Jun 28, 2017

Unfortunately the Spark connector for EventHubs doesn't support the connection string. I'll update the GQL library instead.

Copy link
Contributor Author

@c-w c-w Jun 28, 2017

Oh wow, the nodejs EventHubs library only supports the connection-string. Looks like we'll have to provide both unfortunately.

Copy link
Contributor

@erikschlegel erikschlegel Jun 28, 2017

Spark can just parse the connection string which is just delimited ;

Copy link
Contributor

@erikschlegel erikschlegel Jun 28, 2017

Azure only gives me the connection string so it's going to need to get parsed one way or another 😞

Copy link
Contributor Author

@c-w c-w Jun 28, 2017

Can't you fetch the shared access policy and get the details from there? It's all exposed in the Azure portal (deep link).

image

Copy link
Contributor Author

@c-w c-w Jun 28, 2017

You should be able to get the auth keys via the EventHubs API: https://docs.microsoft.com/en-us/rest/api/eventhub/eventhubs#EventHubs_RegenerateKeys

@@ -42,6 +43,12 @@ object StreamProviderFactory {
new File(settings.progressDir, Constants.EventHubProgressDir).getPath)
)
)
.withFactories(
List(
Copy link
Contributor

@erikschlegel erikschlegel Jun 28, 2017

this is going to mean that @Smarker will have to add a record into the streams table upon a called to CreateOrReplaceSite, as long as the eventhub environment is properly setup.

Copy link
Contributor

@kevinhartman kevinhartman left a comment

LGTM

@c-w c-w force-pushed the customevents branch from b9798ab to 36790f8 Jun 28, 2017
@c-w c-w dismissed erikschlegel’s stale review Jun 28, 2017

Parsing the connection string here would be wrong.

@c-w c-w force-pushed the customevents branch from 36790f8 to dcad908 Jun 28, 2017
@c-w c-w merged commit 8cddda4 into master Jun 28, 2017
2 checks passed
@c-w c-w deleted the customevents branch Jun 28, 2017
@c-w c-w removed the in progress label Jun 28, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

3 participants