Skip to content
This repository has been archived by the owner on Mar 7, 2018. It is now read-only.

Implement custom event pipeline #36

Merged
merged 2 commits into from Jun 28, 2017
Merged

Implement custom event pipeline #36

merged 2 commits into from Jun 28, 2017

Conversation

c-w
Copy link
Contributor

@c-w c-w commented Jun 27, 2017

Test EventHub to play with this.

Resolves #30

Copy link
Contributor

@erikschlegel erikschlegel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! please see my comment below re: consolidating the EH related environment so it's inline with graphql.

ConnectorConfig(
"CustomEvents",
Map (
"policyName" -> envOrElse("CUSTOMEVENTS_EH_POLICY_NAME", "project-fortis-spark"),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

from an environment standpoint Im keeping things consistent with GQL, and setting only three vars

--from-literal=PUBLISH_EVENTS_EVENTHUB_CONNECTION_STRING="${eh_conn_str}" --from-literal=PUBLISH_EVENTS_EVENTHUB_PATH="${eh_path}" --from-literal=PUBLISH_EVENTS_EVENTHUB_PARTITION="${eh_consumer_group}"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately the Spark connector for EventHubs doesn't support the connection string. I'll update the GQL library instead.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh wow, the nodejs EventHubs library only supports the connection-string. Looks like we'll have to provide both unfortunately.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Spark can just parse the connection string which is just delimited ;

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Azure only gives me the connection string so it's going to need to get parsed one way or another 😞

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't you fetch the shared access policy and get the details from there? It's all exposed in the Azure portal (deep link).

image

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should be able to get the auth keys via the EventHubs API: https://docs.microsoft.com/en-us/rest/api/eventhub/eventhubs#EventHubs_RegenerateKeys

@@ -42,6 +43,12 @@ object StreamProviderFactory {
new File(settings.progressDir, Constants.EventHubProgressDir).getPath)
)
)
.withFactories(
List(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is going to mean that @Smarker will have to add a record into the streams table upon a called to CreateOrReplaceSite, as long as the eventhub environment is properly setup.

Copy link
Contributor

@kevinhartman kevinhartman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@c-w c-w dismissed erikschlegel’s stale review June 28, 2017 20:01

Parsing the connection string here would be wrong.

@c-w c-w merged commit 8cddda4 into master Jun 28, 2017
@c-w c-w deleted the customevents branch June 28, 2017 20:20
@c-w c-w removed the in progress label Jun 28, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants