Skip to content
This repository has been archived by the owner. It is now read-only.

Implement custom event pipeline #36

Merged
merged 2 commits into from Jun 28, 2017

Conversation

Projects
None yet
3 participants
@c-w
Copy link
Member

commented Jun 27, 2017

Test EventHub to play with this.

Resolves #30

@c-w c-w added the in progress label Jun 27, 2017

@c-w c-w requested a review from kevinhartman Jun 28, 2017

@c-w c-w force-pushed the customevents branch from 810064e to b9798ab Jun 28, 2017

@erikschlegel
Copy link
Contributor

left a comment

Thanks! please see my comment below re: consolidating the EH related environment so it's inline with graphql.

ConnectorConfig(
"CustomEvents",
Map (
"policyName" -> envOrElse("CUSTOMEVENTS_EH_POLICY_NAME", "project-fortis-spark"),

This comment has been minimized.

Copy link
@erikschlegel

erikschlegel Jun 28, 2017

Contributor

from an environment standpoint Im keeping things consistent with GQL, and setting only three vars

--from-literal=PUBLISH_EVENTS_EVENTHUB_CONNECTION_STRING="${eh_conn_str}" --from-literal=PUBLISH_EVENTS_EVENTHUB_PATH="${eh_path}" --from-literal=PUBLISH_EVENTS_EVENTHUB_PARTITION="${eh_consumer_group}"

This comment has been minimized.

Copy link
@c-w

c-w Jun 28, 2017

Author Member

Unfortunately the Spark connector for EventHubs doesn't support the connection string. I'll update the GQL library instead.

This comment has been minimized.

Copy link
@c-w

c-w Jun 28, 2017

Author Member

Oh wow, the nodejs EventHubs library only supports the connection-string. Looks like we'll have to provide both unfortunately.

This comment has been minimized.

Copy link
@erikschlegel

erikschlegel Jun 28, 2017

Contributor

Spark can just parse the connection string which is just delimited ;

This comment has been minimized.

Copy link
@erikschlegel

erikschlegel Jun 28, 2017

Contributor

Azure only gives me the connection string so it's going to need to get parsed one way or another 😞

This comment has been minimized.

Copy link
@c-w

c-w Jun 28, 2017

Author Member

Can't you fetch the shared access policy and get the details from there? It's all exposed in the Azure portal (deep link).

image

This comment has been minimized.

Copy link
@c-w

c-w Jun 28, 2017

Author Member

You should be able to get the auth keys via the EventHubs API: https://docs.microsoft.com/en-us/rest/api/eventhub/eventhubs#EventHubs_RegenerateKeys

@@ -42,6 +43,12 @@ object StreamProviderFactory {
new File(settings.progressDir, Constants.EventHubProgressDir).getPath)
)
)
.withFactories(
List(

This comment has been minimized.

Copy link
@erikschlegel

erikschlegel Jun 28, 2017

Contributor

this is going to mean that @Smarker will have to add a record into the streams table upon a called to CreateOrReplaceSite, as long as the eventhub environment is properly setup.

@kevinhartman
Copy link
Contributor

left a comment

LGTM

@c-w c-w force-pushed the customevents branch from b9798ab to 36790f8 Jun 28, 2017

Parsing the connection string here would be wrong.

@c-w c-w force-pushed the customevents branch from 36790f8 to dcad908 Jun 28, 2017

@c-w c-w merged commit 8cddda4 into master Jun 28, 2017

2 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details

@c-w c-w deleted the customevents branch Jun 28, 2017

@c-w c-w removed the in progress label Jun 28, 2017

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.