Feature: Webhooks for Workflow and Task Statuses

[CollinDewey]

Pull Request type

• Bugfix
• Feature
• Refactoring (no functional changes, no api changes)
• Build related changes
• WHOSUSING.md
• Other (please describe):

NOTE: Please remember to run ./gradlew spotlessApply to fix any format violations.

Changes in this PR

Describe the new behavior from this PR, and why it's needed
Issue Netflix/conductor-community#212
We require a notification published from Conductor to a callback URL when certain task statuses are reached.

This continues the changes proposed by @charybr in Netflix/conductor-community#228, which originates from the changes made by @techyragu in CiscoM31/conductor#26.

We have added functionally to allow the user to specify a URL to send task and workflow statuses to respectively. The url can be specified in the configuration file used when the user’s specific conductor instance is being spun up. Below is an example of the properties added to the config:

conductor.status-notifier.notification.url=http://example.com
conductor.status-notifier.notification.endpointTask=
conductor.status-notifier.notification.endpointWorkflow=
conductor.status-notifier.notification.subscribedTaskStatuses=SCHEDULED

The endpointTask property is where task notifications will be sent. The endpointWorkflow property is where workflow notifications will be sent. The subscribedTaskStatuses property should be set to SCHEDULED as that is the only state change currently being listened for, but this implementation allows for future enhancements to be added for other task notification types to be listened for and specified in this property to be received at the endpointTask endpoint.

The information that will be sent in a task notification is shown in the following example:

Expand

"data": {
	"correlationId": null,
	"domain": null,
	"endTime": "1970-01-01T00:00:00.000Z",
	"executionTime": 0,
	"externalInputPayloadStoragePath": null,
	"externalOutputPayloadStoragePath": null,
	"input": "{\"asyncComplete\":false,\"http_request\":{\"method\":\"GET\",\"uri\":\"https://datausa.io/api/data?drilldowns=Nation&measures=Population\"}}",
	"output": "{}",
	"queueWaitTime": 31,
	"reasonForIncompletion": null,
	"referenceTaskName": "get_population_data",
	"retryCount": 0,
	"scheduledTime": "2024-04-01T22:21:19.112Z",
	"startTime": "2024-04-01T22:21:19.143Z",
	"status": "SCHEDULED",
	"taskDefName": "get_population_data",
	"taskDescription": null,
	"taskId": "4dea6f89-df79-4b6e-8952-96728e061885",
	"taskType": "HTTP",
	"updateTime": "1970-01-01T00:00:00.000Z",
	"workflowId": "b3e5c928-3d26-4f79-bf41-9a02c9d96239",
	"workflowPriority": 0,
	"workflowTaskType": "HTTP",
	"workflowType": "testwf"
}

Whenever a workflow is completed, a notification will be sent out to the specified webhook. The information that will be sent in a workflow notification is shown in the following example:

Expand

"data": {
	"correlationId": null,
	"endTime": "2024-04-01T22:21:21.443Z",
	"event": null,
	"executionTime": 3419,
	"externalInputPayloadStoragePath": null,
	"externalOutputPayloadStoragePath": null,
	"failedReferenceTaskNames": "",
	"failedTaskNames": [],
	"input": "{}",
	"inputSize": 2,
	"output": "{\"data\":[{\"ID Nation\":\"01000US\",\"Nation\":\"United States\",\"ID Year\":2021,\"Year\":\"2021\",\"Population\":329725481,\"Slug Nation\":\"united-states\"},{\"ID Nation\":\"01000US\",\"Nation\":\"United States\",\"ID Year\":2020,\"Year\":\"2020\",\"Population\":326569308,\"Slug Nation\":\"united-states\"},{\"ID Nation\":\"01000US\",\"Nation\":\"United States\",\"ID Year\":2019,\"Year\":\"2019\",\"Population\":324697795,\"Slug Nation\":\"united-states\"},{\"ID Nation\":\"01000US\",\"Nation\":\"United States\",\"ID Year\":2018,\"Year\":\"2018\",\"Population\":322903030,\"Slug Nation\":\"united-states\"},{\"ID Nation\":\"01000US\",\"Nation\":\"United States\",\"ID Year\":2017,\"Year\":\"2017\",\"Population\":321004407,\"Slug Nation\":\"united-states\"},{\"ID Nation\":\"01000US\",\"Nation\":\"United States\",\"ID Year\":2016,\"Year\":\"2016\",\"Population\":318558162,\"Slug Nation\":\"united-states\"},{\"ID Nation\":\"01000US\",\"Nation\":\"United States\",\"ID Year\":2015,\"Year\":\"2015\",\"Population\":316515021,\"Slug Nation\":\"united-states\"},{\"ID Nation\":\"01000US\",\"Nation\":\"United States\",\"ID Year\":2014,\"Year\":\"2014\",\"Population\":314107084,\"Slug Nation\":\"united-states\"},{\"ID Nation\":\"01000US\",\"Nation\":\"United States\",\"ID Year\":2013,\"Year\":\"2013\",\"Population\":311536594,\"Slug Nation\":\"united-states\"}],\"source\":[{\"measures\":[\"Population\"],\"annotations\":{\"source_name\":\"Census Bureau\",\"source_description\":\"The American Community Survey (ACS) is conducted by the US Census and sent to a portion of the population every year.\",\"dataset_name\":\"ACS 5-year Estimate\",\"dataset_link\":\"http://www.census.gov/programs-surveys/acs/\",\"table_id\":\"B01003\",\"topic\":\"Diversity\",\"subtopic\":\"Demographics\"},\"name\":\"acs_yg_total_population_5\",\"substitutions\":[]}]}",
	"outputSize": 1633,
	"priority": 0,
	"reasonForIncompletion": null,
	"startTime": "2024-04-01T22:21:18.024Z",
	"status": "COMPLETED",
	"statusNotifier": null,
	"updateTime": "2024-04-01T22:21:21.443Z",
	"version": 1,
	"workflowId": "b3e5c928-3d26-4f79-bf41-9a02c9d96239",
	"workflowType": "testwf"
}

This functionality is needed to efficiently process requests when workflows are being managed through a 3rd party service. This functionality is also useful to ascertain which tasks commonly fail, and where a long workflow is in execution at any given time.

Alternatives considered

Describe alternative implementation you have considered
An alternative to pushing webhook notifications would be client applications pulling a workflow or task status from Conductor. This approach does not scale if hundreds or thousands of workflows need to be monitored, calling upon each status of which would become a very expensive operation.

---

Authors @BabyBlue0214 @EJ-S @ToastedTaco @CollinDewey

[v1r3n]

As suggested, the status publisher should maintain its own properties the workflow def should not have those.

[v1r3n]

task status listener does not work fully and prob. should be a separate PR later.

[charybr]

thank you @CollinDewey for continuing this.

[charybr]

thanks

[CollinDewey]

thank you @CollinDewey for continuing this.

Thank you. This was continued by a group of 4 people, listed at the end of the initial PR message. (#115 (comment))