Unified log / API to retrieve all workflow invocations + errors

[dylburger]

I'd like to access a log of all workflow executions (my event history) or errors so I can run analysis on them and observe patterns. The log would need to contain metadata like:

timestamp of execution
duration of workflow execution
success / failure state
workflow id
deployment id
(any other relevant data from steps.trigger.context)
timestamp of error (if error)
error object (if present)

[HaimBendanan]

Additionally, we would like to get the console logs via API. It would allow better integration with data warehouse systems.
The classic use case scenario is when you need to investigate why something happened in a complex workflow, but the logs are not available anymore (since only the last 100 workflow instances are kept). Having the ability to push the logs to an external system easily would be great. We already can push the logs to s3 with .emit, but we are looking for something more general that would not require users to do anything, just logging to console

[mroy-seedbox]

It would be amazing if there were an easy way to setup sinks for those logs, such as Amazon Kinesis.

An API would also be fine (with the ability to retrieve logs for a given date range).