Consolidate the writing of logs into a common place (kafka topic, etc) across the Ingest/Enrich pipeline so that triage of problems becomes more straightforward. In particular with the async processing, when the data is not successfully processed all the way through it is challenging to know where the failure is.
Consolidate the writing of logs into a common place (kafka topic, etc) across the Ingest/Enrich pipeline so that triage of problems becomes more straightforward. In particular with the async processing, when the data is not successfully processed all the way through it is challenging to know where the failure is.