What would you like to happen?
Description:
Apache Hop already provides logging and error handling features at different levels, but it still lacks a centralized and standardized mechanism to capture failures with full execution context.
The proposal is to add a new metadata type, or an equivalent feature, that allows defining an error trigger for pipelines and workflows. Whenever a failure occurs, this trigger should invoke a dedicated error-handling pipeline and send a structured payload containing as much execution context as possible.
Objective:
Enable full observability and analysis of failures, with enough information for auditing, troubleshooting, monitoring, and possible reprocessing.
Expected error event payload:
project and environment;
source workflow or pipeline name;
transform or action that failed;
error message;
stack trace or technical cause, when available;
timestamp;
error status and level;
relevant parameters and variables;
log channel / execution id;
error row metadata, when applicable;
row data that caused the error, when available.
Expected behavior:
configure a reusable error pipeline through metadata;
allow usage at both pipeline and workflow levels;
support the distinction between row-level errors, pipeline execution errors, and workflow errors;
send partial payloads when some details are not available;
allow future extension for sensitive data masking, payload limits, and retry policies.
Benefits:
centralized error handling;
standardized observability;
faster diagnosis;
better operational governance;
foundation for alerts, auditing, and reprocessing.
Note:
The goal is not to replace the current logging mechanisms, but to complement them with a dedicated enriched failure capture feature.
Issue Priority
Priority: 3
Issue Component
Component: Transforms, Component: Metadata
What would you like to happen?
Description:
Apache Hop already provides logging and error handling features at different levels, but it still lacks a centralized and standardized mechanism to capture failures with full execution context.
The proposal is to add a new metadata type, or an equivalent feature, that allows defining an error trigger for pipelines and workflows. Whenever a failure occurs, this trigger should invoke a dedicated error-handling pipeline and send a structured payload containing as much execution context as possible.
Objective:
Enable full observability and analysis of failures, with enough information for auditing, troubleshooting, monitoring, and possible reprocessing.
Expected error event payload:
project and environment;
source workflow or pipeline name;
transform or action that failed;
error message;
stack trace or technical cause, when available;
timestamp;
error status and level;
relevant parameters and variables;
log channel / execution id;
error row metadata, when applicable;
row data that caused the error, when available.
Expected behavior:
configure a reusable error pipeline through metadata;
allow usage at both pipeline and workflow levels;
support the distinction between row-level errors, pipeline execution errors, and workflow errors;
send partial payloads when some details are not available;
allow future extension for sensitive data masking, payload limits, and retry policies.
Benefits:
centralized error handling;
standardized observability;
faster diagnosis;
better operational governance;
foundation for alerts, auditing, and reprocessing.
Note:
The goal is not to replace the current logging mechanisms, but to complement them with a dedicated enriched failure capture feature.
Issue Priority
Priority: 3
Issue Component
Component: Transforms, Component: Metadata