Data Flow Logs
Article Type: Concept Audience: Solution Architects, Solution Engineers, Project Managers, System Administrators Module: System/Orchestration Applies to Versions: All Versions
Data Flow Logs provide comprehensive execution visibility for Data Flows within the Fuuz platform, capturing detailed information about every node execution, error occurrence, and informational message generated during data flow operations. This centralized logging system enables teams to monitor integrations in real-time, diagnose execution failures, validate deployments, analyze performance patterns, and maintain compliance with audit requirements.
The Data Flow Logs screen serves as the primary interface for accessing execution history across all data flows, providing searchable, filterable views with drill-down capabilities into individual log entries. Each log entry contains node-level attribution, execution context, metadata, and timestamps that enable comprehensive analysis of integration behavior over time.
Note: Data Flow Logs are scoped per application and per environment. Logs provide read-only execution records that cannot be manually created, modified, or deleted, ensuring audit trail integrity.
- Data Flow Deployment Log: Individual log entry (record) capturing a single event during data flow execution, including timestamp, message, node context, and metadata.
- Log Type: Categorization of log entries by severity and purpose: Info (informational), log (custom logging), Error (execution failure), ThrowError (explicit error), or other custom types.
- Node Context: Attribution information identifying which specific node within a data flow generated the log entry, including Node Id, Node Name, and Node Type.
- Execution Message Id: Unique identifier correlating all log entries generated during a single data flow execution instance.
- Metadata: Structured JSON data containing detailed execution context, error information, debug flags, runtime configuration, and other technical details.
- Data Flow Version: Specific version of the data flow that was executing when the log entry was generated, enabling behavior tracking across deployments.
- Payload: JSON data being processed at the time the log was generated (when applicable and captured).
- Short Message: Truncated version of the log message displayed in table view for quick scanning.
- Orchestration Engine: Core execution engine that runs data flows and generates log entries at key execution points.
- Log Generator: Component within the orchestration engine that creates log records asynchronously during flow execution to avoid performance impact.
- Log Database: Persistent storage for log records with full relational context linking to data flows, deployments, versions, and users.
- Query Engine: Filtering and search system optimized for performance, executing against read replicas with support for multi-dimensional filtering.
- Data Flow Logs Screen: User interface providing searchable table view and detailed log inspection capabilities.
When a data flow executes, the orchestration engine generates logs based on node configuration and execution results:
| Step | Description |
|---|---|
| 1. Execution Trigger | Data flow deployment initiates execution via schedule, manual trigger, or event-driven activation |
| 2. Node Processing | Orchestration engine processes nodes sequentially or in parallel according to flow logic |
| 3. Log Event Generation | Nodes generate log events based on configuration: explicit log nodes, errors, query operations, transformations |
| 4. Asynchronous Capture | Log generator captures events asynchronously to avoid impacting flow execution performance |
| 5. Context Enrichment | System adds execution context: timestamps, user attribution, data flow references, node details, version information |
| 6. Persistence | Log records are persisted to database with full relational integrity |
| 7. Availability | Logs become available for viewing in Data Flow Logs screen within seconds of generation |
Each DataFlowDeploymentLog record maintains relationships with multiple platform entities:
| Related Entity | Relationship Purpose |
|---|---|
| DataFlow | Links to data flow definition, enables navigation to flow configuration and understanding of intended behavior |
| DataFlowVersion | Identifies specific version executing, critical for comparing behavior across versions and deployment tracking |
| DataFlowDeployment | References deployment configuration including schedules, triggers, and environment settings |
| User (Executed By) | Tracks which user triggered execution, important for audit trails and understanding execution context |
| User (Created By) | Records system user that created log entry, typically orchestration engine service account |
| DataFlowDeploymentLogType | Categorizes log entry by type (Info, Error, log, ThrowError), enables filtering and severity-based analysis |
| SubstitutedByDataFlow | Optional reference when data flow substitution occurred, showing which flow actually executed instead of configured flow |
- Integration Failure Diagnosis: Solution engineer receives report that inventory sync integration stopped working overnight. Filter logs by data flow name and log type "Error" for past 24 hours. Examine error messages revealing API authentication token expired. Navigate to deployment configuration to update credentials.
- Deployment Validation: Solution architect deploys new version (0.0.5) of order processing flow to production. Filter logs by data flow name and version 0.0.5 from deployment timestamp forward. Verify expected log messages appear from custom log nodes. Compare error rates between version 0.0.4 and 0.0.5 to confirm no regression.
- Scheduled Execution Monitoring: Project manager validates that hourly data synchronization runs on schedule. Filter logs by data flow name, set date range to past week. Review Created At timestamps to confirm executions occur every hour as configured. Identify gap in execution pattern indicating missed schedule.
- Performance Analysis: Solution engineer investigates why data flow execution time increased from 2 minutes to 15 minutes. Filter logs by data flow name and execution message ID to see all logs from single execution. Analyze timestamp gaps between sequential log entries to identify which query node consumes excessive time.
- Compliance Audit Trail: Administrator provides evidence that required data transformations executed during audit period. Filter logs by data flow name and date range covering audit timeframe. Export log entries showing execution timestamps, user attribution, and completion status. Demonstrate consistent execution pattern meeting compliance requirements.
- Error Pattern Identification: Solution engineer notices increasing customer complaints about missing data. Filter logs by log type "Error" across all data flows for past month. Identify recurring error message pattern affecting specific query node. Cross-reference error occurrences with deployment dates to correlate with recent database schema change.
- Node-Level Debugging: Developer troubleshoots complex transformation logic in multi-node data flow. Filter logs by specific node name to isolate messages from problematic transformation. Review metadata section showing transformation input and output values. Identify data type mismatch causing calculation errors.
- Cross-Version Comparison: Solution architect evaluates whether version upgrade resolved intermittent timeout issues. Filter logs showing errors from version 0.0.3 during past month. Compare against error logs from version 0.0.4 deployed two weeks ago. Confirm timeout errors eliminated in new version.
- Real-Time Monitoring: Operations team monitors critical integration during business hours. Continuously refresh Data Flow Logs screen filtered to specific high-priority data flow. Watch for error log entries appearing in real-time. Immediately investigate and resolve issues before they impact operations.
- Knowledge Transfer: New solution engineer onboards to maintain existing integrations. Review historical logs from production data flows to understand normal execution patterns. Examine custom log messages documenting business logic decisions. Use logs as living documentation of actual system behavior.
The Data Flow Logs screen is available in the App Admin navigation — open the Data Flow Logs screen to begin.
The filter panel at the top of the screen provides multi-dimensional filtering capabilities:
| Filter | Type | Description |
|---|---|---|
| Start Date | Date Picker | Beginning of date range to search. Click calendar icon to select date. |
| End Date | Date Picker | End of date range to search. Leave empty to search through current time. |
| Log Type | Multi-Select | Filter by one or more log types: Info, Error, log, ThrowError, etc. |
| Data Flow | Lookup | Select one or more data flows to filter results. Begin typing to search. |
| Substituted By Data Flow | Lookup | Filter by substitute data flow when flow substitution is configured. |
| Data Flow Version | Lookup | Filter by specific version number to compare behavior across versions. |
| Node Id | Text | Enter specific node ID to find all logs from that node. |
| Node Name | Text | Search for logs from nodes with specific names or name patterns. |
| Node Type | Text | Filter by node type: query, log, ThrowError, RewriteError, etc. |
| Message | Text | Search for specific text within log messages. |
- Search: Blue magnifying glass icon button executes query based on filter criteria. Refreshes log table with matching results.
- Create (Disabled): Grayed out plus icon — logs cannot be manually created.
- Update (Disabled): Grayed out pencil icon — logs are read-only and cannot be modified.
- Delete (Disabled): Grayed out trash icon — logs cannot be manually deleted to ensure audit integrity.
The main table displays log entries with the following columns (default view):
| Column | Description |
|---|---|
| Created At | Timestamp when log entry was created (format: L h:mm:ss.SSS A with milliseconds). Clickable link to detail view. |
| Data Flow Deployment Log Type | Type categorization: Info, log, Error, ThrowError, or other custom types |
| Node Type | Type of node that generated log: query, log, ThrowError, RewriteError, etc. |
| Node Name | Human-readable name assigned to node (e.g., "Query: SCHED", "Log: DC CREATE MAIN WO SCHEDULE") |
| Data Flow | Name of data flow that generated this log. Clickable link to data flow configuration. |
| Substituted By Data Flow | If flow substitution occurred, shows which flow actually executed. Typically null unless substitution configured. |
| Data Flow Version | Version number of data flow that was executing (e.g., 0.0.2, 0.0.5). Enables version comparison. |
| Node Id | Unique identifier of node within data flow. Useful for debugging and precise node identification. |
| Short Message | Truncated log message for quick scanning. Click timestamp to see complete message in detail view. |
Table Behavior: Results sorted by Created At in descending order (newest first) by default. Maximum 500 records displayed per query. Use filters to narrow results for queries that would exceed this limit.
Clicking on a log entry's Created At timestamp navigates to the Data Flow Deployment Log detail page, displaying comprehensive information organized in sections:
| Section | Information Displayed |
|---|---|
| Info Section | Error status (true/false), execution message ID, node ID, node name, data flow deployment link |
| Message Section | Complete message text including error details, status information, or custom log messages (untruncated) |
| Metadata Section | Structured JSON data: error details, message transformers, debug flags, execution context, line numbers, stack traces |
| Payload Section | Data payload at time log was generated (when applicable and captured) |
| Context Section | Execution context information (when applicable) |
| History Section | Created timestamp and user, updated timestamp and user, with clickable links to user records |
Navigation from Detail View: Click data flow deployment link to view deployment configuration. Click user email links to view user information. Use browser back button or breadcrumb navigation to return to log list.
Clicking on a log entry's Data Flow link will open the data flow detail page, displaying comprehensive information about the data flow that generated the log. This page is also organized in sections to aid in troubleshooting:
Data Flow Logs are categorized into distinct types indicating severity and purpose:
| Type | Description and Usage |
|---|---|
| Info | Informational messages about normal operations. Indicates events occurred as expected, marks significant milestones. No action required but provides valuable execution context. |
| log | Generated by explicit log nodes within data flows. Custom messages defined by designer for debugging, monitoring, or audit trails. Content and frequency controlled by flow configuration. |
| Error | Indicates error during execution preventing node from completing successfully. Contains detailed error messages, stack traces, execution context. Does not necessarily mean entire flow failed if error handling logic exists. |
| ThrowError | Explicitly thrown errors where designer intentionally triggers error condition. Used for validation failures, business rule violations, situations requiring execution halt. May include custom error messages as part of error handling strategy. |
Each log entry's metadata section contains structured JSON providing deep execution insights:
- Error Information (when present): Error type classification, line numbers where errors occurred in code, detailed error message text, stack traces for debugging, catchable/fatal error indicators, message transformers specifying error formatting
- Debug Flags: Boolean values indicating enabled debugging features: database operation debug mode, modality-based redirects, execution flow markers, message transformers, other runtime configuration settings
- Execution Context: Line and column positions in data flow definition, extension configurations, accessor details, variable states, configuration values, other runtime conditions
The Execution Message Id serves as correlation key linking all logs from a single data flow run. This enables:
- Complete Execution Narrative: View all logs sharing same execution ID to see entire execution sequence from start to finish, including successful steps before failures
- Performance Analysis: Analyze timestamp gaps between sequential logs from same execution to identify slow nodes and bottlenecks
- Context Understanding: Review data transformations, decision points, and state changes leading to specific outcomes within single execution instance
Each log entry records the data flow version that was executing, creating natural audit trail of behavior changes:
- Version Comparison: Filter logs by version number to compare error patterns, execution times, and behavior between old and new versions after deployment
- Deployment Validation: Verify new version generates expected log messages, handles errors correctly, performs within requirements by examining logs from test executions
- Regression Detection: Identify unexpected behavior changes by comparing log patterns before and after version deployment
When data flow substitution occurs (one flow replaces another during execution), logs record both the originally configured flow and the substitute flow that actually executed. The "Substituted By Data Flow" field shows which flow ran instead of configured flow. This transparency ensures logs accurately represent what occurred while maintaining connection to original deployment intent. Substitution is typically used for testing, gradual rollouts, or environment-specific implementations.
- Read Replica Execution: Queries execute against read replicas to avoid impacting transactional operations
- Indexed Fields: Frequently queried fields (timestamps, data flow ID, log type) are indexed for fast filtering
- Result Limit: 500-record maximum per query maintains responsive performance. Use filters to narrow results for high-volume queries.
- Sort Order: Default descending sort by Created At (newest first) optimized for recent log review
Important: Data Flow Logs capture execution events but do not automatically implement comprehensive logging. Logging behavior depends on data flow configuration:
- Automatic Logs: System automatically generates Info logs for significant execution events, Error logs for execution failures, and integration logs for API operations when using integration nodes
- Custom Logging: Add explicit log nodes within data flows to capture execution details, intermediate data, processing steps, business events, decision points
- Error Handling Logs: Configure error handling nodes to capture failure details and stack traces
- Real-Time Availability: Logs appear in Data Flow Logs screen within seconds of generation, enabling real-time monitoring
Data Flow Logs access is controlled through Fuuz's policy-based permission system:
- Default Access Levels: App Administrators and Developers have full access to view logs. Web Access Users have no default access and require explicit policy grants.
- Read-Only Nature: All users with access have read-only permissions. Logs cannot be created, modified, or deleted manually regardless of user role.
- Data Flow Scoping: Users can only view logs for data flows they have permission to access, ensuring sensitive integration logs remain protected
- Audit Trail: User attribution in logs shows which users initiated executions, creating accountability for triggered operations
- Per Application: All logs are scoped to specific applications — each application has independent log storage
- Per Environment: Logs exist separately in build, QA, and production environments
- Retention Policies: Log retention may be subject to platform retention policies. Consult with system administrators regarding specific retention periods.
- No Cross-Environment Replication: Logs do not automatically transfer between environments
- Related KB Article: Flow-Schedules
- Related KB Article: Integration Logs
- Related KB Article: App Span Event Logs
- Related KB Article: Data Model Deployment Logs
- Related KB Article: Access-Control
- Platform Information: www.fuuz.com
| Issue | Cause | Resolution |
|---|---|---|
| No logs appearing in results | Date range too restrictive, data flow hasn't executed, too many filters applied, insufficient permissions | Verify Start/End Date are correct and cover expected execution period. Clear all filters and search again. Confirm data flow has executed by checking deployment schedule. Verify you have permissions to view this data flow. Remove filters one at a time to identify which is too restrictive. |
| Error messages are unclear or truncated | Viewing only Short Message column in table view, error metadata requires deeper analysis | Click timestamp to open full detail view. Review complete Message section for full error text. Examine Metadata section for additional error context including line numbers and error types. Navigate to data flow deployment to review node configuration. Check related logs from same execution ID. |
| Cannot find specific execution instance | Execution outside selected date range, execution completed without generating logs, logs archived per retention policy, execution failed to start | Expand date range to cover wider time period. Verify execution actually occurred by checking schedules or other system logs. Check if retention policies may have archived older entries. Review deployment status to ensure it's active. Search using Execution Message Id if known. |
| Too many log results to review effectively | Date range too broad, high-frequency data flow, insufficient filtering, multiple data flows selected | Narrow date range to more specific time period. Apply Log Type filters to focus on specific categories (e.g., only errors). Filter by specific Node Name or Node Type to examine particular nodes. Use Message filter to search for specific text patterns. Consider 500-row limit and adjust filters to see most relevant results first. |
| Log detail view missing expected information | Payload/Context sections may be null if not applicable, node configuration determines logged information | Understand that null values in Payload and Context sections are normal for certain log types. Review Metadata section which typically contains most detailed information. Examine related log entries from same execution to gather complete context. If more detailed logging needed, consider adding custom log nodes to data flow. Check data flow configuration to understand what information should be available. |
| Performance degradation when querying logs | Very broad queries across large time ranges on high-volume flows | Use specific filters to reduce query scope. Filter by data flow name before expanding time ranges. Narrow date range to essential period. Apply log type filters to reduce result set. Use node filters when investigating specific components. |
| Data flow executed but no custom logs appear | No log nodes configured in data flow, log nodes not in execution path, conditional logic skipping log nodes | Review data flow design to verify log nodes are present and in execution path. Check for conditional logic that might bypass log nodes. Look for system-generated Info or Error logs to confirm execution occurred. Add explicit log nodes to data flow if more logging needed. |
| Cannot correlate logs from same execution | Don't know Execution Message Id to filter by | Click on any log from desired execution to view detail. Copy Execution Message Id from Info section. Return to log list and filter by that Execution Message Id. All logs sharing same ID are from same execution instance. |
- Start Broad, Then Narrow: Begin with broad filters (date range, data flow name), then progressively narrow with log type, node filters, and message content. This helps understand overall patterns before drilling into specifics.
- Use Descriptive Node Names: When designing data flows, use descriptive node names that clearly indicate purpose. This makes logs immediately interpretable without referencing flow design (e.g., "Query: Inventory Lookup" vs "Node1").
- Implement Strategic Logging: Add explicit log nodes at key decision points, major state changes, external system interactions, and data transformations. Balance comprehensive visibility with avoiding excessive log volume.
- Document Common Errors: Maintain knowledge base of frequently occurring error patterns, root causes, and resolutions. Reference specific error messages from logs to accelerate future troubleshooting.
- Establish Monitoring Routines: Schedule regular log reviews for critical data flows: daily error log checks, weekly execution frequency validation, monthly performance trend analysis.
- Use Version Comparison: When deploying new data flow versions, filter logs by version number to compare behavior. Validate that new versions maintain expected patterns and don't introduce new errors.
- Leverage Execution Correlation: When troubleshooting failures, use Execution Message Id to view all logs from failed execution. This reveals complete context including successful steps before failure.
- Monitor Real-Time: For critical integrations during deployment or high-risk periods, keep Data Flow Logs screen open with appropriate filters and refresh periodically to catch issues immediately.
- Understand Metadata: Invest time learning metadata structure for your common error types. Metadata often contains technical details needed for root cause analysis that aren't in message text.
- Coordinate with Team: Establish team conventions for node naming, log message formats, and error handling patterns. Consistency across data flows makes logs more valuable for entire team.
| Version | Date | Editor | Description |
|---|---|---|---|
| 1.0 | 2024-12-28 | Fuuz Documentation Team | Initial Release |
- Flow-Schedules
- Debugging-and-Testing-our-Fuuz-Data-Flows
- Debugging-Data-Flow-Node-Types
- How-to-on-Best-Practices-for-Designing-Flows-in-Fuuz
- Python-and-Fuuz-Data-Flows
- Access-Control
Source: support.fuuz.com