diff --git a/docs/assets/flows/abort-icon.png b/docs/assets/flows/abort-icon.png new file mode 100644 index 000000000..93f667811 Binary files /dev/null and b/docs/assets/flows/abort-icon.png differ diff --git a/docs/assets/flows/abort-setting.png b/docs/assets/flows/abort-setting.png new file mode 100644 index 000000000..14c7f2171 Binary files /dev/null and b/docs/assets/flows/abort-setting.png differ diff --git a/docs/assets/flows/abort.png b/docs/assets/flows/abort.png new file mode 100644 index 000000000..24aa36d31 Binary files /dev/null and b/docs/assets/flows/abort.png differ diff --git a/docs/assets/flows/execute-manual.png b/docs/assets/flows/execute-manual.png new file mode 100644 index 000000000..6a2b618a9 Binary files /dev/null and b/docs/assets/flows/execute-manual.png differ diff --git a/docs/assets/flows/exisiting-doc-light.png b/docs/assets/flows/existing-doc-light.png similarity index 100% rename from docs/assets/flows/exisiting-doc-light.png rename to docs/assets/flows/existing-doc-light.png diff --git a/docs/assets/flows/manuall-flow-light.png b/docs/assets/flows/manual-flow-light.png similarity index 100% rename from docs/assets/flows/manuall-flow-light.png rename to docs/assets/flows/manual-flow-light.png diff --git a/docs/assets/flows/manual-list.png b/docs/assets/flows/manual-list.png new file mode 100644 index 000000000..8812098b7 Binary files /dev/null and b/docs/assets/flows/manual-list.png differ diff --git a/docs/assets/flows/note-flow.png b/docs/assets/flows/note-flow.png new file mode 100644 index 000000000..72f7e02fc Binary files /dev/null and b/docs/assets/flows/note-flow.png differ diff --git a/docs/assets/flows/sort-light-85.png b/docs/assets/flows/sort-light-85.png index 0cede2ce9..fc5f619a9 100644 Binary files a/docs/assets/flows/sort-light-85.png and b/docs/assets/flows/sort-light-85.png differ diff --git a/docs/assets/flows/sort-light-86.png b/docs/assets/flows/sort-light-86.png index cd81fb9cd..de082900b 100644 Binary files a/docs/assets/flows/sort-light-86.png and b/docs/assets/flows/sort-light-86.png differ diff --git a/docs/flows/abort.md b/docs/flows/abort.md new file mode 100644 index 000000000..d38f26151 --- /dev/null +++ b/docs/flows/abort.md @@ -0,0 +1,17 @@ +# Abort Flow + +Users can abort a flow that is currently triggered or running to stop further execution. + +**Step 1:** Open the flow execution that is currently in progress and select the **settings** icon. + +![chart](.././assets/flows/abort-setting.png) + +**Step 2:** Click on the **abort** to stop the execution. + +![chart](.././assets/flows/abort.png) + +A success message appears confirming the flow is aborted. + +Alternatively, you can abort the flow from the Executions tab by clicking the **Abort** icon. + +![chart](.././assets/flows/abort-icon.png) diff --git a/docs/flows/activate-flow.md b/docs/flows/activate-flow.md new file mode 100644 index 000000000..47689591a --- /dev/null +++ b/docs/flows/activate-flow.md @@ -0,0 +1,9 @@ +# Activate Flow + +Users can reactivate a flow that was previously deactivated. Once reactivated, the flow’s triggers become active again, allowing it to run automatically based on the defined conditions. + +**Step 1**: Click the **vertical ellipsis (⋮)** next to the flow that you want to activate, then click on **Activate** from the dropdown menu. + +![activate](.././assets/flows/activate-flow-light.png) + +After clicking the **Activate** button, a success notification appears confirming the activation. diff --git a/docs/flows/add-flow.md b/docs/flows/add-flow.md new file mode 100644 index 000000000..a57fb5443 --- /dev/null +++ b/docs/flows/add-flow.md @@ -0,0 +1,44 @@ +# Create a New Flow + +Flows allow you to automate actions based on data operations, anomaly detection, or manual execution. When you create a new Flow, Qualytics automatically adds the base nodes you need to begin configuring the automation. + +!!! note + To create a Flow, you must have **Manager** role. For more details, see the [Team Permissions guide](../settings/security/team-permissions.md){target="_blank"}. + +**Step 1**: Click on the **Add Flow** button from the top right corner. + +![addflow](.././assets/flows/addflow-light-4.png) + +A modal window will appear, allowing you to create a new Flow. +Every Flow begins with two default nodes: + +- **Flow Node** – Defines the Flow’s name, description, and activation state +- **Trigger Node** – Defines when the Flow will start + +![flowchart](.././assets/flows/flowchart-light-5.png) + +## What You Configure Next + +A Flow consists of three main node types. After creating a new Flow, you will configure each one: + +### **1. Flow Entry** + +Defines the Flow’s general information such as its name and description. + +!!! info + If you want to learn more about flow entry, check out the [Flow Entry](../flows/flow-entry.md) guide. + +### **2. Trigger Node** + +Controls *when* the Flow starts (operation completes, anomaly detected, manual, etc.). + +!!! info + For more detailed steps, you can review the [Trigger Node Documentation](../flows/trigger-node.md). + +### **3. Actions Node** + +Defines *what happens* after the Flow is triggered (operations, notifications, HTTP calls, etc.). + +!!! info + To explore this in more depth, take a look at the [Actions Node Documentation](../flows/overview-action.md). + \ No newline at end of file diff --git a/docs/flows/clone-a-flow.md b/docs/flows/clone-a-flow.md new file mode 100644 index 000000000..3bdcdf6ac --- /dev/null +++ b/docs/flows/clone-a-flow.md @@ -0,0 +1,20 @@ +# Clone a Flow + +Users can duplicate existing flows to simplify the reuse and modification of flow configurations for similar scenarios. + +**Step 1**: Click on the existing flow you want to clone. + +![panel](.././assets/flows/existing-doc-light.png) + +**Step 2**: A new window will open displaying the flow's detailed configuration. Click the settings icon and select **Clone**. + +![panel](.././assets/flows/clone-light.png) + +**Step 3:** After selecting the clone button, click the **Publish** button to publish it. + +![panel](.././assets/flows/publish-light-82.png) + +!!! important + Users with the **Manager Role** can create, edit, clone, or publish Flows. For details, see the [Team Permission ](../settings/security/team-permissions.md){target="_blank"}. + +After clicking the **Publish** button, a success notification appears confirming that the flow has been successfully added. diff --git a/docs/flows/clone-an-action.md b/docs/flows/clone-an-action.md new file mode 100644 index 000000000..30d825a97 --- /dev/null +++ b/docs/flows/clone-an-action.md @@ -0,0 +1,14 @@ +# Clone an Action + +Users can duplicate an existing action in just a few clicks. Cloning an action allows you to quickly replicate its configuration without manually setting it up again. + +**Step 1:** Click the vertical ellipsis (**⋮**) on the action you want to clone, then select the **Clone** option from the menu. + +!!! info + You can add a maximum of **10 Operation** Actions (such as Catalog, Profile, or Scan) in a single flow. + +![vertical](.././assets/flows/vertical-light.png) + +**Step 2:** After clicking the Clone option, a cloned action will be created. + +![clone](.././assets/flows/clones-light.png) diff --git a/docs/flows/deactivate-flow.md b/docs/flows/deactivate-flow.md new file mode 100644 index 000000000..86aa8b2f2 --- /dev/null +++ b/docs/flows/deactivate-flow.md @@ -0,0 +1,25 @@ +# Deactivate Flow + +Users can deactivate a flow to pause its triggers by disabling it. This prevents the flow from being executed until it is reactivated. + +!!! info + To deactivate a Flow, you must have **Manager** role. Only users with this role can perform the deactivation action. + For more details about permissions, refer to the [permissions documentation](../settings/security/team-permissions.md){target="_blank"}. + +**Step 1**: Click the **vertical ellipsis (⋮)** next to the flow that you want to deactivate, then click on **Deactivate** from the dropdown menu. + +![deactivate](.././assets/flows/deactivate-light-95.png) + +!!! note + If you accidentally deactivate a flow, you can easily reactivate it by following the steps in the [Activate Flow guide](../flows/activate-flow.md){target="_blank"}. + +After clicking the **Deactivate** button, a success notification appears confirming the deactivation. + +!!! warning + Deactivating a flow only pauses its triggers — not deletes it. You can reactivate it anytime, or see the [Delete Flow](../flows/delete-flow.md){target="_blank"} guide to remove it permanently. + +## FAQ + +**1>.** What happens if I deactivate a Flow? + +Once deactivated, the Flow’s triggers stop executing until you reactivate it. Existing executions will not be affected. \ No newline at end of file diff --git a/docs/flows/delete-flow.md b/docs/flows/delete-flow.md new file mode 100644 index 000000000..e507627d5 --- /dev/null +++ b/docs/flows/delete-flow.md @@ -0,0 +1,25 @@ +# Delete Flow + +**Delete Flow** feature allows you to permanently remove unwanted or outdated flows from the system. This helps in maintaining a clean and organized list of active flows. + +!!! info + To delete a Flow, you must have **Manager** role. Only users with this role can perform the deletion action. + For more details about permissions, refer to the [permissions documentation](../settings/security/team-permissions.md){target="_blank"}. + + +**Step 1**: Click the **vertical ellipsis (⋮)** next to the flow that you want to delete, then click on **Delete** from the dropdown menu. + +![delete](.././assets/flows/delete-light-91.png) + +After clicking the delete button, a confirmation modal window **Delete Flow** will appear. + +![delete](.././assets/flows/delete-light-92.png) + +**Step 2**: Click on the **Delete** button to delete the flow. + +![delete](.././assets/flows/delete-light-93.png) + +After clicking the **Delete** button, a success notification appears confirming the deletion. + +!!! warning + This action cannot be undone. If you only want to pause the flow instead of deleting it, see the [Deactivate Flow guide](../flows/deactivate-flow.md){target="_blank"}. \ No newline at end of file diff --git a/docs/flows/edit-flow.md b/docs/flows/edit-flow.md new file mode 100644 index 000000000..9490ad7cf --- /dev/null +++ b/docs/flows/edit-flow.md @@ -0,0 +1,31 @@ +# Edit Flow + +**Edit Flow** feature lets users update existing flows by modifying configurations or adding actions. + +!!! info + To edit a Flow, you must have **Manager** role. Only users with this role can perform the edit action. + For more details about permissions, refer to the [permissions documentation](../settings/security/team-permissions.md){target="_blank"}. + +**Step 1:** Click the flow you want to edit. + +![panel](.././assets/flows/panel-light-87.png) + +**Step 2:** After clicking the flow, a new window will open displaying the flow's detailed configuration. Click on the boxes you want to edit. + +For demonstration purposes we have selected the Flow node. + +![result](.././assets/flows/result-light-88.png) + +!!! tip + If you make changes that you don’t want to keep, you can undo them by discarding the unsaved configuration or refreshing the page before publishing. + +**Step 3:** Click the **`Save`** button to apply the updates. + +![save](.././assets/flows/save-light-89.png) + +**Step 4:** After clicking the **`Save`** button, click the **`Publish`** button located in the top right corner to finalize and publish the changes. + +!!! warning + If the flow is not published, the setup will not be saved. Always publish the flow after editing to ensure your changes are retained. + +![publish](.././assets/flows/publish-light-82.png) \ No newline at end of file diff --git a/docs/flows/execute-manual-flows.md b/docs/flows/execute-manual-flows.md new file mode 100644 index 000000000..4880a0a79 --- /dev/null +++ b/docs/flows/execute-manual-flows.md @@ -0,0 +1,29 @@ +# Execute Manual Flows + +Manual execution gives you complete control over when a Flow should run. You can start a manual Flow directly from the Flows list or from inside the Flow editor. + +## Method 1: Execute From the Flows List + +**Step 1:** Locate the manual flow in your list of flows. + +![manual-flow](.././assets/flows/manual-flow-light.png) + +**Step 2:** Click the **vertical ellipsis (⋮)** next to the manual flow you wish to execute, then select **"Execute"** from the dropdown menu to trigger the flow. + +![manual-flow](.././assets/flows/verticall-light.png) + +After clicking the **Execute** button, a success notification appears confirming that the flow has been successfully executed. + +## Method 2: Execute From Inside the Flow Editor + +You can also run a manual Flow directly from its editor page. + +**Step 1:** Click on the manual flow in your list to open it. + +![manual-flow](.././assets/flows/manual-list.png) + +**Step 2:** You will be navigated to the Flow editor page; click the **Execute** button. + +![manual-flow](.././assets/flows/execute-manual.png) + +After clicking the **Execute** button, a success notification appears confirming that the flow has been successfully executed. \ No newline at end of file diff --git a/docs/flows/flow-entry.md b/docs/flows/flow-entry.md new file mode 100644 index 000000000..88273b925 --- /dev/null +++ b/docs/flows/flow-entry.md @@ -0,0 +1,34 @@ +# Flow Entry + +The **Flow Entry** is the starting point of every Flow in Qualytics. Think of it as the “root” of your automation — it defines the Flow’s name, purpose, and whether it is active. Once this node is set, you can continue building the rest of the Flow (Triggers, Actions, etc.). + +## Trigger Types + +A Flow begins when one of its Triggers activates. Here are the four available trigger types: + +| Trigger Type | Description | +|--------------|-------------| +| [**Operation Completes**](../flows/trigger-node.md#operation-completes){target="_blank"} | Starts the Flow when a catalog, scan, or profile operation finishes. | +| [**Anomalous Table & File Detected**](../flows/trigger-node.md#anomalous-table-and-file-detected){target="_blank"} | Starts the Flow when anomalies are detected in a table or file. | +| [**Anomaly Detected**](../flows/trigger-node.md#anomaly-detected){target="_blank"} | Starts the Flow when a single anomaly event occurs. | +| [**Manual**](../flows/trigger-node.md#manual){target="_blank"} | Starts the Flow only when the user manually executes it. | + +## Configure the Flow Node + +**Step 1:** Click on the **Flow** node. + +![flow](.././assets/flows/flow-light-6.png) + +A panel will appear on the right-hand side, allowing you to: + +| No. | Field Name | Description | +| :---- | :---- | :---- | +| 1. | **Name** | Enter the name for the flow. | +| 2. | **Description** | Provide a brief description of the flow (optional) to clarify its purpose or functionality. | +| 3. | **Deactivated** | Check the box to deactivate the flow. If selected, the flow won't start even if the trigger conditions are met. | + +![flow](.././assets/flows/flowsetting-light-7.png) + +**Step 2:** Once the details are filled in, click the **Save** button to save the flow settings. + +![save](.././assets/flows/save-light-8.png) \ No newline at end of file diff --git a/docs/flows/flows.md b/docs/flows/flows.md deleted file mode 100644 index 06c8f04ba..000000000 --- a/docs/flows/flows.md +++ /dev/null @@ -1,909 +0,0 @@ -# Flows - -Flows enable users to create pipelines by chaining actions and configuring how they are triggered. Triggers can be set based on predefined events and filters, offering a flexible and efficient way to automate processes. These actions can be notifications or operations, allowing users to inform various notification channels or execute tasks based on specific operations. - -## Navigation to Flows - -**Step 1**: Log in to your Qualytics account and click on **Flows** on the left side panel of the interface. - -![flows](.././assets/flows/flow-light-1.png) - -You will navigate to the Flows interface, where you can add and manage flows. At the top, you will see two tabs: - -* **Definitions:** Displays a list of all flows along with details like triggers, actions, tags, and the last triggered time. - -![definition](.././assets/flows/definitions-light-2.png) - -* **Executions:** Provides the execution history of flows, including their status and timestamps. - -![execution](.././assets/flows/execution-light-3.png) - -## Add Flow - -**Step 1**: Click on the **Add Flow** button from the top right corner. - -![addflow](.././assets/flows/addflow-light-4.png) - -A modal window, **Add Flow**, will appear, providing options to create a flow. Each flow starts by default with two nodes: **Flow** and **Trigger**. - -![flowchart](.././assets/flows/flowchart-light-5.png) - -### Flow - -**Step 1:** Click on the **Flow** node. - -![flow](.././assets/flows/flow-light-6.png) - -A panel will appear on the right-hand side, allowing you to: - -| No. | Field Name | Description | -| :---- | :---- | :---- | -| 1. | **Name** | Enter the name for the flow. | -| 2. | **Description** | Provide a brief description of the flow (optional) to clarify its purpose or functionality. | -| 3. | **Deactivated** | Check the box to deactivate the flow. If selected, the flow won't start even if the trigger conditions are met. | - -![flow](.././assets/flows/flowsetting-light-7.png) - -**Step 2:** Once the details are filled in, click the **Save** button to save the flow settings. - -![save](.././assets/flows/save-light-8.png) - -### Trigger - -**Step 1:** After completing the **"Flow"** node setup, users can click on the **"Trigger"** node. - -![trigger](.././assets/flows/trigger-light-9.png) - -A panel will appear on the right-hand side, enabling users to define when the flow should start. The panel provides four options for initiating the flow. Users can choose one of the following options: - -* Operation Completes. - -* Anomalous Table and File Detection. - -* Anomaly Detected. - -* Manual - -![triggersetting](.././assets/flows/triggersetting-light-10.png) - -#### Operation Completes - -This type of flow is triggered whenever an operation, such as a catalog, profile, or scan, is completed on a source datastore. Upon completion, teams are promptly notified through in-app messages and, if configured, via external notification channels such as email, Slack, Microsoft Teams, and others. For example, the team is notified whenever the catalog operation is completed, helping them proceed with the profile operation on the datastore. - -![operation](.././assets/flows/operation-light-11.png) - -**Filter Conditions** - -Filters can be set to narrow down which operations should trigger the flow execution: - -1. **Source Datastore Tags**: The flow is triggered only for source datastores that have all the selected tags assigned. - -2. **Source Datastores**: The flow is triggered only for the selected source datastores. - -3. **Operation Types**: The flow is triggered only for operations that match one or more of the selected types. - -4. **Operation Status**: The flow is triggered for operations with a status of either Success or Failure. - -![operation](.././assets/flows/operation-light-12.png) - -After defining the conditions, users must click the **Save** button to finalize the trigger configuration. - -![save](.././assets/flows/save-light-8.png) - -#### Anomalous Table and File Detected - -This flow is triggered when anomalies are detected within a specific table, file and check rule types. It includes information about the number of anomalies found and the specific scan target within the datastore. This is useful for assessing the overall health of a particular datastore. - -![table](.././assets/flows/table-light-14.png) - -**Filter Conditions** - -Users can optionally set filters to specify which tables or files should trigger the flow execution. - -1. **Tables / Files Tags**: Only tables or files with all the selected tags assigned will trigger the flow. - -2. **Source Datastores**: The flow is triggered only for the selected source datastores. - -3. **Check Rule Types**: Only anomalies identified by one or more of the selected check rule types will initiate the flow. - -![table](.././assets/flows/table-light-15.png) - -After defining the conditions, users must click the **Save** button to finalize the trigger configuration. - -![save](.././assets/flows/save-light-8.png) - -#### Anomaly Detected - -This type of flow is triggered when any single anomaly is identified in the data. The flow message typically includes the type of anomaly detected and the datastore where it was found. It provides specific information about the anomaly type, which helps quickly understand the issue's nature. - -![anomaly](.././assets/flows/anomaly-light-17.png) - -**Filter Condition** - -Users can define specific conditions to determine when the flow should be initiated. - -1. **Anomaly’s Tags**: Only anomalies with all selected tags assigned will trigger the flow. - -2. **Source Datastores**: Only triggered when anomalies are detected in the selected datastores. - -3. **Check Rule Types**: Only anomalies identified by one or more of the selected check rule types will initiate the flow. - -4. **Anomaly Weight (Min)**: Only anomalies with a weight equal to or greater than the specified value will trigger the flow. - -![anomaly](.././assets/flows/anomaly-light-18.png) - -**Step 2:** Once the filter conditions are set, users must click the **Save** button to finalize the configuration. - -![save](.././assets/flows/save-light-8.png) - -#### Manual - -The flow starts only when the user manually triggers it. It doesn’t depend on any automatic conditions or detections, giving the user full control. - -![manual](.././assets/flows/manual-light-20.png) - -Once selected, users must click the **Save** button to confirm the manual trigger configuration. - -![save](.././assets/flows/save-light-8.png) - -Hover over the **filter tooltip** in trigger nodes to view the applied conditions such as tags, datastores, and operation types. This provides quick visibility into how each trigger is configured. - -![filter-tooltip](.././assets/flows/filter-tooltip-light.png) - -### Actions - -Actions define the specific steps the system will execute after a flow is triggered. They allow users to automate tasks, send notifications, or interact with external systems. - -**Step 1:** After completing the **"Trigger"** node setup, users can click on the **"Actions"** node. - -![action](.././assets/flows/action-light-22.png) - -A panel will appear on the right-hand side displaying the list of available actions. These actions define what the system will execute after the flow is triggered. The actions are categorized into three groups: - -* Operations. - -* Notifications. - -* HTTP. - -![actionlist](.././assets/flows/actionlist-light-23.png) - -!!! info - Inline summaries are shown within action nodes, displaying key details based on the action type—for example, datastore names for operations, Slack or Teams channels for notifications, and webhook URLs for HTTP actions. This enhancement provides quick clarity during flow configuration. - -#### Operation - -Users can execute specific operations when the trigger activates. They can choose from the following options: - -* Catalog. - -* Profile. - -* Scan. - -* Export. - -* Materialize. - -![operations](.././assets/flows/operations-light-24.png) - -**Catalog** - -**Step 1:** Click on **Catalog.** - -![catalog](.././assets/flows/catalog-light-25.png) - -A panel **Catalog Settings** will appear on the right-hand side. This window allows you to configure the catalog operation. - -| No. | Field | Description | -| :---- | :---- | :---- | -| 1. | Source Datastore | Select the source datastore to catalog. | -| 2. | Prune | Checkbox to enable or disable the removal of named collections (tables, views, files, etc.) that no longer exist in the datastore. | -| 3. | Recreate | Checkbox to enable or disable the recreation of previously deleted named collections in Qualytics for the catalog. | -| 4. | Include | Checkboxes to select Tables, Views, or both, specifying the resources to include in the catalog. | - -![catalog](.././assets/flows/catalog-light-26.png) - -**Step 2:** After configuring the settings, click Save to apply and proceed with the catalog operation. - -![save](.././assets/flows/save-light-8.png) - -**Profile** - -**Step 1:** Click on **Profile.** - -![profile](.././assets/flows/profile-light-28.png) - -A panel **Profile Settings** will appear on the right-hand side. This window allows you to configure the Profile operation. - -![profile](.././assets/flows/profile-light-29.png) - -| No. | Field | Description | -| :---- | :---- | :---- | -| 1. | Source Datastore | Select the source datastore to profile. | -| 2. | Select Tables | Allows users to select all tables, specific tables, or tables associated with selected tags to profile. | -| 3. | Read Settings | Configure the starting point for profiling and set a maximum record limit per table for profiling. | -| 4. | Inference Settings | Set the level of automated checks and decide whether inferred checks should be saved in draft mode. | - -![profile](.././assets/flows/profile-light-30.png) - -**Step 2:** Click Save to finalize the profile configuration. - -![save](.././assets/flows/save-light-8.png) - -**Scan** - -**Step 1:** Click on **Scan.** - -![scan](.././assets/flows/scan-light-32.png) - -A panel **Scan Settings** will appear on the right-hand side. This window allows you to configure the Scan operation. - -![scan](.././assets/flows/scan-light-33.png) - -**Source Datastore:** Select the datastore to be scanned. - -![scan](.././assets/flows/scan-light-34.png) - -**Select Tables:** Choose all tables, specific tables, or tables associated with selected tags to include in the scan. - -![scan](.././assets/flows/scan-light-35.png) - -**Select Check Categories:** Select categories of checks to include, such as table properties (Metadata) or value checks (Data Integrity). - -![scan](.././assets/flows/scan-light-36.png) - -**Read Settings:** Define the scan strategy: incremental scans updated records; full scans process all records. - -![scan](.././assets/flows/scan-light-39.png) - -**Starting Threshold:** Set a starting point for scanning based on an incremental identifier. - -![scan](.././assets/flows/scan-light-37.png) - -**Record Limit:** Specify the maximum number of records to scan per table. - -![scan](.././assets/flows/scan-light-38.png) - -**Scan Settings:** Choose how to manage duplicate or recurring anomalies by archiving overlaps or reactivating previously archived anomalies with fingerprint tracking. - -![scan](.././assets/flows/scan-settings-light.png) - -**Anomaly Rollup Threshold:** Set the Rollup Threshold to limit how many anomalies are created per check. When the limit is reached, anomalies will be merged into one for easier management. - -![rollup](.././assets/flows/rollup-light-39.png) - -**Enrichment Source Record Limit:** Define the number of source records to include in the enrichment operation. - -![scan](.././assets/flows/scan-light-40.png) - -**Step 2:** Click Save to finalize the scan configuration. - -![save](.././assets/flows/save-light-43.png) - -**Export** - -**Step 1:** Click on **Export.** - -![export](.././assets/flows/export-light.png) - -A panel **Export Settings** will appear on the right-hand side. This window allows you to configure the Export settings. - -![panel](.././assets/flows/export-setting-light.png) - -**Source Datastore:** Select the datastore to export data from. - -![source](.././assets/flows/source-light.png) - -**Select file patterns to export:** **All** (all file patterns, including future ones), **Specific** (manually chosen file patterns), or **Tag** (file patterns based on selected tags). - -![profile](.././assets/flows/profiles-light.png) - -**Select Metadata:** Choose metadata to export **anomalies**, **quality checks**, or **field profiles**. Anomalies detect data issues, quality checks validate data, and field profiles store field metadata. - -![exportt](.././assets/flows/exportt-light.png) - -**Step 2:** Click Save to finalize the export configuration. - -![save](.././assets/flows/save-light-8.png) - -Export nodes display the asset type in their titles (e.g., “Export Anomalies”) to help you identify the exported content easily. - -![export-status](.././assets/flows/export-status-light.png) - -**Materialize** - -**Step 1:** Click on **Materialize.** - -![materialize](.././assets/flows/materialize-light.png) - -A panel **Materialize Settings** will appear on the right-hand side. This window allows you to configure the Materialize settings. - -![setting](.././assets/flows/setting-light.png) - -**Source Datastore:** Select the datastore to materialize data from. - -![source](.././assets/flows/sourcee-light.png) - -**Select Tables:** Choose which tables (all, specific, or tagged) to extract from your source datastore and export to the enrichment datastore. - -![select](.././assets/flows/select-light.png) - -**Read Settings:** Select the record limit to control how much data is materialized per table. - -![read](.././assets/flows/read-light.png) - -**Step 2:** Click Save to finalize the materialize configuration. - -![save](.././assets/flows/saveee-light.png) - -#### Notification - -Users can configure the application to send notifications through various channels. The available notification options include: - -* In App. - -* Email. - -* Slack. - -* Microsoft Teams. - -* PagerDuty. - -![notification](.././assets/flows/notification-light-44.png) - -**In App** - -This will send an app notification to all users that use Qualytics. Users can set a custom message using variables and modify the standard text. - -**Step 1:** Click on **In App.** - -![notification](.././assets/flows/notification-light-45.png) - -A panel **In App Settings** will appear on the right-hand side, allowing you to configure the notification message. - -![notification](.././assets/flows/notification-light-46.png) - -**Message:** Enter your custom message using variables in the Message field, where you can specify the content of the notification that will be sent out. - -![notification](.././assets/flows/notification-light-47.png) - -!!! tip - You can write your custom notification message by utilizing the autocomplete feature. This feature allows you to easily insert internal variables such as `{{ flow_name }}`, `{{ container_name }}`, and `{{ datastore_name }}`. As you start typing, the autocomplete will suggest and recommend relevant variables in the dropdown. - -**Step 2:** After configuring the message, click **Save** to finalize the settings. - -![save](.././assets/flows/save-light-8.png) - -**Email** - -Adding email notifications allows users to receive timely updates or alerts directly in their inbox. By setting up notifications with specific triggers and channels, you can ensure that you are promptly informed about critical events, such as operation completions or detected anomalies. This proactive approach allows you to take immediate action when necessary, helping to address issues quickly and maintain the smooth and efficient operation of your processes. - -**Step 1:** Click on **Email.** - -![notification](.././assets/flows/notification-light-49.png) - -A panel **Email Settings** will appear on the right-hand side, allowing you to add email addresses, specify an email subject, and configure the notification message. - -![notification](.././assets/flows/notification-light-50.png) - -| No. | Field | Description | -| :---- | :---- | :---- | -| 1. | Email Address | Enter the email address where the notification should be sent. | -| 2. | Email Subject | Enter the subject line of the notification email to help recipients identify its purpose. | -| 3. | Message | Text area to customize the notification message content with dynamic placeholders like **`{{flow_name}}`**, **`{{operation_type}}`**, and **`{{operation_result}}`**. | - -![notification](.././assets/flows/notification-light-51.png) - -**Step 2:** Click the Test Notification button to send a test email to the provided address. If the email is successfully sent, you will receive a confirmation message indicating **Notification successfully sent.** - -![test](.././assets/flows/test-light-52.png) - -**Step 3:** Once all fields are configured, click the **Save** button to finalize the email notification setup. - -![save](.././assets/flows/save-light-8.png) - -**Slack** - -Qualytics integrates with Slack to deliver real-time notifications on scan completions, anomalies, and operational statuses, ensuring teams stay informed and can act quickly. With this integration, users receive instant alerts for system events, monitor scan results, and manage data anomalies directly within Slack. They can view notifications, acknowledge issues, and take necessary actions without switching platforms. - -**Step 1**: Click on **Slack.** - -![click-slack](.././assets/flows/click-slack-light.png) - -A **Slack Settings** panel appears on the right side of the screen. - -![slack-settings](.././assets/flows/slack-settings-light.png) - -| No. | Field | Description | -| :---- | :---- | :---- | -| **1.** | Channel | Choose the channel where notifications should be sent using the **Channel** dropdown. For demonstration purposes, the channel **#demo** is selected. | -| **2.** | Preview | Shows a preview of the Slack notification that will be sent when the flow runs. | - -![slack-options](.././assets/flows/slack-options-light.png) - -**Step 2:** Click the **Test Notification** button to send a sample notification to the selected Slack channel. - -![test-notification](.././assets/flows/test-notification-light.png) - -A prompt appears stating **Notification successfully sent** once the notification is successfully delivered. - -![successfully-notified](.././assets/flows/successfully-notified-light.png) - -**Step 3:** Once the notification is successfully sent, check your connected Slack workspace to ensure it is linked to Qualytics. You will see the test notification in the selected Slack channel. - -!!! note - Each trigger generates a different type of Slack notification message. The content and format of the message vary based on the specific trigger event. - -![anomaly-detected](.././assets/flows/anomaly-detected.png) - -**Step 4:** After confirming that the notification was received successfully, return and click the Save button. - -![save](.././assets/flows/save-light-8.png) - -#### Examples of Trigger Messages - -Trigger messages in Slack provide real-time notifications for various system events, ensuring timely awareness and action. Each trigger message follows a unique format and conveys different types of information based on the operation performed. Below are examples highlighting distinct scenarios: - -**Scenario 1: Scan Completion Notification** - -When a data cataloging or scan operation completes successfully, a notification is sent to Slack. The message includes details such as the dataset name, operation type (e.g., Catalog Operation), and the result of the operation. - -![scan-completed](.././assets/flows/scan-completed.png) - -**Scenario 2: Anomalous Table or File Detected** - -When a scan detects a critical data anomaly, Slack sends a detailed notification highlighting the issue. The notification includes the dataset name, flow (such as Quality Monitor), and source datastore. It also provides a summary of the anomaly, specifying the number of records that differ between datasets and the container where the discrepancy was found. Additionally, the message offers an option to view detailed results. - -![anomalous-scan](.././assets/flows/anomalous-scan.png) - -**Scenario 3: Anomaly Detected** - -When a scan detects record anomalies, Slack sends a notification highlighting the affected container, flow, and source datastore. It specifies the number of records that differ between datasets and provides options to view or acknowledge the anomaly. - -![anomaly-detected](.././assets/flows/anomaly-detected.png) - -#### Managing Qualytics Alerts in Slack - -Qualytics Slack integration enables real-time monitoring and quick action on data quality issues directly from Slack. This guide outlines the different types of alerts and the actions you can take without leaving Slack. - -**When an Operation Success or failure** - -**Step 1:** A Slack notification confirms the scan completion with a **Success/failure** status. - -For demonstration purposes we are using Success operation. - -![scan-completed](.././assets/flows/scan-completed.png) - -**Step 2:** Click **View Operation** to be redirected automatically to the result section in Qualytics. - -![view-operation](.././assets/flows/view-operation.png) - -**When an Anomalous File or Table is Detected** - -**Step 1:** A Slack alert notifies about anomalies in a dataset. - -![anomalous-scan](.././assets/flows/anomalous-scan.png) - -**Step 2:** Click **View Results** to examine the identified discrepancies directly in Qualytics. - -![view-results](.././assets/flows/view-results.png) - -**When a Record Anomaly is Detected** - -If a **shape or record anomaly** is found, you'll receive a Slack notification. You can take the following actions: - -![anomaly-detected](.././assets/flows/anomaly-detected.png) - -* **View Anomaly** – Click on view anomaly to open the details in Qualytics to investigate further. - -![view-anomaly](.././assets/flows/view-anomaly.png) - -* **Acknowledge** – Click on Acknowledge to mark it as reviewed to avoid duplicate alerts. - -![acknowledge-anomaly](.././assets/flows/acknowledge-anomaly.png) - -* **Horizontal ellipsis(⋯)** – Click on horizontal ellipsis. - -![horizontal-ellipsis](.././assets/flows/horizontal-ellipsis.png) - - A dropdown will open with option **comment** and **archive** : - -![comment-archive](.././assets/flows/comment-archive.png) - -| No. | Action | Description | -| :---- | :---- | :---- | -| **1.** | Comment | Add Comment to collaborate with your team. | -| **2.** | Archive | Archive if no further action is needed. | - -**Microsoft Teams** - -**Step 1:** Click on **Microsoft Teams.** - -![notification](.././assets/flows/notification-light-59.png) - -A panel **Microsoft Teams Settings** will appear on the right-hand side, allowing you to add a webhook url and configure the notification message. - -![notification](.././assets/flows/notification-light-60.png) - -| No. | Field | Description | -| :---- | :---- | :---- | -| 1. | Teams Webhook URL | Enter the Teams webhook URL where the notification should be sent. | -| 2. | Message | Text area to customize the notification message content with dynamic placeholders like **`{{flow_name}}`**, **`{{operation_type}}`**, and **`{{operation_result}}`**. | - -![notification](.././assets/flows/notification-light-61.png) - -**Step 2:** Click the **"Test Notification"** button to send a test message to the provided **“Webhook URL”.** If the message is successfully sent, you will receive a confirmation notification indicating **"Notification successfully sent".** - -![test](.././assets/flows/test-light-62.png) - -**Step 3:** Once all fields are configured, click the **Save** button to finalize the Microsoft Teams notification setup. - -![save](.././assets/flows/save-light-8.png) - -**PagerDuty** - -Integrating PagerDuty with Qualytics ensures that your team gets instant alerts for critical data events and system issues. With this connection, you can automatically receive real-time notifications about anomalies, operation completions and other important events directly in your PagerDuty account. By categorizing alerts based on severity, it ensures the right people are notified at the right time, speeding up decision-making and resolving incidents efficiently. This helps your team respond quickly to issues, reducing downtime and keeping data operations on track. - -**Step 1:** Click on **PagerDuty.** - -![notification](.././assets/flows/notification-light-64.png) - -A **PagerDuty Settings** panel will appear on the right-hand side, enabling users to configure and send PagerDuty notifications. - -![notification](.././assets/flows/notification-light-65.png) - -**Integration Key:** Enter the **Integration Key** where you want the notification to be sent. - -![notification](.././assets/flows/notification-light-66.png) - -**Severity:** Select the appropriate PagerDuty severity level to categorize incidents based on their urgency and impact. The available severity levels are: - -* **Info:** For informational messages that don't require immediate action but provide helpful context. - -* **Warning:** For potential issues that may need attention but aren't immediately critical. - -* **Error:** For significant problems that require prompt resolution to prevent disruption. - -* **Critical:** For urgent issues that demand immediate attention due to their severe impact on system operations. - -![notification](.././assets/flows/notification-light-67.png) - -**Message:** Enter your custom message using variables in the Message field, where you can specify the content of the notification that will be sent out. - -![notification](.././assets/flows/notification-light-68.png) - -!!! tip - You can write your custom notification message by utilizing the autocomplete feature. This feature allows you to easily insert internal variables such as `{{ flow_name }}`, `{{ operation_type }}`, and `{{ datastore_name }}`. As you start typing, the autocomplete will suggest and recommend relevant variables in the dropdown. - -**Step 2:** Click on the **Test notification** button to check if the integration key is functioning correctly. Once the test notification is sent, you will see a success message, **"Notification successfully sent."** - -![test](.././assets/flows/test-light-69.png) - -**Step 3:** Once you have entered all the values, then click on the **Save** button. - -![save](.././assets/flows/save-light-8.png) - -#### HTTP - -Users can connect to external apps for notifications using one of these services: - -* Webhook. - -* HTTP Action. - -![notification](.././assets/flows/notification-light-71.png) - -**Webhook** - -Qualytics allows you to connect external apps for notifications using webhooks, making it easy to stay updated in real time. When you set up a webhook, it sends an instant alert to the connected app whenever a specific event or condition occurs. This means you can quickly receive notifications about important events as they happen and respond right away. By using webhook notifications, you can keep your system running smoothly, keep everyone informed, and manage your operations more efficiently. - -**Step 1:** Click on **Webhook.** - -![notification](.././assets/flows/notification-light-72.png) - -A **Webhook Settings** panel will appear on the right-hand side, enabling users to configure and send webhook notifications. - -![notification](.././assets/flows/notification-light-73.png) - -| No. | Field | Description | -| :---- | :---- | :---- | -| 1. | Webhook URL | Enter the desired **"Webhook URL"** of the target system where you want to receive notifications. | -| 2. | Message | Text area to customize the notification message content with dynamic placeholders like **`{{flow_name}}`**, **`{{operation_type}}`**, and **`{{operation_result}}`**. | - -![notification](.././assets/flows/notification-light-74.png) - -**Step 2**: Click on the **"Test HTTP"** button to send a test notification to the webhook URL you provided. If the webhook URL is correct, you will receive a confirmation message saying **"Notification successfully sent."** This indicates that the webhook is functioning correctly. - -![test](.././assets/flows/test-light-75.png) - -**Step 3:** Once you have entered all the values, then click on the **Save** button. - -![save](.././assets/flows/save-light-8.png) - -**HTTP Action** - -Integrating HTTP Action notifications allows users to receive timely updates or alerts directly to a specified server endpoint. By setting up HTTP Action notifications with specific trigger conditions, you can ensure that you are instantly informed about critical events, such as operation completions or anomalies detected. This approach enables you to take immediate action when necessary, helping to address issues quickly and maintain the smooth and efficient operation of your processes. - -**Step 1:** Click on **HTTP Action.** - -![notification](.././assets/flows/notification-light-77.png) - -An **HTTP Action Settings** panel will appear on the right-hand side, enabling users to configure and send HTTP Action notifications. - -![notification](.././assets/flows/notification-light-78.png) - -**Step 2:** Enter the following detail where you want the notification to be sent. - -**1. Action URL:** Enter the **“Action URL”** in this field. It specifies the server endpoint for the HTTP request and defines where data will be sent or retrieved. It must be correctly formatted and accessible, including the protocol (http or https), domain, and path. - -**2. HTTP Verbs:** HTTP verbs specify the actions performed on server resources. Common verbs include: - -* **POST:** Use POST to send data to the server to create something new. For example, it's used for submitting forms or uploading files. The server processes this data and creates a new resource. -* **PUT:** Updates or creates a resource, replacing it entirely if it already exists. For example, updating a user’s profile information or creating a new record with specific details. -* **GET:** Retrieves data from the server without making any modifications. For example, requesting a webpage or fetching user details from a database. - -**3. Username:** Enter the username needed for authentication. - -**4. Auth Type:** This field specifies how to authenticate requests. Choose the method that fits your needs: - -* **Basic:** Uses a username and password sent with each request. Example: **“Authorization: Basic ”.** -* **Bearer:** Uses a token included in the request header to access resources. Example: **“Authorization: Bearer < token >”.** -* **Digest:** Provides a more secure authentication method by using a hashed combination of the username, password, and request details. Example: **Authorization: Digest username=" ", realm=" ", nonce=" ", uri=" ", response=" ".** - -**5. Secret:** Enter the password or token used for authentication. This is paired with the **Username** and **Auth Type** to securely access the server. Keep the secret confidential to ensure security. - -**6. Message:** Enter your custom message using variables in the Message field, where you can specify the content of the notification that will be sent out. - -![notification](.././assets/flows/notification-light-79.png) - -!!! tip - You can write your custom notification message by utilizing the autocomplete feature. This feature allows you to easily insert internal variables such as `{{ flow_name }}`, `{{ operation_type }}`, and `{{ datastore_name }}`. As you start typing, the autocomplete will suggest and recommend relevant variables in the dropdown. - -**Step 3:** Click the **"Test HTTP"** button to verify the correctness of the Action URL. If the URL is correct, a confirmation message saying **"Notification successfully sent"** will appear, confirming that the HTTP action is set up and functioning properly. - -![test](.././assets/flows/test-light-80.png) - -**Step 4:** Once you have entered all the values, then click on the **Save** button. - -![save](.././assets/flows/save-light-8.png) - -**Step 3:** After completing all the required details in the **"Add Flow"** section, click on the **Publish** button to finalize the process. - -![publish](.././assets/flows/publish-light-82.png) - -After clicking the **Publish** button, a success notification appears confirming that the flow has been successfully added. - -## View Created Flows - -Once a flow is added, it will be visible in the **Definitions** tab, where you can view all the created flows. - -![panel](.././assets/flows/panel-light-84.png) - -## Clone a Flow - -Users can duplicate existing flows to simplify the reuse and modification of flow configurations for similar scenarios. - -**Step 1**: Click on the existing flow you want to clone. - -![panel](.././assets/flows/exisiting-doc-light.png) - -**Step 2**: A new window will open displaying the flow's detailed configuration. Click the settings icon and select **Clone**. - -![panel](.././assets/flows/clone-light.png) - -**Step 3:** After selecting the clone button, click the **Publish** button to publish it. - -![panel](.././assets/flows/publish-light-82.png) - -After clicking the **Publish** button, a success notification appears confirming that the flow has been successfully added. - -## Sort Flows - -Qualytics allows you to sort your flows by **Created Date** and **Name** to easily organize and prioritize them according to your needs. - -![sort](.././assets/flows/sort-light-85.png) - -Whatever sorting option is selected, you can arrange the data either in ascending or descending order by clicking the caret button next to the selected sorting criteria. - -![sort](.././assets/flows/sort-light-86.png) - -## Execute Manual Flows - -Users can start a manual flow from the vertical ellipsis menu for greater flexibility in executing flows. - -**Step 1:** Locate the manual flow in your list of flows. - -![manual-flow](.././assets/flows/manuall-flow-light.png) - -**Step 2:** Click the **vertical ellipsis (⋮)** next to the manual flow you wish to execute, then select **"Execute"** from the dropdown menu to trigger the flow. - -![manual-flow](.././assets/flows/verticall-light.png) - -After clicking the **Execute** button, a success notification appears confirming that the flow has been successfully executed. - -## Manage Flows - -**Manage Flow** allows users to edit, delete, deactivate or activate flows. Users can update configurations, remove outdated flows, or pause triggers to maintain an organized and efficient workflow system. - -### Edit Flow - -**Edit Flow** feature lets users update existing flows by modifying configurations or adding actions. - -**Step 1:** Click the flow you want to edit. - -![panel](.././assets/flows/panel-light-87.png) - -**Step 2:** After clicking the flow, a new window will open displaying the flow's detailed configuration. Click on the boxes you want to edit. - -For demonstration purposes we have selected the Flow node. - -![result](.././assets/flows/result-light-88.png) - -**Step 3:** Click the **`Save`** button to apply the updates. - -![save](.././assets/flows/save-light-89.png) - -**Step 4:** After clicking the **`Save`** button, click the **`Publish`** button located in the top right corner to finalize and publish the changes. - -![publish](.././assets/flows/publish-light-82.png) - -### Delete Flow - -**Delete Flow** feature allows you to permanently remove unwanted or outdated flows from the system. This helps in maintaining a clean and organized list of active flows. - -**Step 1**: Click the **vertical ellipsis (⋮)** next to the flow that you want to delete, then click on **Delete** from the dropdown menu. - -![delete](.././assets/flows/delete-light-91.png) - -After clicking the delete button, a confirmation modal window **Delete Flow** will appear. - -![delete](.././assets/flows/delete-light-92.png) - -**Step 2**: Click on the **Delete** button to delete the flow. - -![delete](.././assets/flows/delete-light-93.png) - -After clicking the **Delete** button, a success notification appears confirming the deletion. - -### Deactivate Flow - -Users can deactivate a flow to pause its triggers by disabling it. This prevents the flow from being executed until it is reactivated. - -**Step 1**: Click the **vertical ellipsis (⋮)** next to the flow that you want to deactivate, then click on **Deactivate** from the dropdown menu. - -![deactivate](.././assets/flows/deactivate-light-95.png) - -After clicking the **Deactivate** button, a success notification appears confirming the deactivation. - -### Activate Flow - -Users can reactivate a flow that was previously deactivated. Once reactivated, the flow’s triggers become active again, allowing it to run automatically based on the defined conditions. - -**Step 1**: Click the **vertical ellipsis (⋮)** next to the flow that you want to activate, then click on **Activate** from the dropdown menu. - -![activate](.././assets/flows/activate-flow-light.png) - -After clicking the **Activate** button, a success notification appears confirming the activation. - -## Clone an Action - -Users can duplicate an existing action in just a few clicks. Cloning an action allows you to quickly replicate its configuration without manually setting it up again. - -**Step 1:** Click the vertical ellipsis (**⋮**) on the action you want to clone, then select the **Clone** option from the menu. - -![vertical](.././assets/flows/vertical-light.png) - -**Step 2:** After clicking the Clone option, a cloned action will be created. - -![clone](.././assets/flows/clones-light.png) - -## Flows Execution - -Execution tab allows users to view the execution history and current status of a flow. It provides detailed timestamps, status updates, and a comprehensive record of flow executions for efficient tracking and analysis. - -Click on the **Execution** tab. - -![executions](.././assets/flows/executions-light-99.png) - -You will be navigated to the **Execution** tab, where you can view the complete execution history of all created flows. - -![executions](.././assets/flows/executions-light-100.png) - -### See a Flow Execution - -Users can view flow execution in real-time by clicking on the desired flow operation. The page shows detailed operations but does not allow editing. - -**Step 1:** Click on the flow operation you want to view. - -![manual-flow](.././assets/flows/manuall-flow-light.png) - - After clicking, the user will navigate to the selected flow operation details. The page displays all operational details in real-time. Note that this page is for viewing only, and no edits can be made here. - -![flow](.././assets/flows/flow-light-1001.png) - -### Understanding Flow States - -On the bottom-right corner, there is a **Legend** indicating the possible states of an action, such as: - -* **Success** (Green) - -* **Failure** (Red) - -* **Aborted** (Orange) - -* **Skipped** (Yellow) - -* **Running** (Blue with dotted lines animation) - -* **Pending** (Gray) - -![chart](.././assets/flows/chart-light-101.png) - -If a step is running, you will see a **dot-line animation**, signaling that the step is in progress. -Once completed, the Action box will change its color to reflect the final state. - -![chart](.././assets/flows/chart-light-102.png) - -### Accessing Operation Results - -To view detailed results of specific operations: - -**Step 1:** Click the **Top Right Arrow** button within the action operation box. - -![chart](.././assets/flows/chart-light-103.png) - -**Step 2:** You will navigate to the Activity page, where a Result Modal will open, displaying in-depth details of the operation. - -![result](.././assets/flows/result-light-104.png) - -### Delete Flow Execution - -**Step 1:** Click the Delete icon next to the flow execution you want to remove. - -![delete](.././assets/flows/delete-light-105.png) - -A confirmation modal window **Delete Flow Execution** will appear. - -![delete](.././assets/flows/delete-light-106.png) - -**Step 2:** Click on the **Delete** button to delete the flow execution. - -![delete](.././assets/flows/delete-light-107.png) - -After clicking the **Delete** button, a success notification appears confirming the deletion. - -## Filter and Sort - -**Filter and Sort** in the **`Executions`** tab help organize flow execution data. Users can sort by creation date or duration and filter by flow name, status, or trigger type for quick access to specific details. - -### Sort - -**Sort By** feature allows users to organize executions by **Created Date** or **Duration**, simplifying the process of reviewing flow executions based on their creation or runtime. - -![sort](.././assets/flows/sort-light-109.png) - -### Filter - -**Filter** feature allows users to refine flow execution results based on specific criteria. By clicking the filter icon, users can choose from the following options: - -{% include-markdown "components/general-props/typos.md" - start='' - end='' -%} -![filter](.././assets/flows/flows-filter.png) - -| No. | Filter | Description | -| :---- | :---- | :---- | -| 1. | Flows | Select a specific flow to view its executions. | -| 2. | Status | Filter executions by their completion status (e.g., success, failure and running). | -| 3. | Trigger When | Filter executions based on their trigger condition. | - -![filter](.././assets/flows/filter-light-110.png) - -## Operations - -In the Activity tab, users can easily identify flow executions. The **`Flow`** column shows the flow name and includes a button to redirect users to the flow's operation. This feature is available in **Explore Activities, Datastore Activity,** and **Container Activity**. - -![explore](.././assets/flows/explore-light-111.png) \ No newline at end of file diff --git a/docs/flows/http.md b/docs/flows/http.md new file mode 100644 index 000000000..0397e8296 --- /dev/null +++ b/docs/flows/http.md @@ -0,0 +1,89 @@ +# HTTP + +Users can connect to external apps for notifications using one of these services: + +* Webhook. + +* HTTP Action. + +![notification](.././assets/flows/notification-light-71.png) + +## Webhook + +Qualytics allows you to connect external apps for notifications using webhooks, making it easy to stay updated in real time. When you set up a webhook, it sends an instant alert to the connected app whenever a specific event or condition occurs. This means you can quickly receive notifications about important events as they happen and respond right away. By using webhook notifications, you can keep your system running smoothly, keep everyone informed, and manage your operations more efficiently. + +**Step 1:** Click on **Webhook.** + +![notification](.././assets/flows/notification-light-72.png) + +A **Webhook Settings** panel will appear on the right-hand side, enabling users to configure and send webhook notifications. + +![notification](.././assets/flows/notification-light-73.png) + +| No. | Field | Description | +| :---- | :---- | :---- | +| 1. | Webhook URL | Enter the desired **"Webhook URL"** of the target system where you want to receive notifications. | +| 2. | Message | Text area to customize the notification message content with dynamic placeholders like **`{{flow_name}}`**, **`{{operation_type}}`**, and **`{{operation_result}}`**. | + +![notification](.././assets/flows/notification-light-74.png) + +**Step 2**: Click on the **"Test HTTP"** button to send a test notification to the webhook URL you provided. If the webhook URL is correct, you will receive a confirmation message saying **"Notification successfully sent."** This indicates that the webhook is functioning correctly. + +![test](.././assets/flows/test-light-75.png) + +**Step 3:** Once you have entered all the values, then click on the **Save** button. + +![save](.././assets/flows/save-light-8.png) + +## HTTP Action + +Integrating HTTP Action notifications allows users to receive timely updates or alerts directly to a specified server endpoint. By setting up HTTP Action notifications with specific trigger conditions, you can ensure that you are instantly informed about critical events, such as operation completions or anomalies detected. This approach enables you to take immediate action when necessary, helping to address issues quickly and maintain the smooth and efficient operation of your processes. + +**Step 1:** Click on **HTTP Action.** + +![notification](.././assets/flows/notification-light-77.png) + +An **HTTP Action Settings** panel will appear on the right-hand side, enabling users to configure and send HTTP Action notifications. + +![notification](.././assets/flows/notification-light-78.png) + +**Step 2:** Enter the following detail where you want the notification to be sent. + +**1. Action URL:** Enter the **“Action URL”** in this field. It specifies the server endpoint for the HTTP request and defines where data will be sent or retrieved. It must be correctly formatted and accessible, including the protocol (http or https), domain, and path. + +**2. HTTP Verbs:** HTTP verbs specify the actions performed on server resources. Common verbs include: + +* **POST:** Use POST to send data to the server to create something new. For example, it's used for submitting forms or uploading files. The server processes this data and creates a new resource. +* **PUT:** Updates or creates a resource, replacing it entirely if it already exists. For example, updating a user’s profile information or creating a new record with specific details. +* **GET:** Retrieves data from the server without making any modifications. For example, requesting a webpage or fetching user details from a database. + +**3. Username:** Enter the username needed for authentication. + +**4. Auth Type:** This field specifies how to authenticate requests. Choose the method that fits your needs: + +* **Basic:** Uses a username and password sent with each request. Example: **“Authorization: Basic ”.** +* **Bearer:** Uses a token included in the request header to access resources. Example: **“Authorization: Bearer < token >”.** +* **Digest:** Provides a more secure authentication method by using a hashed combination of the username, password, and request details. Example: **Authorization: Digest username=" ", realm=" ", nonce=" ", uri=" ", response=" ".** + +**5. Secret:** Enter the password or token used for authentication. This is paired with the **Username** and **Auth Type** to securely access the server. Keep the secret confidential to ensure security. + +**6. Message:** Enter your custom message using variables in the Message field, where you can specify the content of the notification that will be sent out. + +![notification](.././assets/flows/notification-light-79.png) + +!!! tip + You can write your custom notification message by utilizing the autocomplete feature. This feature allows you to easily insert internal variables such as `{{ flow_name }}`, `{{ operation_type }}`, and `{{ datastore_name }}`. As you start typing, the autocomplete will suggest and recommend relevant variables in the dropdown. + +**Step 3:** Click the **"Test HTTP"** button to verify the correctness of the Action URL. If the URL is correct, a confirmation message saying **"Notification successfully sent"** will appear, confirming that the HTTP action is set up and functioning properly. + +![test](.././assets/flows/test-light-80.png) + +**Step 4:** Once you have entered all the values, then click on the **Save** button. + +![save](.././assets/flows/save-light-8.png) + +**Step 5:** After completing all the required details in the **"Add Flow"** section, click on the **Publish** button to finalize the process. + +![publish](.././assets/flows/publish-light-82.png) + +After clicking the **Publish** button, a success notification appears confirming that the flow has been successfully added. \ No newline at end of file diff --git a/docs/flows/notification.md b/docs/flows/notification.md new file mode 100644 index 000000000..051c71f2d --- /dev/null +++ b/docs/flows/notification.md @@ -0,0 +1,253 @@ +# Notification + +Users can configure the application to send notifications through various channels. The available notification options include: + +* In App. + +* Email. + +* Slack. + +* Microsoft Teams. + +* PagerDuty. + +![notification](.././assets/flows/notification-light-44.png) + +## In App + +This will send an app notification to all users that use Qualytics. Users can set a custom message using variables and modify the standard text. + +**Step 1:** Click on **In App.** + +![notification](.././assets/flows/notification-light-45.png) + +A panel **In App Settings** will appear on the right-hand side, allowing you to configure the notification message. + +![notification](.././assets/flows/notification-light-46.png) + +**Message:** Enter your custom message using variables in the Message field, where you can specify the content of the notification that will be sent out. + +![notification](.././assets/flows/notification-light-47.png) + +!!! tip + You can write your custom notification message by utilizing the autocomplete feature. This feature allows you to easily insert internal variables such as `{{ flow_name }}`, `{{ container_name }}`, and `{{ datastore_name }}`. As you start typing, the autocomplete will suggest and recommend relevant variables in the dropdown. + +**Step 2:** After configuring the message, click **Save** to finalize the settings. + +![save](.././assets/flows/save-light-8.png) + +## Email + +Adding email notifications allows users to receive timely updates or alerts directly in their inbox. By setting up notifications with specific triggers and channels, you can ensure that you are promptly informed about critical events, such as operation completions or detected anomalies. This proactive approach allows you to take immediate action when necessary, helping to address issues quickly and maintain the smooth and efficient operation of your processes. + +**Step 1:** Click on **Email.** + +![notification](.././assets/flows/notification-light-49.png) + +A panel **Email Settings** will appear on the right-hand side, allowing you to add email addresses, specify an email subject, and configure the notification message. + +![notification](.././assets/flows/notification-light-50.png) + +| No. | Field | Description | +| :---- | :---- | :---- | +| 1. | Email Address | Enter the email address where the notification should be sent. | +| 2. | Email Subject | Enter the subject line of the notification email to help recipients identify its purpose. | +| 3. | Message | Text area to customize the notification message content with dynamic placeholders like **`{{flow_name}}`**, **`{{operation_type}}`**, and **`{{operation_result}}`**. | + +![notification](.././assets/flows/notification-light-51.png) + +**Step 2:** Click the Test Notification button to send a test email to the provided address. If the email is successfully sent, you will receive a confirmation message indicating **Notification successfully sent.** + +![test](.././assets/flows/test-light-52.png) + +**Step 3:** Once all fields are configured, click the **Save** button to finalize the email notification setup. + +![save](.././assets/flows/save-light-8.png) + +## Slack + +Qualytics integrates with Slack to deliver real-time notifications on scan completions, anomalies, and operational statuses, ensuring teams stay informed and can act quickly. With this integration, users receive instant alerts for system events, monitor scan results, and manage data anomalies directly within Slack. They can view notifications, acknowledge issues, and take necessary actions without switching platforms. + +**Step 1**: Click on **Slack.** + +![click-slack](.././assets/flows/click-slack-light.png) + +A **Slack Settings** panel appears on the right side of the screen. + +![slack-settings](.././assets/flows/slack-settings-light.png) + +| No. | Field | Description | +| :---- | :---- | :---- | +| **1.** | Channel | Choose the channel where notifications should be sent using the **Channel** dropdown. For demonstration purposes, the channel **#demo** is selected. | +| **2.** | Preview | Shows a preview of the Slack notification that will be sent when the flow runs. | + +![slack-options](.././assets/flows/slack-options-light.png) + +**Step 2:** Click the **Test Notification** button to send a sample notification to the selected Slack channel. + +![test-notification](.././assets/flows/test-notification-light.png) + +A prompt appears stating **Notification successfully sent** once the notification is successfully delivered. + +![successfully-notified](.././assets/flows/successfully-notified-light.png) + +**Step 3:** Once the notification is successfully sent, check your connected Slack workspace to ensure it is linked to Qualytics. You will see the test notification in the selected Slack channel. + +!!! note + Each trigger generates a different type of Slack notification message. The content and format of the message vary based on the specific trigger event. + +![anomaly-detected](.././assets/flows/anomaly-detected.png) + +**Step 4:** After confirming that the notification was received successfully, return and click the Save button. + +![save](.././assets/flows/save-light-8.png) + +## Examples of Trigger Messages + +Trigger messages in Slack provide real-time notifications for various system events, ensuring timely awareness and action. Each trigger message follows a unique format and conveys different types of information based on the operation performed. Below are examples highlighting distinct scenarios: + +**Scenario 1: Scan Completion Notification** + +When a data cataloging or scan operation completes successfully, a notification is sent to Slack. The message includes details such as the dataset name, operation type (e.g., Catalog Operation), and the result of the operation. + +![scan-completed](.././assets/flows/scan-completed.png) + +**Scenario 2: Anomalous Table or File Detected** + +When a scan detects a critical data anomaly, Slack sends a detailed notification highlighting the issue. The notification includes the dataset name, flow (such as Quality Monitor), and source datastore. It also provides a summary of the anomaly, specifying the number of records that differ between datasets and the container where the discrepancy was found. Additionally, the message offers an option to view detailed results. + +![anomalous-scan](.././assets/flows/anomalous-scan.png) + +**Scenario 3: Anomaly Detected** + +When a scan detects record anomalies, Slack sends a notification highlighting the affected container, flow, and source datastore. It specifies the number of records that differ between datasets and provides options to view or acknowledge the anomaly. + +![anomaly-detected](.././assets/flows/anomaly-detected.png) + +## Managing Qualytics Alerts in Slack + +Qualytics Slack integration enables real-time monitoring and quick action on data quality issues directly from Slack. This guide outlines the different types of alerts and the actions you can take without leaving Slack. + +**When an Operation Success or failure** + +**Step 1:** A Slack notification confirms the scan completion with a **Success/failure** status. + +For demonstration purposes we are using Success operation. + +![scan-completed](.././assets/flows/scan-completed.png) + +**Step 2:** Click **View Operation** to be redirected automatically to the result section in Qualytics. + +![view-operation](.././assets/flows/view-operation.png) + +**When an Anomalous File or Table is Detected** + +**Step 1:** A Slack alert notifies about anomalies in a dataset. + +![anomalous-scan](.././assets/flows/anomalous-scan.png) + +**Step 2:** Click **View Results** to examine the identified discrepancies directly in Qualytics. + +![view-results](.././assets/flows/view-results.png) + +**When a Record Anomaly is Detected** + +If a **shape or record anomaly** is found, you'll receive a Slack notification. You can take the following actions: + +![anomaly-detected](.././assets/flows/anomaly-detected.png) + +* **View Anomaly** – Click on view anomaly to open the details in Qualytics to investigate further. + +![view-anomaly](.././assets/flows/view-anomaly.png) + +* **Acknowledge** – Click on Acknowledge to mark it as reviewed to avoid duplicate alerts. + +![acknowledge-anomaly](.././assets/flows/acknowledge-anomaly.png) + +* **Horizontal ellipsis(⋯)** – Click on horizontal ellipsis. + +![horizontal-ellipsis](.././assets/flows/horizontal-ellipsis.png) + + A dropdown will open with option **comment** and **archive** : + +![comment-archive](.././assets/flows/comment-archive.png) + +| No. | Action | Description | +| :---- | :---- | :---- | +| **1.** | Comment | Add Comment to collaborate with your team. | +| **2.** | Archive | Archive if no further action is needed. | + +## Microsoft Teams + +**Step 1:** Click on **Microsoft Teams.** + +![notification](.././assets/flows/notification-light-59.png) + +A panel **Microsoft Teams Settings** will appear on the right-hand side, allowing you to add a webhook url and configure the notification message. + +![notification](.././assets/flows/notification-light-60.png) + +| No. | Field | Description | +| :---- | :---- | :---- | +| 1. | Teams Webhook URL | Enter the Teams webhook URL where the notification should be sent. | +| 2. | Message | Text area to customize the notification message content with dynamic placeholders like **`{{flow_name}}`**, **`{{operation_type}}`**, and **`{{operation_result}}`**. | + +![notification](.././assets/flows/notification-light-61.png) + +**Step 2:** Click the **"Test Notification"** button to send a test message to the provided **“Webhook URL”.** If the message is successfully sent, you will receive a confirmation notification indicating **"Notification successfully sent".** + +![test](.././assets/flows/test-light-62.png) + +**Step 3:** Once all fields are configured, click the **Save** button to finalize the Microsoft Teams notification setup. + +![save](.././assets/flows/save-light-8.png) + +## PagerDuty + +Integrating PagerDuty with Qualytics ensures that your team gets instant alerts for critical data events and system issues. With this connection, you can automatically receive real-time notifications about anomalies, operation completions and other important events directly in your PagerDuty account. By categorizing alerts based on severity, it ensures the right people are notified at the right time, speeding up decision-making and resolving incidents efficiently. This helps your team respond quickly to issues, reducing downtime and keeping data operations on track. + +**Step 1:** Click on **PagerDuty.** + +![notification](.././assets/flows/notification-light-64.png) + +A **PagerDuty Settings** panel will appear on the right-hand side, enabling users to configure and send PagerDuty notifications. + +![notification](.././assets/flows/notification-light-65.png) + +**Integration Key:** Enter the **Integration Key** where you want the notification to be sent. + +![notification](.././assets/flows/notification-light-66.png) + +**Severity:** Select the appropriate PagerDuty severity level to categorize incidents based on their urgency and impact. The available severity levels are: + +* **Info:** For informational messages that don't require immediate action but provide helpful context. + +* **Warning:** For potential issues that may need attention but aren't immediately critical. + +* **Error:** For significant problems that require prompt resolution to prevent disruption. + +* **Critical:** For urgent issues that demand immediate attention due to their severe impact on system operations. + +![notification](.././assets/flows/notification-light-67.png) + +**Message:** Enter your custom message using variables in the Message field, where you can specify the content of the notification that will be sent out. + +![notification](.././assets/flows/notification-light-68.png) + +!!! tip + You can write your custom notification message by utilizing the autocomplete feature. This feature allows you to easily insert internal variables such as `{{ flow_name }}`, `{{ operation_type }}`, and `{{ datastore_name }}`. As you start typing, the autocomplete will suggest and recommend relevant variables in the dropdown. + +**Step 2:** Click on the **Test notification** button to check if the integration key is functioning correctly. Once the test notification is sent, you will see a success message, **"Notification successfully sent."** + +![test](.././assets/flows/test-light-69.png) + +**Step 3:** Once you have entered all the values, then click on the **Save** button. + +![save](.././assets/flows/save-light-8.png) + +## FAQ + +**1>.** Can I test notifications before publishing? + +Yes. Each notification channel—Email, Slack, Teams, PagerDuty, and HTTP—includes a Test Notification button that allows you to send a sample message before publishing the flow. \ No newline at end of file diff --git a/docs/flows/operations.md b/docs/flows/operations.md new file mode 100644 index 000000000..917826f3e --- /dev/null +++ b/docs/flows/operations.md @@ -0,0 +1,170 @@ +# Operation + +Users can execute specific operations when the trigger activates. They can choose from the following options: + +* Catalog. + +* Profile. + +* Scan. + +* Export. + +* Materialize. + +![operations](.././assets/flows/operations-light-24.png) + +!!! Warning + Only initialized datastores appear in the **Source Datastore** dropdown across all operation types. A datastore becomes initialized after it has successfully completed **Catalog**, **Profile**, and **Scan** runs at least once within the **Action Node Operation** section while setting up your flow. + +![note](.././assets/flows/note-flow.png) + +## Catalog + +**Step 1:** Click on **Catalog.** + +![catalog](.././assets/flows/catalog-light-25.png) + +A panel **Catalog Settings** will appear on the right-hand side. This window allows you to configure the catalog operation. + +| No. | Field | Description | +| :---- | :---- | :---- | +| 1. | Source Datastore | Select the source datastore to catalog. | +| 2. | Prune | Checkbox to enable or disable the removal of named collections (tables, views, files, etc.) that no longer exist in the datastore. | +| 3. | Recreate | Checkbox to enable or disable the recreation of previously deleted named collections in Qualytics for the catalog. | +| 4. | Include | Checkboxes to select Tables, Views, or both, specifying the resources to include in the catalog. | + +![catalog](.././assets/flows/catalog-light-26.png) + +**Step 2:** After configuring the settings, click Save to apply and proceed with the catalog operation. + +![save](.././assets/flows/save-light-8.png) + +## Profile + +**Step 1:** Click on **Profile.** + +![profile](.././assets/flows/profile-light-28.png) + +A panel **Profile Settings** will appear on the right-hand side. This window allows you to configure the Profile operation. + +![profile](.././assets/flows/profile-light-29.png) + +| No. | Field | Description | +| :---- | :---- | :---- | +| 1. | Source Datastore | Select the source datastore to profile. | +| 2. | Select Tables | Allows users to select all tables, specific tables, or tables associated with selected tags to profile. | +| 3. | Read Settings | Configure the starting point for profiling and set a maximum record limit per table for profiling. | +| 4. | Inference Settings | Set the level of automated checks and decide whether inferred checks should be saved in draft mode. | + +![profile](.././assets/flows/profile-light-30.png) + +**Step 2:** Click Save to finalize the profile configuration. + +![save](.././assets/flows/save-light-8.png) + +## Scan + +**Step 1:** Click on **Scan.** + +![scan](.././assets/flows/scan-light-32.png) + +A panel **Scan Settings** will appear on the right-hand side. This window allows you to configure the Scan operation. + +![scan](.././assets/flows/scan-light-33.png) + +**Source Datastore:** Select the datastore to be scanned. + +![scan](.././assets/flows/scan-light-34.png) + +**Select Tables:** Choose all tables, specific tables, or tables associated with selected tags to include in the scan. + +![scan](.././assets/flows/scan-light-35.png) + +**Select Check Categories:** Select categories of checks to include, such as table properties (Metadata) or value checks (Data Integrity). + +![scan](.././assets/flows/scan-light-36.png) + +**Read Settings:** Define the scan strategy: incremental scans updated records; full scans process all records. + +![scan](.././assets/flows/scan-light-39.png) + +**Starting Threshold:** Set a starting point for scanning based on an incremental identifier. + +![scan](.././assets/flows/scan-light-37.png) + +**Record Limit:** Specify the maximum number of records to scan per table. + +![scan](.././assets/flows/scan-light-38.png) + +**Scan Settings:** Choose how to manage duplicate or recurring anomalies by archiving overlaps or reactivating previously archived anomalies with fingerprint tracking. + +![scan](.././assets/flows/scan-settings-light.png) + +**Anomaly Rollup Threshold:** Set the Rollup Threshold to limit how many anomalies are created per check. When the limit is reached, anomalies will be merged into one for easier management. + +![rollup](.././assets/flows/rollup-light-39.png) + +**Enrichment Source Record Limit:** Define the number of source records to include in the enrichment operation. + +![scan](.././assets/flows/scan-light-40.png) + +**Step 2:** Click Save to finalize the scan configuration. + +![save](.././assets/flows/save-light-43.png) + +## Export + +**Step 1:** Click on **Export.** + +![export](.././assets/flows/export-light.png) + +A panel **Export Settings** will appear on the right-hand side. This window allows you to configure the Export settings. + +![panel](.././assets/flows/export-setting-light.png) + +**Source Datastore:** Select the datastore to export data from. + +![source](.././assets/flows/source-light.png) + +**Select file patterns to export:** **All** (all file patterns, including future ones), **Specific** (manually chosen file patterns), or **Tag** (file patterns based on selected tags). + +![profile](.././assets/flows/profiles-light.png) + +**Select Metadata:** Choose metadata to export **anomalies**, **quality checks**, or **field profiles**. Anomalies detect data issues, quality checks validate data, and field profiles store field metadata. + +![exportt](.././assets/flows/exportt-light.png) + +**Step 2:** Click Save to finalize the export configuration. + +![save](.././assets/flows/save-light-8.png) + +Export nodes display the asset type in their titles (e.g., “Export Anomalies”) to help you identify the exported content easily. + +![export-status](.././assets/flows/export-status-light.png) + +## Materialize + +**Step 1:** Click on **Materialize.** + +![materialize](.././assets/flows/materialize-light.png) + +A panel **Materialize Settings** will appear on the right-hand side. This window allows you to configure the Materialize settings. + +![setting](.././assets/flows/setting-light.png) + +**Source Datastore:** Select the datastore to materialize data from. + +![source](.././assets/flows/sourcee-light.png) + +**Select Tables:** Choose which tables (all, specific, or tagged) to extract from your source datastore and export to the enrichment datastore. + +![select](.././assets/flows/select-light.png) + +**Read Settings:** Select the record limit to control how much data is materialized per table. + +![read](.././assets/flows/read-light.png) + +**Step 2:** Click Save to finalize the materialize configuration. + +![save](.././assets/flows/saveee-light.png) \ No newline at end of file diff --git a/docs/flows/overview-action.md b/docs/flows/overview-action.md new file mode 100644 index 000000000..633aebdbc --- /dev/null +++ b/docs/flows/overview-action.md @@ -0,0 +1,41 @@ +# Action Node + +Actions define the specific steps the system will execute after a flow is triggered. They allow users to automate tasks, send notifications, or interact with external systems. + +**Step 1:** After completing the **"Trigger"** node setup, users can click on the **"Actions"** node. + +![action](.././assets/flows/action-light-22.png) + +A panel will appear on the right-hand side displaying the list of available actions. These actions define what the system will execute after the flow is triggered. The actions are categorized into three groups: + +* Operations. + +* Notifications. + +* HTTP. + +![actionlist](.././assets/flows/actionlist-light-23.png) + +!!! info + Inline summaries are shown within action nodes, displaying key details based on the action type—for example, datastore names for operations, Slack or Teams channels for notifications, and webhook URLs for HTTP actions. This enhancement provides quick clarity during flow configuration. + +## Operations + +!!! note + For more detailed information, review the [operations documentation](../flows/operations.md){target="_blank"}. + +## Notifications + +!!! note + For more detailed information, review the [notifications documentation](../flows/notification.md){target="_blank"}. + +## HTTP + +!!! note + For more detailed information, review the [http documentation](../flows/http.md){target="_blank"}. + +## FAQ + +**1>.** Can I have multiple actions under a single flow? + +Yes. You can chain multiple actions—such as **notifications, operations,** or **HTTP steps**—under a single flow to perform sequential or parallel tasks. \ No newline at end of file diff --git a/docs/flows/overview.md b/docs/flows/overview.md new file mode 100644 index 000000000..f3ae689cd --- /dev/null +++ b/docs/flows/overview.md @@ -0,0 +1,120 @@ +# Flows + +Flows in Qualytics help you automate everyday data tasks so you don’t have to do them manually. Each Flow is a chain of connected steps that runs automatically — it starts when something happens (like a data scan finishing) and then performs one or more actions, such as sending a notification or running another check. This makes your work faster, more consistent, and less prone to errors, keeping your data processes running smoothly even when you’re not monitoring them. + +## Why You Need Flows + +Modern data pipelines involve dozens of repetitive steps — from scanning data and profiling tables to alerting the right people when something breaks. Doing all this manually can be time-consuming and prone to errors. + +**Flows** automate this entire process by: + +- **Reducing manual work:** Automatically run follow-up operations when specific events occur. +- **Maintaining consistency:** Ensure all datastores follow the same workflow and rules. +- **Improving response time:** Notify teams instantly about scan results or detected anomalies. +- **Centralizing control:** Manage triggers and actions from one place instead of multiple scripts or manual tasks. + +## How Flows Work + +Every Flow in Qualytics is made up of three key parts that work together to automate your process: + +### Flow Node + +Defines the name, purpose, and activation state of your Flow. +This is where you decide whether the Flow is active or paused. + +### Trigger Node + +Decides *when* your Flow starts. +Triggers can be: + +- **Operation-based:** Starts after a Catalog, Profile, or Scan completes. +- **Anomaly-based:** Starts when Qualytics detects data issues. +- **Manual:** Starts only when you trigger it yourself. + +### Action Node(s) + +Controls *what happens next* once the trigger fires. +Actions can: + +- Run operations like **Scan** or **Export**. +- Send alerts via **Email**, **Slack**, **Teams**, or **PagerDuty**. +- Call external systems using **webhooks** or **API requests**. + +Each Flow runs in order — from **Trigger → Action(s)** — and you can track its progress or results in the **Execution tab**. + +## Example + +Imagine you manage a retail company’s product database that gets updated every night. + +You can create a Flow like this: + +- **Trigger:** When a **Catalog Operation Completes** on your datastore. +- **Action 1:** Automatically run a **Profile** to detect schema or data type changes. +- **Action 2:** Run a **Scan** to check for anomalies in product prices or availability. +- **Action 3:** Send a **Slack Notification** to your team if anomalies are detected. + +This setup ensures every new data load is automatically checked, issues are flagged instantly, and your team can respond right away — without any manual effort. + +## Navigation to Flows + +**Step 1**: Log in to your Qualytics account and click on **Flows** on the left side panel of the interface. + +![flows](.././assets/flows/flow-light-1.png) + +You will navigate to the Flows interface, where you can add and manage flows. At the top, you will see two tabs: + +* **Definitions:** Displays a list of all flows along with details like triggers, actions, tags, and the last triggered time. + +![definition](.././assets/flows/definitions-light-2.png) + +* **Executions:** Provides the execution history of flows, including their status and timestamps. + +![execution](.././assets/flows/execution-light-3.png) + +## Add Flow + +Allows users to create a new Flow, starting with default nodes to define triggers and actions. + +**Step 1**: Click on the **Add Flow** button from the top right corner. + +!!! note + For more detailed information, review the [add flow](../flows/add-flow.md){target="_blank"} documentation. + +## Flow Node + +Defines the core of a Flow, including its name, description, and activation status. + +**Step 1:** Click on the **Flow** node. + +!!! note + For more detailed information, review the [flow entry](../flows/flow-entry.md){target="_blank"} documentation. + +## Trigger Node + +Determines when the Flow should start, based on events, conditions, or manual activation. + +**Step 1:** After completing the **"Flow"** node setup, users can click on the **"Trigger"** node. + +!!! note + For more detailed information, review the [trigger node](../flows/trigger-node.md){target="_blank"} documentation. + +## Actions Node + +Actions define the specific steps the system will execute after a flow is triggered. They allow users to automate tasks, send notifications, or interact with external systems. + +!!! note + For more detailed information, review the [action node overview](../flows/overview-action.md){target="_blank"} documentation. + +## View and Track Flow Execution + +Once a flow is added, it will be visible in the **Definitions** tab, where you can view all the created flows. + +!!! note + For more detailed information, review the [view and track flow execution](../flows/view-created-flows.md){target="_blank"} documentation. + +## Execution Tab + +Execution tab allows users to view the execution history and current status of a flow. It provides detailed timestamps, status updates, and a comprehensive record of flow executions for efficient tracking and analysis. + +!!! note + For more detailed information, review the [execution tab](../flows/view-created-flows.md#accessing-the-execution-tab){target="_blank"} documentation. \ No newline at end of file diff --git a/docs/flows/sort-flow.md b/docs/flows/sort-flow.md new file mode 100644 index 000000000..d661433ab --- /dev/null +++ b/docs/flows/sort-flow.md @@ -0,0 +1,9 @@ +# Sort Flows + +Qualytics allows you to sort your flows by **Created Date**, **Name**, and **Last Triggered** to easily organize and prioritize them according to your needs. + +![sort](.././assets/flows/sort-light-85.png) + +Whatever sorting option is selected, you can arrange the data either in ascending or descending order by clicking the caret button next to the selected sorting criteria. + +![sort](.././assets/flows/sort-light-86.png) diff --git a/docs/flows/trigger-node.md b/docs/flows/trigger-node.md new file mode 100644 index 000000000..1d10364a5 --- /dev/null +++ b/docs/flows/trigger-node.md @@ -0,0 +1,120 @@ +# Trigger + +This is where you can configure how the Flow will be activated. It can be activated automatically by configuring the settings, or you can choose to activate it manually. + +**Step 1:** After completing the **"Flow"** node setup, users can click on the **"Trigger"** node. + +![trigger](.././assets/flows/trigger-light-9.png) + +A panel will appear on the right-hand side, enabling users to define when the flow should start. The panel provides four options for initiating the flow. Users can choose one of the following options: + +* Operation Completes. + +* Anomalous Table and File Detection. + +* Anomaly Detected. + +* Manual + +![triggersetting](.././assets/flows/triggersetting-light-10.png) + +## Important Information About Triggers + +!!! note + Each Flow can have only **one trigger**. If you want to execute a different action or change the trigger, you’ll need to **clone** the Flow and configure the new trigger. + + [Click here to learn how to clone a Flow](../flows/clone-a-flow.md){target="_blank"}. + +## Operation Completes + +This type of flow is triggered whenever an operation, such as a catalog, profile, or scan, is completed on a source datastore. Upon completion, teams are promptly notified through in-app messages and, if configured, via external notification channels such as email, Slack, Microsoft Teams, and others. For example, the team is notified whenever the catalog operation is completed, helping them proceed with the profile operation on the datastore. + +![operation](.././assets/flows/operation-light-11.png) + +**Filter Conditions** + +Filters can be set to narrow down which operations should trigger the flow execution: + +1. **Source Datastore Tags**: The flow is triggered only for source datastores that have all the selected tags assigned. + +2. **Source Datastores**: The flow is triggered only for the selected source datastores. + +3. **Operation Types**: The flow is triggered only for operations that match one or more of the selected types. + +4. **Operation Status**: The flow is triggered for operations with a status of either Success or Failure. + +![operation](.././assets/flows/operation-light-12.png) + +Once the conditions are set, click the **Save** button to finalize the trigger configuration. + +![save](.././assets/flows/save-light-8.png) + +## Anomalous Table and File Detected + +This flow is triggered when anomalies are detected within a specific table, file and check rule types. It includes information about the number of anomalies found and the specific scan target within the datastore. This is useful for assessing the overall health of a particular datastore. + +![table](.././assets/flows/table-light-14.png) + +**Filter Conditions** + +Users can optionally set filters to specify which tables or files should trigger the flow execution. + +1. **Tables / Files Tags**: Only tables or files with all the selected tags assigned will trigger the flow. + +2. **Source Datastores**: The flow is triggered only for the selected source datastores. + +3. **Check Rule Types**: Only anomalies identified by one or more of the selected check rule types will initiate the flow. + +![table](.././assets/flows/table-light-15.png) + +After defining the conditions, users must click the **Save** button to finalize the trigger configuration. + +![save](.././assets/flows/save-light-8.png) + +## Anomaly Detected + +This type of flow is triggered when any single anomaly is identified in the data. The flow message typically includes the type of anomaly detected and the datastore where it was found. It provides specific information about the anomaly type, which helps quickly understand the issue's nature. + +![anomaly](.././assets/flows/anomaly-light-17.png) + +**Filter Condition** + +Users can define specific conditions to determine when the flow should be initiated. + +1. **Anomaly’s Tags**: Only anomalies with all selected tags assigned will trigger the flow. + +2. **Source Datastores**: Only triggered when anomalies are detected in the selected datastores. + +3. **Check Rule Types**: Only anomalies identified by one or more of the selected check rule types will initiate the flow. + +4. **Anomaly Weight (Min)**: Only anomalies with a weight equal to or greater than the specified value will trigger the flow. + +![anomaly](.././assets/flows/anomaly-light-18.png) + +**Step 2:** Once the filter conditions are set, users must click the **Save** button to finalize the configuration. + +![save](.././assets/flows/save-light-8.png) + +## Manual + +The flow starts only when the user manually triggers it. It doesn’t depend on any automatic conditions or detections, giving the user full control. + +!!! note + Here you can manually activate a Flow. For a full walkthrough of manual execution, refer to + [Execute Manual Flows](../flows/execute-manual-flows.md){target="_blank"} documentation. + +![manual](.././assets/flows/manual-light-20.png) + +Once selected, users must click the **Save** button to confirm the manual trigger configuration. + +![save](.././assets/flows/save-light-8.png) + +Hover over the **filter tooltip** in trigger nodes to view the applied conditions such as tags, datastores, and operation types. This provides quick visibility into how each trigger is configured. + +![filter-tooltip](.././assets/flows/filter-tooltip-light.png) + +## FAQ + +**1>.** What’s the difference between a Flow and a Trigger? + +A Flow is the overall automation pipeline, while a Trigger determines when the flow starts — such as after operation completion, anomaly detection, or manually. \ No newline at end of file diff --git a/docs/flows/view-created-flows.md b/docs/flows/view-created-flows.md new file mode 100644 index 000000000..698395d96 --- /dev/null +++ b/docs/flows/view-created-flows.md @@ -0,0 +1,54 @@ +# View and Track Flow Executions + +The **Execution** tab allows users to view the execution history and current status of a flow. It provides detailed timestamps, status updates, and a comprehensive record of flow executions for efficient tracking and analysis. + +## Accessing the Execution Tab + +**Step 1**: Click on the **Execution** tab. + +![executions](.././assets/flows/executions-light-99.png) + +Once clicked, you will be navigated to the **Execution** tab, where you can view the complete execution history of all created flows. + +![executions](.././assets/flows/executions-light-100.png) + +## See a Flow Execution + +Users can view flow execution in real-time by clicking on the desired flow operation. The page will show detailed operations, but no edits can be made here. + +**Step 1**: Click on the flow operation you want to view. + +![manual-flow](.././assets/flows/manuall-flow-light.png) + +After clicking, you will navigate to the selected flow operation details page. The page displays all operational details in real-time. Please note, this page is **view-only**, and no edits can be made. + +![flow](.././assets/flows/flow-light-1001.png) + +## Understanding Flow States + +On the bottom-right corner, there is a **Legend** indicating the possible states of an action, such as: + +- **Success** (Green) +- **Failure** (Red) +- **Aborted** (Orange) +- **Skipped** (Yellow) +- **Running** (Blue with dotted lines animation) +- **Pending** (Gray) + +![chart](.././assets/flows/chart-light-101.png) + +If a step is running, you will see a **dot-line animation**, signaling that the step is in progress. Once the action completes, the **Action** box will change its color to reflect the final state. + +![chart](.././assets/flows/chart-light-102.png) + +## Accessing Operation Results + +To view detailed results of specific operations: + +**Step 1**: Click the **Top Right Arrow** button within the action operation box. + +![chart](.././assets/flows/chart-light-103.png) + +**Step 2**: You will navigate to the **Activity page**, where a **Result Modal** will open, displaying in-depth details of the operation. + +![result](.././assets/flows/result-light-104.png) diff --git a/mkdocs.yml b/mkdocs.yml index 4029b4ecc..5a26d35cd 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -211,7 +211,26 @@ nav: - Checks: explore/checks.md - Anomalies: explore/anomalies.md - Flows: - - Overview: flows/flows.md + - Overview: flows/overview.md + - Add Flow: flows/add-flow.md + - Flow Entry: flows/flow-entry.md + - Trigger Node: flows/trigger-node.md + - Actions Node: + - Overview: flows/overview-action.md + - Operations: flows/operations.md + - Notifications: flows/notification.md + - HTTP: flows/http.md + - View and Track Flow Executions: flows/view-created-flows.md + - Manage Flow: + - Clone a Flows: flows/clone-a-flow.md + - Execute Manual Flow: flows/execute-manual-flows.md + - Abort Flow: flows/abort.md + - Sort Flow: flows/sort-flow.md + - Edit Flow: flows/edit-flow.md + - Delete Flow: flows/delete-flow.md + - Deactivate Flow: flows/deactivate-flow.md + - Activate Flows: flows/activate-flow.md + - Clone An Action: flows/clone-an-action.md - Tags: - Overview: tags/overview.md - External Tags: tags/external-tag.md