Skip to content

Platform: DAG options #355

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Dec 1, 2024
Merged

Platform: DAG options #355

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 32 additions & 5 deletions platform/workflows.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -245,12 +245,39 @@ There are two ways to create a custom workflow:
- Checking this box reprocesses all documents in the source location on every workflow run.
- Unchecking this box causes only new documents that are added to the source location since the last workflow run to be processed on future runs. Previously processed documents are not processed again, even if those documents' contents change.

8. In the pipeline designer, click the **Source** node. In the **Source** pane, select the source location. Then click **Save**.
8. The workflow begins with the following layout:


```mermaid
flowchart LR
Source-->Partitioner-->Destination
```

The following workflow layouts are also valid:

```mermaid
flowchart LR
Source-->Partitioner-->Chunker-->Destination
```
```mermaid
flowchart LR
Source-->Partitioner-->Chunker-->Embedder-->Destination
```
```mermaid
flowchart LR
Source-->Partitioner-->Enrichment-->Chunker-->Destination
```
```mermaid
flowchart LR
Source-->Partitioner-->Enrichment-->Chunker-->Embedder-->Destination
```

9. In the pipeline designer, click the **Source** node. In the **Source** pane, select the source location. Then click **Save**.

![Workflow designer](/img/platform/Workflow-Designer.png)

9. Click the **Destination** node. In the **Destination** pane, select the destination location. Then click **Save**.
10. As needed, add more nodes by clicking the plus icon (recommended) or **Add Node** button:
10. Click the **Destination** node. In the **Destination** pane, select the destination location. Then click **Save**.
11. As needed, add more nodes by clicking the plus icon (recommended) or **Add Node** button:

![Add node to workflow](/img/platform/Workflow-Add-Node.png)

Expand All @@ -269,8 +296,8 @@ There are two ways to create a custom workflow:

To delete a node, click that node, and then click the trash can icon above it.

11. Click **Save**.
12. If you did not set the workflow to run on a schedule, you can [run the worklow](#edit-delete-or-run-a-workflow) now.
12. Click **Save**.
13. If you did not set the workflow to run on a schedule, you can [run the worklow](#edit-delete-or-run-a-workflow) now.

#### Custom workflow node types

Expand Down