Skip to content

Commit

Permalink
example: extra sections in Text operator
Browse files Browse the repository at this point in the history
  • Loading branch information
jvallesm committed Jun 24, 2024
1 parent 391bb98 commit 9a3511d
Show file tree
Hide file tree
Showing 4 changed files with 23 additions and 1 deletion.
10 changes: 10 additions & 0 deletions operator/text/v0/.compogen/extra-chunk.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
The `strategy field defines the chunking method through the `chunk-method`
property. The following methods are implemented:

- Token
- Markdown
- Recursive

Please refer to
[`tasks.json`](https://github.com/instill-ai/component/blob/main/operator/text/v0/config/tasks.json)
for more information.
1 change: 1 addition & 0 deletions operator/text/v0/.compogen/extra-intro.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
The Text operator is particularly important for RAG applications.
11 changes: 11 additions & 0 deletions operator/text/v0/README.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ It can carry out the following tasks:
- [Split By Token](#split-by-token)
- [Chunk Text](#chunk-text)

The Text operator is particularly important for RAG applications.


## Release Stage
Expand Down Expand Up @@ -99,6 +100,16 @@ Chunk text with different strategies.
| Number of Text Chunks | `chunk-num` | integer | Total number of output text chunks |


The `strategy field defines the chunking method through the `chunk-method`
property. The following methods are implemented:

- Token
- Markdown
- Recursive

Please refer to
[`tasks.json`](https://github.com/instill-ai/component/blob/main/operator/text/v0/config/tasks.json)
for more information.



Expand Down
2 changes: 1 addition & 1 deletion operator/text/v0/main.go
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
//go:generate compogen readme ./config ./README.mdx
//go:generate compogen readme ./config ./README.mdx --extraContents intro=.compogen/extra-intro.mdx --extraContents TASK_CHUNK_TEXT=.compogen/extra-chunk.mdx
package text

import (
Expand Down

0 comments on commit 9a3511d

Please sign in to comment.