-
Notifications
You must be signed in to change notification settings - Fork 15
docs: add use cases tutorials for AI workflows #392
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
jumski
wants to merge
1
commit into
main
Choose a base branch
from
claude/add-use-cases-tutorials-011CUrzyLUcAZQ8EY4BCi1eC
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| /home/jumski/Code/pgflow-dev/claude/source/.claude/README.md |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| /home/jumski/Code/pgflow-dev/claude/source/.claude/agents |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| /home/jumski/Code/pgflow-dev/claude/source/.claude/commands |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| /home/jumski/Code/pgflow-dev/claude/source/.claude/core |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| /home/jumski/Code/pgflow-dev/claude/source/.claude/guides |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| /home/jumski/Code/pgflow-dev/claude/source/.claude/reference |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| /home/jumski/Code/pgflow-dev/claude/source/.claude/settings.json |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| /home/jumski/Code/pgflow-dev/claude/source/.claude/skills |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| /home/jumski/Code/pgflow-dev/claude/source/.mcp.json |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| /home/jumski/Code/pgflow-dev/claude/source/CLAUDE.md | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| /home/jumski/Code/pgflow-dev/claude/source/pkgs/cli/CLAUDE.md |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| /home/jumski/Code/pgflow-dev/claude/source/pkgs/client/CLAUDE.md |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| /home/jumski/Code/pgflow-dev/claude/source/pkgs/edge-worker/CLAUDE.md |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| /home/jumski/Code/pgflow-dev/claude/source/pkgs/website/CLAUDE.md |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
191 changes: 191 additions & 0 deletions
191
pkgs/website/src/content/docs/tutorials/use-cases/automatic-embeddings.mdx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,191 @@ | ||
| --- | ||
| title: Automatic Embeddings | ||
| description: Generate embeddings automatically with database triggers and pgflow | ||
| sidebar: | ||
| order: 1 | ||
| --- | ||
|
|
||
| Generate vector embeddings automatically when new content is added to the database. | ||
|
|
||
| ## Setup | ||
|
|
||
| Install dependencies: | ||
|
|
||
| ```bash | ||
| pnpm add ai @ai-sdk/openai | ||
| ``` | ||
|
|
||
| Enable pgvector extension in a migration: | ||
|
|
||
| ```sql | ||
| create extension if not exists vector; | ||
| ``` | ||
|
|
||
| ## Database Schema | ||
|
|
||
| ```sql | ||
| create table documents ( | ||
| id bigserial primary key, | ||
| content text not null, | ||
| created_at timestamptz default now() | ||
| ); | ||
|
|
||
| create table document_chunks ( | ||
| id bigserial primary key, | ||
| document_id bigint references documents(id) on delete cascade, | ||
| content text not null, | ||
| embedding vector(1536) | ||
| ); | ||
| ``` | ||
|
|
||
| ## Task Functions | ||
|
|
||
| **Split document into chunks:** | ||
|
|
||
| ```typescript | ||
| // supabase/functions/_tasks/splitChunks.ts | ||
| export default async function splitChunks(content: string) { | ||
| const chunks = content | ||
| .trim() | ||
| .split('.') | ||
| .filter(chunk => chunk.trim().length > 0) | ||
| .map(chunk => chunk.trim()); | ||
|
|
||
| return chunks; | ||
| } | ||
| ``` | ||
|
|
||
| **Generate embeddings for chunks:** | ||
|
|
||
| ```typescript | ||
| // supabase/functions/_tasks/generateEmbeddings.ts | ||
| import { openai } from '@ai-sdk/openai'; | ||
| import { embedMany } from 'ai'; | ||
|
|
||
| export default async function generateEmbeddings(chunks: string[]) { | ||
| const { embeddings } = await embedMany({ | ||
| model: openai.textEmbeddingModel('text-embedding-3-small'), | ||
| values: chunks, | ||
| }); | ||
|
|
||
| return chunks.map((chunk, i) => ({ | ||
| content: chunk, | ||
| embedding: embeddings[i], | ||
| })); | ||
| } | ||
| ``` | ||
|
|
||
| **Save chunks with embeddings:** | ||
|
|
||
| ```typescript | ||
| // supabase/functions/_tasks/saveChunks.ts | ||
| import { createClient } from 'jsr:@supabase/supabase-js'; | ||
|
|
||
| export default async function saveChunks(input: { | ||
| documentId: number; | ||
| chunks: Array<{ content: string; embedding: number[] }>; | ||
| }) { | ||
| const supabaseUrl = Deno.env.get('SUPABASE_URL'); | ||
| const supabaseKey = Deno.env.get('SUPABASE_SERVICE_ROLE_KEY'); | ||
|
|
||
| if (!supabaseUrl || !supabaseKey) { | ||
| throw new Error('Missing Supabase credentials'); | ||
| } | ||
|
|
||
| const supabase = createClient(supabaseUrl, supabaseKey); | ||
|
|
||
| const rows = input.chunks.map(chunk => ({ | ||
| document_id: input.documentId, | ||
| content: chunk.content, | ||
| embedding: JSON.stringify(chunk.embedding), | ||
| })); | ||
|
|
||
| const { data } = await supabase | ||
| .from('document_chunks') | ||
| .insert(rows) | ||
| .select() | ||
| .throwOnError(); | ||
|
|
||
| return data; | ||
| } | ||
| ``` | ||
|
|
||
| ## Flow Definition | ||
|
|
||
| ```typescript | ||
| // supabase/functions/_flows/generate_embeddings.ts | ||
| import { Flow } from 'npm:@pgflow/dsl'; | ||
| import splitChunks from '../_tasks/splitChunks.ts'; | ||
| import generateEmbeddings from '../_tasks/generateEmbeddings.ts'; | ||
| import saveChunks from '../_tasks/saveChunks.ts'; | ||
|
|
||
| type Input = { | ||
| documentId: number; | ||
| content: string; | ||
| }; | ||
|
|
||
| export default new Flow<Input>({ slug: 'generateEmbeddings' }) | ||
| .step({ slug: 'split' }, ({ run }) => | ||
| splitChunks(run.content) | ||
| ) | ||
| .array({ slug: 'chunks', dependsOn: ['split'] }, ({ split }) => | ||
| split | ||
| ) | ||
| .map({ slug: 'embed', dependsOn: ['chunks'] }, ({ chunks }) => | ||
| generateEmbeddings([chunks.item]) | ||
| ) | ||
| .step({ slug: 'save', dependsOn: ['embed'] }, ({ run, embed }) => | ||
| saveChunks({ | ||
| documentId: run.documentId, | ||
| chunks: embed.flat(), | ||
| }) | ||
| ); | ||
| ``` | ||
|
|
||
| The flow uses `.array()` to convert the chunks array into parallel tasks, then `.map()` to embed each chunk, and finally aggregates results to save all chunks at once. | ||
|
|
||
| ## Database Trigger | ||
|
|
||
| ```sql | ||
| create or replace function trigger_embedding_flow() | ||
| returns trigger as $$ | ||
| begin | ||
| perform pgflow.start_flow( | ||
| flow_slug => 'generateEmbeddings', | ||
| input => json_build_object( | ||
| 'documentId', new.id, | ||
| 'content', new.content | ||
| ) | ||
| ); | ||
| return new; | ||
| end; | ||
| $$ language plpgsql; | ||
|
|
||
| create trigger documents_embedding_trigger | ||
| after insert on documents | ||
| for each row | ||
| execute function trigger_embedding_flow(); | ||
| ``` | ||
|
|
||
| ## Compile and Deploy | ||
|
|
||
| ```bash | ||
| npx pgflow@latest compile supabase/functions/_flows/generate_embeddings.ts | ||
| npx supabase migrations up --local | ||
| ``` | ||
|
|
||
| ## Usage | ||
|
|
||
| Insert a document - embeddings generate automatically: | ||
|
|
||
| ```sql | ||
| insert into documents (content) values ( | ||
| 'PostgreSQL is a powerful database. It supports extensions. pgvector enables vector search.' | ||
| ); | ||
|
|
||
| -- Check chunks | ||
| select id, content, embedding[1:5] as embedding_sample | ||
| from document_chunks; | ||
| ``` | ||
|
|
||
| The trigger starts the flow automatically, which splits content into chunks, generates embeddings in parallel using `.map()`, and saves them to the database. |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
CLAUDE.md is committed as a symlink to
/home/jumski/Code/pgflow-dev/claude/source/CLAUDE.md, which only exists on the author’s machine. In any other checkout the target is missing, leaving the doc unreadable and leaking a local path, so any process expecting this file will fail; the link should be made relative or the actual content checked in.Useful? React with 👍 / 👎.