-
-
Notifications
You must be signed in to change notification settings - Fork 894
Added use cases section in the docs #2641
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
987092c
e38c062
f43468e
62b3c9d
1c14727
829eef2
77ab65b
59ac4f7
001e094
ce7e7ad
74f7139
60060c6
9e64af1
8184ea2
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,159 @@ | ||
| --- | ||
| title: "Data processing & ETL workflows" | ||
| sidebarTitle: "Data processing & ETL" | ||
| description: "Learn how to use Trigger.dev for data processing and ETL (Extract, Transform, Load), including web scraping, database synchronization, batch enrichment and more." | ||
| --- | ||
|
|
||
| import UseCasesCards from "/snippets/use-cases-cards.mdx"; | ||
|
|
||
| ## Overview | ||
|
|
||
| Build complex data pipelines that process large datasets without timeouts. Handle streaming analytics, batch enrichment, web scraping, database sync, and file processing with automatic retries and progress tracking. | ||
|
|
||
| ## Featured examples | ||
|
|
||
| <CardGroup cols={3}> | ||
| <Card | ||
| title="Realtime CSV importer" | ||
| icon="book" | ||
| href="/guides/example-projects/realtime-csv-importer" | ||
| > | ||
| Import CSV files with progress streamed live to frontend. | ||
| </Card> | ||
| <Card title="Web scraper with BrowserBase" icon="book" href="/guides/examples/scrape-hacker-news"> | ||
| Scrape websites using BrowserBase and Puppeteer. | ||
| </Card> | ||
| <Card | ||
| title="Supabase database webhooks" | ||
| icon="book" | ||
| href="/guides/frameworks/supabase-edge-functions-database-webhooks" | ||
| > | ||
| Trigger tasks from Supabase database webhooks. | ||
| </Card> | ||
| </CardGroup> | ||
|
|
||
| ## Benefits of using Trigger.dev for data processing & ETL workflows | ||
|
|
||
| **Process datasets for hours without timeouts:** Handle multi-hour transformations, large file processing, or complete database exports. No execution time limits. | ||
|
|
||
| **Parallel processing with built-in rate limiting:** Process thousands of records simultaneously while respecting API rate limits. Scale efficiently without overwhelming downstream services. | ||
|
|
||
| **Stream progress to your users in real-time:** Show row-by-row processing status updating live in your dashboard. Users see exactly where processing is and how long remains. | ||
|
|
||
| ## Production use cases | ||
|
|
||
| <CardGroup cols={1}> | ||
| <Card title="MagicSchool AI customer story" href="https://trigger.dev/customers/magicschool-ai-customer-story"> | ||
|
|
||
| Read how MagicSchool AI uses Trigger.dev to generate insights from millions of student interactions. | ||
|
|
||
| </Card> | ||
|
|
||
| <Card title="Comp AI customer story" href="https://trigger.dev/customers/comp-ai-customer-story"> | ||
|
|
||
| Read how Comp AI uses Trigger.dev to automate evidence collection at scale, powering their open source, AI-driven compliance platform. | ||
|
|
||
| </Card> | ||
| <Card title="Midday customer story" href="https://trigger.dev/customers/midday-customer-story"> | ||
|
|
||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Fix verb agreement: "Midday use" → "Midday uses". Line 58 has a subject-verb agreement error. Apply this diff: -Read how Midday use Trigger.dev to sync large volumes of bank transactions in their financial management platform.
+Read how Midday uses Trigger.dev to sync large volumes of bank transactions in their financial management platform.
🧰 Tools🪛 LanguageTool[style] ~58-~58: Three successive sentences begin with the same word. Consider rewording the sentence or use a thesaurus to find a synonym. (ENGLISH_WORD_REPEAT_BEGINNING_RULE) 🤖 Prompt for AI Agents |
||
| Read how Midday use Trigger.dev to sync large volumes of bank transactions in their financial management platform. | ||
|
|
||
| </Card> | ||
| </CardGroup> | ||
|
|
||
| ## Example workflow patterns | ||
|
|
||
| <Tabs> | ||
| <Tab title="CSV file import"> | ||
| Simple CSV import pipeline. Receives file upload, parses CSV rows, validates data, imports to database with progress tracking. | ||
|
|
||
| <div align="center"> | ||
|
|
||
| ```mermaid | ||
| graph TB | ||
| A[importCSV] --> B[parseCSVFile] | ||
| B --> C[validateRows] | ||
| C --> D[bulkInsertToDB] | ||
| D --> E[notifyCompletion] | ||
| ``` | ||
|
|
||
| </div> | ||
| </Tab> | ||
|
|
||
| <Tab title="Multi-source ETL pipeline"> | ||
| **Coordinator pattern with parallel extraction**. Batch triggers parallel extraction from multiple sources (APIs, databases, S3), transforms and validates data, loads to data warehouse with monitoring. | ||
|
|
||
| <div align="center"> | ||
|
|
||
| ```mermaid | ||
| graph TB | ||
| A[runETLPipeline] --> B[coordinateExtraction] | ||
| B --> C[batchTriggerAndWait] | ||
|
|
||
| C --> D[extractFromAPI] | ||
| C --> E[extractFromDatabase] | ||
| C --> F[extractFromS3] | ||
|
|
||
| D --> G[transformData] | ||
| E --> G | ||
| F --> G | ||
|
|
||
| G --> H[validateData] | ||
| H --> I[loadToWarehouse] | ||
| ``` | ||
|
|
||
| </div> | ||
| </Tab> | ||
|
|
||
| <Tab title="Parallel web scraping"> | ||
| **Coordinator pattern with browser automation**. Launches headless browsers in parallel to scrape multiple pages, extracts structured data, cleans and normalizes content, stores in database. | ||
|
|
||
| <div align="center"> | ||
|
|
||
| ```mermaid | ||
| graph TB | ||
| A[scrapeSite] --> B[coordinateScraping] | ||
| B --> C[batchTriggerAndWait] | ||
|
|
||
| C --> D[scrapePage1] | ||
| C --> E[scrapePage2] | ||
| C --> F[scrapePageN] | ||
|
|
||
| D --> G[cleanData] | ||
| E --> G | ||
| F --> G | ||
|
|
||
| G --> H[normalizeData] | ||
| H --> I[storeInDatabase] | ||
| ``` | ||
|
|
||
| </div> | ||
| </Tab> | ||
|
|
||
| <Tab title="Batch data enrichment"> | ||
| **Coordinator pattern with rate limiting**. Fetches records needing enrichment, batch triggers parallel API calls with configurable concurrency to respect rate limits, validates enriched data, updates database. | ||
|
|
||
| <div align="center"> | ||
|
|
||
| ```mermaid | ||
| graph TB | ||
| A[enrichRecords] --> B[fetchRecordsToEnrich] | ||
| B --> C[coordinateEnrichment] | ||
| C --> D[batchTriggerAndWait] | ||
|
|
||
| D --> E[enrichRecord1] | ||
| D --> F[enrichRecord2] | ||
| D --> G[enrichRecordN] | ||
|
|
||
| E --> H[validateEnrichedData] | ||
| F --> H | ||
| G --> H | ||
|
|
||
| H --> I[updateDatabase] | ||
| ``` | ||
|
|
||
| </div> | ||
| </Tab> | ||
| </Tabs> | ||
|
|
||
| <UseCasesCards /> | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,147 @@ | ||
| --- | ||
| title: "Marketing workflows" | ||
| sidebarTitle: "Marketing" | ||
| description: "Learn how to use Trigger.dev for marketing workflows, including drip campaigns, behavioral triggers, personalization engines, and AI-powered content workflows" | ||
| --- | ||
|
|
||
| import UseCasesCards from "/snippets/use-cases-cards.mdx"; | ||
|
|
||
| ## Overview | ||
|
|
||
| Build marketing workflows from email drip sequences to orchestrating full multi-channel campaigns. Handle multi-day sequences, behavioral triggers, dynamic content generation, and build live analytics dashboards. | ||
|
|
||
| ## Featured examples | ||
|
|
||
| <CardGroup cols={3}> | ||
| <Card | ||
| title="Email sequences with Resend" | ||
| icon="book" | ||
| href="/guides/examples/resend-email-sequence" | ||
| > | ||
| Send multi-day email sequences with wait delays between messages. | ||
| </Card> | ||
| <Card | ||
| title="Product image generator" | ||
| icon="book" | ||
| href="/guides/example-projects/product-image-generator" | ||
| > | ||
| Transform product photos into professional marketing images using Replicate. | ||
| </Card> | ||
| <Card | ||
| title="Human-in-the-loop workflow" | ||
| icon="book" | ||
| href="/guides/example-projects/human-in-the-loop-workflow" | ||
| > | ||
| Approve marketing content using a human-in-the-loop workflow. | ||
| </Card> | ||
| </CardGroup> | ||
|
|
||
| ## Benefits of using Trigger.dev for marketing workflows | ||
|
|
||
| **Delays without idle costs:** Wait hours or weeks between steps. Waits over 5 seconds are automatically checkpointed and don't count towards compute usage. Perfect for drip campaigns and scheduled follow-ups. | ||
|
|
||
| **Guaranteed delivery:** Messages send exactly once, even after retries. Personalized content isn't regenerated on failure. | ||
|
|
||
| **Scale without limits:** Process thousands in parallel while respecting rate limits. Send to entire segments without overwhelming APIs. | ||
|
|
||
| ## Production use cases | ||
|
|
||
| <Card title="Icon customer story" href="https://trigger.dev/customers/icon-customer-story"> | ||
|
|
||
| Read how Icon uses Trigger.dev to process and generate thousands of videos per month for their AI-driven video creation platform. | ||
|
|
||
| </Card> | ||
|
|
||
| ## Example workflow patterns | ||
|
|
||
| <Tabs> | ||
| <Tab title="Drip email campaign"> | ||
| Simple drip campaign. User signs up, waits specified delay, sends personalized email, tracks engagement. | ||
|
|
||
| <div align="center"> | ||
|
|
||
| ```mermaid | ||
| graph TB | ||
| A[userCreateAccount] --> B[sendWelcomeEmail] | ||
| B --> C[wait.for 24h] | ||
| C --> D[sendProductTipsEmail] | ||
| D --> E[wait.for 7d] | ||
| E --> F[sendFeedbackEmail] | ||
|
|
||
| ``` | ||
|
|
||
| </div> | ||
| </Tab> | ||
|
|
||
| <Tab title="Multi-channel campaigns"> | ||
| **Router pattern with delay orchestration**. User action triggers campaign, router selects channel based on preferences (email/SMS/push), coordinates multi-day sequence with delays between messages, tracks engagement across channels. | ||
|
|
||
| <div align="center"> | ||
|
|
||
| ```mermaid | ||
| graph TB | ||
| A[startCampaign] --> B[fetchUserProfile] | ||
| B --> C[selectChannel] | ||
| C --> D{Preferred<br/>Channel?} | ||
|
|
||
| D -->|Email| E[sendEmail1] | ||
| D -->|SMS| F[sendSMS1] | ||
| D -->|Push| G[sendPush1] | ||
|
|
||
| E --> H[wait.for 2d] | ||
| F --> H | ||
| G --> H | ||
|
|
||
| H --> I[sendFollowUp] | ||
| I --> J[trackConversion] | ||
| ``` | ||
|
|
||
| </div> | ||
| </Tab> | ||
|
|
||
| <Tab title="AI content with approval"> | ||
| **Supervisor pattern with approval gate**. Generates AI marketing content (images, copy, assets), pauses with wait.forToken for human review, applies revisions if needed, publishes to channels after approval. | ||
|
|
||
| <div align="center"> | ||
|
|
||
| ```mermaid | ||
| graph TB | ||
| A[createCampaignAssets] --> B[generateAIContent] | ||
| B --> C[wait.forToken approval] | ||
| C --> D{Approved?} | ||
|
|
||
| D -->|Yes| E[publishToChannels] | ||
| D -->|Needs revision| F[applyFeedback] | ||
| F --> B | ||
| ``` | ||
|
|
||
| </div> | ||
| </Tab> | ||
|
|
||
| <Tab title="Survey response enrichment"> | ||
| **Coordinator pattern with enrichment**. User completes survey, batch triggers parallel enrichment from CRM/analytics, analyzes and scores responses, updates customer profiles, triggers personalized follow-up campaigns. | ||
|
|
||
| <div align="center"> | ||
|
|
||
| ```mermaid | ||
| graph TB | ||
| A[processSurveyResponse] --> B[coordinateEnrichment] | ||
| B --> C[batchTriggerAndWait] | ||
|
|
||
| C --> D[fetchCRMData] | ||
| C --> E[fetchAnalytics] | ||
| C --> F[fetchBehaviorData] | ||
|
|
||
| D --> G[analyzeAndScore] | ||
| E --> G | ||
| F --> G | ||
|
|
||
| G --> H[updateCRMProfile] | ||
| H --> I[triggerFollowUp] | ||
| ``` | ||
|
|
||
| </div> | ||
| </Tab> | ||
| </Tabs> | ||
|
|
||
| <UseCasesCards /> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add hyphen to "open source" when used as adjective modifier.
Line 54 should use "open-source" (hyphenated) when modifying "compliance platform."
Apply this diff:
📝 Committable suggestion
🧰 Tools
🪛 LanguageTool
[grammar] ~54-~54: Use a hyphen to join words.
Context: ...collection at scale, powering their open source, AI-driven compliance platform. ...
(QB_NEW_EN_HYPHEN)
🤖 Prompt for AI Agents