Feat: Lightpanda extension #2192
Conversation
🦋 Changeset detectedLatest commit: 60f3357 The changes in this PR will be included in the next version bump. Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
WalkthroughThis change introduces support and documentation for the Lightpanda browser extension within the Trigger.dev ecosystem. A new build extension, ✨ Finishing Touches
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
🧹 Nitpick comments (5)
docs/config/config-file.mdx (1)
431-434: Let’s retry locating the file with a simpler command:#!/bin/bash find docs -type f -iname 'lightpanda.mdx'packages/build/src/extensions/lightpanda.ts (1)
5-9: Consider expanding configuration options.The current configuration is limited but functional. Consider future extensibility for common use cases.
Potential improvements:
- Support for additional architectures if Lightpanda expands support
- Configurable installation path for advanced users
- Support for stable releases when available (currently only nightly)
Also applies to: 11-11
docs/guides/examples/lightpanda.mdx (2)
92-94: Fix formatting and grammar.Missing comma and formatting issue in the proxy example.
-_Example using a German IP :_ +_Example using a German IP:_ -`wss://cloud.lightpanda.io/ws?proxy=datacenter&country=de&token=TOKEN``` +``` +wss://cloud.lightpanda.io/ws?proxy=datacenter&country=de&token=TOKEN +```
108-108: Fix grammar: Use "Set up" instead of "Setup"."Setup" is a noun; the verb should be "Set up".
-- Setup the [Lightpanda build extension](/config/extensions/lightpanda) +- Set up the [Lightpanda build extension](/config/extensions/lightpanda)Also applies to: 151-151
docs/introduction.mdx (1)
96-97: Keep the build-extension table alphabetically sorted for easier scanning.The newly added
lightpandarow sits betweenpuppeteerandffmpeg.
All other rows are alphabetically ordered; moving this row afterffmpegpreserves that convention and prevents merge conflicts in future additions.-| puppeteer | Use Puppeteer with Trigger.dev | [Learn more](/config/extensions/puppeteer) | -| lightpanda | Use Lightpanda Browser with Trigger.dev | [Learn more](/config/extensions/lightpanda) | -| ffmpeg | Use FFmpeg with Trigger.dev | [Learn more](/config/extensions/ffmpeg) | +| puppeteer | Use Puppeteer with Trigger.dev | [Learn more](/config/extensions/puppeteer) | +| ffmpeg | Use FFmpeg with Trigger.dev | [Learn more](/config/extensions/ffmpeg) | +| lightpanda | Use Lightpanda Browser with Trigger.dev | [Learn more](/config/extensions/lightpanda) |
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (3)
docs/images/intro-lightpanda.jpgis excluded by!**/*.jpgreferences/v3-catalog/src/trigger/lightpandaTask.tsis excluded by!references/**references/v3-catalog/trigger.config.tsis excluded by!references/**
📒 Files selected for processing (10)
apps/webapp/app/routes/_app.orgs.$organizationSlug.projects.$projectParam.env.$envParam._index/route.tsx(1 hunks)docs/config/config-file.mdx(1 hunks)docs/config/extensions/lightpanda.mdx(1 hunks)docs/config/extensions/overview.mdx(1 hunks)docs/docs.json(2 hunks)docs/guides/examples/lightpanda.mdx(1 hunks)docs/guides/introduction.mdx(1 hunks)docs/introduction.mdx(2 hunks)packages/build/package.json(3 hunks)packages/build/src/extensions/lightpanda.ts(1 hunks)
🧰 Additional context used
📓 Path-based instructions (1)
`apps/webapp/**/*`: The webapp at apps/webapp is a Remix 2.1 app that uses Node.js v20. Use zod extensively in the webapp.
apps/webapp/**/*: The webapp at apps/webapp is a Remix 2.1 app that uses Node.js v20.
Use zod extensively in the webapp.
apps/webapp/app/routes/_app.orgs.$organizationSlug.projects.$projectParam.env.$envParam._index/route.tsx
🧠 Learnings (1)
docs/introduction.mdx (1)
Learnt from: CR
PR: triggerdotdev/trigger.dev#0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-06-24T08:07:09.989Z
Learning: Build extensions in 'trigger.config.ts' allow customization of the build process, such as copying files, including packages, installing system dependencies, and enabling TypeScript features.
🪛 LanguageTool
docs/guides/examples/lightpanda.mdx
[uncategorized] ~92-~92: Possible missing comma found.
Context: ...iki/ISO_3166-1_alpha-2) country code. Example using a German IP : ```wss://cloud.li...
(AI_HYDRA_LEO_MISSING_COMMA)
[uncategorized] ~103-~103: A comma might be missing here.
Context: ... using Lightpanda Using the Lightpanda binary we will dump the HTML for a provided UR...
(AI_EN_LECTOR_MISSING_PUNCTUATION_COMMA)
[grammar] ~108-~108: The word “Setup” is a noun. The verb is spelled with a space.
Context: ...ggering the task. ### Prerequisites - Setup the [Lightpanda build extension](/confi...
(NOUN_VERB_CONFUSION)
[uncategorized] ~148-~148: Do not mix variants of the same word (‘initialise’ and ‘initialize’) within a single text.
Context: ... use a Lightpanda CDP server This task initialises a Lightpanda CDP server to allow you to...
(EN_WORD_COHERENCY)
[grammar] ~151-~151: The word “Setup” is a noun. The verb is spelled with a space.
Context: ...y via Trigger.dev. ### Prerequisites - Setup the [Lightpanda build extension](/confi...
(NOUN_VERB_CONFUSION)
[style] ~154-~154: Consider a more concise word here.
Context: ...ask will have to launch a child process in order to have the websocket available to scrape ...
(IN_ORDER_TO_PREMIUM)
🔇 Additional comments (10)
docs/config/extensions/overview.mdx (1)
55-56: LGTM – new Lightpanda row looks correctThe link path and description render correctly and follow the table’s existing style – no issues spotted.
docs/guides/introduction.mdx (2)
96-97: Confirm exported identifier casingThe table lists the extension as
lightpanda. Ensure the build package exports exactly the same lowercase identifier (e.g.export const lightpanda = …). A mismatch would break consumer imports like:import { lightpanda } from "@trigger.dev/build/extensions/core";
86-87: Check that the Lightpanda hero image is committedBroken images silently hurt the docs UX. Please confirm
public/images/intro-lightpanda.jpg(or equivalent) exists and is picked up by the docs build.#!/bin/bash # Locate the Lightpanda hero image fd --strip-cwd-prefix 'intro-lightpanda.jpg' | head -n 20apps/webapp/app/routes/_app.orgs.$organizationSlug.projects.$projectParam.env.$envParam._index/route.tsx (1)
718-719: New Lightpanda example link wired correctlyPath, description and external-link behaviour align with existing entries – looks good.
docs/docs.json (1)
81-81: LGTM! Navigation structure updated correctly.The Lightpanda extension and example entries are properly placed in the documentation navigation structure, following the existing patterns and alphabetical ordering.
Also applies to: 362-362
docs/config/extensions/lightpanda.mdx (1)
1-31: Documentation structure and content look good.The documentation follows the established pattern for extension documentation with clear configuration instructions and proper references to examples.
packages/build/package.json (1)
34-34: LGTM! Package exports configured correctly.The Lightpanda extension exports are properly configured following the same pattern as other extensions, with correct paths for both CommonJS and ESM module systems.
Also applies to: 66-68, 196-206
packages/build/src/extensions/lightpanda.ts (1)
17-19: LGTM! Extension structure and lifecycle integration.The extension properly integrates with the build lifecycle, correctly skips dev environments, and follows the established pattern for build extensions.
Also applies to: 43-55
docs/guides/examples/lightpanda.mdx (1)
1-86: Comprehensive examples with good coverage.The guide provides excellent coverage of different Lightpanda usage patterns with detailed code examples and clear documentation. The three examples effectively demonstrate the versatility of the integration.
Also applies to: 101-244
docs/introduction.mdx (1)
86-87: Double-check that the new image asset has been added and exported.
img="/images/intro-lightpanda.jpg"will break at build time if the file hasn’t been committed or the path/case differs (S3/Netlify deploys are case-sensitive).
Please verify the asset exists indocs/public/images(or the equivalent assets dir) and matches the exact filename.#!/bin/bash # Confirm the Lightpanda hero image is present fd intro-lightpanda.jpg docs | head -n 10
There was a problem hiding this comment.
Actionable comments posted: 0
♻️ Duplicate comments (6)
docs/guides/examples/lightpanda.mdx (6)
45-48: Comprehensive URL validation still missing (see prior review).
Previous feedback already requested adding avalidateUrlhelper and replacing this simple truthy check. The current code is still vulnerable to malformed / malicious input.
124-127: Comprehensive URL validation still missing (see prior review).
Same issue as in Example #1 – please reuse the sharedvalidateUrlhelper.
136-136: execSync remains injection-prone and lacks error handling.
The earlier review already suggested wrapping the call intry/catch, validating the URL, and usingJSON.stringify(or similar) to prevent shell expansion.
193-196: Comprehensive URL validation still missing (see prior review).
Identical concern as above – adopt the shared helper.
230-233: Child-process cleanup logic unchanged.
The previous review recommended a safer, two-phase termination (SIGTERM→ delay →SIGKILL) and handling in thecatchblock to avoid zombies.
240-242: Ensure process cleanup in error branch.
IfspawnLightpandasucceeds but a later step throws, the Lightpanda process is left running. Integrate the cleanup suggested in prior feedback.
🧹 Nitpick comments (2)
docs/guides/examples/lightpanda.mdx (2)
64-71: Filter‐out null / empty hrefs for cleaner results.
getAttribute('href')can returnnull; returning these values forces every consumer to re-filter. Consider:- return Array.from(document.querySelectorAll('a')).map(row => { - return row.getAttribute('href') - }) + return Array.from(document.querySelectorAll('a')) + .map(a => a.getAttribute('href')) + .filter(Boolean)
109-109: Minor wording fixes for docs.-### Prerequisites -- Setup the [Lightpanda build extension](/config/extensions/lightpanda) +### Prerequisites +- Set up the [Lightpanda build extension](/config/extensions/lightpanda)Similarly, prefer “initializes” (or “starts”) over “initialises” for consistency with US spelling used elsewhere.
Also applies to: 152-152
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
docs/guides/examples/lightpanda.mdx(1 hunks)
🧰 Additional context used
🪛 LanguageTool
docs/guides/examples/lightpanda.mdx
[uncategorized] ~93-~93: Possible missing comma found.
Context: ...iki/ISO_3166-1_alpha-2) country code. Example using a German IP : ```wss://cloud.li...
(AI_HYDRA_LEO_MISSING_COMMA)
[uncategorized] ~104-~104: A comma might be missing here.
Context: ... using Lightpanda Using the Lightpanda binary we will dump the HTML for a provided UR...
(AI_EN_LECTOR_MISSING_PUNCTUATION_COMMA)
[grammar] ~109-~109: The word “Setup” is a noun. The verb is spelled with a space.
Context: ...ggering the task. ### Prerequisites - Setup the [Lightpanda build extension](/confi...
(NOUN_VERB_CONFUSION)
[uncategorized] ~149-~149: Do not mix variants of the same word (‘initialise’ and ‘initialize’) within a single text.
Context: ... use a Lightpanda CDP server This task initialises a Lightpanda CDP server to allow you to...
(EN_WORD_COHERENCY)
[grammar] ~152-~152: The word “Setup” is a noun. The verb is spelled with a space.
Context: ...y via Trigger.dev. ### Prerequisites - Setup the [Lightpanda build extension](/confi...
(NOUN_VERB_CONFUSION)
[style] ~155-~155: Consider a more concise word here.
Context: ...ask will have to launch a child process in order to have the websocket available to scrape ...
(IN_ORDER_TO_PREMIUM)
nrigaudiere
force-pushed
the
feat/lightpanda-extension
branch
from
6ded313 to
60f3357
Compare
There was a problem hiding this comment.
Actionable comments posted: 0
♻️ Duplicate comments (5)
packages/build/src/extensions/lightpanda.ts (2)
11-57: LGTM! Security improvements have been implemented.The extension implementation looks solid with proper type definitions and build lifecycle integration. I can see the security improvements from previous reviews have been incorporated:
- Error handling with
|| (echo "Failed to download..." && exit 1)- Retry logic with
--retry 3- Binary verification with version check
The code follows good practices for build extensions and properly handles environment variable setup.
23-42: Consider implementing Docker layer optimization.Based on previous review discussions, combining all RUN statements into a single instruction would reduce Docker layers and improve build efficiency. The current approach uses multiple RUN statements which creates unnecessary layers.
docs/guides/examples/lightpanda.mdx (3)
45-48: Address URL validation security concern.The URL validation is insufficient and could lead to security issues. This was flagged in previous reviews.
Apply the comprehensive URL validation suggested in previous reviews:
const validateUrl = (url: string): void => { if (!url || typeof url !== 'string') { throw new Error('URL is required and must be a string') } try { new URL(url) } catch { throw new Error('Invalid URL format') } if (!url.startsWith('http://') && !url.startsWith('https://')) { throw new Error('URL must use http or https protocol') } }Replace the existing validation:
- if (!payload.url) { - logger.warn('Please define the payload url') - throw new Error('payload.url is undefined') - } + validateUrl(payload.url)
136-136: Fix command injection vulnerability.The execSync call is vulnerable to command injection as flagged in previous reviews.
Apply the security fix from previous reviews:
- const e = execSync(`${process.env.LIGHTPANDA_BROWSER_PATH} fetch --dump ${payload.url}`) + // Validate URL format to prevent command injection + const urlPattern = /^https?:\/\/.+/ + if (!urlPattern.test(payload.url)) { + throw new Error('Invalid URL format') + } + + try { + const result = execSync(`${process.env.LIGHTPANDA_BROWSER_PATH} fetch --dump ${JSON.stringify(payload.url)}`, { + encoding: 'utf8', + timeout: 30000 // 30 second timeout + }) + return { + message: result, + } + } catch (error) { + logger.error('Lightpanda fetch failed', { error }) + throw new Error(`Failed to fetch page: ${error}`) + }
231-233: Improve process cleanup as suggested in previous reviews.The current cleanup approach may not handle all scenarios properly.
Apply the improved cleanup logic:
// Stop Lightpanda's CDP Server - lpProcess.stdout.destroy() - lpProcess.stderr.destroy() - lpProcess.kill() + if (lpProcess && !lpProcess.killed) { + lpProcess.kill('SIGTERM') + + // Give process time to gracefully shutdown + await new Promise(resolve => setTimeout(resolve, 1000)) + + // Force kill if still running + if (!lpProcess.killed) { + lpProcess.kill('SIGKILL') + } + }Also add cleanup in the catch block:
} catch (e: any) { + // Ensure process cleanup on error + if (lpProcess && !lpProcess.killed) { + lpProcess.kill('SIGKILL') + } throw new Error(e) }
🧹 Nitpick comments (1)
docs/guides/introduction.mdx (1)
72-72: Consider adding the missing article for better readability.The static analysis tool correctly identifies a missing article. Consider this minor improvement:
-| [Lightpanda](/guides/examples/lightpanda) | Use Lightpanda browser (or cloud version) to get a webpage's content. | +| [Lightpanda](/guides/examples/lightpanda) | Use Lightpanda browser (or the cloud version) to get a webpage's content. |
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (3)
docs/images/intro-lightpanda.jpgis excluded by!**/*.jpgreferences/v3-catalog/src/trigger/lightpandaTask.tsis excluded by!references/**references/v3-catalog/trigger.config.tsis excluded by!references/**
📒 Files selected for processing (11)
.changeset/rare-mails-fail.md(1 hunks)apps/webapp/app/routes/_app.orgs.$organizationSlug.projects.$projectParam.env.$envParam._index/route.tsx(1 hunks)docs/config/config-file.mdx(1 hunks)docs/config/extensions/lightpanda.mdx(1 hunks)docs/config/extensions/overview.mdx(1 hunks)docs/docs.json(2 hunks)docs/guides/examples/lightpanda.mdx(1 hunks)docs/guides/introduction.mdx(1 hunks)docs/introduction.mdx(2 hunks)packages/build/package.json(3 hunks)packages/build/src/extensions/lightpanda.ts(1 hunks)
🚧 Files skipped from review as they are similar to previous changes (8)
- docs/config/extensions/overview.mdx
- docs/config/config-file.mdx
- apps/webapp/app/routes/_app.orgs.$organizationSlug.projects.$projectParam.env.$envParam._index/route.tsx
- docs/docs.json
- .changeset/rare-mails-fail.md
- docs/introduction.mdx
- docs/config/extensions/lightpanda.mdx
- packages/build/package.json
🧰 Additional context used
🪛 LanguageTool
docs/guides/examples/lightpanda.mdx
[uncategorized] ~93-~93: Possible missing comma found.
Context: ...iki/ISO_3166-1_alpha-2) country code. Example using a German IP : ```wss://cloud.li...
(AI_HYDRA_LEO_MISSING_COMMA)
[uncategorized] ~104-~104: A comma might be missing here.
Context: ... using Lightpanda Using the Lightpanda binary we will dump the HTML for a provided UR...
(AI_EN_LECTOR_MISSING_PUNCTUATION_COMMA)
[grammar] ~109-~109: The word “Setup” is a noun. The verb is spelled with a space.
Context: ...ggering the task. ### Prerequisites - Setup the [Lightpanda build extension](/confi...
(NOUN_VERB_CONFUSION)
[uncategorized] ~149-~149: Do not mix variants of the same word (‘initialise’ and ‘initialize’) within a single text.
Context: ... use a Lightpanda CDP server This task initialises a Lightpanda CDP server to allow you to...
(EN_WORD_COHERENCY)
[grammar] ~152-~152: The word “Setup” is a noun. The verb is spelled with a space.
Context: ...y via Trigger.dev. ### Prerequisites - Setup the [Lightpanda build extension](/confi...
(NOUN_VERB_CONFUSION)
[style] ~155-~155: Consider a more concise word here.
Context: ...ask will have to launch a child process in order to have the websocket available to scrape ...
(IN_ORDER_TO_PREMIUM)
docs/guides/introduction.mdx
[uncategorized] ~72-~72: You might be missing the article “the” here.
Context: ... | Use Lightpanda browser (or cloud version) to get a web...
(AI_EN_LECTOR_MISSING_DETERMINER_THE)
🔇 Additional comments (1)
docs/guides/examples/lightpanda.mdx (1)
1-246: Excellent comprehensive documentation, but security issues need attention.This is a very well-structured guide that provides three practical examples for different Lightpanda usage patterns. The documentation is thorough and will be valuable for users.
However, the security concerns from previous reviews still need to be addressed before this goes live.
Closes N/A
✅ Checklist
Testing
pnpm devcommand in./docsto make sure that the additions to the documentation were working and compliant with the overall look and feel of the documentation.Changelog
The main purpose of that PR is to add the Lightpanda documentation and examples. It has 3 working examples with different use cases (fetching a page, using Lightpanda's cloud and using Lightpanda browser with Puppeteer). This PR also provides the code of the extension which gets the nightly build (only version currently available) and gives users ability to run Lightpanda in their tasks.
Screenshots
💯