add utils regexes list #16745
add utils regexes list #16745
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎ 3 Skipped Deployments
|
|
Caution Review failedThe pull request is closed. WalkthroughA new Pipedream action module was introduced to extract matches from a text input using a list of regular expressions. The module defines input properties, processes each regex, collects match details, and returns results in both map and list formats, including match text, groups, and positions. Additionally, the package version was incremented. Changes
Sequence Diagram(s)sequenceDiagram
participant User
participant PipedreamAction
User->>PipedreamAction: Provide input text and list of regex strings
PipedreamAction->>PipedreamAction: For each regex, compile and match against input
PipedreamAction->>PipedreamAction: Collect matches (text, groups, positions)
PipedreamAction->>User: Return results as map and list with summary
Poem
Note ⚡️ AI Code Reviews for VS Code, Cursor, WindsurfCodeRabbit now has a plugin for VS Code, Cursor and Windsurf. This brings AI code reviews directly in the code editor. Each commit is reviewed immediately, finding bugs before the PR is raised. Seamless context handoff to your AI code agent ensures that you can easily incorporate review feedback. Note ⚡️ Faster reviews with cachingCodeRabbit now supports caching for code and dependencies, helping speed up reviews. This means quicker feedback, reduced wait times, and a smoother review experience overall. Cached data is encrypted and stored securely. This feature will be automatically enabled for all accounts on May 30th. To opt out, configure 📜 Recent review detailsConfiguration used: CodeRabbit UI 📒 Files selected for processing (1)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
components/pipedream_utils/actions/extract-by-regular-expressions-list/extract-by-regular-expressions-list.mjs (1)
75-79: Consider adding match examples in the returned output.For improved user experience, consider including a few example matches in the summary or adding a separate field with sample matches to help users verify their regex patterns are working as expected.
return { map: resultMap, list: resultList, + examples: totalMatches ? getSampleMatches(resultList, 3) : [] };You could implement a helper method like:
getSampleMatches(resultList, count) { const samples = []; for (const matches of resultList) { if (matches.length > 0) { samples.push(matches[0].match); if (samples.length >= count) break; } } return samples; }
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
Cache: Disabled due to data retention organization setting
Knowledge Base: Disabled due to data retention organization setting
📒 Files selected for processing (1)
components/pipedream_utils/actions/extract-by-regular-expressions-list/extract-by-regular-expressions-list.mjs(1 hunks)
⏰ Context from checks skipped due to timeout of 90000ms (4)
- GitHub Check: Publish TypeScript components
- GitHub Check: pnpm publish
- GitHub Check: Lint Code Base
- GitHub Check: Verify TypeScript components
🔇 Additional comments (5)
components/pipedream_utils/actions/extract-by-regular-expressions-list/extract-by-regular-expressions-list.mjs (5)
1-8: Component metadata and imports look appropriate.The action is well-named and descriptive, following Pipedream's naming conventions. Version 0.0.1 is correct for a new component.
9-21: Props definitions are comprehensive.The prop definitions provide helpful descriptions and appropriate types. The regex description includes a link to documentation which is a nice touch for users.
27-33: Good handling of empty or invalid regex strings.The code correctly handles empty regex strings by pushing an empty array to preserve the order in the results list, which is important for consistency.
41-48: Thorough match extraction with detailed information.The code efficiently uses
matchAll()and effectively extracts all relevant information including the matched text, capture groups, and positions. The null coalescing operator for groups is a nice touch for handling cases where no named groups exist.
60-74: Robust output summary implementation.The summary export provides useful information about the number of matches found, with appropriate handling for both match and no-match scenarios.
| const re = rStr.startsWith("/") | ||
| ? buildRegExp(rStr, [ | ||
| "g", | ||
| ]) | ||
| : new RegExp(rStr, "g"); |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Consider adding RegExp validation and error handling.
The code handles different regex formats but doesn't protect against invalid regex patterns that might throw exceptions.
const re = rStr.startsWith("/")
? buildRegExp(rStr, [
"g",
])
: new RegExp(rStr, "g");
+
+ // Add try/catch to handle invalid regex patternsConsider wrapping the RegExp creation in a try/catch block to gracefully handle syntax errors in user-provided regex patterns:
try {
const re = rStr.startsWith("/")
? buildRegExp(rStr, ["g"])
: new RegExp(rStr, "g");
// Process matches...
} catch (error) {
console.log(`Invalid regex pattern: ${rStr}. Error: ${error.message}`);
resultMap[rStr] = [];
resultList.push([]);
continue;
}🤖 Prompt for AI Agents
In
components/pipedream_utils/actions/extract-by-regular-expressions-list/extract-by-regular-expressions-list.mjs
around lines 35 to 39, the RegExp creation does not handle invalid regex
patterns which can throw exceptions. Wrap the RegExp creation code in a
try/catch block to catch syntax errors from user-provided regex strings. In the
catch block, log the error message indicating the invalid pattern, assign an
empty array to the corresponding resultMap entry, push an empty array to
resultList, and continue processing the next regex pattern.
| import buildRegExp from "../../common/text/buildRegExp.mjs"; | ||
| import pipedream_utils from "../../pipedream_utils.app.mjs"; | ||
| export default { | ||
| name: "Formatting - [Text] Extract by Regular Expressions List (Regex)", | ||
| description: "Find matches for regular expressions. Returns all matched groups with start and end position.", | ||
| key: "pipedream_utils-extract-by-regular-expressions-list", | ||
| version: "0.0.1", | ||
| type: "action", | ||
| props: { | ||
| pipedream_utils, | ||
| input: { | ||
| type: "string", | ||
| label: "Input", | ||
| description: "Text you would like to find a pattern from", | ||
| }, | ||
| regExpStrings: { | ||
| type: "string[]", | ||
| label: "Regular Expressions", | ||
| description: "An array of [regex strings](https://www.w3schools.com/js/js_regexp.asp) (e.g. `/foo/g`, `/bar/i`)", | ||
| }, | ||
| }, | ||
| methods: { | ||
| getAllResults(input) { | ||
| const resultMap = {}; | ||
| const resultList = []; | ||
|
|
||
| for (const rStr of this.regExpStrings) { | ||
| if (typeof rStr !== "string" || !rStr.length) { | ||
| // still push an empty array to preserve order | ||
| resultMap[rStr] = []; | ||
| resultList.push([]); | ||
| continue; | ||
| } | ||
|
|
||
| const re = rStr.startsWith("/") | ||
| ? buildRegExp(rStr, [ | ||
| "g", | ||
| ]) | ||
| : new RegExp(rStr, "g"); | ||
|
|
||
| const matches = [ | ||
| ...input.matchAll(re), | ||
| ].map((m) => ({ | ||
| match: m[0], | ||
| groups: m.groups ?? {}, | ||
| startPosition: m.index, | ||
| endPosition: m.index + m[0].length, | ||
| })); | ||
|
|
||
| resultMap[rStr] = matches; | ||
| resultList.push(matches); | ||
| } | ||
|
|
||
| return { | ||
| resultMap, | ||
| resultList, | ||
| }; | ||
| }, | ||
| }, | ||
| async run({ $ }) { | ||
| const { | ||
| resultMap, | ||
| resultList, | ||
| } = this.getAllResults(this.input); | ||
|
|
||
| const totalMatches = resultList.reduce((sum, arr) => sum + arr.length, 0); | ||
|
|
||
| $.export( | ||
| "$summary", | ||
| totalMatches | ||
| ? `Found ${totalMatches} matches across ${Object.keys(resultMap).length} patterns` | ||
| : "No matches found", | ||
| ); | ||
|
|
||
| return { | ||
| map: resultMap, | ||
| list: resultList, | ||
| }; | ||
| }, | ||
| }; |
There was a problem hiding this comment.
💡 Verification agent
🧩 Analysis chain
Consider adding protection against ReDoS attacks.
Regular expressions provided by users could potentially cause Regular Expression Denial of Service (ReDoS) attacks if they contain patterns with catastrophic backtracking.
Consider implementing:
- A timeout mechanism for regex execution
- Complexity analysis of user-provided regexes
- A maximum limit on input length for large regex lists
🌐 Web query:
What are best practices to prevent ReDoS (Regular Expression Denial of Service) attacks in JavaScript?
💡 Result:
Best Practices to Prevent ReDoS Attacks in JavaScript
Regular Expression Denial of Service (ReDoS) vulnerabilities occur when poorly structured regular expressions allow attackers to supply input that causes excessive backtracking. This can hang or crash JavaScript applications, especially in single-threaded Node.js environments[3][4].
Below are the best practices to prevent and mitigate ReDoS attacks in JavaScript:
1. Avoid Vulnerable Patterns
- Do not nest quantifiers: Patterns like
(a+)+,(a|a)+, and similar possess exponential worst-case behavior and are the most notorious for catastrophic backtracking[3]. - Avoid quantified overlapping disjunctions: For example,
(a|b|ab)+can be problematic when input allows ambiguous matches[3]. - Be wary of adjacent quantified patterns: Patterns like
(\d+\d+)can also cause issues[3].
2. Use Bounded Quantifiers and Possessive Patterns
- Bound repetitions: Always prefer quantifiers like
{min,max}over unlimited+or*. For example,/a{1,5}/is safer than/a+/[7]. - Use possessive quantifiers or atomic groups: In regex engines that support them, these prevent backtracking within certain subpatterns. While JavaScript’s regex does not natively support possessive quantifiers, structuring patterns to avoid ambiguous matches helps[7].
3. Validate and Sanitize Input Used in Regex Construction
- Never include untrusted user input directly in a regex pattern. If you must, always escape special characters using libraries like
lodash.escaperegexp[5]. - Sanitize and filter user input: Remove or encode regex metacharacters to prevent injection attacks[5].
4. Limit Input Length
- Restrict the length of user-provided data processed by regexes. This lessens the potential impact of any slow pattern match[5].
5. Use Proven and Reviewed Patterns
- Rely on regexes from trusted sources: Use vetted patterns, such as from the [OWASP Validation Regex Repository][3].
- Regularly audit third-party dependencies: Many open source packages include regexes that might be exploitable[4].
6. Set Timeouts or Monitor Execution
- Implement regular expression timeouts: In environments that support it, abort regex matches that take too long[6]. While JavaScript engines often lack built-in timeout controls for regex, you can run regex operations in Web Workers or separate processes, kill them after a threshold, or use libraries that provide safer execution environments.
7. Test and Visualize Regex Performance
- Bench test critical regexes: Use fuzzers or input generators to test regexes with increasingly long inputs to detect performance cliffs[8].
- Visualize or lint regexes: Use regex debuggers or linters to spot problematic patterns.
8. Educate Developers and Review Regular Expressions
- Train teams to recognize “evil regex” patterns.
- Code review should include scrutiny of regexes for both correctness and performance concerns.
Summary Table: Common ReDoS Mitigation Strategies
| Strategy | Description |
|---|---|
| Avoid nested/ambiguous quantifiers | Prevents exponential backtracking |
| Bound repetitions | Use {min,max} quantifiers |
| Escape user input in regexes | Prevents injection attacks |
| Limit input length | Reduces surface for DoS |
| Use vetted regex patterns | Reduces likelihood of vulnerabilities |
| Set regex execution timeouts | Mitigates impact if match runs too long |
| Audit and test regexes | Ensures performance and avoids vulnerable patterns |
Key Takeaway:
Avoid complex, nested, or ambiguous regex patterns, escape and sanitize all user input used in regex construction, limit the size of input, and prefer reviewed, bounded, and safe regexes. Regularly test and audit regex use—and remember that in JavaScript’s single-threaded environment, a problematic regex can block your entire app[3][4][5][7].
Citations:
- 1: https://owasp.org/www-community/attacks/Regular_expression_Denial_of_Service_-_ReDoS
- 2: https://blog.logrocket.com/protect-against-regex-denial-of-service-redos-attacks/
- 3: https://learn.snyk.io/lesson/redos/
- 4: https://www.sonarsource.com/blog/vulnerable-regular-expressions-javascript/
- 5: https://docs.guardrails.io/docs/vulnerabilities/javascript/insecure_use_of_regular_expressions
- 6: https://www.regular-expressions.info/redos.html
- 7: https://plainenglish.io/blog/preventing-regular-expression-backtracking-in-javascript
- 8: https://learn.microsoft.com/en-us/archive/msdn-magazine/2010/may/security-briefs-regular-expression-denial-of-service-attacks-and-defenses
- 9: https://blog.superhuman.com/how-to-eliminate-regular-expression-denial-of-service/
- 10: https://www.imperva.com/learn/ddos/regular-expression-denial-of-service-redos/
I’d like to inspect how buildRegExp constructs regexes to confirm if it includes any sanitization or limits:
🏁 Script executed:
#!/bin/bash
# Locate and display the buildRegExp implementation
fd buildRegExp.mjsLength of output: 75
🏁 Script executed:
#!/bin/bash
# Display the contents of buildRegExp to check for sanitization or safety mechanisms
sed -n '1,200p' components/pipedream_utils/common/text/buildRegExp.mjsLength of output: 522
Mitigate ReDoS Risks in User-Supplied Regexes
The current implementation in
components/pipedream_utils/actions/extract-by-regular-expressions-list/extract-by-regular-expressions-list.mjs
and its use of buildRegExp allows arbitrary, unchecked patterns to be executed via String.prototype.matchAll(). In a single-threaded environment, a malicious or pathological regex (e.g. nested quantifiers) can trigger catastrophic backtracking and block the event loop.
Suggested fixes:
- Enforce a maximum input length (e.g. 10 KB) in
run()orgetAllResults()before any matching. - Pre-validate each
regExpStringsentry against a simple complexity check (reject patterns with nested quantifiers like(a+)+). - Offload heavy regex matching into a worker or separate process with a timeout/abort controller to kill long-running matches.
- (Optional) Escape or whitelist safe patterns when possible, using a library like
lodash.escapeRegExpfor literal inputs.
File locations to update:
- components/pipedream_utils/actions/extract-by-regular-expressions-list/extract-by-regular-expressions-list.mjs
- components/pipedream_utils/common/text/buildRegExp.mjs
🤖 Prompt for AI Agents
In
components/pipedream_utils/actions/extract-by-regular-expressions-list/extract-by-regular-expressions-list.mjs
lines 1 to 80, the current code executes user-provided regexes without
safeguards, risking ReDoS attacks. To fix this, add a maximum input length check
(e.g., 10 KB) in the run() or getAllResults() method to reject overly large
inputs. Implement a pre-validation step on each regex string to detect and
reject patterns with nested quantifiers or other high-complexity constructs.
Consider offloading regex matching to a worker or separate process with a
timeout to abort long-running matches. Optionally, sanitize or escape regex
strings when possible to avoid injection of unsafe patterns.
WHY
Summary by CodeRabbit