Skip to content

add utils regexes list #16745

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
May 21, 2025
Merged

add utils regexes list #16745

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
877ee2a
add utils regexes list
andrewjschuang May 21, 2025
4c389cc
bump package.json
andrewjschuang May 21, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
80 changes: 80 additions & 0 deletions ...utils/actions/extract-by-regular-expressions-list/extract-by-regular-expressions-list.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
import buildRegExp from "../../common/text/buildRegExp.mjs";
import pipedream_utils from "../../pipedream_utils.app.mjs";
export default {
name: "Formatting - [Text] Extract by Regular Expressions List (Regex)",
description: "Find matches for regular expressions. Returns all matched groups with start and end position.",
key: "pipedream_utils-extract-by-regular-expressions-list",
version: "0.0.1",
type: "action",
props: {
pipedream_utils,
input: {
type: "string",
label: "Input",
description: "Text you would like to find a pattern from",
},
regExpStrings: {
type: "string[]",
label: "Regular Expressions",
description: "An array of [regex strings](https://www.w3schools.com/js/js_regexp.asp) (e.g. `/foo/g`, `/bar/i`)",
},
},
methods: {
getAllResults(input) {
const resultMap = {};
const resultList = [];

for (const rStr of this.regExpStrings) {
if (typeof rStr !== "string" || !rStr.length) {
// still push an empty array to preserve order
resultMap[rStr] = [];
resultList.push([]);
continue;
}

const re = rStr.startsWith("/")
? buildRegExp(rStr, [
"g",
])
: new RegExp(rStr, "g");
Comment on lines +35 to +39
Copy link
Contributor

@coderabbitai coderabbitai bot May 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Consider adding RegExp validation and error handling.

The code handles different regex formats but doesn't protect against invalid regex patterns that might throw exceptions.

 const re = rStr.startsWith("/")
   ? buildRegExp(rStr, [
     "g",
   ])
   : new RegExp(rStr, "g");
+ 
+ // Add try/catch to handle invalid regex patterns

Consider wrapping the RegExp creation in a try/catch block to gracefully handle syntax errors in user-provided regex patterns:

try {
  const re = rStr.startsWith("/")
    ? buildRegExp(rStr, ["g"])
    : new RegExp(rStr, "g");
  
  // Process matches...
} catch (error) {
  console.log(`Invalid regex pattern: ${rStr}. Error: ${error.message}`);
  resultMap[rStr] = [];
  resultList.push([]);
  continue;
}
🤖 Prompt for AI Agents 
In
components/pipedream_utils/actions/extract-by-regular-expressions-list/extract-by-regular-expressions-list.mjs
around lines 35 to 39, the RegExp creation does not handle invalid regex
patterns which can throw exceptions. Wrap the RegExp creation code in a
try/catch block to catch syntax errors from user-provided regex strings. In the
catch block, log the error message indicating the invalid pattern, assign an
empty array to the corresponding resultMap entry, push an empty array to
resultList, and continue processing the next regex pattern.


const matches = [
...input.matchAll(re),
].map((m) => ({
match: m[0],
groups: m.groups ?? {},
startPosition: m.index,
endPosition: m.index + m[0].length,
}));

resultMap[rStr] = matches;
resultList.push(matches);
}

return {
resultMap,
resultList,
};
},
},
async run({ $ }) {
const {
resultMap,
resultList,
} = this.getAllResults(this.input);

const totalMatches = resultList.reduce((sum, arr) => sum + arr.length, 0);

$.export(
"$summary",
totalMatches
? `Found ${totalMatches} matches across ${Object.keys(resultMap).length} patterns`
: "No matches found",
);

return {
map: resultMap,
list: resultList,
};
},
};
Comment on lines +1 to +80
Copy link
Contributor

@coderabbitai coderabbitai bot May 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Verification agent

🧩 Analysis chain

Consider adding protection against ReDoS attacks.

Regular expressions provided by users could potentially cause Regular Expression Denial of Service (ReDoS) attacks if they contain patterns with catastrophic backtracking.

Consider implementing:

  1. A timeout mechanism for regex execution
  2. Complexity analysis of user-provided regexes
  3. A maximum limit on input length for large regex lists

🌐 Web query:

What are best practices to prevent ReDoS (Regular Expression Denial of Service) attacks in JavaScript?

💡 Result:

Best Practices to Prevent ReDoS Attacks in JavaScript

Regular Expression Denial of Service (ReDoS) vulnerabilities occur when poorly structured regular expressions allow attackers to supply input that causes excessive backtracking. This can hang or crash JavaScript applications, especially in single-threaded Node.js environments[3][4].

Below are the best practices to prevent and mitigate ReDoS attacks in JavaScript:

1. Avoid Vulnerable Patterns

  • Do not nest quantifiers: Patterns like (a+)+, (a|a)+, and similar possess exponential worst-case behavior and are the most notorious for catastrophic backtracking[3].
  • Avoid quantified overlapping disjunctions: For example, (a|b|ab)+ can be problematic when input allows ambiguous matches[3].
  • Be wary of adjacent quantified patterns: Patterns like (\d+\d+) can also cause issues[3].

2. Use Bounded Quantifiers and Possessive Patterns

  • Bound repetitions: Always prefer quantifiers like {min,max} over unlimited + or *. For example, /a{1,5}/ is safer than /a+/[7].
  • Use possessive quantifiers or atomic groups: In regex engines that support them, these prevent backtracking within certain subpatterns. While JavaScript’s regex does not natively support possessive quantifiers, structuring patterns to avoid ambiguous matches helps[7].

3. Validate and Sanitize Input Used in Regex Construction

  • Never include untrusted user input directly in a regex pattern. If you must, always escape special characters using libraries like lodash.escaperegexp[5].
  • Sanitize and filter user input: Remove or encode regex metacharacters to prevent injection attacks[5].

4. Limit Input Length

  • Restrict the length of user-provided data processed by regexes. This lessens the potential impact of any slow pattern match[5].

5. Use Proven and Reviewed Patterns

  • Rely on regexes from trusted sources: Use vetted patterns, such as from the [OWASP Validation Regex Repository][3].
  • Regularly audit third-party dependencies: Many open source packages include regexes that might be exploitable[4].

6. Set Timeouts or Monitor Execution

  • Implement regular expression timeouts: In environments that support it, abort regex matches that take too long[6]. While JavaScript engines often lack built-in timeout controls for regex, you can run regex operations in Web Workers or separate processes, kill them after a threshold, or use libraries that provide safer execution environments.

7. Test and Visualize Regex Performance

  • Bench test critical regexes: Use fuzzers or input generators to test regexes with increasingly long inputs to detect performance cliffs[8].
  • Visualize or lint regexes: Use regex debuggers or linters to spot problematic patterns.

8. Educate Developers and Review Regular Expressions

  • Train teams to recognize “evil regex” patterns.
  • Code review should include scrutiny of regexes for both correctness and performance concerns.

Summary Table: Common ReDoS Mitigation Strategies

Strategy Description
Avoid nested/ambiguous quantifiers Prevents exponential backtracking
Bound repetitions Use {min,max} quantifiers
Escape user input in regexes Prevents injection attacks
Limit input length Reduces surface for DoS
Use vetted regex patterns Reduces likelihood of vulnerabilities
Set regex execution timeouts Mitigates impact if match runs too long
Audit and test regexes Ensures performance and avoids vulnerable patterns

Key Takeaway:
Avoid complex, nested, or ambiguous regex patterns, escape and sanitize all user input used in regex construction, limit the size of input, and prefer reviewed, bounded, and safe regexes. Regularly test and audit regex use—and remember that in JavaScript’s single-threaded environment, a problematic regex can block your entire app[3][4][5][7].

Citations:

I’d like to inspect how buildRegExp constructs regexes to confirm if it includes any sanitization or limits:

🏁 Script executed:

#!/bin/bash
# Locate and display the buildRegExp implementation
fd buildRegExp.mjs

Length of output: 75

🏁 Script executed:

#!/bin/bash
# Display the contents of buildRegExp to check for sanitization or safety mechanisms
sed -n '1,200p' components/pipedream_utils/common/text/buildRegExp.mjs

Length of output: 522

Mitigate ReDoS Risks in User-Supplied Regexes

The current implementation in
components/pipedream_utils/actions/extract-by-regular-expressions-list/extract-by-regular-expressions-list.mjs
and its use of buildRegExp allows arbitrary, unchecked patterns to be executed via String.prototype.matchAll(). In a single-threaded environment, a malicious or pathological regex (e.g. nested quantifiers) can trigger catastrophic backtracking and block the event loop.

Suggested fixes:

  • Enforce a maximum input length (e.g. 10 KB) in run() or getAllResults() before any matching.
  • Pre-validate each regExpStrings entry against a simple complexity check (reject patterns with nested quantifiers like (a+)+).
  • Offload heavy regex matching into a worker or separate process with a timeout/abort controller to kill long-running matches.
  • (Optional) Escape or whitelist safe patterns when possible, using a library like lodash.escapeRegExp for literal inputs.

File locations to update:

  • components/pipedream_utils/actions/extract-by-regular-expressions-list/extract-by-regular-expressions-list.mjs
  • components/pipedream_utils/common/text/buildRegExp.mjs
🤖 Prompt for AI Agents 
In
components/pipedream_utils/actions/extract-by-regular-expressions-list/extract-by-regular-expressions-list.mjs
lines 1 to 80, the current code executes user-provided regexes without
safeguards, risking ReDoS attacks. To fix this, add a maximum input length check
(e.g., 10 KB) in the run() or getAllResults() method to reject overly large
inputs. Implement a pre-validation step on each regex string to detect and
reject patterns with nested quantifiers or other high-complexity constructs.
Consider offloading regex matching to a worker or separate process with a
timeout to abort long-running matches. Optionally, sanitize or escape regex
strings when possible to avoid injection of unsafe patterns.

2 changes: 1 addition & 1 deletion components/pipedream_utils/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@pipedream/pipedream_utils",
"version": "0.0.3",
"version": "0.0.4",
"description": "Pipedream Utils Components",
"main": "pipedream_utils.app.mjs",
"keywords": [
Expand Down
Loading