|
| 1 | +# Regex Replace Preprocessor |
| 2 | + |
| 3 | +The Regex Replace preprocessor performs regular expression-based find and replace operations on entry data. This is useful for sanitizing sensitive data, normalizing log formats, or transforming data before ingestion. |
| 4 | + |
| 5 | +The Regex Replace preprocessor Type is `regexreplace`. |
| 6 | + |
| 7 | +## Supported Options |
| 8 | + |
| 9 | +* `Regex` (string, required): The regular expression pattern to match against entry data. Supports standard Go regular expression syntax including named capture groups. |
| 10 | +* `Replacement` (string, required): The replacement string. Can reference capture groups using `$1`, `$2`, etc. for numbered groups or `${name}` for named groups. |
| 11 | +* `Case-Sensitive` (boolean, optional): When set to `true`, the regex matching is case-sensitive. When `false` (the default), matching is case-insensitive. |
| 12 | + |
| 13 | +## Common Use Cases |
| 14 | + |
| 15 | +The regexreplace preprocessor is commonly used for: |
| 16 | + |
| 17 | +* Sanitizing sensitive data |
| 18 | +* Normalizing log formats across different sources |
| 19 | +* Stripping or replacing unwanted characters or patterns |
| 20 | +* Transforming data for easier downstream processing |
| 21 | + |
| 22 | +### Example: Redacting Numbers |
| 23 | + |
| 24 | +To redact all numbers from log entries (e.g., for removing phone numbers, IDs, or other sensitive numeric data): |
| 25 | + |
| 26 | +``` |
| 27 | +Phone: 123-456-7890, Age: 25, ID: 987654321 |
| 28 | +``` |
| 29 | + |
| 30 | +Use the following configuration: |
| 31 | + |
| 32 | +``` |
| 33 | +[Preprocessor "redact-numbers"] |
| 34 | + Type=regexreplace |
| 35 | + Regex=`\d+` |
| 36 | + Replacement=`REDACTED` |
| 37 | +``` |
| 38 | + |
| 39 | +The result is: |
| 40 | + |
| 41 | +``` |
| 42 | +Phone: REDACTED-REDACTED-REDACTED, Age: REDACTED, ID: REDACTED |
| 43 | +``` |
| 44 | + |
| 45 | +### Example: Case-Insensitive Replacement |
| 46 | + |
| 47 | +To replace all occurrences of a word regardless of case: |
| 48 | + |
| 49 | +``` |
| 50 | +This is a TEST string with Test words |
| 51 | +``` |
| 52 | + |
| 53 | +Use the following configuration: |
| 54 | + |
| 55 | +``` |
| 56 | +[Preprocessor "normalize-test"] |
| 57 | + Type=regexreplace |
| 58 | + Regex=`test` |
| 59 | + Replacement=`example` |
| 60 | + Case-Sensitive=false |
| 61 | +``` |
| 62 | + |
| 63 | +The result is: |
| 64 | + |
| 65 | +``` |
| 66 | +This is a example string with example words |
| 67 | +``` |
| 68 | + |
| 69 | +### Example: Modifying JSON Fields |
| 70 | + |
| 71 | +Given JSON log data where you want to modify a specific field: |
| 72 | + |
| 73 | +``` |
| 74 | +{"name":"john","age":30,"city":"new york"} |
| 75 | +``` |
| 76 | + |
| 77 | +Use named capture groups to extract and modify the value: |
| 78 | + |
| 79 | +``` |
| 80 | +[Preprocessor "modify-json"] |
| 81 | + Type=regexreplace |
| 82 | + Regex=`"name":"(?P<n>[^"]*)"` |
| 83 | + Replacement=`"name":"${n}_modified"` |
| 84 | + Case-Sensitive=true |
| 85 | +``` |
| 86 | + |
| 87 | +The result is: |
| 88 | + |
| 89 | +``` |
| 90 | +{"name":"john_modified","age":30,"city":"new york"} |
| 91 | +``` |
| 92 | + |
0 commit comments