Skip to content

2.3.0
Compare
Choose a tag to compare
@asifsmohammed asifsmohammed released this 06 Jun 21:41
· 764 commits to main since this release
2.3.0
ce1017e
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.

2023-06-06 Version 2.3.0

Features

  • Supports S3 as a sink. (#1048, #2732)
  • Tagging events. (#629)
  • Adds a function for getting the length of a string using Data Prepper Expressions. (#2639)
  • Adds a function to determine if an IP address is in a CIDR block. (#2625)
  • Support for expressions to extract string or numerical values in Data Prepper Expressions. (#2686, #2685)
  • Read Parquet files using a codec for the S3 source and other codec-compatible sources. (#2445)
  • Read Avro files using a codec for the S3 source and other codec-compatible sources. (#2527)
  • Adds an obfuscate processor to help obfuscate data. (#1952)
  • Support parsing User-Agent headers with a user_agent processor. (#2618)
  • Support expressions in add_entries processor. (#2672)
  • Support adding event data from event metadata in add_entries processor. (#2687)
  • Adds a tail sampling processor. (#2572)
  • Adds when conditions for many difference processors. (#2613)

Enhancements

  • Support OpenSearch composable index templates. (#1275)
  • Adds input codec plugin concept to Data Prepper. (#1532)
  • Improves error messages on authorization failures with OpenSearch clusters. (#2657, #2655)
  • Clearer metrics when S3 source does not detect events in S3 objects. (#2699)
  • Allow override the Log4j configuration file when running Data Prepper. (#2720)
  • Pipeline creation succeeds even when sinks are not ready. (#2656)
  • Support compression in OTel sources. (#1176)
  • Support gzip compression in HTTP source. (#1399)
  • Support functions in Data Prepper Expression language. (#2626)
  • Support SNS to SQS messages in S3 source. (#2604)
  • Support shared AWS credentials across pipeline components. (#2751, #2764)
  • Initial extension plugin support. (#2636, #2637)

Bug Fixes

  • Improves how the opensearch sink retries requests to OpenSearch with a maximum wait time. (#2641)
  • Improves how the opensearch sink initializes by continuing to retry failure requests. (#2769)
  • Fixes an exception when writing to OpenSearch. (#2654)
  • Improves error logs for document failures in opensearch sink. (#2597)
  • Fixes issues with S3 key prefixes in the opensearch S3 DLQ. (#2581)
  • Updates Random Cut Forest dependency to reduce noise. (#2783)
  • Fixes to writing to the Data Prepper buffer in the S3 source. (#2701)
  • Fixes S3 source backoff which was too aggressive in the requests it made. (#2568)

Security

Maintenance

  • DataPrepper ExpressionEvaluator should not be type specific. (#2703)
  • Change JsonStringBuilder in JacksonEvent to be non static for ease-of-use. (#2665)
Assets 2
0 Join discussion