Making NormalizedDateSegmentNameGenerator use joda time to keep consistency other time handling

[timsants]

Context

For a Kafka ingestion use case, the time column was configured with a time format that was working as expected. But when configuring a minion task leveraging the SegmentProcessorFramework, a date format exception was thrown with the same exact time column/format:

Caught exception while parsing simple date format: yyyy-MM-dd'T'HH:mm:ss.SSSSSSSSSZ with value: 2023-03-07T21:18:05.072784135Z at org.apache.pinot.segment.spi.creator.name.NormalizedDateSegmentNameGenerator.getNormalizedDate(NormalizedDateSegmentNameGenerator.java:142)

Specifically the realtime ingestion was using SegmentColumnarIndexCreator, which uses joda time DateFormatter for parsing time format into epoch millis (See SegmentColumnarIndexCreator).

But the minion task was configured to use the NormalizedDateSegmentNameGenerator which failed since it uses another date time format library, java.text SimpleDateFormat. This library seems to be stricter that the joda time library and if a time column values has literal ‘Z’ in it, the time format must have the Z in single quotes. E.g. "2023-01-01T12:00:00.111111111Z" needs a time format of "yyyy-MM-dd'T'HH:mm:ss.SSSSSSSSS’Z’" otherwise it will fail.

Changes

This PR makes the NormalizedDateSegmentNameGenerator also use the joda date time formatter. Joda is no longer maintained and java.text seems to be outdated. This PR aims to unify the time library until we are ready to make the switch to use the newer java.time library.

Instructions:

1. The PR has to be tagged with at least one of the following labels (*):
   1. feature
   2. bugfix
   3. performance
   4. ui
   5. backward-incompat
   6. release-notes (**)
2. Remove these instructions before publishing the PR.

(*) Other labels to consider:

• testing
• dependencies
• docker
• kubernetes
• observability
• security
• code-style
• extension-point
• refactor
• cleanup

(**) Use release-notes label for scenarios like:

• New configuration options
• Deprecation of configurations
• Signature changes to public methods/interfaces
• New plugins added or old plugins removed

[codecov-commenter]

Codecov Report

Merging #10721 (37aef31) into master (ab4e8fa) will increase coverage by 1.79%.
The diff coverage is 78.57%.

@@             Coverage Diff              @@
##             master   #10721      +/-   ##
============================================
+ Coverage     68.41%   70.20%   +1.79%     
- Complexity     6518     6529      +11     
============================================
  Files          2164     2164              
  Lines        116410   116412       +2     
  Branches      17609    17609              
============================================
+ Hits          79637    81725    +2088     
+ Misses        31136    28990    -2146     
- Partials       5637     5697      +60     

Flag	Coverage Δ
integration1	24.03% <0.00%> (-0.02%)	⬇️
integration2	23.55% <0.00%> (?)
unittests1	67.69% <78.57%> (+<0.01%)	⬆️
unittests2	13.65% <0.00%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...eator/name/NormalizedDateSegmentNameGenerator.java	83.60% <76.92%> (+0.27%)	⬆️
.../org/apache/pinot/spi/data/DateTimeFormatSpec.java	83.06% <100.00%> (+0.13%)	⬆️

... and 171 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[snleee]

Thank you for quickly addressing this!

Can you add the testing case where the format works in joda while it's not working in java to cover this case?

[timsants]

Thank you for quickly addressing this!

Can you add the testing case where the format works in joda while it's not working in java to cover this case?

Will do. Also I checked out the other references and it doesn't seem like they will be impacted by this case since they are formatting a constant string.

[snleee]

LGTM

Can you double check on the tests?