Skip to content

Fix expiry timeout bug in LocalIntermediateDataManager#12722

Merged
kfaraz merged 1 commit intoapache:masterfrom
kfaraz:fix_lidm_timeout
Jul 1, 2022
Merged

Fix expiry timeout bug in LocalIntermediateDataManager#12722
kfaraz merged 1 commit intoapache:masterfrom
kfaraz:fix_lidm_timeout

Conversation

@kfaraz
Copy link
Contributor

@kfaraz kfaraz commented Jun 30, 2022

Description

Fixes one of the bugs reported in #12692

As observed in this comment,
the expiry timeout is compared against the current time but the condition is reversed.
This means that as soon as a supervisor finishes, it gets cleaned up, irrespective of the specified
intermediaryPartitionTimeout period.

Changes

  • Fix the condition
  • Add tests to verify the new correct behaviour
  • Reduce the default expiry timeout from P1D to PT5M
    to retain current behaviour in case of default configs.

Upgrade effects

On upgrade,

  1. Clusters that have overridden intermediaryPartitionCleanupPeriodSec
    but not intermediaryPartitionTimeout might notice intermediate data
    from completed supervisors lingering for up to 5 more minutes.
  2. Clusters that have overridden intermediaryPartitionTimeout but not
    intermediaryPartitionCleanupPeriodSec might notice intermediate data
    lingering for up to the specified partition timeout duration (i.e. their partition
    timeout will now start getting honored).
  3. Clusters that have not overridden these configs will not notice any change.

This PR has:

  • been self-reviewed.
  • added documentation for new or modified features or behaviors.
  • added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • added or updated version, license, or notice information in licenses.yaml
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • added integration tests.
  • been tested in a test Druid cluster.

@kfaraz kfaraz merged commit f5b5cb9 into apache:master Jul 1, 2022
@kfaraz kfaraz deleted the fix_lidm_timeout branch July 1, 2022 10:59
@kfaraz
Copy link
Contributor Author

kfaraz commented Jul 1, 2022

Thanks for the review, @clintropolis !

@abhishekagarwal87 abhishekagarwal87 added this to the 24.0.0 milestone Aug 26, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants