Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

KAFKA-15905 Restarts of MirrorCheckpointTask should not permanently i… #15910

Merged
merged 15 commits into from
May 23, 2024

Commits on May 9, 2024

  1. KAFKA-15905 Restarts of MirrorCheckpointTask should not permanently i…

    …nterrupt offset translation
    
    MirrorCheckpointTask reloads the last checkpoint at start,
    OffsetSyncStore stores OffsetSyncs before reading till end.
    
    Add test case simulating restarted task where the store is reinitialized
    with later OffsetSyncs and check that emitted Checkpoint do not rewind.
    
    Also addresses KAFKA-16622 Mirromaker2 first Checkpoint not emitted until
    consumer group fully catches up once because the OffsetSyncStore store
    is populated before reading to log end.
    
    Co-Authored-By: Adrian Preston <prestona@uk.ibm.com>
    edoardocomar and prestona committed May 9, 2024
    Configuration menu
    Copy the full SHA
    66031ee View commit details
    Browse the repository at this point in the history

Commits on May 10, 2024

  1. MirrorCheckpointTask report errors when loading checkpoints

    Co-Authored-By: Adrioan Preston <prestona@uk.ibm.com>
    edoardocomar and prestona committed May 10, 2024
    Configuration menu
    Copy the full SHA
    4ad350f View commit details
    Browse the repository at this point in the history

Commits on May 14, 2024

  1. If CheckpointTask cannot read checkpoints at startup,

    use previous OffsetSyncStore "pessimistic" load logic
    
    Co-Authored-By: Adrian Preston <prestona@uk.ibm.com>
    edoardocomar and prestona committed May 14, 2024
    Configuration menu
    Copy the full SHA
    64a60b4 View commit details
    Browse the repository at this point in the history

Commits on May 15, 2024

  1. Configuration menu
    Copy the full SHA
    e33edd2 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    2146ed0 View commit details
    Browse the repository at this point in the history

Commits on May 16, 2024

  1. Load checkpoints asynchronously.

    Use a wrapper to the checkpointsPerGroup map to ensure thread safety on restart.
    
    The callback rethrows the error as a way to stop the KafkaBasedLog looping forever in readtoEnd.
    This occurs when unauthorized to READ from the topic, as KafkaBasedLog retries indefinitely.
    Unauthorized to DESCRIBE instead fails immediately before fetching.
    
    All such failures result in a warning log message about degraded offset translation.
    edoardocomar committed May 16, 2024
    Configuration menu
    Copy the full SHA
    7115d89 View commit details
    Browse the repository at this point in the history

Commits on May 17, 2024

  1. Configuration menu
    Copy the full SHA
    cc27e4c View commit details
    Browse the repository at this point in the history
  2. one more unit test

    edoardocomar committed May 17, 2024
    Configuration menu
    Copy the full SHA
    e415e67 View commit details
    Browse the repository at this point in the history

Commits on May 20, 2024

  1. Renamed CheckpointsStore to CheckpointStore

    for consistency with OffsetSyncStore
    
    CheckpointStore started debug/trace message
    edoardocomar committed May 20, 2024
    Configuration menu
    Copy the full SHA
    56db4ed View commit details
    Browse the repository at this point in the history

Commits on May 21, 2024

  1. Configuration menu
    Copy the full SHA
    36b7bb1 View commit details
    Browse the repository at this point in the history
  2. avoid warning

    edoardocomar committed May 21, 2024
    Configuration menu
    Copy the full SHA
    dfeca99 View commit details
    Browse the repository at this point in the history
  3. avoid warning

    edoardocomar committed May 21, 2024
    Configuration menu
    Copy the full SHA
    2fdc9f9 View commit details
    Browse the repository at this point in the history
  4. avoid warning

    edoardocomar committed May 21, 2024
    Configuration menu
    Copy the full SHA
    9896653 View commit details
    Browse the repository at this point in the history

Commits on May 22, 2024

  1. debug log message oops

    edoardocomar committed May 22, 2024
    Configuration menu
    Copy the full SHA
    1d0363b View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    b6eeaba View commit details
    Browse the repository at this point in the history