Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

coord,persist: teach coord about persisted streams that are involved in sources #9403

Closed
wants to merge 2 commits into from
Closed

coord,persist: teach coord about persisted streams that are involved in sources #9403

wants to merge 2 commits into from

Commits on Dec 2, 2021

  1. coord,persist: rename PersistDetails to TablePersistDetails 

    They are table-specific and we want to introduce a SourcePersistDetails.
    @aljoscha
    aljoscha committed Dec 2, 2021
    Configuration menu
    Copy the full SHA
    fd2b61c View commit details
    Browse the repository at this point in the history

  2. coord,persist: teach coord about persisted streams that are involved … 

    …in sources

Before, the coordinator would only know the "prefix" that was used to
form the names of persistent streams that are involved in, say, the
persistent Kafka source.

Now, we keep a mapping from a _logical name_ to the physical stream name
in the coordinator, for all persisted streams that are involved in a
source.

This also changes what persist information is sent to workers. Before,
we would send a single persist name. Now, we send a new
`SourcePersistDesc`, which contains the mapping of streams. In future
commits, we will add more information to this.

We need to do this to allow the coordinator to:
 - determine a common seal timestamp when restarting
 - negotiate/figure out if a requested as_of matches the compaction
   frontier/since of a persisted source
 - compact persisted streams when necessary

We will do these additional changes in follow-up commits.
    @aljoscha
    aljoscha committed Dec 2, 2021
    Configuration menu
    Copy the full SHA
    d3ce736 View commit details
    Browse the repository at this point in the history