Skip to content
This repository was archived by the owner on Mar 12, 2026. It is now read-only.
This repository was archived by the owner on Mar 12, 2026. It is now read-only.

Tracking issue about todo in distributed wal #366

@Rachelint

Description

@Rachelint

Description
There are some enhancements in wal needing to do in the futures, just mark them now.

Proposal

On message

  • Now manifest based on wal need to scan twice from storage while doing snapshot. Add a cache for this may be better.
  • In wal's message queue implementation, we now sync the snapshot to remote in every flushing. Indeed, we should just need to do the sync work while flushing after dropping table.
  • Now, the desigin of wal on message queue is based on the thread model of one writer for one table, maybe we should remove this assumption.
  • Define some iterators with more explicit semantics.
  • Now we init region(heavy work with network IO) even if it is just created, we should change MessageQueue to know whether it is a new region.
  • We should disable the ttl of topic in kafka, but it is not supported in rskafka now. It is important so I plan to use another crate to finish it temporarily.

On OBKV

  • Rename the table name prefix.
  • Change the meta table's sharding key to table id.

Manifest(on wal)

  • Use the table id as region id.

Additional context

Metadata

Metadata

Assignees

No one assigned

    Labels

    A-analytic-engineArea: Analytic EnginefeatureNew feature or requesttracking issueIssue tracks progress for something

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions