Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor the LogEntryVecDeque in the log.rs #800

Closed
Phoenix500526 opened this issue Apr 30, 2024 · 1 comment · Fixed by #704
Closed

Refactor the LogEntryVecDeque in the log.rs #800

Phoenix500526 opened this issue Apr 30, 2024 · 1 comment · Fixed by #704
Assignees
Labels
good first issue Good for newcomers

Comments

@Phoenix500526
Copy link
Collaborator

After the pr #704 being merged, we got a LogEntryVecDeque implementation like this:

/// For the leader, there should never be a gap between snapshot and entries
pub(super) struct Log<C: Command> {
    /// Log entries, should be persisted
    /// Note that the logical index in `LogEntry` is different from physical index
    entries: LogEntryVecDeque<C>,
    /// The last log index that has been compacted
    pub(super) base_index: LogIndex,
    /// The last log term that has been compacted
    pub(super) base_term: u64,
    /// Index of highest log entry known to be committed
    pub(super) commit_index: LogIndex,
    /// Index of highest log entry sent to after sync. `last_as` should always be less than or equal to `last_exe`.
    pub(super) last_as: LogIndex,
    /// Index of highest log entry sent to speculatively exe. `last_exe` should always be greater than or equal to `last_as`.
    pub(super) last_exe: LogIndex,
    /// Contexts of fallback log entries
    pub(super) fallback_contexts: HashMap<LogIndex, FallbackContext<C>>,
    /// Tx to send log entries to persist task
    log_tx: mpsc::UnboundedSender<Arc<LogEntry<C>>>,
    /// Entries to keep in memory
    entries_cap: usize,
}

struct LogEntryVecDeque<C: Command> {
    /// A VecDeque to store log entries, it will be serialized and persisted
    entries: VecDeque<Arc<LogEntry<C>>>,
    /// entry size of each item in entries
    entry_size: VecDeque<u64>,
    /// the right index of the batch (offset)
    /// batch_range: [i, i + batch_index[i]]
    batch_index: VecDeque<usize>,
    /// the first entry idx of the current batch window
    first_entry_at_last_batch: usize,
    /// the current batch window size
    last_batch_size: u64,
    /// Batch size limit
    batch_limit: u64,
}

As you can see, batch_index stores an offset. With this offset, we can calculate a log batch of size less than or equal to batch_size for any given index i (FYI: issue #368 ). We record the offset instead of the right bound in that the log may be compressed. However, there is a method named li_to_pi in Log, which can transform the logical index to the physical index (The compression only changes the physical index ):

/// Transform logical index to physical index of `self.entries`
    fn li_to_pi(&self, i: LogIndex) -> usize {
        assert!(
            i > self.base_index,
            "can't access the log entry whose index is not bigger than {}, might have been compacted",
            self.base_index
        );
        (i - self.base_index - 1).numeric_cast()
    }

Therefore, we think we can do some refactoring stuff: move li_to_pi and relevant fields, like base_index and base_term into LogEntryVecDeque.

@Phoenix500526 Phoenix500526 added the good first issue Good for newcomers label Apr 30, 2024
Copy link

👋 Thanks for opening this issue!

Reply with the following command on its own line to get help or engage:

  • /contributing-agreement : to print Contributing Agreements.
  • /assignme : to assign this issue to you.

Phoenix500526 pushed a commit to GFX9/Xline that referenced this issue May 28, 2024
Closes: xline-kv#368, xline-kv#800
Signed-off-by: Phoeniix Zhao <Phoenix500526@163.com>
@Phoenix500526 Phoenix500526 linked a pull request May 28, 2024 that will close this issue
@Phoenix500526 Phoenix500526 self-assigned this May 28, 2024
Phoenix500526 pushed a commit to GFX9/Xline that referenced this issue May 28, 2024
Closes: xline-kv#368, xline-kv#800
Signed-off-by: Phoeniix Zhao <Phoenix500526@163.com>
Phoenix500526 pushed a commit to GFX9/Xline that referenced this issue May 28, 2024
Closes: xline-kv#368, xline-kv#800
Signed-off-by: Phoeniix Zhao <Phoenix500526@163.com>
Phoenix500526 pushed a commit to Phoenix500526/Xline that referenced this issue Jun 4, 2024
Closes: xline-kv#368, xline-kv#800
Signed-off-by: Phoeniix Zhao <Phoenix500526@163.com>
Phoenix500526 pushed a commit to Phoenix500526/Xline that referenced this issue Jun 12, 2024
Closes: xline-kv#368, xline-kv#800
Signed-off-by: Phoeniix Zhao <Phoenix500526@163.com>
mergify bot pushed a commit that referenced this issue Jun 13, 2024
Closes: #368, #800
Signed-off-by: Phoeniix Zhao <Phoenix500526@163.com>
@mergify mergify bot closed this as completed in #704 Jun 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good first issue Good for newcomers
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant