how to implement log_store interface?

[cklxiaocui]

I use sqlite3 to store log entires,but I do not understand how to implement log_store interface,include issues:

• how compact function work? inmem_log_store.h implement it by remove data from map, I implement:

private:
    static ptr<log_entry> make_clone(const ptr<log_entry> &entry);

    // Dummy entry for index 0.
    ptr<log_entry> dummy_entry_;
    db_entry_helper entry_helper; //操作数据库
    std::atomic<ulong> start_idx_; //the start log id
    std::atomic<ulong> last_idx_;  //the last log id

bool indb_log_store::compact(ulong last_log_index) {
    info("compact log from [%u] to [%u], cur last_log_index:[%d]", start_idx_.load(), last_log_index, last_idx_.load());

    entry_helper.removeRange("id", start_idx_, last_log_index);

    // WARNING:
    //   Even though nothing has been erased,
    //   we should set `start_idx_` to new index.
    start_idx_ = last_log_index + 1;
    auto last = entry_helper.selectLast("id");
    if (!last->empty()) {
        last_idx_ = last->at(0).id_;
    } else {
        last_idx_ = last_log_index + 1;
    }
    info("start_idx:[%d], last_indx[%d]", start_idx_.load(), last_idx_.load());
}

• The operation log is as follows

方日志 3号日志 leader log
2019-11-12T18:51:13.792_873+08:00 [c313] [INFO] trying to sync snapshot with last index 15 to peer 1, its last log idx 9	[handle_snapshot_sync.cxx:111, create_sync_snapshot_req()]
2019-11-12T18:51:13.794_468+08:00 [7067] [WARN] peer 1 declined snapshot: p->get_next_log_idx(): 10, log_store_->next_slot(): 17	[handle_snapshot_sync.cxx:317, handle_install_snapshot_resp()]
2019-11-12T18:51:13.795_047+08:00 [a3ce] [WARN] declined append: peer 1, prev next log idx 12, resp next 12, new next log idx 11	[handle_append_entries.cxx:676, handle_append_entries_resp()]
2019-11-12T18:51:13.795_317+08:00 [a3ce] [ERRO] bad log_idx 10 for retrieving the term value, will ignore this log req	[raft_server.cxx:1063, term_for_log()]
2019-11-12T18:51:13.795_322+08:00 [a3ce] [ERRO] last snapshot 0x7ff0ec003fe0, log_idx 10, snapshot last_log_idx 15	[raft_server.cxx:1066, term_for_log()]
2019-11-12T18:51:13.795_325+08:00 [a3ce] [ERRO] log_store_->start_index() 11	[raft_server.cxx:1068, term_for_log()]
2019-11-12T18:51:13.795_412+08:00 [c313] [WARN] declined append: peer 1, prev next log idx 11, resp next 12, new next log idx 10



//发送方日志 1号日志 fflower logs
2019-11-12T18:51:13.814_681+08:00 [42d1] [WARN] [LOG XX] req log idx: 11, req log term: 1, my last log idx: 11, my log (11) term: 0	[handle_append_entries.cxx:391, handle_append_entries()]
2019-11-12T18:51:13.814_683+08:00 [42d1] [WARN] deny, req term 1, my term 1, req log idx 11, my log idx 11	[handle_append_entries.cxx:398, handle_append_entries()]
2019-11-12T18:51:13.814_685+08:00 [42d1] [WARN] snp idx 15 term 1	[handle_append_entries.cxx:402, handle_append_entries()]
2019-11-12T18:51:13.814_845+08:00 [8b66] [ERRO] bad log_idx 10 for retrieving the term value, will ignore this log req	[raft_server.cxx:1063, term_for_log()]
2019-11-12T18:51:13.814_848+08:00 [8b66] [ERRO] last snapshot 0x7f213c0013c0, log_idx 10, snapshot last_log_idx 15	[raft_server.cxx:1066, term_for_log()]
2019-11-12T18:51:13.814_850+08:00 [8b66] [ERRO] log_store_->start_index() 16	[raft_server.cxx:1068, term_for_log()]

• My trouble

1. in compact function , how to update start_idx_ and last_idx_
2. in compact function, Whether or not to remove log entries from db

[greensky00]

Hi @cklxiaocui

The semantic of compact is to remove all logs up to given last_log_index (including itself), so your implementation looks ok.

Isn't it because of race condition between calling removeRange and updating last_idx_? If you increase log level to 5 and get more detailed logs, that will be helpful. Thanks.

[greensky00]

+) Also you should return true once compaction is done correctly.

[cklxiaocui]

thanks a lot, the log_store interface be called by asio In multithreaded mode，There are many boundary values to deal with，I need to see the source code of nuraft .