Fix a bug in FlushJob picking more memtables beyond synced WALs #9142
Conversation
riversand963
force-pushed
the
fix-flush-more-memtables-than-wal
branch
from
18864e3
to
602f058
Compare
|
@riversand963 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator. |
riversand963
force-pushed
the
fix-flush-more-memtables-than-wal
branch
from
602f058
to
d78271d
Compare
|
@riversand963 has updated the pull request. You must reimport the pull request before landing. |
|
@riversand963 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator. |
riversand963
force-pushed
the
fix-flush-more-memtables-than-wal
branch
from
d78271d
to
c9b150b
Compare
|
@riversand963 has updated the pull request. You must reimport the pull request before landing. |
|
@riversand963 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator. |
riversand963
force-pushed
the
fix-flush-more-memtables-than-wal
branch
from
c9b150b
to
405e1ab
Compare
|
@riversand963 has updated the pull request. You must reimport the pull request before landing. |
|
@riversand963 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator. |
riversand963
force-pushed
the
fix-flush-more-memtables-than-wal
branch
from
405e1ab
to
caa0182
Compare
|
@riversand963 has updated the pull request. You must reimport the pull request before landing. |
|
@riversand963 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator. |
|
Thanks @ajkr for the review! |
|
@riversand963 merged this pull request in 1e8322c. |
After RocksDB 6.19 and before this PR, RocksDB FlushJob may pick more memtables to flush beyond synced WALs.
This can be problematic if there are multiple column families, since it can prematurely advance the flushed column
family's log_number. Should subsequent attempts fail to sync the latest WALs and the database goes
through a recovery, it may detect corrupted WAL number below the flushed column family's log number
and complain about column family inconsistency.
To fix, we record the maximum memtable ID of the column family being flushed. Then we call SyncClosedLogs()
so that all closed WALs at the time when memtable ID is recorded will be synced.
I also disabled a unit test temporarily due to reasons described in #9151
Test plan:
make check