Merge branch develop into master

[GanZiheng]

Development Task

Merge develop into master part by part.

Development Progress

The pull requests merged at develop branch are as follows, I would try to apply them into master.

• vlog

  • [dev] vlog: add iterator with vlog read support #84 @zhangjinpeng1987 txn: finish transaction #143
  • [dev] vlog: fix persistence and never panic on drop #88 @zhangjinpeng1987 *: port Drop implementation #127

• memtable

  • [dev] add memtable and WAL #45 @GanZiheng agate: finish memtable #126
  • [dev] agate: support immutable memtable #46 @GanZiheng agate: finish memtable #126

• levels

  • [dev] levels: add L0 flush #47 @zhangjinpeng1987 flush memtable to L0 #138
  • [dev] levels: add L0 to base compactor #50 @zhangjinpeng1987 db: start bg compaction threads after init agate #148
  • [dev] levels: dump levels and shuffle requests #52 @GanZiheng levels: finish levels #129
  • [dev] levels: finish compaction #54 @GanZiheng levels: finish levels #129
  • [dev] levels: fix compaction stall in sanitizer test #75 @GanZiheng levels: finish levels #129
  • [dev] agate: fix lock order in compaction #73 @GanZiheng levels: finish levels #129
  • [dev] levels: add tests for compaction #77 @GanZiheng levels: finish levels #129
  • [dev] levels: add level test #78 @GanZiheng levels: finish levels #129
  • [dev] compaction: support multi-thread compact #101 @GanZiheng levels: finish levels #129

• txn

  • [dev] txn: implement watermark #57 @GanZiheng txn: add watermark #132
  • [dev] txn: add MVCC and transaction #67 @GanZiheng
    • txn: finish oracle #136
    • txn: finish transaction #143
  • [dev] txn: add managed-db tests #72 @GanZiheng txn: finish transaction #143

• agate

  • [dev] agate: delete unused files upon drop #58 @GanZiheng *: port Drop implementation #127
  • [dev] agate: persist tables after close DB #62 @GanZiheng *: port Drop implementation #127
  • [dev] agate: add manifest and persistence #60 @GanZiheng Add manifest #123
  • [dev] db: optimize database write #87 @GanZiheng agate: support write batch #170
  • [dev] agate: support write batch #91 @GanZiheng agate: support write batch #170

• bench

  • [dev] feat: optimize for benchmark #86 @GanZiheng bench: add iterator bench #169
  • [dev] iterator: add bench #90 @GanZiheng bench: add iterator bench #169
  • [dev] bench: add benchmark script #93 @GanZiheng bench: add bench script #167
  • [dev] bench: improve performance #94 @GanZiheng bench: add bench script #167
  • [dev] bench: add rocksdb bench #96 @GanZiheng bench: add bench script #167
  • bench: add iterate test #99 @GanZiheng bench: add bench script #167
  • bench: sync wal in rocksdb writing #102 @GanZiheng bench: add bench script #167

[GanZiheng]

@skyzh PTAL, thx.

[GanZiheng]

I think we may not need to apply these prs one by one. I would like to port Badger and create pr by module, such as levels, txn and so on.

[skyzh]

That's cool! I think it's also okay to port module by module. But sometimes, reviewers might not be knowing what some code is doing without looking ahead for future PRs. I'd be okay with any solution.