Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Large transactional writes leave large memtables #1906

Open
JDNdeveloper opened this issue Jul 25, 2019 · 0 comments
Open

Large transactional writes leave large memtables #1906

JDNdeveloper opened this issue Jul 25, 2019 · 0 comments
Assignees
Labels

Comments

@JDNdeveloper
Copy link
Contributor

@JDNdeveloper JDNdeveloper commented Jul 25, 2019

After inserting 1M rows using a single distributed transaction (e.g. YSQL COPY) it was observed that the IntentsDB and RegularDB memtables were much larger than the configured memstore size limit. The memstore size limit was 128MB, but the IntentsDB memtable was 793MB and the RegularDB memtable was 477MB.

The issue has two causes:

  1. Intents are applied in a single rocksdb write batch, meaning the memtable can grow as large as the size of the write batch which could exceed the memstore size limit. The fix for this will be to batch the regular record writes when intents are applied, and similarly to batch the deletion of the intents on distributed txn cleanup.
  2. Memtables are only checked for whether they exceed the memstore size limit (forcing a flush) before a write is performed, meaning if a large single write is processed it could leave the memtables larger than the configured maximum memstore size until another write on the rocksdb is processed. The fix for this will be to move this check until after the write is performed, to ensure it is flushed if the current write exceeded the memstore size limit.
@JDNdeveloper JDNdeveloper self-assigned this Jul 25, 2019
@JDNdeveloper JDNdeveloper added this to To Do in YBase features via automation Jul 25, 2019
@JDNdeveloper JDNdeveloper moved this from To Do to In progress in YBase features Jul 30, 2019
@JDNdeveloper JDNdeveloper moved this from In progress to To Do in YBase features Aug 20, 2019
@ndeodhar ndeodhar assigned bmatican and unassigned JDNdeveloper Dec 2, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
YBase features
  
To Do
2 participants
You can’t perform that action at this time.