Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: TSKV Optimizations to reduce the memory usage. #1199

Merged
merged 3 commits into from May 25, 2023

Conversation

zipper-meng
Copy link
Member

Which issue does this PR close?

Uncertainly, but lower probability: #1141

Rationale for this change

  1. Fix some typos.
  2. Fix logs print when file created or write finished in compaction and flushing.
  3. Prevent too frequently flush job executions invoked from WAL total size checking.
  4. Prevent too many old compact requests stack in the compact channel (the previous compaction sometimes costs more than 1 minute).

What changes are included in this PR?

Log print

  • When a new file created in flush/compaction:
    Flush/Compaction: File: {} been created (level={}).
    
  • When a file write finished in flush/compaction:
    Flush/Compaction: File: {} write finished (level: {}, {} B).
    

Reduce the frequency of external flush job executions.

A force flush job will be invoked when the total file size of WALs is too large (by default the max file size is 2G), if the flush job lasted over the check interval, it starts the second flush job unnecessarily.

Now the flush job by WAL file size check will never be invoked in one minute.

Group compaction job executions.

Since the compaction job sometimes lasts more than 1 minute, the compact channel may be full stacked by old compact requests 1 minute ago.

Now the compact jobs is grouped, the new compact request will be added to the group if previous compaction group is not completed.

Are there any user-facing changes?

@roseboy-liu roseboy-liu merged commit fcd2760 into cnosdb:main May 25, 2023
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants