Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

db: make DB.Ingest return bytes ingested into L0 #1600

Closed
sumeerbhola opened this issue Mar 24, 2022 · 0 comments · Fixed by #1653
Closed

db: make DB.Ingest return bytes ingested into L0 #1600

sumeerbhola opened this issue Mar 24, 2022 · 0 comments · Fixed by #1653

Comments

@sumeerbhola
Copy link
Collaborator

This is needed for improvements to admission control that will use byte based tokens, that is part of cockroachdb/cockroach#75066 (see the prototype code in https://github.com/cockroachdb/cockroach/blob/ec4149e310f9b383f87cc7ff8776258d72927394/pkg/util/admission/work_queue.go#L911).
This is complicated by #25, since the actual placement into a Version will be asynchronously done. However, it is viable to produce an estimate at the time of Ingest based on the current spans of L0 files and memtables. This may be an over-estimate since (a) a memtable can be split into multiple sstables when flushing, and an ingested sstable may be able to slip through a gap, (b) an overlapping L0 file may get compacted down to Lbase by the time the ingest is applied to produce a new Version. An estimate is acceptable for admission control.
@sumeerbhola sumeerbhola mentioned this issue Mar 30, 2022
Closed
@sumeerbhola sumeerbhola mentioned this issue Apr 15, 2022
Merged
sumeerbhola added a commit to sumeerbhola/pebble that referenced this issue Apr 19, 2022
@sumeerbhola
db: add IngestOperationStats for admission control
18ad83e 
To allow for async ingestion (being implemented) we don't reuse
TableIngestInfo, and allow an approximate value for the bytes
ingested into L0. These will be used in admission control to
more precisely account for how much load an operation is adding
to L0 (prototype is in
https://github.com/cockroachdb/cockroach/blob/ec4149e310f9b383f87cc7ff8776258d72927394/pkg/util/admission/work_queue.go#L911).

Fixes cockroachdb#1600
sumeerbhola added a commit that referenced this issue Apr 19, 2022
@sumeerbhola
db: add IngestOperationStats for admission control
4a0889f 
To allow for async ingestion (being implemented) we don't reuse
TableIngestInfo, and allow an approximate value for the bytes
ingested into L0. These will be used in admission control to
more precisely account for how much load an operation is adding
to L0 (prototype is in
https://github.com/cockroachdb/cockroach/blob/ec4149e310f9b383f87cc7ff8776258d72927394/pkg/util/admission/work_queue.go#L911).

Fixes #1600
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

db: add IngestOperationStats for admission control sumeerbhola/pebble
1 participant
@sumeerbhola