Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ChunksIterator method to Series interface. #5882

Closed
wants to merge 1 commit into from

Conversation

@bwplotka
Copy link
Member

@bwplotka bwplotka commented Aug 13, 2019

Continuation from: prometheus-junkyard/tsdb#665

Changes:

  • Adjusts Iterator Interface of Series e.g Seek
  • Reuse single interface instead of 3 same ones spread across 3 packages.
  • Refactored tests
  • Added ChunkIterator to iterate over chunks if requested by the client.

Fixes: #5871

@bwplotka bwplotka force-pushed the add-chunks-to-queriers-iterator-tsdb branch 2 times, most recently from f460245 to 1999ce8 Aug 13, 2019
@bwplotka bwplotka marked this pull request as ready for review Aug 13, 2019
tsdb/querier.go Outdated
// Err returns the current error.
Err() error
func (s *chunkSeries) ChunkIterator() ChunkIterator {
// TODO(bwplotka): This is wrong. We probably have to be strict in terms of min/maxt and tombstones.
Copy link
Member Author

@bwplotka bwplotka Aug 13, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO/TO decide:

When we want to apply tombstones in ChunkIterator? :/

TBH it might be the most efficient to pass intervals upstream e.g in streamed remote read protocol as well (???)

cc @brian-brazil

Copy link
Member Author

@bwplotka bwplotka Aug 13, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two options:

A) Naive: Rewrite chunk on the fly applying mint,maxt and dranges. It is must have for Select interface which assumes things are scoped to given time range and tombstones are tolerated.
B) Ignore mint,maxt level. Maybe allow ChunkIterator to return deleted intervals as well. This probably means extending the remote read chunked protocol here for passing tombstones intervals as well -> let client side to apply intervals.. However, we leak "deleted" metrics. If they were deleted because of GDPR or something, this is kind of wrong (:

I think overall we need to do A here. Thoughts @brian-brazil @tomwilkie @codesome ?

Copy link
Contributor

@brian-brazil brian-brazil Aug 14, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Definitely A, for general data security we don't want to be returning deleted data. It'd also push that complexity onto all readers, and they'd probably forget it.

Copy link
Member Author

@bwplotka bwplotka Aug 15, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It'd also push that complexity onto all readers, and they'd probably forget it.

You would NOT put it to readers you mean?

Makes sense to me.. Let's do it then here.

@bwplotka
Copy link
Member Author

@bwplotka bwplotka commented Aug 13, 2019

Addressed all comments from the previous PR: prometheus-junkyard/tsdb#665 PTAL @brian-brazil @krasi-georgiev

tsdb/head.go Outdated
@@ -1617,6 +1617,8 @@ func newMemSeries(lset labels.Labels, id uint64, chunkRange int64) *memSeries {
ref: id,
chunkRange: chunkRange,
nextAt: math.MinInt64,
// This is to make sure iterator.At without advancing returns default values.
Copy link
Contributor

@brian-brazil brian-brazil Aug 14, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not valid to call At without advancing.

Copy link
Contributor

@krasi-georgiev krasi-georgiev Aug 15, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe calling At before Next or Seek should cause iterator.Error
In memSafeIterator add a var initialized and when false memSafeIterator.At returns default values and causes memSafeIterator.Err.

Either way the logic should be in the iterator itself.

Copy link
Contributor

@krasi-georgiev krasi-georgiev left a comment

so many iterators now, might run out of names soon 😄

@@ -1,5 +1,8 @@
## master / unreleased

- [CHANGE] `chunks.MergeOverlappingChunks` moved to `tsdb.MergeOverlappingChunks`
- [CHANGE] `Series` interface allows return chunk iterator that allows iterating over encoded chunks.
Copy link
Contributor

@krasi-georgiev krasi-georgiev Aug 15, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you think it is worth it to still keep track of these?
Is it helpful to you or the cortex team?
cc @tomwilkie

I mean do we still need a CHANGELOG file at all? How would it be useful?

tsdb/querier.go Outdated
@@ -1175,6 +1358,19 @@ func (it *deletedIterator) At() (int64, float64) {
return it.it.At()
}

func (it *deletedIterator) Seek(t int64) bool {
if atT, _ := it.At(); t >= atT {
return false
Copy link
Contributor

@krasi-georgiev krasi-georgiev Aug 15, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seek should return true when t==atT

If possible you can add the TestDeletedIterator to TestSeriesIterators and delete the TestDeletedIterator test or add a seek test to TestDeletedIterator

tsdb/querier.go Outdated
// ChunkIterator iterates over the chunk of a time series.
type ChunkIterator interface {
// Seek advances the iterator forward to the given timestamp.
// It advances to the chunk with min time at t or first chunk with min time after t.
Copy link
Contributor

@krasi-georgiev krasi-georgiev Aug 15, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a behavior description similar to the chunk.Iterator?
I mean what hapens in the different scenarios when requested time is the same or smaller than the current At.

var r []tsdbutil.Sample
if tc.seek != 0 {
testutil.Equals(t, tc.seekSuccess, it.Seek(tc.seek))
testutil.Equals(t, tc.seekSuccess, it.Seek(tc.seek)) // Next one should be noop.
Copy link
Contributor

@krasi-georgiev krasi-georgiev Aug 15, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
testutil.Equals(t, tc.seekSuccess, it.Seek(tc.seek)) // Next one should be noop.
testutil.Equals(t, tc.seekSuccess, it.Seek(tc.seek)) // Next one should be noop.
testutil.Equals(t, false, it.Seek(tc.seek-1)) // Seek in the past should be noop and return false.

var r []chunks.Meta
if tc.seek != 0 {
testutil.Equals(t, tc.seekSuccess, it.Seek(tc.seek))
testutil.Equals(t, tc.seekSuccess, it.Seek(tc.seek)) // Next one should be noop.
Copy link
Contributor

@krasi-georgiev krasi-georgiev Aug 15, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
testutil.Equals(t, tc.seekSuccess, it.Seek(tc.seek)) // Next one should be noop.
testutil.Equals(t, tc.seekSuccess, it.Seek(tc.seek)) // Next one should be noop.
testutil.Equals(t, false, it.Seek(tc.seek-1)) // Seek in the past should be noop and return false.

}
}

func (it *verticalMergeSeriesIterator) Seek(t int64) bool {
if it.initialized && it.curT >= t {
return true
Copy link
Contributor

@krasi-georgiev krasi-georgiev Aug 15, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hm this doesn't look right. This would return true when it.curT is 150 and the requested time is 100

I thought we want the following behaviour:
Iterator with minT:100, maxT:200, curT:150

Seek(150) - returns true and the Seek is noop, doesn't advance.
Seek(100) - returns false and is noop.
Seek(160) - returns true and advances the iterator to 160 or the next available timestamp.
Seek(210) - returns false and exhausts the iterator.
When it.Err is not nil return false in all cases.

Maybe putting an example like this in the iterator description would make things a bir more clear for what is expected in the different scenarios.

if t > it.maxt {
// Exhaust iterator.
it.i = len(it.chunks)
Copy link
Contributor

@krasi-georgiev krasi-georgiev Aug 15, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe it is just my style, but I would think it will be easier to follow with an explicit var it.exhausted = true. Can leave as is if you prefer.

return false
}
return true
return t <= it.maxt
}
if err := it.cur.Err(); err != nil {
Copy link
Contributor

@krasi-georgiev krasi-georgiev Aug 15, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if err := it.cur.Err(); err != nil {
if it.cur.Err() != nil {

tsdb/querier.go Outdated
@@ -940,7 +957,7 @@ func (it *chainedSeriesIterator) Next() bool {
if it.cur.Next() {
return true
}
if err := it.cur.Err(); err != nil {
if it.cur.Err() != nil {
Copy link
Contributor

@krasi-georgiev krasi-georgiev Aug 15, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I completely forgot we can do that 🥇 nice reminder, thanks!

@@ -1053,8 +1304,9 @@ func TestChunkSeriesIterator_DoubleSeek(t *testing.T) {
}

res := newChunkSeriesIterator(chkMetas, nil, 2, 8)
testutil.Assert(t, res.Seek(1) == true, "")
testutil.Assert(t, res.Seek(2) == true, "")
testutil.Assert(t, res.Seek(1), "")
Copy link
Contributor

@krasi-georgiev krasi-georgiev Aug 15, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't think we even need this test anymore as we can cover it in TestSeriesIterators

@bwplotka
Copy link
Member Author

@bwplotka bwplotka commented Nov 28, 2019

Need to get back to this soon (:

@bwplotka bwplotka force-pushed the add-chunks-to-queriers-iterator-tsdb branch from dbe23b0 to 17c05f2 Feb 3, 2020
@bwplotka
Copy link
Member Author

@bwplotka bwplotka commented Feb 3, 2020

Getting back to this, expect changes soon: WIP.

@brian-brazil @codesome do you know why we have two places with storage interfaces?
tsdb/Querier have some: https://github.com/prometheus/prometheus/blob/master/tsdb/querier.go#L33
And storage has almost the same: https://github.com/prometheus/prometheus/blob/master/storage/interface.go

Since we merged TSDB with Prometheus, do you think it makes sense to unify those together?

@brian-brazil
Copy link
Contributor

@brian-brazil brian-brazil commented Feb 3, 2020

It's historical, but we probably want to think about the impact on the remote storage code too.

@bwplotka bwplotka force-pushed the add-chunks-to-queriers-iterator-tsdb branch from 17c05f2 to ad7d8e4 Feb 4, 2020
@bwplotka
Copy link
Member Author

@bwplotka bwplotka commented Feb 5, 2020

We can literally remove and deduplicate lot's of iterators.. expect big PR 😢

Should be ready for review EOW

bwplotka added a commit that referenced this issue Feb 6, 2020
This is part of #5882 that can be done to simplify things.
All todos I added will be fixed in follow up PRs.

## Changes

* querier.Querier, querier.SeriesSet, and querier.Series interfaces merged
with storage interface.go. All imports that.
* querier.SeriesIterator replaced by chunkenc.Iterator
* Added chunkenc.Iterator.Seek method and tests for xor implementation (?)
* Since we properly handle SelectParams for Select methods I adjusted min max
based on that. This should help in terms of performance for queries with functions like offset.
* added Seek to deletedIterator and test.

No logic was changed, only different source of abstractions, so no need for benchmarks.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
bwplotka added a commit that referenced this issue Feb 7, 2020
This is part of #5882 that can be done to simplify things.
All todos I added will be fixed in follow up PRs.

## Changes

* querier.Querier, querier.Appender, querier.SeriesSet, and querier.Series interfaces merged
with storage interface.go. All imports that.
* querier.SeriesIterator replaced by chunkenc.Iterator
* Added chunkenc.Iterator.Seek method and tests for xor implementation (?)
* Since we properly handle SelectParams for Select methods I adjusted min max
based on that. This should help in terms of performance for queries with functions like offset.
* added Seek to deletedIterator and test.
* storage/tsdb was removed as it was only a unnecessary glue with incompatible structs.

No logic was changed, only different source of abstractions, so no need for benchmarks.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
bwplotka added a commit that referenced this issue Feb 7, 2020
This is part of #5882 that can be done to simplify things.
All todos I added will be fixed in follow up PRs.

## Changes

* querier.Querier, querier.Appender, querier.SeriesSet, and querier.Series interfaces merged
with storage interface.go. All imports that.
* querier.SeriesIterator replaced by chunkenc.Iterator
* Added chunkenc.Iterator.Seek method and tests for xor implementation (?)
* Since we properly handle SelectParams for Select methods I adjusted min max
based on that. This should help in terms of performance for queries with functions like offset.
* added Seek to deletedIterator and test.
* storage/tsdb was removed as it was only a unnecessary glue with incompatible structs.

No logic was changed, only different source of abstractions, so no need for benchmarks.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
bwplotka added a commit that referenced this issue Feb 7, 2020
This is part of #5882 that can be done to simplify things.
All todos I added will be fixed in follow up PRs.

## Changes

* querier.Querier, querier.Appender, querier.SeriesSet, and querier.Series interfaces merged
with storage interface.go. All imports that.
* querier.SeriesIterator replaced by chunkenc.Iterator
* Added chunkenc.Iterator.Seek method and tests for xor implementation (?)
* Since we properly handle SelectParams for Select methods I adjusted min max
based on that. This should help in terms of performance for queries with functions like offset.
* added Seek to deletedIterator and test.
* storage/tsdb was removed as it was only a unnecessary glue with incompatible structs.

No logic was changed, only different source of abstractions, so no need for benchmarks.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
bwplotka added a commit that referenced this issue Feb 7, 2020
This is part of #5882 that can be done to simplify things.
All todos I added will be fixed in follow up PRs.

## Changes

* querier.Querier, querier.Appender, querier.SeriesSet, and querier.Series interfaces merged
with storage interface.go. All imports that.
* querier.SeriesIterator replaced by chunkenc.Iterator
* Added chunkenc.Iterator.Seek method and tests for xor implementation (?)
* Since we properly handle SelectParams for Select methods I adjusted min max
based on that. This should help in terms of performance for queries with functions like offset.
* added Seek to deletedIterator and test.
* storage/tsdb was removed as it was only a unnecessary glue with incompatible structs.

No logic was changed, only different source of abstractions, so no need for benchmarks.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
bwplotka added a commit that referenced this issue Feb 7, 2020
This is part of #5882 that can be done to simplify things.
All todos I added will be fixed in follow up PRs.

## Changes

* querier.Querier, querier.Appender, querier.SeriesSet, and querier.Series interfaces merged
with storage interface.go. All imports that.
* querier.SeriesIterator replaced by chunkenc.Iterator
* Added chunkenc.Iterator.Seek method and tests for xor implementation (?)
* Since we properly handle SelectParams for Select methods I adjusted min max
based on that. This should help in terms of performance for queries with functions like offset.
* added Seek to deletedIterator and test.
* storage/tsdb was removed as it was only a unnecessary glue with incompatible structs.

No logic was changed, only different source of abstractions, so no need for benchmarks.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
bwplotka added a commit that referenced this issue Feb 7, 2020
This is part of #5882 that can be done to simplify things.
All todos I added will be fixed in follow up PRs.

## Changes

* querier.Querier, querier.Appender, querier.SeriesSet, and querier.Series interfaces merged
with storage interface.go. All imports that.
* querier.SeriesIterator replaced by chunkenc.Iterator
* Added chunkenc.Iterator.Seek method and tests for xor implementation (?)
* Since we properly handle SelectParams for Select methods I adjusted min max
based on that. This should help in terms of performance for queries with functions like offset.
* added Seek to deletedIterator and test.
* storage/tsdb was removed as it was only a unnecessary glue with incompatible structs.

No logic was changed, only different source of abstractions, so no need for benchmarks.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
bwplotka added a commit that referenced this issue Feb 7, 2020
This is part of #5882 that can be done to simplify things.
All todos I added will be fixed in follow up PRs.

## Changes

* querier.Querier, querier.Appender, querier.SeriesSet, and querier.Series interfaces merged
with storage interface.go. All imports that.
* querier.SeriesIterator replaced by chunkenc.Iterator
* Added chunkenc.Iterator.Seek method and tests for xor implementation (?)
* Since we properly handle SelectParams for Select methods I adjusted min max
based on that. This should help in terms of performance for queries with functions like offset.
* added Seek to deletedIterator and test.
* storage/tsdb was removed as it was only a unnecessary glue with incompatible structs.

No logic was changed, only different source of abstractions, so no need for benchmarks.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
bwplotka added a commit that referenced this issue Feb 7, 2020
This is part of #5882 that can be done to simplify things.
All todos I added will be fixed in follow up PRs.

## Changes

* querier.Querier, querier.Appender, querier.SeriesSet, and querier.Series interfaces merged
with storage interface.go. All imports that.
* querier.SeriesIterator replaced by chunkenc.Iterator
* Added chunkenc.Iterator.Seek method and tests for xor implementation (?)
* Since we properly handle SelectParams for Select methods I adjusted min max
based on that. This should help in terms of performance for queries with functions like offset.
* added Seek to deletedIterator and test.
* storage/tsdb was removed as it was only a unnecessary glue with incompatible structs.

No logic was changed, only different source of abstractions, so no need for benchmarks.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
…terface.

* Added Seek to chunks.Iterator interface for iterating over chunks.
* Mock, NewTestSeries, SampleSeriesIterator and ChunkSeriesIterator are now available from storage package and reuses instead of
being recreated in many places. NewConcreteSeries was created to replace concreteSeries.
*

Added simple `verticalMergeChunkIterator` implementation - no tests so far.

Reusing same vertical merging strategy in both compact and querier.
* Refactored TestSeriesIterator.
* Simplified; merged seek and non seek cases together. Added explicit min/max only for chunk series iterator, where it is relevant.
* Adjusted all seek implementation to match edge case requirement (double seek, failed seek + next).
Added more tests for old iterators; pulled Free's changes from the PR #329
Fixed series iterator tests.

Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
@bwplotka bwplotka force-pushed the add-chunks-to-queriers-iterator-tsdb branch from ad0106b to c530b4b Feb 26, 2020
brancz
Copy link
Member

brancz commented on c530b4b Feb 27, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

label querier?

Copy link
Contributor

@krasi-georgiev krasi-georgiev left a comment

I think it is ok,

The Labeled interface name is strange, but I also can't think of a better one

for isNext {
chk := iter.At()

if chk.Chunk == nil {
Copy link
Contributor

@krasi-georgiev krasi-georgiev Feb 27, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of this I would expect that iter.Next() would return false and the error would be in iter.Err()

return s.(SeriesSet), w, err
}
func (q *querierAdapter) SelectSorted(params *SelectParams, matchers ...*labels.Matcher) (SeriesSet, Warnings, error) {
s, w, err := q.SelectSorted(params, matchers...)
Copy link
Contributor

@krasi-georgiev krasi-georgiev Feb 27, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

recursion by mistake? the same below

Suggested change
s, w, err := q.SelectSorted(params, matchers...)
s, w, err := q.genericQuerier.SelectSorted(params, matchers...)

Copy link
Member Author

@bwplotka bwplotka Feb 28, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, lol

// LabelValues returns all potential values for a label name.
// It is not safe to use the strings beyond the lifefime of the querier.
LabelValues(name string) ([]string, Warnings, error)
// ChunkedQuerier provides querying access over time series' chunks of a fixed
Copy link
Contributor

@krasi-georgiev krasi-georgiev Feb 27, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not ChunksQuerier?

Suggested change
// ChunkedQuerier provides querying access over time series' chunks of a fixed
// ChunksQuerier provides querying access over time series' chunks of a fixed

for context I would also like to see a small comment on where it can be used - in this case for remote read chunks streaming.

Copy link
Member Author

@bwplotka bwplotka Feb 28, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can do (:

//// It advances to the chunk with min time at t or first chunk with min time after t.
//Seek(t int64) bool
// At returns the current meta.
// It depends on implementation if the chunk is populated or not.
Copy link
Contributor

@krasi-georgiev krasi-georgiev Feb 27, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is that? Why not just return non empty chunks?

Copy link
Member Author

@bwplotka bwplotka Feb 28, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think you know what I mean here. Populated is if you have only Ref or if you have actually Bytes of chunk there (:

ChunkIteratorFn func() chunks.Iterator
}

func NewTestSeries(lset labels.Labels, samples ...[]tsdbutil.Sample) Series {
Copy link
Contributor

@krasi-georgiev krasi-georgiev Feb 27, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is only for testing right? so why not put in a test package or a test file?

Copy link
Member Author

@bwplotka bwplotka Feb 28, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because we want to reuse this across all unit test, even those not in this package.


type genericSeriesSet interface {
Next() bool
At() Labeled
Copy link
Contributor

@krasi-georgiev krasi-georgiev Feb 27, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
At() Labeled
At() Series

Copy link
Member Author

@bwplotka bwplotka Feb 28, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nope, we need Labelled, but also because it's actually genericSeries ;p

@bwplotka
Copy link
Member Author

@bwplotka bwplotka commented Feb 28, 2020

label querier?

Labeled

I like labelled as it is "something" that has labels (:

boyskila added a commit to boyskila/prometheus that referenced this issue Mar 3, 2020
This is part of prometheus#5882 that can be done to simplify things.
All todos I added will be fixed in follow up PRs.

* querier.Querier, querier.Appender, querier.SeriesSet, and querier.Series interfaces merged
with storage interface.go. All imports that.
* querier.SeriesIterator replaced by chunkenc.Iterator
* Added chunkenc.Iterator.Seek method and tests for xor implementation (?)
* Since we properly handle SelectParams for Select methods I adjusted min max
based on that. This should help in terms of performance for queries with functions like offset.
* added Seek to deletedIterator and test.
* storage/tsdb was removed as it was only a unnecessary glue with incompatible structs.

No logic was changed, only different source of abstractions, so no need for benchmarks.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
boyskila added a commit to boyskila/prometheus that referenced this issue Mar 3, 2020
This is part of prometheus#5882 that can be done to simplify things.
All todos I added will be fixed in follow up PRs.

* querier.Querier, querier.Appender, querier.SeriesSet, and querier.Series interfaces merged
with storage interface.go. All imports that.
* querier.SeriesIterator replaced by chunkenc.Iterator
* Added chunkenc.Iterator.Seek method and tests for xor implementation (?)
* Since we properly handle SelectParams for Select methods I adjusted min max
based on that. This should help in terms of performance for queries with functions like offset.
* added Seek to deletedIterator and test.
* storage/tsdb was removed as it was only a unnecessary glue with incompatible structs.

No logic was changed, only different source of abstractions, so no need for benchmarks.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
Signed-off-by: blalov <boiskila@gmail.com>
bwplotka added a commit that referenced this issue Mar 12, 2020
I found during work on #5882 that
we do so many repetitions because of this, for not good reason. I think
I found a good balance between convenience and readability with just one method.
Smaller the interface = better.

Also I don't know what TestSelectSorted was testing, but now it's testing sorting.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
bwplotka added a commit that referenced this issue Mar 12, 2020
I found during work on #5882 that
we do so many repetitions because of this, for not good reason. I think
I found a good balance between convenience and readability with just one method.
Smaller the interface = better.

Also I don't know what TestSelectSorted was testing, but now it's testing sorting.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
bwplotka added a commit that referenced this issue Mar 12, 2020
I found during work on #5882 that
we do so many repetitions because of this, for not good reason. I think
I found a good balance between convenience and readability with just one method.
Smaller the interface = better.

Also I don't know what TestSelectSorted was testing, but now it's testing sorting.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
bwplotka added a commit that referenced this issue Mar 12, 2020
I found during work on #5882 that
we do so many repetitions because of this, for not good reason. I think
I found a good balance between convenience and readability with just one method.
Smaller the interface = better.

Also I don't know what TestSelectSorted was testing, but now it's testing sorting.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
bwplotka added a commit that referenced this issue Mar 12, 2020
I found during work on #5882 that
we do so many repetitions because of this, for not good reason. I think
I found a good balance between convenience and readability with just one method.
Smaller the interface = better.

Also I don't know what TestSelectSorted was testing, but now it's testing sorting.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
bwplotka added a commit that referenced this issue Mar 12, 2020
I found during work on #5882 that
we do so many repetitions because of this, for not good reason. I think
I found a good balance between convenience and readability with just one method.
Smaller the interface = better.

Also I don't know what TestSelectSorted was testing, but now it's testing sorting.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
bwplotka added a commit that referenced this issue Mar 12, 2020
I found during work on #5882 that
we do so many repetitions because of this, for not good reason. I think
I found a good balance between convenience and readability with just one method.
Smaller the interface = better.

Also I don't know what TestSelectSorted was testing, but now it's testing sorting.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
bwplotka added a commit that referenced this issue Mar 12, 2020
I found during work on #5882 that
we do so many repetitions because of this, for not good reason. I think
I found a good balance between convenience and readability with just one method.
Smaller the interface = better.

Also I don't know what TestSelectSorted was testing, but now it's testing sorting.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
bwplotka added a commit that referenced this issue Mar 12, 2020
I found during work on #5882 that
we do so many repetitions because of this, for not good reason. I think
I found a good balance between convenience and readability with just one method.
Smaller the interface = better.

Also I don't know what TestSelectSorted was testing, but now it's testing sorting.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
bwplotka added a commit that referenced this issue Mar 13, 2020
…uirement for remote read to sort response.

This is technically BREAKING CHANGE, but it was like this from the beginning: I just notice that we rely in
Prometheus on remote read being sorted. This is because we use selected data from remote reads in MergeSeriesSet
which rely on sorting.

I found during work on #5882 that
we do so many repetitions because of this, for not good reason. I think
I found a good balance between convenience and readability with just one method.
Smaller the interface = better.

Also I don't know what TestSelectSorted was testing, but now it's testing sorting.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
bwplotka added a commit that referenced this issue Mar 13, 2020
…uirement for remote read to sort response.

This is technically BREAKING CHANGE, but it was like this from the beginning: I just notice that we rely in
Prometheus on remote read being sorted. This is because we use selected data from remote reads in MergeSeriesSet
which rely on sorting.

I found during work on #5882 that
we do so many repetitions because of this, for not good reason. I think
I found a good balance between convenience and readability with just one method.
Smaller the interface = better.

Also I don't know what TestSelectSorted was testing, but now it's testing sorting.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
bwplotka added a commit that referenced this issue Mar 13, 2020
…uirement for remote read to sort response.

This is technically BREAKING CHANGE, but it was like this from the beginning: I just notice that we rely in
Prometheus on remote read being sorted. This is because we use selected data from remote reads in MergeSeriesSet
which rely on sorting.

I found during work on #5882 that
we do so many repetitions because of this, for not good reason. I think
I found a good balance between convenience and readability with just one method.
Smaller the interface = better.

Also I don't know what TestSelectSorted was testing, but now it's testing sorting.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
bwplotka added a commit that referenced this issue Mar 13, 2020
…uirement for remote read to sort response.

This is technically BREAKING CHANGE, but it was like this from the beginning: I just notice that we rely in
Prometheus on remote read being sorted. This is because we use selected data from remote reads in MergeSeriesSet
which rely on sorting.

I found during work on #5882 that
we do so many repetitions because of this, for not good reason. I think
I found a good balance between convenience and readability with just one method.
Smaller the interface = better.

Also I don't know what TestSelectSorted was testing, but now it's testing sorting.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
@bwplotka
Copy link
Member Author

@bwplotka bwplotka commented Mar 16, 2020

OK, let's close this in favor of clean version of same thing: #6990

@bwplotka bwplotka closed this Mar 16, 2020
@bwplotka bwplotka deleted the add-chunks-to-queriers-iterator-tsdb branch Mar 16, 2020
cstyan added a commit to cstyan/prometheus that referenced this issue Mar 18, 2020
…uirement for remote read to sort response.

This is technically BREAKING CHANGE, but it was like this from the beginning: I just notice that we rely in
Prometheus on remote read being sorted. This is because we use selected data from remote reads in MergeSeriesSet
which rely on sorting.

I found during work on prometheus#5882 that
we do so many repetitions because of this, for not good reason. I think
I found a good balance between convenience and readability with just one method.
Smaller the interface = better.

Also I don't know what TestSelectSorted was testing, but now it's testing sorting.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
roidelapluie added a commit to roidelapluie/prometheus that referenced this issue Mar 20, 2020
…uirement for remote read to sort response.

This is technically BREAKING CHANGE, but it was like this from the beginning: I just notice that we rely in
Prometheus on remote read being sorted. This is because we use selected data from remote reads in MergeSeriesSet
which rely on sorting.

I found during work on prometheus#5882 that
we do so many repetitions because of this, for not good reason. I think
I found a good balance between convenience and readability with just one method.
Smaller the interface = better.

Also I don't know what TestSelectSorted was testing, but now it's testing sorting.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

4 participants