Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: Make dedicated iterator package #13273

Merged
merged 5 commits into from
Jul 3, 2024
Merged

chore: Make dedicated iterator package #13273

merged 5 commits into from
Jul 3, 2024

Conversation

chaudum
Copy link
Contributor

@chaudum chaudum commented Jun 20, 2024

What this PR does / why we need it:

Iterators of various types are widely used throughout the Loki code base. With the recent code additions of the bloom filters, a new set of utility functions for iterators emerged in github.com/grafana/loki/v3/pkg/storage/bloom/v1. The package defines interfaces for various common types, but also provides implementations to create and compose new iterators that implement these interfaces. This new package uses Go Generics.

  • However, at the current state, there are multiple iterator interfaces and implementations, which adds cognitive overhead in understanding how they work and which ones should be used. The idea is to unify them into a single iterator lib for Loki.
  • As a first step towards a single iterator library for the Loki code base, this PR moves the utilities from the pkg/storage/bloom/v1 package to the pkg/iter/v2 package.
  • Second, it changes the existing EntryIterator and SampleIterator iterators to "inherit" v2.CloseIterator in order to expose the same function names.
  • And lastly, naming conventions of iterator interfaces and structs are unified.

The basic iterator interface (defined in pkg/iter/v2/interface.go) looks like so:

// Usage:
//
//	for it.Next() {
//	    curr := it.At()
//	    // do something
//	}
//	if it.Err() != nil {
//	    // do something
//	}
type Iterator[T any] interface {
	Next() bool
	Err() error
	At() T
}

Discussion

I proposed to use the Err() function for getting the optional error, because it was used in the pkg/storage/bloom/v1 package. However, in existing iterators for entries and samples, this function was named Error().
I don't have strong arguments for or against one or the other name.

Next steps

This PR intentionally leaves the duplicate implementation of various iterator types, such a heap sort iterators, slice iterators, or peeking iterators. These should be consolidated in a follow up PR.

Checklist

  • Reviewed the CONTRIBUTING.md guide (required)
  • Documentation added
  • Tests updated
  • Title matches the required conventional commits format, see here
    • Note that Promtail is considered to be feature complete, and future development for logs collection will be in Grafana Alloy. As such, feat PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.
  • Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
  • For Helm chart changes bump the Helm chart version in production/helm/loki/Chart.yaml and update production/helm/loki/CHANGELOG.md and production/helm/loki/README.md. Example PR
  • If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

@chaudum chaudum force-pushed the chaudum/iter-module branch 2 times, most recently from 04e8778 to 5a4036a Compare June 21, 2024 06:37
@chaudum chaudum marked this pull request as ready for review June 21, 2024 06:46
@chaudum chaudum requested a review from a team as a code owner June 21, 2024 06:46
@chaudum chaudum changed the title chore: Make dedicated iterator module chore: Make dedicated iterator package Jun 21, 2024
@grobinson-grafana
Copy link
Contributor

Sounds great to me! ❤️ I think let's use Err() as it seems idiomatic (see context.Context).

@chaudum chaudum force-pushed the chaudum/iter-module branch 2 times, most recently from f7ccd9d to eab8542 Compare July 1, 2024 06:32
Copy link
Contributor

@cyriltovena cyriltovena left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
Iterators with added functionality are named by their additional
function as prefix plus the `Iterator` suffix, e.g. `CloseIterator`, or
`PeekIterator`.

Implementations (structs) use the same prefix, but the `Iter` suffix,
e.g. `PeekIter`, or `MapIter`.

Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
@chaudum chaudum merged commit d8cc1ce into main Jul 3, 2024
60 checks passed
@chaudum chaudum deleted the chaudum/iter-module branch July 3, 2024 08:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants