Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(blooms): Add metrics for per-tenant tasks progress to planner #13078

Merged
merged 4 commits into from
Jun 20, 2024

Conversation

salvacorts
Copy link
Contributor

@salvacorts salvacorts commented May 30, 2024

What this PR does / why we need it:
This PR adds two new metrics to track the build process of each tenant.

Special notes for your reviewer:

  • We already expose a per-tnenat metric for the queue size:
    queueLength: promauto.With(registerer).NewGaugeVec(prometheus.GaugeOpts{
    Namespace: metricsNamespace,
    Subsystem: subsystem,
    Name: "queue_length",
    Help: "Number of queries in the queue.",
    }, []string{"user"}),

Checklist

  • Reviewed the CONTRIBUTING.md guide (required)
  • Documentation added
  • Tests updated
  • Title matches the required conventional commits format, see here
    • Note that Promtail is considered to be feature complete, and future development for logs collection will be in Grafana Alloy. As such, feat PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.
  • Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
  • For Helm chart changes bump the Helm chart version in production/helm/loki/Chart.yaml and update production/helm/loki/CHANGELOG.md and production/helm/loki/README.md. Example PR
  • If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

@salvacorts salvacorts marked this pull request as ready for review May 30, 2024 12:27
@salvacorts salvacorts requested a review from a team as a code owner May 30, 2024 12:27
@salvacorts salvacorts force-pushed the salvacorts/bloom-refactor/planner-progress-tracking branch from e99a459 to a0c5a5e Compare May 30, 2024 13:27
@@ -149,6 +151,8 @@ func (p *Planner) running(ctx context.Context) error {
}

case <-inflightTasksTicker.C:
// TODO(salvacorts): Is think this won't evaluate while the runOne is running.
// We should run the runOne in a goroutine and check the context inside (as we already do).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I would rather run the ticker in a new goroutine

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you! Done

Comment on lines +284 to +285
p.metrics.tenantTasksPlanned.WithLabelValues(tenant).Set(0)
p.metrics.tenantTasksCompleted.WithLabelValues(tenant).Set(0)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: should we remove the metric with the tenant label completely, instead of just resetting it to 0?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, we could then use a Counter

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we remove the metric with the tenant label completely, instead of just resetting it to 0?

AFAICT a metric cannot be explicitly removed. Can it?

we could then use a Counter

That was my first idea but a counter would make it a bit difficult to see for a given iteration, how many "new" tasks would be created. It would require using increase but setting the interval is difficult. Instead, by using a gauge, you can see the current set of tasks.

Also note that the gauge doesn't change often, but only when the build is planned.

@pull-request-size pull-request-size bot added size/M and removed size/S labels Jun 5, 2024
@salvacorts salvacorts requested a review from chaudum June 5, 2024 09:18
@salvacorts salvacorts merged commit 9289493 into main Jun 20, 2024
60 checks passed
@salvacorts salvacorts deleted the salvacorts/bloom-refactor/planner-progress-tracking branch June 20, 2024 09:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants