Pushdown Prometheus Predicates #8742 #25957

jirislav · 2025-06-09T00:37:43Z

Description

Implements pushdown for Prometheus predicates, as requested in #8742.

Additional context and related issues

Release notes

( ) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
(x) Release notes are required, with the following suggested text:

## Prometheus Connector
* Support predicate pushdown for equality, non-equality, LIKE & IN operators. (#8742)

cla-bot · 2025-06-09T00:37:47Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2025-06-09T08:04:27Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2025-06-09T10:12:22Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

plugin/trino-prometheus/src/main/java/io/trino/plugin/prometheus/expression/RewriteIn.java

plugin/trino-prometheus/src/test/java/io/trino/plugin/prometheus/LoggedRequest.java

cla-bot · 2025-06-09T13:24:46Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

jirislav · 2025-06-09T13:27:25Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

I have the CLA signed and sent like 6 hours ago, not sure how long it usually takes to propagate to the related failing CI job.

jirislav · 2025-06-24T19:05:54Z

@ebyhr could you please review & merge? I see you made recent changes to Prometheus Connector as well.

ebyhr

Just skimmed.

...in/trino-prometheus/src/main/java/io/trino/plugin/prometheus/expression/RewriteEqAndNeq.java

plugin/trino-prometheus/src/main/java/io/trino/plugin/prometheus/expression/RewriteHelpers.java

plugin/trino-prometheus/src/main/java/io/trino/plugin/prometheus/PrometheusTableHandle.java

ebyhr · 2025-06-24T21:58:47Z

plugin/trino-prometheus/src/test/java/io/trino/plugin/prometheus/TestPrometheusIntegration.java

+            assertThat(request.extractPromQL()).isEqualTo("up{job=\"prometheus\"}[1d]");
+        });
+
+        assertThat(results).hasSizeBetween(1, 2);


Why is the result size non-deterministic? Please leave a code comment.

Prometheus reports the up metric every 15 seconds, so we expect at least one row, but at most two rows if we filter for the last 15 seconds (edge case). Thanks for asking, I have added a code comment.

plugin/trino-prometheus/src/test/java/io/trino/plugin/prometheus/TestPrometheusIntegration.java

plugin/trino-prometheus/src/test/java/io/trino/plugin/prometheus/TestPrometheusTableHandle.java

plugin/trino-prometheus/src/test/java/io/trino/plugin/prometheus/TestPrometheusIntegration.java

plugin/trino-prometheus/src/test/java/io/trino/plugin/prometheus/PrometheusServer.java

chenjian2664 · 2025-06-25T06:56:54Z

plugin/trino-prometheus/src/test/java/io/trino/plugin/prometheus/PrometheusServer.java

    private static final int PROMETHEUS_PORT = 9090;
-    private static final String DEFAULT_VERSION = "v2.15.1";


Does this changes required?

Your question is ambiguous, what change exactly?

If you mean the change of scope from private to public, then yes, we're using this constant in the TestPrometheusIntegration class.

If you mean the change of name from DEFAULT_VERSION to DEFAULT_PROMETHEUS_VERSION, then it's required to avoid ambiguity of what container exactly belongs this version to (prometheus vs. nginx).

plugin/trino-prometheus/src/main/java/io/trino/plugin/prometheus/expression/RewriteHelpers.java

plugin/trino-prometheus/src/test/java/io/trino/plugin/prometheus/RequestsRecorder.java

plugin/trino-prometheus/src/main/java/io/trino/plugin/prometheus/PrometheusTableHandle.java

plugin/trino-prometheus/src/main/java/io/trino/plugin/prometheus/PrometheusSplitManager.java

chenjian2664 · 2025-06-27T12:57:04Z

plugin/trino-prometheus/src/main/java/io/trino/plugin/prometheus/PrometheusMetadata.java

        PrometheusTableHandle tableHandle = ((PrometheusTableHandle) handle)
-                .withPredicate(constraint.getSummary());
+                .withPredicate(constraint.getSummary())
+                .withExpressions(rewriter.rewrite(session, expression, constraint.getAssignments()).orElseGet(ImmutableList::of));
        return Optional.of(new ConstraintApplicationResult<>(tableHandle, constraint.getSummary(), constraint.getExpression(), false));


After updating tableHandle , it looks like remainingFilter and remainingExpression still reference the previous state. Is that intentional?

Yes, this is intentional because we can't push-down arbitrary operation for the labels map (see the testLabelLikeAndEqAndNeqNegatedPushDown test to see why - though it's just a single edge-case example).

Yes, Trino will potentially filter something that was already pushed down, but the real value of this PR is that Prometheus doesn't have to return all the timeseries anymore and Trino doesn't have to filter them all. From now, Prometheus will always perform very efficient timeseries filtering based on provided label selectors and Trino will go just through a fraction of data compared to previous version.

In the future, someone might add support for the OR operator, or NOT(x AND y) by spitting the query into multiple Prometheus requests, potentially covering all the logical edge-cases for label filter pushdown. At that point, I think we could safely remove all the filters on label values in the above discussed lines, but I'd argue it's out-of-scope for this PR, as this itself is a major improvement already.

plugin/trino-prometheus/src/main/java/io/trino/plugin/prometheus/expression/RewriteHelpers.java

plugin/trino-prometheus/src/main/java/io/trino/plugin/prometheus/expression/RewriteIn.java

To inspect compilation warnings, see https://github.com/trinodb/trino/actions/runs/15529824872/job/43716197311?pr=25957#step:5:149.

github-actions bot added docs prometheus Prometheus connector labels Jun 9, 2025

jirislav mentioned this pull request Jun 9, 2025

Trino-Prometheus connector: Enable filter labels pushdown to the api/v1/query endpoint in addition to the timestamp and values filters #8742

Open

jirislav force-pushed the pushdown-promql-label-filter branch from 01a2274 to 62d8bd7 Compare June 9, 2025 08:04

jirislav force-pushed the pushdown-promql-label-filter branch from 62d8bd7 to dc6eaf4 Compare June 9, 2025 10:12

etolbakov reviewed Jun 9, 2025

View reviewed changes

plugin/trino-prometheus/src/main/java/io/trino/plugin/prometheus/expression/RewriteIn.java Outdated Show resolved Hide resolved

etolbakov reviewed Jun 9, 2025

View reviewed changes

plugin/trino-prometheus/src/test/java/io/trino/plugin/prometheus/LoggedRequest.java Outdated Show resolved Hide resolved

jirislav force-pushed the pushdown-promql-label-filter branch from dc6eaf4 to b2a6f1f Compare June 9, 2025 13:24

jirislav force-pushed the pushdown-promql-label-filter branch from b2a6f1f to 6028e80 Compare June 23, 2025 18:58

cla-bot bot added the cla-signed label Jun 23, 2025

jirislav assigned ebyhr Jun 24, 2025

ebyhr removed their assignment Jun 24, 2025

ebyhr reviewed Jun 24, 2025

View reviewed changes

chenjian2664 reviewed Jun 25, 2025

View reviewed changes

jirislav force-pushed the pushdown-promql-label-filter branch 4 times, most recently from a666bd4 to 6c3d0bc Compare June 27, 2025 11:04

chenjian2664 reviewed Jun 27, 2025

View reviewed changes

jirislav added 2 commits June 27, 2025 16:49

Push-Down Prometheus Predicates trinodb#8742

310de2e

Resolve all compilation warnings for Prometheus Connector

3b8a38d

To inspect compilation warnings, see https://github.com/trinodb/trino/actions/runs/15529824872/job/43716197311?pr=25957#step:5:149.

jirislav force-pushed the pushdown-promql-label-filter branch from 6c3d0bc to 3b8a38d Compare June 27, 2025 14:49

		private static final int PROMETHEUS_PORT = 9090;
		private static final String DEFAULT_VERSION = "v2.15.1";

Pushdown Prometheus Predicates #8742 #25957

Are you sure you want to change the base?

Pushdown Prometheus Predicates #8742 #25957

Uh oh!

Conversation

jirislav commented Jun 9, 2025

Description

Additional context and related issues

Release notes

Uh oh!

cla-bot bot commented Jun 9, 2025

Uh oh!

cla-bot bot commented Jun 9, 2025

Uh oh!

cla-bot bot commented Jun 9, 2025

Uh oh!

Uh oh!

Uh oh!

cla-bot bot commented Jun 9, 2025

Uh oh!

jirislav commented Jun 9, 2025

Uh oh!

jirislav commented Jun 24, 2025

Uh oh!

ebyhr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ebyhr Jun 24, 2025

Choose a reason for hiding this comment

Uh oh!

jirislav Jun 26, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chenjian2664 Jun 25, 2025

Choose a reason for hiding this comment

Uh oh!

jirislav Jun 26, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chenjian2664 Jun 27, 2025

Choose a reason for hiding this comment

Uh oh!

jirislav Jun 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jirislav Jun 27, 2025 •

edited

Loading