ENG-46252 : Pinot time range filtering support #224

anujgoyal1 · 2024-07-22T04:44:25Z

No description provided.

kotharironak · 2024-07-22T12:00:51Z

query-service/src/main/resources/configs/common/application.conf

+#         additionalTenantFilters = [
+#           {
+#             "tenantId" : "testTenant",
+#             "timeRangeAndFilters" : [


What does multiple timeRangeAndFilters mean here?
So, different filters for multiple time ranges, right?

kotharironak · 2024-07-22T12:10:38Z

query-service/src/main/resources/configs/common/application.conf

+#             "timeRangeAndFilters" : [
+#               {
+#                 "start_time_millis" : 1,
+#                 "endTimeMillis" : 1,


endTimeMillis -> end_time_millis
This will be an epoch time, right?

Yes, but I'm thinking of making start_time_millis as starTimeMillis to keep it consistent with the other keys

kotharironak · 2024-07-22T12:13:36Z

query-service/src/main/resources/configs/common/application.conf

@@ -82,6 +82,18 @@ service.config = {
      requestHandlerInfo = {
        tenantColumnName = tenant_id
        startTimeAttributeName = "EVENT.startTime"
+#         additionalTenantFilters = [


additionalTenantFilters -> scopedAdditionalFilters?
Just a thought: Do you think we might need to extend that in the future?

since it's already under a table scope, I think we not need add another scope here.
I don't think we need to further extend it to query scope(Its not even available in query as we discussed)

kotharironak · 2024-07-22T13:19:17Z

...ice-impl/src/main/java/org/hypertrace/core/query/service/AdditionalHandlerFiltersConfig.java

+      try {
+        JsonNode jsonNode = objectMapper.readTree(filterJson);
+
+        Filter.Builder personBuilder = Filter.newBuilder();


personBuilder -> fitlerBuilder

kotharironak · 2024-07-22T13:19:42Z

query-service-impl/src/main/java/org/hypertrace/core/query/service/ExecutionContext.java

@@ -3,9 +3,6 @@
 import static org.hypertrace.core.query.service.QueryFunctionConstants.QUERY_FUNCTION_COUNT;
 import static org.hypertrace.core.query.service.QueryRequestUtil.getAlias;
 import static org.hypertrace.core.query.service.QueryRequestUtil.getLogicalColumnName;
-import static org.hypertrace.core.query.service.api.Expression.ValueCase.ATTRIBUTE_EXPRESSION;


Did we move this some other place?

kotharironak · 2024-07-22T13:21:17Z

Can we add some tests?

kotharironak · 2024-07-24T10:13:14Z

query-service-impl/src/test/resources/application.conf

+              {
+                "startTimeMillis" = 1
+                "endTimeMillis" = 2
+                "filter" : "{\r\n  \"lhs\": {\r\n    \"columnIdentifier\": {\r\n      \"columnName\": \"SERVICE.name\"\r\n    }\r\n  },\r\n  \"operator\": \"EQ\",\r\n  \"rhs\": {\r\n    \"literal\": {\r\n      \"value\": {\r\n        \"string\": \"dummyService\"\r\n      }\r\n    }\r\n  }\r\n}"


Q: Do we need \r\n? If we just put a valid single line json string, would it work?

Yeah, it would work. Was just added by the json string escaper I was using, got rid of it.

kotharironak · 2024-07-24T10:13:49Z

query-service-impl/src/test/resources/application.conf

+                    "tenantId" = "__default"
+                    "timeRangeAndFilters" = [
+                      {
+                        "filter" : "{\r\n  \"lhs\": {\r\n    \"columnIdentifier\": {\r\n      \"columnName\": \"SERVICE.name\"\r\n    }\r\n  },\r\n  \"operator\": \"EQ\",\r\n  \"rhs\": {\r\n    \"literal\": {\r\n      \"value\": {\r\n        \"string\": \"dummyService\"\r\n      }\r\n    }\r\n  }\r\n}"


Do we need \r\n? If we just put a single string, would it work?

kotharironak · 2024-07-24T10:24:12Z

...ice-impl/src/main/java/org/hypertrace/core/query/service/AdditionalHandlerFiltersConfig.java

+      }
+    }
+
+    private Optional<Filter> buildTimeRangeAndFilters(Optional<String> timeRangeAttributeOptional) {


nit: timeRangeAttributeOptional -> timeRangeAttribute

kotharironak · 2024-07-24T10:31:32Z

...impl/src/test/java/org/hypertrace/core/query/service/pinot/PinotBasedRequestHandlerTest.java

+              config.getString("name"), config.getConfig("requestHandlerInfo"));
+
+      // domain-service-handler only has filter and no time range
+      if (config.getString("name").equals("domain-service-handler")) {


Is this test using the correct handler - domain-service-handler? For this, the original request should be the same after applying filters, right?

I think, it should have been - trace-view-handler, right?

domain-event-view only has filter.
span-event-view only has time range
trace-view has both time range and filter.

kotharironak · 2024-07-24T10:40:27Z

...ice-impl/src/main/java/org/hypertrace/core/query/service/AdditionalHandlerFiltersConfig.java

+        if (this.startTimeMillis.isPresent() && this.endTimeMillis.isPresent()) {
+          filterBuilder.addChildFilter(
+              QueryRequestUtil.createLongFilter(
+                  timeRangeAttributeOptional.get(), Operator.GT, this.endTimeMillis.get()));


Q: Is this check correct? Should endTime be compared with the greater than operator (I don't remember the text you had in your sublime, so just re-checking)?

Yes this is how it's supposed to be as it's an or filter
This is what we discussed.

t3.... t1 t2 ....t4 ((s >= t2 OR s <= t1) OR (API NOT IN (api1, api2)) AND (t3, t4))

satish-mittal · 2024-07-25T05:30:34Z

query-service-impl/src/main/java/org/hypertrace/core/query/service/RequestHandler.java

@@ -27,4 +27,9 @@ public interface RequestHandler {

  /** Handle the request and add rows to the collector. */
  Observable<Row> handleRequest(QueryRequest request, ExecutionContext executionContext);
+
+  default QueryRequest applyAdditionalFilters(


There is already an existing step of query transformation. Look at QueryTransformation. This step of adding filters should go there.
https://github.com/hypertrace/query-service/blob/main/query-service-impl/src/main/java/org/hypertrace/core/query/service/QueryTransformation.java

We started from this, then realised we don't get scope information in query request.
If we apply a filter on Transformation step, it'll be applied on all the scopes and would work as global filter. Which is wrong behaviour
Query Service decides the scope and it's handler based of the query. Hence pushed down these filters to handler

...ice-impl/src/main/java/org/hypertrace/core/query/service/AdditionalHandlerFiltersConfig.java

query-service/src/main/resources/configs/common/application.conf

satish-mittal · 2024-07-25T05:59:58Z

...ice-impl/src/main/java/org/hypertrace/core/query/service/AdditionalHandlerFiltersConfig.java

+
+    private Optional<Filter> buildTimeRangeAndFilters(Optional<String> timeRangeAttribute) {
+      Filter.Builder filterBuilder = Filter.newBuilder();
+      filterBuilder.setOperator(Operator.OR);


why OR? don't we want the filter to be: time > startTime and time < endTime?

#224 (comment)

...ice-impl/src/main/java/org/hypertrace/core/query/service/AdditionalHandlerFiltersConfig.java

satish-mittal · 2024-07-25T06:30:13Z

query-service-impl/src/test/resources/application.conf

@@ -39,6 +39,18 @@ service.config = {
        startTimeAttributeName = "Trace.start_time_millis"
        slowQueryThresholdMs = 0
        percentileAggFunction = "PERCENTILE"
+        additionalTenantFilters = [


There are multiple problems in the way this new config has been defined:

It is is very confusing. Only after going through the code one can understand.

There is a hardcoded assumption that startTimeAttributeName will act as the lhs key for the 2 values below. Makes it very specific and no room for generalising.

It is hardcoded with OR operator.

I would prefer it being defined as follows:

additionalHandlerConfigs = [ operator = "OR" { key = "Trace.start_time_millis" operator = LT value = "max_cutoff" }, { key = "Trace.start_time_millis" operator = GE value = "min_cutoff" }, filter : { columnIdentifiier: ... } ]

Would operator = "AND" possible here?

We would like to apply the time range to startTimeAttributeName. Any other attribute can go to the normal filter expression.

The given time range indicates that we would like to apply a filter within that specified range. is that not clear?

kotharironak

lgtm

kotharironak

LGTM

github-actions · 2024-07-26T07:40:03Z

Unit Test Results

  42 files +2   42 suites +2 11s ⏱️ -1s
325 tests +5 325 ✔️ +5 0 💤 ±0 0 ❌ ±0

Results for commit d136696. ± Comparison against base commit 13e6ccf.

Pinot time range filtering support

accc648

anujgoyal1 changed the title ~~Pinot time range filtering support~~ ENG-46252 : Pinot time range filtering support Jul 22, 2024

kotharironak reviewed Jul 22, 2024

View reviewed changes

Added tests

4a43d00

This comment has been minimized.

Sign in to view

rever execution context

119af0b

anujgoyal1 requested a review from kotharironak July 24, 2024 04:09

fix comment

84b5238

anujgoyal1 marked this pull request as ready for review July 24, 2024 06:41

anujgoyal1 requested a review from a team as a code owner July 24, 2024 06:41

This comment has been minimized.

Sign in to view

kotharironak reviewed Jul 24, 2024

View reviewed changes

anujgoyal1 added 2 commits July 24, 2024 16:12

comment

1b21673

fix test string

3394efc

This comment has been minimized.

Sign in to view

kotharironak approved these changes Jul 24, 2024

View reviewed changes

kotharironak previously approved these changes Jul 24, 2024

View reviewed changes

commit

3bcdc1f

anujgoyal1 dismissed kotharironak’s stale review via 3bcdc1f July 24, 2024 15:30

This comment has been minimized.

Sign in to view

kotharironak previously approved these changes Jul 24, 2024

View reviewed changes

dummy commit

f85068d

anujgoyal1 dismissed kotharironak’s stale review via f85068d July 25, 2024 05:26

satish-mittal reviewed Jul 25, 2024

View reviewed changes

This comment has been minimized.

Sign in to view

satish-mittal reviewed Jul 25, 2024

View reviewed changes

...ice-impl/src/main/java/org/hypertrace/core/query/service/AdditionalHandlerFiltersConfig.java Outdated Show resolved Hide resolved

satish-mittal reviewed Jul 25, 2024

View reviewed changes

query-service/src/main/resources/configs/common/application.conf Outdated Show resolved Hide resolved

satish-mittal reviewed Jul 25, 2024

View reviewed changes

...ice-impl/src/main/java/org/hypertrace/core/query/service/AdditionalHandlerFiltersConfig.java Outdated Show resolved Hide resolved

satish-mittal reviewed Jul 25, 2024

View reviewed changes

...ice-impl/src/main/java/org/hypertrace/core/query/service/AdditionalHandlerFiltersConfig.java Outdated Show resolved Hide resolved

satish-mittal reviewed Jul 25, 2024

View reviewed changes

...ice-impl/src/main/java/org/hypertrace/core/query/service/AdditionalHandlerFiltersConfig.java Outdated Show resolved Hide resolved

dummy commit

195b0f7

satish-mittal reviewed Jul 25, 2024

View reviewed changes

This comment has been minimized.

Sign in to view

hypertrace deleted a comment from kotharironak Jul 25, 2024

Added query filter string generator

cbe7f2f

This comment has been minimized.

Sign in to view

kotharironak reviewed Jul 25, 2024

View reviewed changes

kotharironak previously approved these changes Jul 25, 2024

View reviewed changes

renaming

1e8ed16

anujgoyal1 dismissed kotharironak’s stale review via 1e8ed16 July 25, 2024 11:33

This comment has been minimized.

Sign in to view

Handler filter config test

a58461b

This comment has been minimized.

Sign in to view

anujgoyal1 requested a review from kotharironak July 26, 2024 04:56

kotharironak approved these changes Jul 26, 2024

View reviewed changes

anujgoyal1 merged commit d136696 into main Jul 26, 2024
6 checks passed

anujgoyal1 deleted the additionalFilteringSupport branch July 26, 2024 07:33

ENG-46252 : Pinot time range filtering support #224

ENG-46252 : Pinot time range filtering support #224

Conversation

anujgoyal1 commented Jul 22, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kotharironak commented Jul 22, 2024

This comment has been minimized.

This comment has been minimized.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

satish-mittal Jul 25, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment has been minimized.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment has been minimized.

This comment has been minimized.

kotharironak left a comment

Choose a reason for hiding this comment

This comment has been minimized.

This comment has been minimized.

kotharironak left a comment

Choose a reason for hiding this comment

github-actions bot commented Jul 26, 2024

Unit Test Results

satish-mittal Jul 25, 2024 •

edited

Loading