Support file pruning through JsonPredicates and predicates queryRequest parameters #130

duhizjame · 2023-11-03T18:00:06Z

No description provided.

server/src/main/java/io/whitefox/core/EvalHelper.java

server/src/main/java/io/whitefox/core/JsonPredicates.java

server/src/main/java/io/whitefox/core/EvalHelper.java

server/src/main/java/io/whitefox/core/JsonPredicates.java

server/src/main/java/io/whitefox/core/services/DeltaSharedTable.java

tmnd1991

Still some work to do (I know, this PR is huge!) but very good job until here

server/src/main/java/io/whitefox/core/JsonPredicatesUtils.java

server/src/main/java/io/whitefox/core/services/DeltaSharedTable.java

server/src/main/java/io/whitefox/core/types/predicates/LeafOp.java

server/src/main/java/io/whitefox/core/types/predicates/EvalHelper.java

server/src/main/java/io/whitefox/core/types/predicates/LeafOpDeserializer.java

server/core/src/main/java/io/whitefox/core/ColumnRange.java

buildSrc/src/main/kotlin/whitefox.java-conventions.gradle.kts

server/core/build.gradle.kts

server/app/src/test/java/io/whitefox/api/deltasharing/server/DeltaSharesApiImplTest.java

server/core/src/main/java/io/whitefox/core/types/predicates/BasePrimitiveTypeNames.java

server/core/src/main/java/io/whitefox/core/types/predicates/LeafEvaluationResult.java

duhizjame · 2024-01-30T01:05:52Z

refactored a lot of stuff based on the comments;
notable change: INTEGER type is now formed from string "int" instead of "integer" to match the types from the JsonPredicateUtils
all other changes are mostly refactoring for readability and deduplication of code

agilelab-tmnd1991 · 2024-01-30T16:04:23Z

refactored a lot of stuff based on the comments; notable change: INTEGER type is now formed from string "int" instead of "integer" to match the types from the JsonPredicateUtils all other changes are mostly refactoring for readability and deduplication of code

Is this compliant with the spec? https://github.com/delta-io/delta-sharing/blob/main/PROTOCOL.md#primitive-types

duhizjame · 2024-01-30T16:17:18Z

it should be; based on the actual protocol update in #203(line 585 in the delta protocol) + even delta standalone refers to this type by its short name
io/delta/standalone/types/IntegerType.java

all other usages of IntegerType(the io.whitefox.types one) is our internal usage

…-and-jsonpredicatehints-in-querytable-api

because it's me from another account

duhizjame linked an issue Nov 3, 2023 that may be closed by this pull request

Support parameters predicateHints and jsonPredicateHints in QueryTable API #93

Closed

tmnd1991 self-requested a review November 6, 2023 08:30

tmnd1991 assigned duhizjame Nov 6, 2023

tmnd1991 reviewed Nov 6, 2023

View reviewed changes

tmnd1991 previously requested changes Dec 1, 2023

View reviewed changes

duhizjame force-pushed the feature/93-support-parameters-predicatehints-and-jsonpredicatehints-in-querytable-api branch from ca5bb6c to 9553187 Compare December 8, 2023 19:58

tmnd1991 reviewed Dec 15, 2023

View reviewed changes

server/core/src/main/java/io/whitefox/core/ColumnRange.java Outdated Show resolved Hide resolved

server/core/src/main/java/io/whitefox/core/ColumnRange.java Outdated Show resolved Hide resolved

duhizjame force-pushed the feature/93-support-parameters-predicatehints-and-jsonpredicatehints-in-querytable-api branch from dadbfdc to fa40ab3 Compare December 16, 2023 14:22

tmnd1991 reviewed Dec 19, 2023

View reviewed changes

server/core/src/main/java/io/whitefox/core/ColumnRange.java Outdated Show resolved Hide resolved

server/core/src/main/java/io/whitefox/core/ColumnRange.java Outdated Show resolved Hide resolved

duhizjame force-pushed the feature/93-support-parameters-predicatehints-and-jsonpredicatehints-in-querytable-api branch from eba1ae8 to 429e8a9 Compare December 28, 2023 01:59

duhizjame force-pushed the feature/93-support-parameters-predicatehints-and-jsonpredicatehints-in-querytable-api branch 2 times, most recently from 9fc7980 to 9948574 Compare January 19, 2024 02:50

agilelab-tmnd1991 requested changes Jan 29, 2024

View reviewed changes

duhizjame added 17 commits January 29, 2024 23:22

wip-first draft of JsonPredicates and a test partitioned table

0f0a847

wip - working parsing

1e085cd

wip - partition columns filterable

b33ac8b

formatting

fc4d54c

exception tree

e877d72

naming

038ea31

wip

372e500

wip - draft column range evaluation

5d808f3

typed eval result

f06dc81

Handle non-existing columns in predicate; formatting

0549324

sql filters parsing

2b4597c

wip- sql predicates without error handling

4534458

protocol not compliant

9ad58d9

final protocol align

8d1d413

fixed and added tests

03ebecf

fix test v2

5a63e34

fixed integration test

58ea7bd

duhizjame added 11 commits January 29, 2024 23:24

correct json body

125cef7

removed not needed test

4cf584b

exception tests

bdcd60b

disable some tests on windows

f5a076a

more tests

8f3c7e5

more tests and coverage printer

8c79a25

more tests + spotless

79179fc

disabled test on windows

9322aa0

primitive type names simplified

9807fd1

return types refactoring

fdcb954

refactor everything to ColumnRange

8f18c09

duhizjame force-pushed the feature/93-support-parameters-predicatehints-and-jsonpredicatehints-in-querytable-api branch from 26e9791 to 8f18c09 Compare January 30, 2024 00:06

exception handling from supplied function

2460a95

updated jsqlparser package

e5721ac

Merge branch 'main' into feature/93-support-parameters-predicatehints…

09791b2

…-and-jsonpredicatehints-in-querytable-api

agilelab-tmnd1991 approved these changes Jan 31, 2024

View reviewed changes

agilelab-tmnd1991 changed the title ~~wip-first draft of JsonPredicates and a test partitioned table~~ Support file pruning through JsonPredicates and predicates queryRequest parameters Jan 31, 2024

agilelab-tmnd1991 merged commit 94aefa6 into main Jan 31, 2024

agilelab-tmnd1991 deleted the feature/93-support-parameters-predicatehints-and-jsonpredicatehints-in-querytable-api branch January 31, 2024 10:38

Support file pruning through JsonPredicates and predicates queryRequest parameters #130

Support file pruning through JsonPredicates and predicates queryRequest parameters #130

Uh oh!

Conversation

duhizjame commented Nov 3, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tmnd1991 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

duhizjame commented Jan 30, 2024

Uh oh!

agilelab-tmnd1991 commented Jan 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

duhizjame commented Jan 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

agilelab-tmnd1991 commented Jan 30, 2024 •

edited

Loading

duhizjame commented Jan 30, 2024 •

edited

Loading