Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Compound filter operators with relations #1855

Merged
merged 27 commits into from
Sep 11, 2023

Conversation

islamaliev
Copy link
Contributor

Relevant issue(s)

Resolves #1808

Description

Any time a filter with _and or _or operator that includes relations is applied it will fail.
There were 2 problems:

  • filter is not recursively analysed during mapping phase which would result in some relational fields not being mapped and later panic upon attempt to access them.
  • the filter is not properly split between the scan and select nodes. It would just analyse and split top-level fields.
    These problems are fixed.

Tasks

  • I made sure the code is well commented, particularly hard-to-understand areas.
  • I made sure the repository-held documentation is changed accordingly.
  • I made sure the pull request title adheres to the conventional commit style (the subset used in the project can be found in tools/configs/chglog/config.yml).
  • I made sure to discuss its limitations such as threats to validity, vulnerability to mistake and misuse, robustness to invalidation of assumptions, resource requirements, ...

How has this been tested?

Integration tests

Specify the platform(s) on which this was tested:

  • MacOS

@islamaliev islamaliev added bug Something isn't working area/query Related to the query component labels Sep 7, 2023
@islamaliev islamaliev added this to the DefraDB v0.7 milestone Sep 7, 2023
@islamaliev islamaliev self-assigned this Sep 7, 2023
Copy link
Contributor

@AndrewSisley AndrewSisley left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've gone over everything outside the filter package, which I'll try and do later/tomorrow depending on how much time/energy I have left.

I like how you have done it from what I have seen so far, my requests are all fairly localised I think. If I haven't finished my review by Friday afternoon and someone else has approved (and you have responded to all my comments so far) please feel very free to dismiss my request for changes and merge without my approval.

planner/mapper/mapper.go Outdated Show resolved Hide resolved
planner/type_join.go Show resolved Hide resolved
planner/type_join.go Outdated Show resolved Hide resolved
planner/mapper/mapper.go Outdated Show resolved Hide resolved
planner/mapper/mapper.go Outdated Show resolved Hide resolved
tests/integration/query/one_to_many/with_filter_test.go Outdated Show resolved Hide resolved
Copy link
Contributor

@AndrewSisley AndrewSisley left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me, have gone through everything and am happy, assuming the (small/localised) comments I've left are responded to sensibly :)


// CopyField copies the given field from the provided filter.
// The result filter preserves the structure of the original filter.
func CopyField(filter *mapper.Filter, field mapper.Field) *mapper.Filter {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: This looks like it could be made private?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is actually a question: what makes a function a good candidate for being public? Is it just usage from another package? Some function are just nice utilities that can be used later.

planner/filter/external_conditions.go Outdated Show resolved Hide resolved
planner/filter/merge.go Outdated Show resolved Hide resolved
planner/filter/normalize.go Outdated Show resolved Hide resolved
@codecov
Copy link

codecov bot commented Sep 11, 2023

Codecov Report

Patch coverage: 93.82% and project coverage change: +0.28% 🎉

Comparison is base (a8c253b) 70.18% compared to head (a35edfc) 70.45%.

Impacted file tree graph

@@             Coverage Diff             @@
##           develop    #1855      +/-   ##
===========================================
+ Coverage    70.18%   70.45%   +0.28%     
===========================================
  Files          225      232       +7     
  Lines        23987    24184     +197     
===========================================
+ Hits         16833    17038     +205     
+ Misses        5996     5991       -5     
+ Partials      1158     1155       -3     
Flag Coverage Δ
all-tests 70.45% <93.82%> (+0.28%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed Coverage Δ
planner/type_join.go 74.03% <88.52%> (-1.91%) ⬇️
planner/mapper/mapper.go 86.81% <90.32%> (+1.10%) ⬆️
planner/filter/normalize.go 93.14% <93.14%> (ø)
planner/filter/copy_field.go 95.35% <95.35%> (ø)
planner/filter/complex.go 100.00% <100.00%> (ø)
planner/filter/copy.go 100.00% <100.00%> (ø)
planner/filter/merge.go 100.00% <100.00%> (ø)
planner/filter/remove_field.go 100.00% <100.00%> (ø)
planner/filter/split.go 100.00% <100.00%> (ø)
planner/mapper/targetable.go 82.89% <100.00%> (ø)

... and 6 files with indirect coverage changes


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a8c253b...a35edfc. Read the comment docs.

@islamaliev islamaliev force-pushed the fix/islam/compound-filter-with-relation branch from 222b8d4 to a35edfc Compare September 11, 2023 11:13
@islamaliev islamaliev merged commit 482be74 into develop Sep 11, 2023
16 checks passed
@islamaliev islamaliev deleted the fix/islam/compound-filter-with-relation branch September 11, 2023 13:00
shahzadlone pushed a commit to shahzadlone/defradb that referenced this pull request Feb 23, 2024
## Relevant issue(s)

Resolves sourcenetwork#1808 

## Description

Any time a filter with `_and` or `_or` operator that includes relations
is applied it will fail.
There were 2 problems:
- filter is not recursively analysed during mapping phase which would
result in some relational fields not being mapped and later panic upon
attempt to access them.
- the filter is not properly split between the scan and select nodes. It
would just analyse and split top-level fields.
These problems are fixed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/query Related to the query component bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Compound filter condition that includes a relation panics
2 participants