Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

resolves #1671: match index hints in a query #1638

Merged
merged 25 commits into from
May 14, 2022
Merged

resolves #1671: match index hints in a query #1638

merged 25 commits into from
May 14, 2022

Conversation

pengpeng-lu
Copy link
Contributor

@pengpeng-lu pengpeng-lu commented May 3, 2022 •
edited

  • This PR is to support index hint in a query
  • Add index info to FullUnorderedScanExpression, then when match candidates, the query can be subsumed by a candidate only when the index is included in the hint indexes.

@foundationdb-ci
Copy link
Contributor

AWS CodeBuild CI Report for Linux CentOS 7

  • CodeBuild project: fdb-record-layer-pr-proto3
  • Commit ID: b202230
  • Result: FAILED
  • Error: Error while executing command: ./gradlew --no-daemon --console=plain -b ./build.gradle build destructiveTest sonarqube -PcoreNotStrict -PreleaseBuild=false -PpublishBuild=false -PspotbugsEnableHtmlReport. Reason: exit status 1
  • Build Logs (available for 30 days)

@foundationdb-ci
Copy link
Contributor

AWS CodeBuild CI Report for Linux CentOS 7

  • CodeBuild project: fdb-record-layer-pr-proto2
  • Commit ID: b202230
  • Result: FAILED
  • Error: Error while executing command: ./gradlew --no-daemon --console=plain -b ./build.gradle build destructiveTest sonarqube -PcoreNotStrict -PreleaseBuild=false -PpublishBuild=false -PspotbugsEnableHtmlReport. Reason: exit status 1
  • Build Logs (available for 30 days)

@foundationdb-ci
Copy link
Contributor

AWS CodeBuild CI Report for Linux CentOS 7

  • CodeBuild project: fdb-record-layer-pr-proto2
  • Commit ID: 225021f
  • Result: FAILED
  • Error: Error while executing command: ./gradlew --no-daemon --console=plain -b ./build.gradle build destructiveTest sonarqube -PcoreNotStrict -PreleaseBuild=false -PpublishBuild=false -PspotbugsEnableHtmlReport. Reason: exit status 1
  • Build Logs (available for 30 days)

@foundationdb-ci
Copy link
Contributor

AWS CodeBuild CI Report for Linux CentOS 7

  • CodeBuild project: fdb-record-layer-pr-proto3
  • Commit ID: 225021f
  • Result: FAILED
  • Error: Error while executing command: ./gradlew --no-daemon --console=plain -b ./build.gradle build destructiveTest sonarqube -PcoreNotStrict -PreleaseBuild=false -PpublishBuild=false -PspotbugsEnableHtmlReport. Reason: exit status 1
  • Build Logs (available for 30 days)

@pengpeng-lu pengpeng-lu marked this pull request as ready for review May 3, 2022 23:19
@pengpeng-lu pengpeng-lu changed the title save Match index hints in a query May 3, 2022
normen662
normen662 requested changes May 4, 2022
View reviewed changes
Copy link
Contributor

@normen662 normen662 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi,

the major code path is exactly as discussed. Nice! Please do work on these two bullets:

  • setting the name of the index into the fuse of the MatchCandidates when they get created. That name is available but needs to be threaded into the right place(s).
  • test cases! Please add (at least) a query test case that demonstrates the hinting mechanism, i.e. even though a better index is available the worse one wins because it is hinted. Look into FDBSimpleQueryGraphTest for inspiration on how to do that.

...yer-core/src/main/java/com/apple/foundationdb/record/query/plan/cascades/MatchCandidate.java Outdated
@@ -325,7 +326,9 @@ static Optional<MatchCandidate> fromPrimaryDefinition(@Nonnull final RecordMetaD
@Nonnull
static GroupExpressionRef<RelationalExpression> createBaseRef(@Nonnull RecordMetaData metaData, @Nonnull final Set<String> allAvailableRecordTypes, @Nonnull final Set<String> recordTypesForIndex) {
final var quantifier =
Quantifier.forEach(GroupExpressionRef.of(new FullUnorderedScanExpression(allAvailableRecordTypes)));
Quantifier.forEach(GroupExpressionRef.of(new FullUnorderedScanExpression(allAvailableRecordTypes, metaData.getAllIndexes().stream()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be a singleton set containing just the name of the index this particular MatchCandidate represents.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You will need to pass the name of the index this expansion is for to the expansion visitors that create the MatchCandidate. That name should be passed in here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a little confused here: we can getRecordTypes from both MetaDataPlanContext and RecordMetaData, and the result seems a bit different for these two. Which one should we use in new FullUnorderedScanExpression? Also we can getIndexNames for both, which one should we use?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ones in MetaDataPlanContext are derived from the query being planned. So, the record types being queried ("FROM clause") and the allowed indexes ("INDEX hints"). The ones in RecordMetaData are the ones in the schema.

...re/src/main/java/com/apple/foundationdb/record/query/plan/cascades/RelationalExpression.java Outdated Show resolved Hide resolved
...m/apple/foundationdb/record/query/plan/cascades/expressions/FullUnorderedScanExpression.java Outdated Show resolved Hide resolved
@foundationdb-ci
Copy link
Contributor

AWS CodeBuild CI Report for Linux CentOS 7

  • CodeBuild project: fdb-record-layer-pr-proto3
  • Commit ID: c66bfd7
  • Result: FAILED
  • Error: Error while executing command: ./gradlew --no-daemon --console=plain -b ./build.gradle build destructiveTest sonarqube -PcoreNotStrict -PreleaseBuild=false -PpublishBuild=false -PspotbugsEnableHtmlReport. Reason: exit status 1
  • Build Logs (available for 30 days)

@foundationdb-ci
Copy link
Contributor

AWS CodeBuild CI Report for Linux CentOS 7

  • CodeBuild project: fdb-record-layer-pr-proto2
  • Commit ID: c66bfd7
  • Result: FAILED
  • Error: Error while executing command: ./gradlew --no-daemon --console=plain -b ./build.gradle build destructiveTest sonarqube -PcoreNotStrict -PreleaseBuild=false -PpublishBuild=false -PspotbugsEnableHtmlReport. Reason: exit status 1
  • Build Logs (available for 30 days)

@foundationdb-ci
Copy link
Contributor

AWS CodeBuild CI Report for Linux CentOS 7

  • CodeBuild project: fdb-record-layer-pr-proto2
  • Commit ID: 155c2a0
  • Result: FAILED
  • Error: Error while executing command: ./gradlew --no-daemon --console=plain -b ./build.gradle build destructiveTest sonarqube -PcoreNotStrict -PreleaseBuild=false -PpublishBuild=false -PspotbugsEnableHtmlReport. Reason: exit status 1
  • Build Logs (available for 30 days)

@foundationdb-ci
Copy link
Contributor

AWS CodeBuild CI Report for Linux CentOS 7

  • CodeBuild project: fdb-record-layer-pr-proto3
  • Commit ID: 155c2a0
  • Result: FAILED
  • Error: Error while executing command: ./gradlew --no-daemon --console=plain -b ./build.gradle build destructiveTest sonarqube -PcoreNotStrict -PreleaseBuild=false -PpublishBuild=false -PspotbugsEnableHtmlReport. Reason: exit status 1
  • Build Logs (available for 30 days)

@foundationdb-ci
Copy link
Contributor

AWS CodeBuild CI Report for Linux CentOS 7

  • CodeBuild project: fdb-record-layer-pr-proto2
  • Commit ID: 0afb276
  • Result: FAILED
  • Error: Error while executing command: ./gradlew --no-daemon --console=plain -b ./build.gradle build destructiveTest sonarqube -PcoreNotStrict -PreleaseBuild=false -PpublishBuild=false -PspotbugsEnableHtmlReport. Reason: exit status 1
  • Build Logs (available for 30 days)

@foundationdb-ci
Copy link
Contributor

AWS CodeBuild CI Report for Linux CentOS 7

  • CodeBuild project: fdb-record-layer-pr-proto3
  • Commit ID: 0afb276
  • Result: FAILED
  • Error: Error while executing command: ./gradlew --no-daemon --console=plain -b ./build.gradle build destructiveTest sonarqube -PcoreNotStrict -PreleaseBuild=false -PpublishBuild=false -PspotbugsEnableHtmlReport. Reason: exit status 1
  • Build Logs (available for 30 days)

@foundationdb-ci
Copy link
Contributor

AWS CodeBuild CI Report for Linux CentOS 7

  • CodeBuild project: fdb-record-layer-pr-proto2
  • Commit ID: f875a83
  • Result: SUCCEEDED
  • Error: N/A
  • Build Logs (available for 30 days)

@foundationdb-ci
Copy link
Contributor

AWS CodeBuild CI Report for Linux CentOS 7

  • CodeBuild project: fdb-record-layer-pr-proto3
  • Commit ID: f875a83
  • Result: SUCCEEDED
  • Error: N/A
  • Build Logs (available for 30 days)

@foundationdb-ci
Copy link
Contributor

AWS CodeBuild CI Report for Linux CentOS 7

  • CodeBuild project: fdb-record-layer-pr-proto2
  • Commit ID: e01a1e8
  • Result: FAILED
  • Error: Error while executing command: ./gradlew --no-daemon --console=plain -b ./build.gradle build destructiveTest sonarqube -PcoreNotStrict -PreleaseBuild=false -PpublishBuild=false -PspotbugsEnableHtmlReport. Reason: exit status 1
  • Build Logs (available for 30 days)

@foundationdb-ci
Copy link
Contributor

AWS CodeBuild CI Report for Linux CentOS 7

  • CodeBuild project: fdb-record-layer-pr-proto3
  • Commit ID: e01a1e8
  • Result: FAILED
  • Error: Error while executing command: ./gradlew --no-daemon --console=plain -b ./build.gradle build destructiveTest sonarqube -PcoreNotStrict -PreleaseBuild=false -PpublishBuild=false -PspotbugsEnableHtmlReport. Reason: exit status 1
  • Build Logs (available for 30 days)

@foundationdb-ci
Copy link
Contributor

AWS CodeBuild CI Report for Linux CentOS 7

  • CodeBuild project: fdb-record-layer-pr-proto2
  • Commit ID: 46858e6
  • Result: SUCCEEDED
  • Error: N/A
  • Build Logs (available for 30 days)

@foundationdb-ci
Copy link
Contributor

AWS CodeBuild CI Report for Linux CentOS 7

  • CodeBuild project: fdb-record-layer-pr-proto2
  • Commit ID: 4b56d98
  • Result: SUCCEEDED
  • Error: N/A
  • Build Logs (available for 30 days)

@foundationdb-ci
Copy link
Contributor

AWS CodeBuild CI Report for Linux CentOS 7

  • CodeBuild project: fdb-record-layer-pr-proto3
  • Commit ID: 4b56d98
  • Result: SUCCEEDED
  • Error: N/A
  • Build Logs (available for 30 days)

@foundationdb-ci
Copy link
Contributor

AWS CodeBuild CI Report for Linux CentOS 7

  • CodeBuild project: fdb-record-layer-pr-proto2
  • Commit ID: 55a2a41
  • Result: SUCCEEDED
  • Error: N/A
  • Build Logs (available for 30 days)

@foundationdb-ci
Copy link
Contributor

AWS CodeBuild CI Report for Linux CentOS 7

  • CodeBuild project: fdb-record-layer-pr-proto3
  • Commit ID: 55a2a41
  • Result: SUCCEEDED
  • Error: N/A
  • Build Logs (available for 30 days)

@foundationdb-ci
Copy link
Contributor

AWS CodeBuild CI Report for Linux CentOS 7

  • CodeBuild project: fdb-record-layer-pr-proto2
  • Commit ID: 9fca0c7
  • Result: SUCCEEDED
  • Error: N/A
  • Build Logs (available for 30 days)

@foundationdb-ci
Copy link
Contributor

AWS CodeBuild CI Report for Linux CentOS 7

  • CodeBuild project: fdb-record-layer-pr-proto3
  • Commit ID: 9fca0c7
  • Result: SUCCEEDED
  • Error: N/A
  • Build Logs (available for 30 days)

@normen662 normen662 changed the title Match index hints in a query resolves #1671: match index hints in a query May 14, 2022
@foundationdb-ci
Copy link
Contributor

AWS CodeBuild CI Report for Linux CentOS 7

  • CodeBuild project: fdb-record-layer-pr-proto2
  • Commit ID: c4d86e6
  • Result: SUCCEEDED
  • Error: N/A
  • Build Logs (available for 30 days)

@foundationdb-ci
Copy link
Contributor

AWS CodeBuild CI Report for Linux CentOS 7

  • CodeBuild project: fdb-record-layer-pr-proto3
  • Commit ID: c4d86e6
  • Result: SUCCEEDED
  • Error: N/A
  • Build Logs (available for 30 days)

@normen662 normen662 self-assigned this May 14, 2022
@foundationdb-ci
Copy link
Contributor

AWS CodeBuild CI Report for Linux CentOS 7

  • CodeBuild project: fdb-record-layer-pr-proto2
  • Commit ID: 2cee583
  • Result: SUCCEEDED
  • Error: N/A
  • Build Logs (available for 30 days)

@foundationdb-ci
Copy link
Contributor

AWS CodeBuild CI Report for Linux CentOS 7

  • CodeBuild project: fdb-record-layer-pr-proto3
  • Commit ID: 2cee583
  • Result: SUCCEEDED
  • Error: N/A
  • Build Logs (available for 30 days)

@foundationdb-ci
Copy link
Contributor

AWS CodeBuild CI Report for Linux CentOS 7

  • CodeBuild project: fdb-record-layer-pr-proto2
  • Commit ID: 00b8c7d
  • Result: SUCCEEDED
  • Error: N/A
  • Build Logs (available for 30 days)

@foundationdb-ci
Copy link
Contributor

AWS CodeBuild CI Report for Linux CentOS 7

  • CodeBuild project: fdb-record-layer-pr-proto3
  • Commit ID: 00b8c7d
  • Result: SUCCEEDED
  • Error: N/A
  • Build Logs (available for 30 days)

@sonarcloud
Copy link

sonarcloud bot commented May 14, 2022

SonarCloud Quality Gate failed.    Quality Gate failed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 4 Code Smells

0.0% 0.0% Coverage
0.0% 0.0% Duplication

@foundationdb-ci
Copy link
Contributor

AWS CodeBuild CI Report for Linux CentOS 7

  • CodeBuild project: fdb-record-layer-pr-proto3
  • Commit ID: 893a858
  • Result: SUCCEEDED
  • Error: N/A
  • Build Logs (available for 30 days)

@foundationdb-ci
Copy link
Contributor

AWS CodeBuild CI Report for Linux CentOS 7

  • CodeBuild project: fdb-record-layer-pr-proto2
  • Commit ID: 893a858
  • Result: SUCCEEDED
  • Error: N/A
  • Build Logs (available for 30 days)

@normen662 normen662 merged commit d80d1db into FoundationDB:main May 14, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@MMcM MMcM MMcM left review comments

@hatyo hatyo hatyo left review comments

@normen662 normen662 normen662 approved these changes

Assignees

@normen662 normen662

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@pengpeng-lu @foundationdb-ci @MMcM @hatyo @normen662