Expose DelimitedTermFrequencyTokenFilter #9479

russcam · 2023-08-22T12:00:36Z

Description

This commit exposes Lucene's delimited term frequency token filter to be able to provide term frequencies along with terms.

Related Issues

Resolves #9413

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Relates: opensearch-project#9413 This commit exposes Lucene's delimited term frequency token filter to be able to provide term frequencies along with terms. Signed-off-by: Russ Cam <russcam@canva.com>

github-actions · 2023-08-22T12:05:20Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/23196/
CommitID: 55526ec
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Russ Cam <russcam@canva.com>

github-actions · 2023-08-22T12:12:46Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/23197/
CommitID: 79e8fc5
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

opensearch-trigger-bot · 2023-08-22T12:33:36Z

Compatibility status:

Checks if related components are compatible with change dd75a22

Incompatible components

Incompatible components: [https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/security-analytics.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

github-actions · 2023-08-30T22:48:48Z

Compatibility status:

Checks if related components are compatible with change 76fe75d

Incompatible components

Incompatible components: [https://github.com/opensearch-project/cross-cluster-replication.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

github-actions · 2023-08-30T23:09:58Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/23887/
CommitID: 952f0ca
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-08-30T23:31:48Z

Compatibility status:

Checks if related components are compatible with change 952f0ca

Incompatible components

Incompatible components: [https://github.com/opensearch-project/cross-cluster-replication.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git]

russcam · 2023-08-31T01:24:11Z

Failing tests look to be unrelated to this change:

Tests with failures:
 - org.opensearch.index.shard.RemoteIndexShardTests.testNRTReplicaWithRemoteStorePromotedAsPrimaryCommitRefresh

12039 tests completed, 4 failed, 25 skipped

Signed-off-by: Russ Cam <russcam@canva.com>

github-actions · 2023-08-31T08:20:36Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/23936/
CommitID: 15129d9
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-08-31T08:31:09Z

Compatibility status:

Checks if related components are compatible with change 15129d9

Incompatible components

Incompatible components: [https://github.com/opensearch-project/cross-cluster-replication.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

github-actions · 2023-08-31T08:35:28Z

Compatibility status:

Checks if related components are compatible with change a80e9a0

Incompatible components

Incompatible components: [https://github.com/opensearch-project/cross-cluster-replication.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git]

github-actions · 2023-08-31T08:43:16Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/23938/
CommitID: a80e9a0
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

noCharger · 2023-08-31T15:59:54Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌

URL: https://build.ci.opensearch.org/job/gradle-check/23938/

CommitID: a80e9a0
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

It's the same flaky test as in #9580. Other PRs appear to be skipping this test case and merging exactly where the auto cut occurred: #9631

github-actions · 2023-08-31T17:34:20Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/23983/
CommitID: a80e9a0
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-08-31T18:29:18Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/23988/
CommitID: a80e9a0

opensearch-trigger-bot · 2023-08-31T18:45:15Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-9479-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 1126d2f7caac7ec660975e63527404b9d9f6e087
# Push it to GitHub
git push --set-upstream origin backport/backport-9479-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-9479-to-2.x.

* Expose DelimitedTermFrequencyTokenFilter Relates: opensearch-project#9413 This commit exposes Lucene's delimited term frequency token filter to be able to provide term frequencies along with terms. Signed-off-by: Russ Cam <russcam@canva.com> * fix format violations Signed-off-by: Russ Cam <russcam@canva.com> * fix test and add to changelog Signed-off-by: Russ Cam <russcam@canva.com> * Address PR feedback - Add unit tests for DelimitedTermFrequencyTokenFilterFactory - Remove IllegalArgumentException as caught exception - Add skip to yaml rest tests to skip for version < 2.10 Signed-off-by: Russ Cam <russcam@canva.com> * formatting Signed-off-by: Russ Cam <russcam@canva.com> * Rename filter Signed-off-by: Russ Cam <russcam@canva.com> * update naming in REST tests Signed-off-by: Russ Cam <russcam@canva.com> --------- Signed-off-by: Russ Cam <russcam@canva.com> (cherry picked from commit 1126d2f)

* Expose DelimitedTermFrequencyTokenFilter Relates: opensearch-project#9413 This commit exposes Lucene's delimited term frequency token filter to be able to provide term frequencies along with terms. Signed-off-by: Russ Cam <russcam@canva.com> * fix format violations Signed-off-by: Russ Cam <russcam@canva.com> * fix test and add to changelog Signed-off-by: Russ Cam <russcam@canva.com> * Address PR feedback - Add unit tests for DelimitedTermFrequencyTokenFilterFactory - Remove IllegalArgumentException as caught exception - Add skip to yaml rest tests to skip for version < 2.10 Signed-off-by: Russ Cam <russcam@canva.com> * formatting Signed-off-by: Russ Cam <russcam@canva.com> * Rename filter Signed-off-by: Russ Cam <russcam@canva.com> * update naming in REST tests Signed-off-by: Russ Cam <russcam@canva.com> --------- Signed-off-by: Russ Cam <russcam@canva.com> (cherry picked from commit 1126d2f) Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

* Expose DelimitedTermFrequencyTokenFilter Relates: #9413 This commit exposes Lucene's delimited term frequency token filter to be able to provide term frequencies along with terms. * fix format violations * fix test and add to changelog * Address PR feedback - Add unit tests for DelimitedTermFrequencyTokenFilterFactory - Remove IllegalArgumentException as caught exception - Add skip to yaml rest tests to skip for version < 2.10 * formatting * Rename filter * update naming in REST tests --------- (cherry picked from commit 1126d2f) Signed-off-by: Russ Cam <russcam@canva.com> Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Co-authored-by: Russ Cam <russ.cam@forloop.co.uk>

* Expose DelimitedTermFrequencyTokenFilter Relates: opensearch-project#9413 This commit exposes Lucene's delimited term frequency token filter to be able to provide term frequencies along with terms. Signed-off-by: Russ Cam <russcam@canva.com> * fix format violations Signed-off-by: Russ Cam <russcam@canva.com> * fix test and add to changelog Signed-off-by: Russ Cam <russcam@canva.com> * Address PR feedback - Add unit tests for DelimitedTermFrequencyTokenFilterFactory - Remove IllegalArgumentException as caught exception - Add skip to yaml rest tests to skip for version < 2.10 Signed-off-by: Russ Cam <russcam@canva.com> * formatting Signed-off-by: Russ Cam <russcam@canva.com> * Rename filter Signed-off-by: Russ Cam <russcam@canva.com> * update naming in REST tests Signed-off-by: Russ Cam <russcam@canva.com> --------- Signed-off-by: Russ Cam <russcam@canva.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com>

* Expose DelimitedTermFrequencyTokenFilter Relates: opensearch-project#9413 This commit exposes Lucene's delimited term frequency token filter to be able to provide term frequencies along with terms. Signed-off-by: Russ Cam <russcam@canva.com> * fix format violations Signed-off-by: Russ Cam <russcam@canva.com> * fix test and add to changelog Signed-off-by: Russ Cam <russcam@canva.com> * Address PR feedback - Add unit tests for DelimitedTermFrequencyTokenFilterFactory - Remove IllegalArgumentException as caught exception - Add skip to yaml rest tests to skip for version < 2.10 Signed-off-by: Russ Cam <russcam@canva.com> * formatting Signed-off-by: Russ Cam <russcam@canva.com> * Rename filter Signed-off-by: Russ Cam <russcam@canva.com> * update naming in REST tests Signed-off-by: Russ Cam <russcam@canva.com> --------- Signed-off-by: Russ Cam <russcam@canva.com> Signed-off-by: Ivan Brusic <ivan.brusic@flocksafety.com>

* Expose DelimitedTermFrequencyTokenFilter Relates: opensearch-project#9413 This commit exposes Lucene's delimited term frequency token filter to be able to provide term frequencies along with terms. Signed-off-by: Russ Cam <russcam@canva.com> * fix format violations Signed-off-by: Russ Cam <russcam@canva.com> * fix test and add to changelog Signed-off-by: Russ Cam <russcam@canva.com> * Address PR feedback - Add unit tests for DelimitedTermFrequencyTokenFilterFactory - Remove IllegalArgumentException as caught exception - Add skip to yaml rest tests to skip for version < 2.10 Signed-off-by: Russ Cam <russcam@canva.com> * formatting Signed-off-by: Russ Cam <russcam@canva.com> * Rename filter Signed-off-by: Russ Cam <russcam@canva.com> * update naming in REST tests Signed-off-by: Russ Cam <russcam@canva.com> --------- Signed-off-by: Russ Cam <russcam@canva.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

Expose DelimitedTermFrequencyTokenFilter

55526ec

Relates: opensearch-project#9413 This commit exposes Lucene's delimited term frequency token filter to be able to provide term frequencies along with terms. Signed-off-by: Russ Cam <russcam@canva.com>

russcam mentioned this pull request Aug 22, 2023

Expose DelimitedTermFrequencyTokenFilter #9413

Closed

fix format violations

b529953

Signed-off-by: Russ Cam <russcam@canva.com>

russcam force-pushed the delimited-termfreq-token-filter branch from 79e8fc5 to b529953 Compare August 22, 2023 12:11

Merge branch 'main' into delimited-termfreq-token-filter

15129d9

Signed-off-by: Russ Cam <russcam@canva.com>

russcam requested review from dblock and macohen August 31, 2023 07:53

update naming in REST tests

a80e9a0

Signed-off-by: Russ Cam <russcam@canva.com>

msfroh approved these changes Aug 31, 2023

View reviewed changes

msfroh added the backport 2.x Backport to 2.x branch label Aug 31, 2023

msfroh merged commit 1126d2f into opensearch-project:main Aug 31, 2023
15 checks passed

opensearch-trigger-bot bot added the backport-failed label Aug 31, 2023

reta mentioned this pull request Aug 31, 2023

[Backport] [2.x] Expose DelimitedTermFrequencyTokenFilter (#9479) #9671

Merged

noCharger mentioned this pull request Sep 7, 2023

[DOC] Document delimited term frequency token filter opensearch-project/documentation-website#4986

Closed

4 tasks

noCharger mentioned this pull request Oct 27, 2023

Adding access to noSubMatches and noOverlappingMatches #10765

Closed

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Expose DelimitedTermFrequencyTokenFilter #9479

Expose DelimitedTermFrequencyTokenFilter #9479

russcam commented Aug 22, 2023 •

edited

Loading

github-actions bot commented Aug 22, 2023

github-actions bot commented Aug 22, 2023

opensearch-trigger-bot bot commented Aug 22, 2023

github-actions bot commented Aug 30, 2023

github-actions bot commented Aug 30, 2023

github-actions bot commented Aug 30, 2023

russcam commented Aug 31, 2023 •

edited

Loading

github-actions bot commented Aug 31, 2023

github-actions bot commented Aug 31, 2023

github-actions bot commented Aug 31, 2023

github-actions bot commented Aug 31, 2023

noCharger commented Aug 31, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 31, 2023

github-actions bot commented Aug 31, 2023

opensearch-trigger-bot bot commented Aug 31, 2023

Expose DelimitedTermFrequencyTokenFilter #9479

Expose DelimitedTermFrequencyTokenFilter #9479

Conversation

russcam commented Aug 22, 2023 • edited Loading

Description

Related Issues

Check List

github-actions bot commented Aug 22, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 22, 2023

Gradle Check (Jenkins) Run Completed with:

opensearch-trigger-bot bot commented Aug 22, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Aug 30, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Aug 30, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 30, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

russcam commented Aug 31, 2023 • edited Loading

github-actions bot commented Aug 31, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 31, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Aug 31, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Aug 31, 2023

Gradle Check (Jenkins) Run Completed with:

noCharger commented Aug 31, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 31, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 31, 2023

Gradle Check (Jenkins) Run Completed with:

opensearch-trigger-bot bot commented Aug 31, 2023

russcam commented Aug 22, 2023 •

edited

Loading

russcam commented Aug 31, 2023 •

edited

Loading