Enable numeric sort optimization support for all numeric types #6424

gashutos · 2023-02-21T22:45:38Z

Description

Enable sort optimization for all remaining Numeric types which were not covered in our last PR 6321
The main issue supporting these remaining NumericTypes was its merging capabilities during field mapping change for indices. That is resolved in this PR by introducing wider size comparator during merge.

Below remainig types were enabled as a part of this PR.

Boolean
Bytes
Short
Int
Half_float (it always uses custom comparator though)
Float

The performance gain we had with this is very huge, almost 10x gain on server side latency comparison. I have listed down few benchmarks in testing section.

Performance improvement

Below is server side latency comparison. Units are in ms.
Dataset as of now I have are smaller for float & short types, but improvement in 'x' wise is pretty wide.

NumericType	New sort latency	Old sort latency	OSB workload
SHORT	13	130	eventdata
INT	150	1411	http_logs
FLOAT	15	240	noaa

http_logs is having largest ingested data, so latency we have there higher compare to other workload during sorting. But in case of relative latency w.r.t w/out sort optimization in 'x' multiple wise gives us similar improvement.

Issues Resolved

6326

Tested

Added extensive unit tests to check if merge works properly in case indices have different numeric types sorted.
Tested creating managed service domain and did performance testing, below were the results.
Reindexed indices with different mapping and checked things are working fine with optimization gains.

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: gashutos <gashutos@amazon.com>

github-actions · 2023-02-21T23:09:25Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/11544/
CommitID: dfde04a
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-02-21T23:11:46Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/11547/
CommitID: 5915c85
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

…ling Signed-off-by: gashutos <gashutos@amazon.com>

github-actions · 2023-02-22T13:45:03Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/11574/
CommitID: 4cc18f0
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: gashutos <gashutos@amazon.com>

gashutos · 2023-03-21T05:38:52Z

@nknize pinging.....
I feel this is adding nice optimization....

github-actions · 2023-03-21T19:00:38Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.indices.replication.SegmentReplicationAllocationIT.testAllocationWithDisruption

URL: https://build.ci.opensearch.org/job/gradle-check/12747/
CommitID: dadcb12
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

dblock · 2023-03-27T18:55:05Z

I resolved the CHANGELOG conflict. Code LGTM, let's merge on green.

gashutos · 2023-03-27T19:06:41Z

I resolved the CHANGELOG conflict. Code LGTM, let's merge on green.

Thanks @dblock !

nknize

Looking good! Just a couple nits.

server/src/main/java/org/opensearch/action/search/SearchPhaseController.java

server/src/main/java/org/opensearch/search/sort/SortedWiderNumericSortField.java

.../src/main/java/org/opensearch/index/fielddata/fieldcomparator/IntValuesComparatorSource.java

nknize · 2023-03-27T19:20:15Z

Unrelated precommit failure:

Execution failed for task ':modules:lang-expression:compileTestJava'.

Refired precommit job.

github-actions · 2023-03-27T19:23:42Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/12955/
CommitID: 2ac34a4
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: gashutos <gashutos@amazon.com>

nknize

LGTM once green! Nice work!

gashutos · 2023-03-27T19:38:02Z

LGTM once green! Nice work!

Awesome , thanks Nick !

github-actions · 2023-03-27T20:03:25Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/12958/
CommitID: 5cf1e75

dblock · 2023-03-27T22:44:22Z

@gashutos watch the auto-backport (might need to be done manually if it fails)

* Adding numeric optimization support for all numeric types Signed-off-by: gashutos <gashutos@amazon.com> * modifying CHANGELOG.md Signed-off-by: gashutos <gashutos@amazon.com> * Handling multi-cluster scenario where SortField serialization was failing Signed-off-by: gashutos <gashutos@amazon.com> * Fixing javadoc errors Signed-off-by: gashutos <gashutos@amazon.com> * Fixing nested sort integ tests Signed-off-by: gashutos <gashutos@amazon.com> * Stremlining behaviour of custom comparator tests too Signed-off-by: gashutos <gashutos@amazon.com> * Adding more integ tests for IntValuesComparatorSource & fixing few ITs Signed-off-by: gashutos <gashutos@amazon.com> * Fixing few more integ tests Signed-off-by: gashutos <gashutos@amazon.com> * Streamlining applySortWidening method with CreateSort and avoid modifying cteated objects of sort Signed-off-by: gashutos <gashutos@amazon.com> * Correcting licence header Signed-off-by: gashutos <gashutos@amazon.com> --------- Signed-off-by: gashutos <gashutos@amazon.com> Co-authored-by: Daniel (dB.) Doubrovkine <dblock@amazon.com> (cherry picked from commit e8425fc) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

#6849) * Adding numeric optimization support for all numeric types * modifying CHANGELOG.md * Handling multi-cluster scenario where SortField serialization was failing * Fixing javadoc errors * Fixing nested sort integ tests * Stremlining behaviour of custom comparator tests too * Adding more integ tests for IntValuesComparatorSource & fixing few ITs * Fixing few more integ tests * Streamlining applySortWidening method with CreateSort and avoid modifying cteated objects of sort * Correcting licence header --------- (cherry picked from commit e8425fc) Signed-off-by: gashutos <gashutos@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Daniel (dB.) Doubrovkine <dblock@amazon.com>

…earch-project#6424) * Adding numeric optimization support for all numeric types Signed-off-by: gashutos <gashutos@amazon.com> * modifying CHANGELOG.md Signed-off-by: gashutos <gashutos@amazon.com> * Handling multi-cluster scenario where SortField serialization was failing Signed-off-by: gashutos <gashutos@amazon.com> * Fixing javadoc errors Signed-off-by: gashutos <gashutos@amazon.com> * Fixing nested sort integ tests Signed-off-by: gashutos <gashutos@amazon.com> * Stremlining behaviour of custom comparator tests too Signed-off-by: gashutos <gashutos@amazon.com> * Adding more integ tests for IntValuesComparatorSource & fixing few ITs Signed-off-by: gashutos <gashutos@amazon.com> * Fixing few more integ tests Signed-off-by: gashutos <gashutos@amazon.com> * Streamlining applySortWidening method with CreateSort and avoid modifying cteated objects of sort Signed-off-by: gashutos <gashutos@amazon.com> * Correcting licence header Signed-off-by: gashutos <gashutos@amazon.com> --------- Signed-off-by: gashutos <gashutos@amazon.com> Co-authored-by: Daniel (dB.) Doubrovkine <dblock@amazon.com> Signed-off-by: Valentin Mitrofanov <mitrofmep@gmail.com>

Adding numeric optimization support for all numeric types

dfde04a

Signed-off-by: gashutos <gashutos@amazon.com>

gashutos mentioned this pull request Feb 21, 2023

Enable sort optimization for all NumericTypes #6420

Closed

modifying CHANGELOG.md

5915c85

Signed-off-by: gashutos <gashutos@amazon.com>

Handling multi-cluster scenario where SortField serialization was fai…

4cc18f0

…ling Signed-off-by: gashutos <gashutos@amazon.com>

Fixing javadoc errors

4e2a4ad

Signed-off-by: gashutos <gashutos@amazon.com>

Merge branch 'opensearch-project:main' into main

dadcb12

This was referenced Mar 23, 2023

Time series based workload sort optimization through Lucene segment sorting in DocumentReader #6814

Closed

TimeSeries optimizations in OpenSearch #3734

Open

Merge branch 'main' into main

2ac34a4

nknize requested changes Mar 27, 2023

View reviewed changes

Correcting licence header

5cf1e75

Signed-off-by: gashutos <gashutos@amazon.com>

nknize approved these changes Mar 27, 2023

View reviewed changes

dblock merged commit e8425fc into opensearch-project:main Mar 27, 2023
9 checks passed

dblock added the backport 2.x Backport to 2.x branch label Mar 27, 2023

opensearch-trigger-bot bot mentioned this pull request Mar 27, 2023

[Backport 2.x] Enable numeric sort optimization support for all numeric types #6849

Merged

This was referenced Apr 1, 2023

Fix sort in case of different numeric field types between indices #6926

Merged

Fixing SortField comparison to use equals instead of reference equality #6901

Merged

This was referenced Jun 20, 2023

Enable point based sort optimization for all custom comparators #8167

Closed

Enable Point based optimization for custom comparators #8168

Merged

This was referenced Aug 9, 2023

[FEATURE] Provide sort query with int field type in http_logs opensearch-project/opensearch-benchmark-workloads#94

Closed

Provide sort query with int field type in http_logs opensearch-project/opensearch-benchmark-workloads#95

Merged

This was referenced Sep 14, 2023

[BUG] Fix broken backward compatibility from 2.7 for IndexSorted field indices #10044

Closed

Fix broken backward compatibility from 2.7 for IndexSorted field indices #10045

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable numeric sort optimization support for all numeric types #6424

Enable numeric sort optimization support for all numeric types #6424

gashutos commented Feb 21, 2023 •

edited

github-actions bot commented Feb 21, 2023

github-actions bot commented Feb 21, 2023

github-actions bot commented Feb 22, 2023

gashutos commented Mar 21, 2023

github-actions bot commented Mar 21, 2023

dblock commented Mar 27, 2023

gashutos commented Mar 27, 2023

nknize left a comment

nknize commented Mar 27, 2023

github-actions bot commented Mar 27, 2023

nknize left a comment

gashutos commented Mar 27, 2023

github-actions bot commented Mar 27, 2023

dblock commented Mar 27, 2023

Enable numeric sort optimization support for all numeric types #6424

Enable numeric sort optimization support for all numeric types #6424

Conversation

gashutos commented Feb 21, 2023 • edited

Description

Below remainig types were enabled as a part of this PR.

Performance improvement

Issues Resolved

Tested

Check List

github-actions bot commented Feb 21, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Feb 21, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Feb 22, 2023

Gradle Check (Jenkins) Run Completed with:

gashutos commented Mar 21, 2023

github-actions bot commented Mar 21, 2023

Gradle Check (Jenkins) Run Completed with:

dblock commented Mar 27, 2023

gashutos commented Mar 27, 2023

nknize left a comment

Choose a reason for hiding this comment

nknize commented Mar 27, 2023

github-actions bot commented Mar 27, 2023

Gradle Check (Jenkins) Run Completed with:

nknize left a comment

Choose a reason for hiding this comment

gashutos commented Mar 27, 2023

github-actions bot commented Mar 27, 2023

Gradle Check (Jenkins) Run Completed with:

dblock commented Mar 27, 2023

gashutos commented Feb 21, 2023 •

edited