Fixing KDE evaluate() to return correct ValueAndMagnitude object #128602
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ct for bandwidth = 0 Prior to this change, the evaluate method would return a dummy object for an input with bandwidth = 0. This would occur if the dataset had zero variance (see KDE constructor). This would then cause ChangePointDetector to fail to detect a spike on a dataset containing all equal numbers except for one spike. This change removes the bandwidth = 0 condition for returning a dummy ValueAndMagnitude object. Statistical testing in ChangePointDetector now properly detects the spike in the example mentioned above. Unit tests are added to confirm the bug is fixed.
elasticsearchmachine
added
needs:triage
nielsbauman
added
>bug
:ml
Collaborator
|
Pinging @elastic/ml-core (Team:ML) |
Member
|
@elasticmachine test this please |
Member
|
Thanks for your contribution @Jason-Whitmore and thanks for adding the tests, we will try to review this as soon as possible |
Member
|
@elasticmachine test this please |
tveasey
reviewed
Oct 9, 2025
Contributor
tveasey
left a comment
tveasey
left a comment
There was a problem hiding this comment.
I propose a small tweak.
x-pack/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/aggs/changepoint/KDE.java
Show resolved
Hide resolved
This change enables an early return and avoids numerical issues.
tveasey
approved these changes
Oct 14, 2025
Member
|
@elasticmachine generate changelog |
Member
|
@elasticmachine test this please |
przemekwitek
approved these changes
Oct 21, 2025
Member
|
@elasticmachine test this please |
Member
|
@elasticmachine test this please |
Member
|
@elasticmachine test this please |
Member
|
@elasticmachine test this please |
Contributor
|
@elasticmachine test this please |
Member
|
@elasticmachine test this please |
Member
|
@elasticmachine test this please |
Member
|
@elasticmachine test this please |
weizijun
added a commit
to weizijun/elasticsearch
that referenced
this pull request
Nov 16, 2025
* main: (135 commits)
Mute org.elasticsearch.upgrades.IndexSortUpgradeIT testIndexSortForNumericTypes {upgradedNodes=1} elastic#138130
Mute org.elasticsearch.upgrades.IndexSortUpgradeIT testIndexSortForNumericTypes {upgradedNodes=2} elastic#138129
Mute org.elasticsearch.search.basic.SearchWithRandomDisconnectsIT testSearchWithRandomDisconnects elastic#138128
[DiskBBQ] avoid EsAcceptDocs bug by calling cost before building iterator (elastic#138127)
Log NOT_PREFERRED shard movements (elastic#138069)
Improve bulk loading of binary doc values (elastic#137995)
Add internal action for getting inference fields and inference results for those fields (elastic#137680)
Address issue with DateFieldMapper#isFieldWithinQuery(...) (elastic#138032)
WriteLoadConstraintDecider: Have separate rate limiting for canRemain and canAllocate decisions (elastic#138067)
Adding NodeContext to TransportBroadcastByNodeAction (elastic#138057)
Mute org.elasticsearch.simdvec.ESVectorUtilTests testSoarDistanceBulk elastic#138117
Mute org.elasticsearch.xpack.esql.qa.single_node.GenerativeIT test elastic#137909
Backport batched_response_might_include_reduction_failure version to 8.19 (elastic#138046)
Add summary metrics for tdigest fields (elastic#137982)
Add gp-llm-v2 model ID and inference endpoint (elastic#138045)
Various tracing fixes (elastic#137908)
[ML] Fixing KDE evaluate() to return correct ValueAndMagnitude object (elastic#128602)
Mute org.elasticsearch.xpack.shutdown.NodeShutdownIT testStalledShardMigrationProperlyDetected elastic#115697
[ML] Fix Flaky Audit Message Assertion in testWithDatastream for RegressionIT and ClassificationIT (elastic#138065)
[ML] Fix Non-Deterministic Training Set Selection in RegressionIT testTwoJobsWithSameRandomizeSeedUseSameTrainingSet (elastic#138063)
...
# Conflicts:
# rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/search.vectors/200_dense_vector_docvalue_fields.yml
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
8 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue #127517 describes
ChangePointDetector.getChangeType()failing to detect a spike when all non spike values are constant (for example, the input [51,51,...,51,50001,51...,51]).Prior to this change, the
KDE.evaluate()method would return a special object for an input withbandwidth == 0. This would occur if the values excluding the spike had zero variance (seeKDEconstructor). This would then causeChangePointDetector.getChangeType()andSpikeAndDipDetector.detect()to use the wrong p-value and fail to detect the spike.This change removes the
bandwidth == 0condition for returning a specialValueAndMagnitudeobject. Statistical testing inChangePointDetector.getChangeType()andSpikeAndDipDetector.detect()now properly detect the spike in the example mentioned above.Unit tests are added to confirm the bug is fixed.
Closes #127517