Failing ES Promotion: FTR Configs #48 / Alerts APIs - Trial License/Complete Tier @ess Alerts Compatibility Query should generate a signal-on-legacy-signal with AAD index pattern #176105
Assignees
Labels
failed-es-promotion
failed-test
A test failure on a tracked branch, potentially flaky-test
impact:critical
This issue should be addressed immediately due to a critical level of impact on the product.
legit-flake
Test was triaged and marked as an actual flake.
Team:Detection Engine
Security Solution Detection Engine Area
Team:Detections and Resp
Security Detection Response Team
Team: SecuritySolution
Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc.
Milestone
Comments
mistic
added
blocker
skipped-test
failed-es-promotion
Team:Detection Engine
|
Pinging @elastic/security-detection-engine (Team:Detection Engine) |
mistic
added a commit
that referenced
this issue
Feb 1, 2024
|
Skipped. main: aabf7b7 |
banderror
added
triage_needed
Team:Detections and Resp
|
Pinging @elastic/security-solution (Team: SecuritySolution) |
|
Pinging @elastic/security-detections-response (Team:Detections and Resp) |
banderror
added
the
failed-test
WafaaNasr
pushed a commit
to WafaaNasr/kibana
that referenced
this issue
Feb 6, 2024
|
It appears, aliases not working properly anymore Here is alias configuration for .alerts-security.alerts-default and .siem-signals-default, with count results of indexed documents. Results match, so tests are passing. GET .alerts-security.alerts-default
{
".internal.alerts-security.alerts-default-000001": {
"aliases": {
".alerts-security.alerts-default": {
"is_write_index": true
},
".siem-signals-default": {
"is_write_index": false
}
},
....
},
".siem-signals-default-000001-7.16.0": {
"aliases": {
".alerts-security.alerts-default": {
"is_write_index": false
},
".siem-signals-default": {
"is_write_index": true
}
},
...
},
}
Search results for documents in .alerts-security.alerts-default GET .alerts-security.alerts-default/_count
{
"count": 23,
"_shards": {
"total": 2,
"successful": 2,
"skipped": 0,
"failed": 0
}
}Search results for documents in .siem-signals-default GET .siem-signals-default/_count
{
"count": 23,
"_shards": {
"total": 2,
"successful": 2,
"skipped": 0,
"failed": 0
}
}
But next promotion build https://buildkite.com/elastic/kibana-elasticsearch-snapshot-verify/builds/3614, with snapshot: not working anymore when looking for aliases of .alerts-security.alerts-default, .siem-signals-default is not anymore present GET .alerts-security.alerts-default
{
".internal.alerts-security.alerts-default-000001": {
"aliases": {
".alerts-security.alerts-default": {
"is_write_index": true
},
".siem-signals-default": {
"is_write_index": false
}
},
...
}
}Local Kibana ran with these 2 snapshots is the same. Looks like, there some changes on ES side. |
fkanout
pushed a commit
to fkanout/kibana
that referenced
this issue
Feb 7, 2024
|
pinged ES team in internal slack |
yctercero
added
legit-flake
|
@mistic based on @vitaliidm 's findings and it being a legitimate failure that should be blocking promotion. Should we unskip it so we know when a valid fix is in? |
|
Hi, I'm trying to determine if elasticsearch#104145 would have caused this. It seems likely given the timelines here. I am having a hard time understanding the exact data streams or indices in scope for this test. It seems like this is entirely focused on indices and aliases and not data streams, and the changes in the linked ES ticket should have only fixed a bug in how data stream aliases were returned. To help me, could you describe the specific indices and aliases that are used here throughout the lifetime of the failing test? And then maybe also describe which specific query is failing? It's looking like a timeout but I'm not sure which query is causing that to happen. Also, is this only running on |
This was referenced Feb 7, 2024
|
Summary on tests: 7.16 here is the version, indexed documents are originated from.
So, second test fails, as .alerts-security does not have alias I don't see any alias related failures, here is log from alias creation
POST alias And as mentioned from comments above, aliases are created GET .alerts-security.alerts-default/_alias
{
".internal.alerts-security.alerts-default-000001": {
"aliases": {
".alerts-security.alerts-default": {
"is_write_index": true
},
".siem-signals-default": {
"is_write_index": false
}
}
},
".siem-signals-default-000001-7.16.0": {
"aliases": {
".alerts-security.alerts-default": {
"is_write_index": false
},
".siem-signals-default": {
"is_write_index": true
}
}
}
}
No specific query is failing, and timeout happens because we looking for search results that should be returned from
I think so Here logs from build that DOES FAIL tests No request to create alias for this test though. Something went wrong, probably some GET request did not return results |
|
Looking deeper into logs, I can see put mapping is failing
That prevents, further add alias request to be performed Exactly the same query works fine for the earlier build |
|
Failing PUT mapping query from above comment requestresponse {
"error": {
"root_cause": [
{
"type": "mapper_parsing_exception",
"reason": "Alias [kibana.alert.rule.created_by] is defined both as an alias and a concrete field"
}
],
"type": "mapper_parsing_exception",
"reason": "Alias [kibana.alert.rule.created_by] is defined both as an alias and a concrete field"
},
"status": 400
}
Same query works without errors for ES snapshots dated earlier than 1st of February |
|
We're ruling out elasticsearch#104145. The actual error here is happening before the index alias is retrieved, and this test is working on elasticsearch#103648 touched code that could be in this area and we'll investigate that. |
|
Confirmed fix upstream #176661 |
CoenWarmer
pushed a commit
to CoenWarmer/kibana
that referenced
this issue
Feb 15, 2024
CoenWarmer
pushed a commit
to CoenWarmer/kibana
that referenced
this issue
Feb 15, 2024
## Summary Unskips tests that were skipped due to an upstream change and fixed in elastic/elasticsearch#105298 Addresses elastic#176105, elastic#176117, elastic#176270, elastic#176359, elastic#176360
CoenWarmer
pushed a commit
to CoenWarmer/kibana
that referenced
this issue
Feb 15, 2024
CoenWarmer
pushed a commit
to CoenWarmer/kibana
that referenced
this issue
Feb 15, 2024
## Summary Unskips tests that were skipped due to an upstream change and fixed in elastic/elasticsearch#105298 Addresses elastic#176105, elastic#176117, elastic#176270, elastic#176359, elastic#176360
fkanout
pushed a commit
to fkanout/kibana
that referenced
this issue
Mar 4, 2024
fkanout
pushed a commit
to fkanout/kibana
that referenced
this issue
Mar 4, 2024
## Summary Unskips tests that were skipped due to an upstream change and fixed in elastic/elasticsearch#105298 Addresses elastic#176105, elastic#176117, elastic#176270, elastic#176359, elastic#176360
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Detection Engine - Alerts Integration Tests - ESS Env - Trial License
x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/alerts/trial_license_complete_tier/alerts_compatibility.ts
Alerts APIs - Trial License/Complete Tier @ess Alerts Compatibility Query should generate a signal-on-legacy-signal with AAD index pattern
This failure is preventing the promotion of the current Elasticsearch snapshot.
For more information on the Elasticsearch snapshot process including how to reproduce using the unverified ES build please read the failed promotion annotation. Other important information can be found at:
The text was updated successfully, but these errors were encountered: