Repository navigation
so-search node rapidly detaches from cluster during state updates and recovery #16363
Unanswered
csyntheses
asked this question in
3.0
Replies: 3 comments 6 replies
|
Start with the Elasticsearch log on the manager and search, If you see something of note, put it here, if you can share the log, attach it for review please. |
4 replies
|
Hello, Here are the outputs from the three commands requested:
{
"nodes" : {
"a1Gb8vFXQrqcoxvxFt11bg" : {
"name" : "so-manager",
"jvm" : {
"pools" : {
"old" : {
"used_in_bytes" : 1379939960,
"max_in_bytes" : 11324620800,
"peak_used_in_bytes" : 1379939960,
"peak_max_in_bytes" : 11324620800
}
}
}
}
}
}
Currently the search node has disconnected from the cluster and thats why its not showing up as a node. |
1 reply
|
Hello, |
1 reply
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Version
3.1.0
Installation Method
Security Onion ISO image
Description
other (please provide detail below)
Installation Type
Distributed
Location
on-prem with Internet access
Hardware Specs
Exceeds minimum requirements
CPU
8
RAM
30GiB/78GiB
Storage for /
279Gb
Storage for /nsm
8tb
Network Traffic Collection
span port
Network Traffic Speeds
more than 10Gbps
Status
Yes, all services on all nodes are running OK
Salt Status
Yes, there are salt failures (please provide detail below)
Logs
No, there are no additional clues
Detail
Problem Description
The
so-searchnode recently started dropping out of the active Elasticsearch cluster state during index recovery repeatedly :cluster.routing.allocation.enable: all) or after a cluster service restart (so-elasticsearch-restart), or when shard recovery initiates.so-searchdetaches from the master node and disappears from the node registry (_cat/nodes?v).Work Done & Ruled Out
nc -zv/telnetverified).dmesg; confirmed 0 OOM killer invocations.cluster.publish.timeout,transport.ping_schedule) via the Elasticsearch REST API (_cluster/settings), which returned expected400 illegal_argument_exceptionas these are static settings./opt/so/saltstack/local/pillar/global.slsto injectelasticsearch.config.cluster.publish_timeout: 90sandelasticsearch.config.transport.ping_schedule: 5s, but the search node continues to rapidly drop transport connection during state sync.Request for Support
I am looking for guidance on:
so-managerandso-searchduring state updates.Guidelines
All reactions