Replies: 4 comments 19 replies
-
|
Test on the manager:
Test on the sensor:
Test on both:
|
Beta Was this translation helpful? Give feedback.
-
|
Thanks for your reply! It's one manager search node and one sensor node, so there is no single or multiple search nodes. Here is the logstash log from the manager search node. It's always this two warnings. [2024-03-22T14:56:46,049][WARN ][logstash.outputs.redis ] Failed to flush outgoing items {:outgoing_count=>1, :exception=>"Redis::CommandError", :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/client.rb:> |
Beta Was this translation helpful? Give feedback.
-
|
Do you see any errors/issues in the Elasticsearch log at |
Beta Was this translation helpful? Give feedback.
-
|
Sorry for the late reply. No errors/issues inside the elasticsearch... There are just update role messages. |
Beta Was this translation helpful? Give feedback.
-
|
I have the same problem. There are also no obvious problems in the logs. Try restarting elastic-agent after reboot, or after reboot and starting all containers in so-status. in my case, it helped |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for your answer @tuamortemo, but in my case it doesn't helped... |
Beta Was this translation helpful? Give feedback.
-
|
Tried to fix the problem with the update to 2.4.60. Update works fine, but the issue is the same... |
Beta Was this translation helpful? Give feedback.
-
|
Check your logstash log after the upgrade, do you still see the same errors? If so, what do the pipelines output in these commands: |
Beta Was this translation helpful? Give feedback.
-
|
Hey thanks for the answer. Here is the output of the pipeline. I will check again the errors and send them below. But the problem is still the same. I don't recieve alerts at the managersearch. sudo so-logstash-pipeline-stats manager sudo so-logstash-pipeline-stats search |
Beta Was this translation helpful? Give feedback.
-
|
The managersearch. I skipped the empty logs and duplicated errors. [user@managersearch]$ sudo so-log-check
Checking container "so-elastalert"Reading index mapping 'es_mappings/8/elastalert_error.json' Checking container "so-dockerregistry"time="2024-04-05T15:16:55.969668304Z" level=info msg="Purge uploads finished. Num deleted=0, num errors=0" Checking log file /opt/so/log/kratos/kratos.log{"audience":"audit","error":{"debug":"","message":"request does not have a valid authentication session","reason":"No active session was found in this request.","stack_trace":"\ngithub.com/ory/kratos/session.(*ManagerHTTP).FetchFromRequest\n\t/go/src/github.com/ory/kratos/session/manager_http.go:236\ngithub.com/ory/kratos/session.(*Handler).whoami\n\t/go/src/github.com/ory/kratos/session/handler.go:215\ngithub.com/ory/kratos/x.(*RouterPublic).Handle.NoCacheHandle.func1\n\t/go/src/github.com/ory/kratos/x/nocache.go:21\ngithub.com/julienschmidt/httprouter.(*Router).ServeHTTP\n\t/go/pkg/mod/github.com/julienschmidt/httprouter@v1.3.0/router.go:387\ngithub.com/ory/nosurf.(*CSRFHandler).handleSuccess\n\t/go/pkg/mod/github.com/ory/nosurf@v1.2.7/handler.go:234\ngithub.com/ory/nosurf.(*CSRFHandler).ServeHTTP\n\t/go/pkg/mod/github.com/ory/nosurf@v1.2.7/handler.go:185\ngithub.com/urfave/negroni.(*Negroni).UseHandler.Wrap.func1\n\t/go/pkg/mod/github.com/urfave/negroni@v1.0.0/negroni.go:46\ngithub.com/urfave/negroni.HandlerFunc.ServeHTTP\n\t/go/pkg/mod/github.com/urfave/negroni@v1.0.0/negroni.go:29\ngithub.com/urfave/negroni.middleware.ServeHTTP\n\t/go/pkg/mod/github.com/urfave/negroni@v1.0.0/negroni.go:38\ngithub.com/ory/kratos/x.glob..func1\n\t/go/src/github.com/ory/kratos/x/clean_url.go:15\ngithub.com/urfave/negroni.HandlerFunc.ServeHTTP\n\t/go/pkg/mod/github.com/urfave/negroni@v1.0.0/negroni.go:29\ngithub.com/urfave/negroni.middleware.ServeHTTP\n\t/go/pkg/mod/github.com/urfave/negroni@v1.0.0/negroni.go:38\ngithub.com/ory/kratos/cmd/daemon.servePublic.func1\n\t/go/src/github.com/ory/kratos/cmd/daemon/serve.go:111\ngithub.com/urfave/negroni.HandlerFunc.ServeHTTP\n\t/go/pkg/mod/github.com/urfave/negroni@v1.0.0/negroni.go:29\ngithub.com/urfave/negroni.middleware.ServeHTTP\n\t/go/pkg/mod/github.com/urfave/negroni@v1.0.0/negroni.go:38\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerResponseSize.func1\n\t/go/pkg/mod/github.com/prometheus/client_golang@v1.13.0/prometheus/promhttp/instrument_server.go:284\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1\n\t/go/pkg/mod/github.com/prometheus/client_golang@v1.13.0/prometheus/promhttp/instrument_server.go:142\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func1\n\t/go/pkg/mod/github.com/prometheus/client_golang@v1.13.0/prometheus/promhttp/instrument_server.go:92\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func2\n\t/go/pkg/mod/github.com/prometheus/client_golang@v1.13.0/prometheus/promhttp/instrument_server.go:104\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerRequestSize.func1\n\t/go/pkg/mod/github.com/prometheus/client_golang@v1.13.0/prometheus/promhttp/instrument_server.go:234\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/ory/x/prometheusx.Metrics.Instrument.Metrics.instrumentHandlerStatusBucket.func1\n\t/go/pkg/mod/github.com/ory/x@v0.0.614/prometheusx/metrics.go:115\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2136\ngithub.com/ory/x/prometheusx.(MetricsManager).ServeHTTP\n\t/go/pkg/mod/github.com/ory/x@v0.0.614/prometheusx/middleware.go:41\ngithub.com/urfave/negroni.middleware.ServeHTTP\n\t/go/pkg/mod/github.com/urfave/negroni@v1.0.0/negroni.go:38\ngithub.com/ory/x/metricsx.(Service).ServeHTTP\n\t/go/pkg/mod/github.com/ory/x@v0.0.614/metricsx/middleware.go:272\ngithub.com/urfave/negroni.middleware.ServeHTTP\n\t/go/pkg/mod/github.com/urfave/negroni@v1.0.0/negroni.go:38","status":"Unauthorized","status_code":401},"http_request":{"headers":{"accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,/;q=0.8","accept-encoding":"gzip, deflate, br","accept-language":"de,en-US;q=0.7,en;q=0.3","connection":"close","cookie":"Value is sensitive and has been redacted. To see the value set config key "log.leak_sensitive_values = true" or environment variable "LOG_LEAK_SENSITIVE_VALUES=true".","if-modified-since":"Tue, 19 Mar 2024 18:07:37 GMT","sec-fetch-dest":"document","sec-fetch-mode":"navigate","sec-fetch-site":"none","sec-fetch-user":"?1","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:124.0) Gecko/20100101 Firefox/124.0","x-forwarded-for":"193.196.188.222","x-forwarded-proto":"https","x-real-ip":"193.196.188.222"},"host":"10.100.79.2","method":"GET","path":"/sessions/whoami","query":null,"remote":"172.17.1.1:60654","scheme":"http"},"level":"info","msg":"No valid session found.","service_name":"Ory Kratos","service_version":"v1.1.0","time":"2024-04-11T13:24:34.420818832Z"} Checking log file /opt/so/log/influxdb/influxdb.logts=2024-04-11T10:25:07.757629Z lvl=info msg="http: TLS handshake error from 127.0.0.1:47532: EOF" log_id=0oMRZ5H0000 service=http Checking log file /opt/so/log/telegraf/telegraf.log Checking log file /opt/so/log/logstash/logstash.log[2024-04-11T13:27:00,801][WARN ][logstash.outputs.redis ] Failed to flush outgoing items {:outgoing_count=>1, :exception=>"Redis::CommandError", :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/client.rb:162:in |
Beta Was this translation helpful? Give feedback.
-
|
Same issue with standalone fresh deployment 2.4.60-20240320. |
Beta Was this translation helpful? Give feedback.
-
|
@M4jx if you find a solution before this is answered. Pls let me know |
Beta Was this translation helpful? Give feedback.
-
|
@tthomasb the installation is working for me now (even though it is showing installation failed at the end of the installation process #12806). |
Beta Was this translation helpful? Give feedback.
-
|
I will give it a try. Thank you! |
Beta Was this translation helpful? Give feedback.
-
|
In my case it doesn't work... |
Beta Was this translation helpful? Give feedback.
-
|
Can you restart Logstash and share the whole |
Beta Was this translation helpful? Give feedback.
-
|
Hey @cm-ops, thanks again for the reply. After "Failed to flush outgoing items" and "Failed to send backlog of events to Redis" there are just this two messages all the way down. |
Beta Was this translation helpful? Give feedback.
-
|
I didn't see what you have in your manager and search pipelines in the output above. Looks like the log is cut off some. Should look like below:
Can you take a look in your logstash log and confirm they are the same? |
Beta Was this translation helpful? Give feedback.
-
|
I don't know why it's cut off, but here are the output, after restarting just now. I think it just look fine... [2024-04-15T18:19:10,283][INFO ][logstash.javapipeline ] Starting pipeline {:pipeline_id=>"manager", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, "pipeline.sources"=>["/usr/share/logstash/pipelines/manager/0011_input_endgame.conf", "/usr/share/logstash/pipelines/manager/0012_input_elastic_agent.conf", "/usr/share/logstash/pipelines/manager/0013_input_lumberjack_fleet.conf", "/usr/share/logstash/pipelines/manager/9999_output_redis.conf"], :thread=>"#<Thread:0x295ce61 /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:134 run>"} [2024-04-15T18:19:10,711][INFO ][logstash.javapipeline ] Starting pipeline {:pipeline_id=>"search", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, "pipeline.sources"=>["/usr/share/logstash/pipelines/search/0900_input_redis.conf", "/usr/share/logstash/pipelines/search/9805_output_elastic_agent.conf", "/usr/share/logstash/pipelines/search/9900_output_endgame.conf"], :thread=>"#<Thread:0x435755a1 /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:134 run>"} |
Beta Was this translation helpful? Give feedback.
-
|
Okay, that looks right. Also, let me know what Elasticsearch pipelines are loaded with |
Beta Was this translation helpful? Give feedback.
-
|
Ok sure. |
Beta Was this translation helpful? Give feedback.
-
Version
2.4.40
Installation Method
Security Onion ISO image
Description
configuration
Installation Type
Distributed
Location
on-prem with Internet access
Hardware Specs
Exceeds minimum requirements
CPU
8
RAM
32
Storage for /
70 GB
Storage for /nsm
130 GB
Network Traffic Collection
span port
Network Traffic Speeds
1Gbps to 10Gbps
Status
Yes, all services on all nodes are running OK
Salt Status
No, there are no failures
Logs
Yes, there are additional clues in /opt/so/log/ (please provide detail below)
Detail
Hello there, after a reboot I don't receive any alerts.
The logs on the sensor logs everything fine. The logs of the manager I will provide down here. I also provide some additional information in the comment section. Installed system is a distributed system with one managersearch and one sensor.
sudo so-log-check (just the logs with informations):
Checking container "so-elastalert":
...
Reading index mapping 'es_mappings/8/elastalert_error.json'
Checking container "so-elastic-fleet"
...
Error: context canceled
Checking container "so-dockerregistry"
...
time="2024-03-20T10:55:59.20758766Z" level=info msg="Purge uploads finished. Num deleted=0, num errors=0"
Checking log file /opt/so/log/influxdb/influxdb.log
...
ts=2024-03-21T09:33:02.176099Z lvl=info msg="http: TLS handshake error from 127.0.0.1:54448: EOF" log_id=0o1ambp0000 service=http
Checking log file /opt/so/log/telegraf/telegraf.log
2024-03-21T00:01:30Z E! [inputs.exec] Error in plugin: metric parse error: expected field at 1:19: "influxsize kbytes="
2024-03-21T00:02:00Z E! [inputs.exec] Error in plugin: metric parse error: expected field at 1:19: "influxsize kbytes="
2024-03-21T08:59:30Z E! [inputs.logstash] Error in plugin: Get "http://localhost:9600/_node/stats/pipelines": dial tcp 127.0.0.1:9600: connect: connection refused
Checking log file /opt/so/log/logstash/logstash.log
[2024-03-21T09:33:41,305][WARN ][logstash.outputs.redis ] Failed to flush outgoing items {:outgoing_count=>1, :exception=>"Redis::CommandError", :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/client.rb:162:in
call'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis.rb:270:inblock in send_command'", "org/jruby/ext/monitor/Monitor.java:82:insynchronize'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis.rb:269:insend_command'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/commands/lists.rb:86:inrpush'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-redis-5.0.0/lib/logstash/outputs/redis.rb:152:inflush'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:221:inblock in buffer_flush'", "org/jruby/RubyHash.java:1587:ineach'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:216:inbuffer_flush'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:159:inbuffer_receive'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-redis-5.0.0/lib/logstash/outputs/redis.rb:209:insend_to_redis'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-codec-json-3.1.1/lib/logstash/codecs/json.rb:69:inencode'", "/usr/share/logstash/logstash-core/lib/logstash/codecs/delegator.rb:48:inblock in encode'", "org/logstash/instrument/metrics/AbstractSimpleMetricExt.java:74:intime'", "org/logstash/instrument/metrics/AbstractNamespacedMetricExt.java:68:intime'", "/usr/share/logstash/logstash-core/lib/logstash/codecs/delegator.rb:47:inencode'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-redis-5.0.0/lib/logstash/outputs/redis.rb:123:inreceive'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:104:inblock in multi_receive'", "org/jruby/RubyArray.java:1987:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:104:inmulti_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121:inmulti_receive'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:304:inblock in start_workers'"]}[2024-03-21T09:33:41,305][WARN ][logstash.outputs.redis ] Failed to send backlog of events to Redis {:identity=>"redis://@seconion:6379/0 list:logstash:unparsed", :exception=>#<Redis::CommandError: OOM command not allowed when used memory > 'maxmemory'.>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/client.rb:162:in
call'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis.rb:270:inblock in send_command'", "org/jruby/ext/monitor/Monitor.java:82:insynchronize'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis.rb:269:insend_command'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/commands/lists.rb:86:inrpush'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-redis-5.0.0/lib/logstash/outputs/redis.rb:152:inflush'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:221:inblock in buffer_flush'", "org/jruby/RubyHash.java:1587:ineach'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:216:inbuffer_flush'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:159:inbuffer_receive'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-redis-5.0.0/lib/logstash/outputs/redis.rb:209:insend_to_redis'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-codec-json-3.1.1/lib/logstash/codecs/json.rb:69:inencode'", "/usr/share/logstash/logstash-core/lib/logstash/codecs/delegator.rb:48:inblock in encode'", "org/logstash/instrument/metrics/AbstractSimpleMetricExt.java:74:intime'", "org/logstash/instrument/metrics/AbstractNamespacedMetricExt.java:68:intime'", "/usr/share/logstash/logstash-core/lib/logstash/codecs/delegator.rb:47:inencode'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-redis-5.0.0/lib/logstash/outputs/redis.rb:123:inreceive'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:104:inblock in multi_receive'", "org/jruby/RubyArray.java:1987:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:104:inmulti_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121:inmulti_receive'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:304:inblock in start_workers'"]}Any idea what's going on?
Kind regards
Guidelines
Beta Was this translation helpful? Give feedback.
All reactions