SOC Disconnected (401)

[endpointsec]

So my /nsm was at 99% for a few days when I wasn't monitoring the server and noticed Kibana wasn't available.

I ran so-elastic-clear and removed all data, got nsm available to 50%.

Now, I can't connect to the soc -- looking at sensoroni-server.log I continually see

timestamp=2022-06-29T19:41:03.678766206Z level=warn message="Request did not complete successfully" error="Access denied" requestId=e5f03e75-9541-4077-9f9e-a4e04a2ea9b5 requestor=null
timestamp=2022-06-29T19:41:03.678857248Z level=info message="Handled request" contentLength=115 elapsedMs=0 impl=*server.InfoHandler method=GET path=/api/info remoteAddr=172.17.1.1:60570 requestId=e5f03e75-9541-4077-9f9e-a4e04a2ea9b5 requestor=null sourceIp=10.x.x.x statusCode=401

so-status reports everything green except strelka + wazuh which I have disabled. Any ideas on how to correct this? I would like to avoid re-building the VM.

My IP is permitted to hit the web port, I double checked the iptables. It just hammers the red banner 'Request failued with status code 401'.

[dougburks]

What version of Security Onion are you running?

Have you checked the Elasticsearch logs for additional clues?

Have you tried creating a new user account to see if that allows you to login?

[endpointsec]

I am on 2.3.130, found it in the global.sls. Sorry, the portal shows 0.0.0.

I do not see any direct errors in elasticsearch logs -- indexes continue to be created daily and have good sizes. I performed 'zcat *.gz | grep -i error' but have not seen any errors since 6/29 when disk space was gone and indexes were being marked as read only. Elastic logs have been super small since then.

I have created a new user, same problem.

[dougburks]

Have you removed the read-only setting from the indices?

[endpointsec]

I was not detecting read only locks on indexes with some GET cluster commands but I ran the following commands with no changes to the web interface.

curl -k -XPUT -H "Content-Type: application/json" hxxps://localhost:9200/_cluster/settings -d '{ "transient": { "cluster.routing.allocation.disk.threshold_enabled": false } }'
curl -k -XPUT -H "Content-Type: application/json" hxxps://localhost:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'

Rebooted the VM. Cat'd the elasticsearch log and no hits for warning|error. Not that I review the log normally but nothing is really sticking out to me to indicate a problem. I authenticate into the main portal and continue to see 'Request failed with status code 401' and the SO2 version at the bottom shows 0.0.0. I appreciate your responses Doug and don't mean to waste your time!

the /opt/so/log/soc/sensoroni-server.log continues to show 'level=warn message="Request did not complete successfully" error="Access denied"' from my IP. It does accept existing and new user credentials but no user can actually interact with the portal.

*edit

curl -k hxxps://localhost:9200/_settings?pretty | grep read_only

Only resulted in seeing "read_only" : "false"

[dougburks]

This seems like an unusual problem. You could do a quick installation of Security Onion in a separate VM and then compare that to your existing system side-by-side looking for any differences.

[endpointsec]

Did not compare any files -- backed up my configs/certs over sftp and did a fresh install. Everything working again though I lost Kibana dashboards and some playbook rules I could have tried saving. Don't know what caused that but this may be closed. Thank you support!