Changing Indexer/Dashboard password failure in Docker Wazuh Single-node Deployment

[pro-akim]

Wazuh version	Component	Install type	Install method	Platform
4.5.3-rc1	Wazuh component	Manager/Indexer	Packages	Ubuntu

Description
Having a Wazuh installation in single mode with Docker, when attempting to change the dashboard/indexer password, the dashboard, after the change, returns a status_code 500.

Test E2E Deployment on docker:

• Release 4.5.3 - RC 1 - E2E UX tests - Deployment on docker wazuh#19101

Preconditions

1. Having a Wazuh installation in single mode with Docker

Steps to reproduce

Change Wazuh Indexer/Dashboard password 🔴

1. Stop the services

  root@ip-172-31-86-7:/home/ubuntu/wash-docker/single-node# sudo docker-compose down
  [+] Running 4/4
  ⠿ Container single-node-wazuh.dashboard-1  Removed                                                                                                                                                          10.4s
  ⠿ Container single-node-wazuh.manager-1    Removed                                                                                                                                                           4.0s
  ⠿ Container single-node-wazuh.indexer-1    Removed                                                                                                                                                           0.6s
  ⠿ Network single-node_default              Removed   

2. Generate a hash of a new password

root@ip-172-31-71-197:/home/ubuntu/wazuh-docker/single-node# docker run --rm -ti wazuh/wazuh-indexer:4.5.3 bash /usr/share/wazuh-indexer/plugins/opensearch-security/tools/hash.sh
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
[Password:]
$2y$12$1.ZV4D5PLpnd4suwgTVPMuPjLMurh2Txy1dzBnsxockVqaprEmyi.

3. Replace the new hash in the config/wazuh_indexer/internal_users.yml file

  admin:
    hash: "$2y$12$1.ZV4D5PLpnd4suwgTVPMuPjLMurh2Txy1dzBnsxockVqaprEmyi."
    reserved: true
    backend_roles:
    - "admin"
    description: "Demo admin user"

  kibanaserver:
    hash: "$2y$12$1.ZV4D5PLpnd4suwgTVPMuPjLMurh2Txy1dzBnsxockVqaprEmyi."
    reserved: true
    description: "Demo kibanaserver user"

4. Replace the old password with the new one in the docker-compose.yml file in all INDEXER_PASSWORD occurrences

wazuh.manager:
  environment:
    - INDEXER_URL=https://wazuh.indexer:9200
    - INDEXER_USERNAME=admin
    - INDEXER_PASSWORD=Barcelona!*
    - FILEBEAT_SSL_VERIFICATION_MODE=full
    - SSL_CERTIFICATE_AUTHORITIES=/etc/ssl/root-ca.pem
    - SSL_CERTIFICATE=/etc/ssl/filebeat.pem
    - SSL_KEY=/etc/ssl/filebeat.key
    - API_USERNAME=wazuh-wui
    - API_PASSWORD=MyS3cr37P450r.*-

wazuh.dashboard:
    - INDEXER_USERNAME=admin
    - INDEXER_PASSWORD=Barcelona!*
    - WAZUH_API_URL=https://wazuh.manager
    - DASHBOARD_USERNAME=kibanaserver
    - DASHBOARD_PASSWORD=Barcelona!*        
    - API_USERNAME=wazuh-wui
    - API_PASSWORD=MyS3cr37P450r.*-

5. Start services again

  root@ip-172-31-71-197:/home/ubuntu/wazuh-docker/single-node# docker-compose up -d
  [+] Running 4/4
   ⠿ Network single-node_default              Created                                                                                                                                                           0.1s
   ⠿ Container single-node-wazuh.manager-1    Started                                                                                                                                                           0.7s
   ⠿ Container single-node-wazuh.indexer-1    Started                                                                                                                                                           0.6s
   ⠿ Container single-node-wazuh.dashboard-1  Started     

6. Enter the Wazuh Indexer container

root@ip-172-31-71-197:/home/ubuntu/wazuh-docker/single-node# docker exec -it single-node-wazuh.indexer-1 bash

7. Set the required environment variables to apply the new password

  wazuh-indexer@wazuh:~$ export INSTALLATION_DIR=/usr/share/wazuh-indexer
  wazuh-indexer@wazuh:~$ CACERT=$INSTALLATION_DIR/certs/root-ca.pem
  wazuh-indexer@wazuh:~$ KEY=$INSTALLATION_DIR/certs/admin-key.pem
  wazuh-indexer@wazuh:~$ CERT=$INSTALLATION_DIR/certs/admin.pem
  wazuh-indexer@wazuh:~$ export JAVA_HOME=/usr/share/wazuh-indexer/jdk

8. Run the script to apply the changes

  wazuh-indexer@wazuh:~$ bash /usr/share/wazuh-indexer/plugins/opensearch-security/tools/securityadmin.sh -cd /usr/share/wazuh-indexer/opensearch-security/ -nhnv -cacert  $CACERT -cert $CERT -key $KEY -p 9200 -icl
  **************************************************************************
  ** This tool will be deprecated in the next major release of OpenSearch **
  ** https://github.com/opensearch-project/security/issues/1755           **
  **************************************************************************
  Security Admin v7
  Will connect to localhost:9200 ... done
  Connected as "CN=admin,OU=Wazuh,O=Wazuh,L=California,C=US"
  OpenSearch Version: 2.6.0
  Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
  Clustername: opensearch
  Clusterstate: GREEN
  Number of nodes: 1
  Number of data nodes: 1
  .opendistro_security index already exists, so we do not need to create one.
  Populate config from /usr/share/wazuh-indexer/opensearch-security/
  Will update '/config' with /usr/share/wazuh-indexer/opensearch-security/config.yml 
     SUCC: Configuration for 'config' created or updated
  Will update '/roles' with /usr/share/wazuh-indexer/opensearch-security/roles.yml 
     SUCC: Configuration for 'roles' created or updated
  Will update '/rolesmapping' with /usr/share/wazuh-indexer/opensearch-security/roles_mapping.yml 
     SUCC: Configuration for 'rolesmapping' created or updated
  Will update '/internalusers' with /usr/share/wazuh-indexer/opensearch-security/internal_users.yml 
     SUCC: Configuration for 'internalusers' created or updated
  Will update '/actiongroups' with /usr/share/wazuh-indexer/opensearch-security/action_groups.yml 
     SUCC: Configuration for 'actiongroups' created or updated
  Will update '/tenants' with /usr/share/wazuh-indexer/opensearch-security/tenants.yml 
     SUCC: Configuration for 'tenants' created or updated
  Will update '/nodesdn' with /usr/share/wazuh-indexer/opensearch-security/nodes_dn.yml 
     SUCC: Configuration for 'nodesdn' created or updated
  Will update '/whitelist' with /usr/share/wazuh-indexer/opensearch-security/whitelist.yml 
     SUCC: Configuration for 'whitelist' created or updated
  Will update '/audit' with /usr/share/wazuh-indexer/opensearch-security/audit.yml 
     SUCC: Configuration for 'audit' created or updated
  Will update '/allowlist' with /usr/share/wazuh-indexer/opensearch-security/allowlist.yml 
     SUCC: Configuration for 'allowlist' created or updated
  SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups
","config"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"]) due to: null
  Done with success

9. Enter the dashboard

Details

• Indexer

wazuh-indexer@wazuh:~$ cat /var/log/wazuh-indexer/wazuh-cluster.log | grep -i -E "error|warn"
cat: /var/log/wazuh-indexer/wazuh-cluster.log: No such file or directory

• Indices

root@ip-172-31-93-75:/home/ubuntu/wazuh-docker/single-node# curl https://ec2-3-88-190-235.compute-1.amazonaws.com:9200/_cat/indices/wazuh-alerts-* -u admin:murcia -k
green open wazuh-alerts-4.x-2023.09.22 ToeqcDWTS_CCfchP8TXngg 3 0 408 0 1.1mb 1.1mb

• API

root@ip-172-31-93-75:/home/ubuntu/wazuh-docker/single-node# curl -k -X GET "https://ec2-3-88-190-235.compute-1.amazonaws.com:55000/" -H "Authorization: Bearer $(curl -u wazuh-wui:MyS3cr37P450r.*- -k -X POST 'https://ec2-3-88-190-235.compute-1.amazonaws.com:55000/security/user/authenticate?raw=true')"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   404  100   404    0     0    703      0 --:--:-- --:--:-- --:--:--   702
{"data": {"title": "Wazuh API REST", "api_version": "4.5.3", "revision": 40506, "license_name": "GPL 2.0", "license_url": "https://github.com/wazuh/wazuh/blob/v4.5.3/LICENSE", "hostname": "wazuh.manager", "timestamp": "2023-09-22T14:16:57Z"}, "error": 0}root@ip-172-31-93-75:/home/ubuntu/wazuh-docker/single-node#

• Manager

root@wazuh:/# filebeat test output
elasticsearch: https://wazuh.indexer:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 172.20.0.2
    dial up... OK
  TLS...
    security: server's certificate chain verification is enabled
    handshake... OK
    TLS version: TLSv1.3
    dial up... OK
  talk to server... OK
  version: 7.10.2
root@wazuh:/# cat /var/log/filebeat/filebeat | grep -i -E "error|warn"

root@wazuh:/# cat /var/ossec/logs/ossec.log | grep -i -E "error|warn"
2023/09/22 13:49:56 wazuh-remoted: WARNING: (1408): Invalid ID 001 for the source ip: '34.201.138.149' (name 'unknown').
2023/09/22 13:50:06 wazuh-remoted: WARNING: (1408): Invalid ID 001 for the source ip: '34.201.138.149' (name 'unknown').
2023/09/22 13:50:16 wazuh-remoted: WARNING: (1408): Invalid ID 001 for the source ip: '34.201.138.149' (name 'unknown').
2023/09/22 13:50:26 wazuh-remoted: WARNING: (1408): Invalid ID 001 for the source ip: '34.201.138.149' (name 'unknown').
2023/09/22 13:50:36 wazuh-remoted: WARNING: (1408): Invalid ID 001 for the source ip: '34.201.138.149' (name 'unknown').
2023/09/22 14:10:06 wazuh-authd: WARNING: Duplicate name 'amazonlinux-aarch64', rejecting enrollment. Agent '001' can't be replaced since it is not disconnected.

• Dashboard

wazuh-dashboard@wazuh:~$ su journalctl -u wazuh-dashboard
Try 'su --help' for more information.
wazuh-dashboard@wazuh:~$ cat /usr/share/wazuh-dashboard/data/wazuh/logs/wazuhapp.log | grep -i -E "error|warn"
{"date":"2023-09-22T14:10:37.143Z","level":"error","location":"monitoring:getApiInfo","message":"connect ECONNREFUSED 172.20.0.3:55000"}
wazuh-dashboard@wazuh:~$ 

Expected Result

1. Connection to the dashboard

Actual Result
Status_code 500

Screenshots
Attached in the test

Further research should be done.
Multi-node installation could have the same issue, but the documentation should be defined first.

[teddytpc1]

The cause of the issue is that both passwords are being changed at the same time.

[pro-akim]

Update

The test was performed following the steps of the documentation https://documentation-dev.wazuh.com/v4.5.3-rc1/deployment-options/docker/wazuh-container.html#setting-the-new-password.
It appears that there isn't a step-by-step explanation provided for this process.