viewing the alert that triggered by the wazuh 6 months ago

[mostwanted5]

is it possible to view and analyse the alerts in wazuh dashboard when i re upload a log file like alert.json or archive.json that is deleted long ago from wazuh.

[s-ocando]

Hi @mostwanted5,

Yes, you can use the old alert files to re-inject alerts into your Wazuh dashboard for further analysis. Here's a link to an old blog post that includes detailed instructions on how to do so: Recover your data using Wazuh alerts backups.

This blog post includes a script named recovery.py that you can use to extract old alert files into a new file, for example, recovery.json. To ensure the smooth operation of your cluster, you can configure the script to write these old alerts at a designated Events Per Second (EPS) rate.

Configure Filebeat to read the recovery file by editing /usr/share/filebeat/module/wazuh/alerts/manifest.yml and adding the path to the new file:

module_version: 0.1

var:
  - name: paths
    default:
      - /var/ossec/logs/alerts/alerts.json
      - /tmp/recovery.json
  - name: index_prefix
    default: wazuh-alerts-4.x-

input: config/alerts.yml

ingest_pipeline: ingest/pipeline.json

Restart Filebeat so changes can take effect, and run the script to start writing the recovery file and re-indexing your alerts.

Let us know if you have any further questions.

[mostwanted5]

Hii,
@s-ocando thank you for responding,
But I don't need to recover the logs. I have that log for a specific date stored separately in a different system.I moved that file to the manager and tried to ship that log, but it is not working. Any solution for that

[s-ocando]

Hello,

If you have your alerts stored in a compressed file such as ossec-alerts-30.gz you can use the recovery.py script to extract the file and write the alerts into a new recovery file. By using Filebeat, you can then index these alerts into the Wazuh indexer, thereby making them accessible in the Wazuh dashboard.

Note that we don't use a static file. As the script writes the recovery file recovery.json, Filebeat ships the alerts to the Wazuh indexer.

This process is analogous to the default Wazuh operation: As a new alert is written into alerts.json, Filebeat ships it to the Wazuh indexer.

Please refer to the script's usage section to explore the available options, such as specifying the path to the old alert file.

Could you please provide further information on how you are currently attempting to ship the logs?

[mostwanted5]

Hello,
I tried to ship the log after running the recovery.py script using filebeat by adding the path in its file. But no log is showing in both dashboard and indexer.

[mostwanted5]

or do i need to create a seperate wazuh for viewing this?

[s-ocando]

Hi @mostwanted5,

No, there's no need to create a separate Wazuh instance to re-index the old alerts. Let's troubleshoot the issue:

• Verify if the /tmp/recovery.json file is created and if it contains the old alerts you want to re-index.

• Check the recovery.log looking for possible errors. Note that the script searches for the alerts in the following path for dates between the minimum and maximum timestamp specified in the execution:

 <wazuh-path>/logs/alerts/<YEAR>/<MONTH>/ossec-alerts-<DAY>.json.gz 

The <wazuh-path> is /var/ossec/ by default, but you can change it with the -w option.

Make sure you have the same folder structure as well as the expected file names.

If the recovery script can't find the old alert file, you'll see a message similar to this one in the recovery.log: "wazuh-reinjection: Couldn't find file <your-path>/logs/alerts/2023/May/ossec-alerts-16.json.gz"

• Be sure to add the /tmp/recovery.json file path in the /usr/share/filebeat/module/wazuh/alerts/manifest.yml file and restart Filebeat to apply changes:

systemctl restart filebeat

• Make sure to run the recovery script after you've edited the manifest.yml file and restarted Filebeat.

• In your Wazuh dashboard, look for the re-indexed alerts under the original date they occurred.

[mostwanted5]

Hi @s-ocando,
Thank for you quick response
I done all these steps without an error. Configured the path in filebeat, got the alerts i expected in the recovery.json file. Still not showing. I also tried to change the permission of the file.

[s-ocando]

Check for errors or warnings related to the recovery file under Filebeat logs: grep 'recovery' /var/log/filebeat/* and let us know what you find.

To be sure that all services are working as expected, check if you continue to receive new alerts on your dashboard.

For further debugging, share your manifest.yml and the recovery.log file.

[mostwanted5]

Hii,
this is my filebeat log file
2023-06-01T11:39:05.320+0530 INFO log/input.go:157 Configured paths: [/var/ossec/logs/alerts/alerts.json /tmp/recovery.json]
2023-06-01T11:39:15.321+0530 INFO log/harvester.go:302 Harvester started for file: /tmp/recovery.json
2023-06-01T12:21:56.304+0530 INFO log/harvester.go:333 File is inactive: /tmp/recovery.json. Closing because close_inactive of 5m0s reached.
2023-06-01T15:08:56.041+0530 INFO log/harvester.go:302 Harvester started for file: /tmp/recovery.json
2023-06-01T15:55:03.514+0530 INFO log/harvester.go:333 File is inactive: /tmp/recovery.json. Closing because close_inactive of 5m0s reached.

[mostwanted5]

manifest.yml

module_version: 0.1

var:

• name: paths
  default:
  • /var/ossec/logs/alerts/alerts.json
  • /tmp/recovery.json
• name: index_prefix
  default: wazuh-alerts-4.x-

input: config/alerts.yml

ingest_pipeline: ingest/pipeline.json
~
~

[mostwanted5]

recovery.log file

2023-06-01 15:08:55 wazuh-reinjection: Reading file: /var/ossec/logs/alerts/2023/Apr/ossec-alerts-10.json.gz
2023-06-01 15:50:00 wazuh-reinjection: Extracted 544361 alerts from day 10-Apr-2023

[s-ocando]

Try running the recovery script and lsof periodically to verify if Filebeat is reading the recovery.json file:

lsof -r 2 /tmp/recovery.json

You should get a similar output:

COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF     NODE NAME
filebeat 670 root   11r   REG    8,1   361366 67151046 /tmp/recovery.json

• Is Filebeat reading the file?
• Do you continue to receive alerts in your Wazuh dashboard?
• Did you find any warnings or errors in the Filebeat logs?

Also, verify if all services are up and running:

systemctl is-active filebeat wazuh-indexer wazuh-dashboard wazuh-manager 

[mostwanted5]

when i run lsof i am getting,
OMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
filebeat 27301 root 11r REG 253,0 14562284 68037574 /tmp/recovery.json
python 27338 root 4w REG 253,0 14562284 68037574 /tmp/recovery.json

But nothing showing in dashboard. All the services are active and running.

filebeat log,
2023-06-02T09:49:49.022+0530 INFO instance/beat.go:299 Setup Beat: filebeat; Version: 7.10.2
2023-06-02T09:49:49.026+0530 INFO [publisher] pipeline/module.go:113 Beat name: wazuh
2023-06-02T09:49:49.031+0530 INFO beater/filebeat.go:117 Enabled modules/filesets: wazuh (alerts), ()
2023-06-02T09:49:49.032+0530 INFO instance/beat.go:455 filebeat start running.
2023-06-02T09:49:49.034+0530 INFO memlog/store.go:119 Loading data file of '/var/lib/filebeat/registry/filebeat' succeeded. Active transaction id=628914
2023-06-02T09:49:49.886+0530 INFO memlog/store.go:124 Finished loading transaction log file for '/var/lib/filebeat/registry/filebeat'. Active transaction id=645655
2023-06-02T09:49:49.887+0530 INFO [registrar] registrar/registrar.go:109 States Loaded from registrar: 3
2023-06-02T09:49:49.888+0530 INFO [crawler] beater/crawler.go:71 Loading Inputs: 1
2023-06-02T09:49:49.889+0530 INFO log/input.go:157 Configured paths: [/var/ossec/logs/alerts/alerts.json /tmp/recovery.json]
2023-06-02T09:49:49.889+0530 INFO [crawler] beater/crawler.go:141 Starting input (ID: 14401419307270898487)
2023-06-02T09:49:49.889+0530 INFO [crawler] beater/crawler.go:108 Loading and starting Inputs completed. Enabled inputs: 1
2023-06-02T09:52:29.901+0530 INFO log/harvester.go:302 Harvester started for file: /var/ossec/logs/alerts/alerts.json
2023-06-02T09:52:30.902+0530 INFO [publisher_pipeline_output] pipeline/output.go:143 Connecting to backoff(elasticsearch(https://192.168.17.144:9200))
2023-06-02T09:52:30.902+0530 INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2023-06-02T09:52:30.903+0530 INFO [publisher] pipeline/retry.go:223 done
2023-06-02T09:52:30.930+0530 INFO [esclientleg] eslegclient/connection.go:314 Attempting to connect to Elasticsearch version 7.10.2
2023-06-02T09:52:30.931+0530 INFO [esclientleg] eslegclient/connection.go:314 Attempting to connect to Elasticsearch version 7.10.2
2023-06-02T09:52:30.934+0530 INFO template/load.go:97 Template wazuh already exists and will not be overwritten.
2023-06-02T09:52:30.934+0530 INFO [index-management] idxmgmt/std.go:298 Loaded index template.
2023-06-02T09:52:59.903+0530 INFO log/harvester.go:302 Harvester started for file: /tmp/recovery.json
2023-06-02T10:02:03.450+0530 INFO log/harvester.go:333 File is inactive: /tmp/recovery.json. Closing because close_inactive of 5m0s reached.

[s-ocando]

Hi @mostwanted5,

The Wazuh app includes a default filter that filters the alerts by manager name or cluster name. If the alerts were generated in another manager, if the name of the manager or the Wazuh cluster settings have recently changed, this filter may be preventing you from seeing the alerts.

To overcome this limitation, search for your alerts in the Discovery section. Go to the upper-left menu ☰ and select Discovery.

You can also verify if the index wazuh-alerts-4.x-2023.04.10 is created under ☰ > Dev Tools.

GET _cat/indices/wazuh-alerts-4.x-2023.04*

Let us know if this solves your issue.

[mostwanted5]

Hi @s-ocando,

Thank you for your response.
The manager I am using to view the history logs is the one that triggered those alerts. I tried to look into the Discover tab, as you said, but there are no logs from that source.
GET _cat/indices/wazuh-alerts-4.x-2023.04* , it is also verified

[s-ocando]

Can you confirm if GET _cat/indices/wazuh-alerts-4.x-2023.04* call generated results? Is the wazuh-alerts-4.x-2023.04.10 index present?

[mostwanted5]

it is present there.
i got the result like this
green open wazuh-alerts-4.x-2023.04.13 jXic0hjgTvqVFWYWPNKfHw 3 0 921264 0 613.2mb 613.2mb
green open wazuh-alerts-4.x-2023.04.14 wip8o3E-QOK_JKPeTky2zg 3 0 300856 0 155.2mb 155.2mb
green open wazuh-alerts-4.x-2023.04.17 blXLsl1QSMi_Tg8eTzrg_Q 3 0 2441 0 6.4mb 6.4mb
green open wazuh-alerts-4.x-2023.04.18 InSy2rcsTgyR_T8MN8Q00w 3 0 4530 0 8.5mb 8.5mb
green open wazuh-alerts-4.x-2023.04.19 syriLIBySdiA2YJpAdo1lw 3 0 4561 0 8.7mb 8.7mb
green open wazuh-alerts-4.x-2023.04.30 _OBNK2TcQSGMfPT7mpfG5w 3 0 898 0 2.2mb 2.2mb
green open wazuh-alerts-4.x-2023.04.10 TxYCGzJuQO6Z2N10imh6VA 3 0 1666785 0 1.1gb 1.1gb
green open wazuh-alerts-4.x-2023.04.11 gVkxUhQ4T5iwe3n3E3o09w 3 0 899031 0 695.3mb 695.3mb
green open wazuh-alerts-4.x-2023.04.12 ZL4iR9i4QVuJ9tBitGHbqg 3 0 1182114 0 881mb 881mb
green open wazuh-alerts-4.x-2023.04.02 bOTvrvJdQJSUrmRlGK8RuA 3 0 1004 0 1.9mb 1.9mb
green open wazuh-alerts-4.x-2023.04.24 zkV624rwQqOU3LpGuEZP8Q 3 0 2894 0 7.6mb 7.6mb
green open wazuh-alerts-4.x-2023.04.25 AGqxOIgPS1-nnpZIFKXmMw 3 0 2691 0 6.6mb 6.6mb
green open wazuh-alerts-4.x-2023.04.03 OHLPkm67Q6OuKBxXPtAOKQ 3 0 1289 0 3.2mb 3.2mb
green open wazuh-alerts-4.x-2023.04.26 KGKq7RcTR8ym5GeRsM-X-Q 3 0 2591 0 6.6mb 6.6mb
green open wazuh-alerts-4.x-2023.04.04 acBarUF6TPC2dWtgERPBDg 3 0 3532 0 7.6mb 7.6mb
green open wazuh-alerts-4.x-2023.04.27 fYufxso6SrKcc4h7rl7tCQ 3 0 2568 0 4.1mb 4.1mb
green open wazuh-alerts-4.x-2023.04.05 EdGPy8qySQ2TW0NdG-MqXA 3 0 25472 0 22.6mb 22.6mb
green open wazuh-alerts-4.x-2023.04.06 S7jcPvdxTOWsyzPs-3WiSQ 3 0 256764 0 193.5mb 193.5mb
green open wazuh-alerts-4.x-2023.04.28 adXqD140RLqjU_qGSuUm9A 3 0 2658 0 7mb 7mb
green open wazuh-alerts-4.x-2023.04.07 0_AO0HAGTS2qjUd5TDKuhg 3 0 375867 0 282.1mb 282.1mb
green open wazuh-alerts-4.x-2023.04.29 I4k-sqaSTzei4BcvLyPMgg 3 0 2107 0 4.5mb 4.5mb
green open wazuh-alerts-4.x-2023.04.08 VhAzdYMbSbyk_0xlzTf6WQ 3 0 60775 0 51.8mb 51.8mb
green open wazuh-alerts-4.x-2023.04.09 aF-IxDURT7KjN0QLD2YZ0w 3 0 215074 0 142.3mb 142.3mb
green open wazuh-alerts-4.x-2023.04.20 0OQX7yOCRNqhSaeLKNd4Hw 3 0 3364 0 7.1mb 7.1mb
green open wazuh-alerts-4.x-2023.04.21 IWaFxPB2RGWNULm0oqhjUQ 3 0 3054 0 6.9mb 6.9mb
green open wazuh-alerts-4.x-2023.04.22 awKNo4hmQr2T-N6tbcO1Kg 3 0 1902 0 4.2mb 4.2mb
green open wazuh-alerts-4.x-2023.04.23 GXgJ5NrpTjiQP-wxRMod1g 3 0 990 0 1.9mb 1.9mb
green open wazuh-alerts-4.x-2023.04.01 fpnX3PPxTneo-GNgEJtkHQ 3 0 593 0 1.6mb 1.6mb

[s-ocando]

I see the wazuh-alerts-4.x-2023.04.10 index is present, and I understand per your recovery.log that this index contains the old alerts you want to analyze further. Is that right?

If the index is present, you should be able to see the alerts in the Discovery section. Make sure to select the wazuh-alerts-* index pattern and the time range you want to explore.

Also, judging by the wazuh-alerts-4.x-2023.04.10 index size, it appears to have duplicated alerts. It may be that the alerts were indexed multiple times as you ran the script for testing.

If you think this is the case, consider deleting the index in ☰ > Dev Tools DELETE wazuh-alerts-4.x-2023.04.10 and running the recovery script one more time to create it again.

[mostwanted5]

Hi @s-ocando,

I have received the log in the discover tab. Thank you so much for your help and for your patience in answering my questions. I wanted this functionality because I delete my wazuh log files on a daily basis. Therefore, I need something like this to examine the old logs.

[s-ocando]

You're welcome! I'm very glad that everything is now working as expected. Feel free to ask more questions anytime.