wazuh-agent crash randomly when RPCRT4.dll is loaded.

[4rth4S]

Wazuh version	Component	Install type	Install method	Platform
4.5.0	Wazuh Agent	Agent	Packages	Windows

Collected information about the issue

First

Faulting application name: wazuh-agent.exe, version: 0.0.0.0, time stamp: 0x64cd085b
Faulting module name: RPCRT4.dll, version: 10.0.19041.3208, time stamp: 0xfa2a686f
Exception code: 0xc0000005
Fault offset: 0x000270db
Faulting process id: 0x2368
Faulting application start time: 0x01d9d5a2efc14b3f
Faulting application path: C:\Program Files (x86)\ossec-agent\wazuh-agent.exe
Faulting module path: C:\Windows\System32\RPCRT4.dll
Report Id: e5c96d0e-6f14-40e1-95f0-051641a6d875
Faulting package full name: 
Faulting package-relative application ID: 

Second

Fault bucket 2079041460117890667, type 1
Event Name: APPCRASH
Response: Não disponível
Cab Id: 0

Problem signature:
P1: wazuh-agent.exe
P2: 0.0.0.0
P3: 64cd085b
P4: RPCRT4.dll
P5: 10.0.19041.3208
P6: fa2a686f
P7: c0000005
P8: 000270db
P9: 
P10: 

Attached files:
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WERD34D.tmp.dmp
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WERD513.tmp.WERInternalMetadata.xml
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WERD534.tmp.xml
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WERD532.tmp.csv
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WERD552.tmp.txt

These files may be available here:
\\?\C:\ProgramData\Microsoft\Windows\WER\ReportArchive\AppCrash_wazuh-agent.exe_7bbdc41d485e65bcf429ce682419348734782337_d71b03ac_aa7110f8-1dac-4a2b-b1ba-def9c507ff55

Analysis symbol: 
Rechecking for solution: 0
Report Id: e5c96d0e-6f14-40e1-95f0-051641a6d875
Report Status: 268435456
Hashed bucket: fc259e22111f28034cda3d3157a2e66b
Cab Guid: 0

WER data mentioned in Second

Version=1
EventType=APPCRASH
EventTime=133372559924207711
ReportType=2
Consent=1
UploadTime=133372559929870103
ReportStatus=268435456
ReportIdentifier=aa7110f8-1dac-4a2b-b1ba-def9c507ff55
IntegratorReportIdentifier=e5c96d0e-6f14-40e1-95f0-051641a6d875
Wow64Host=34404
Wow64Guest=332
NsAppName=wazuh-agent.exe
AppSessionGuid=00002368-0000-0016-3f4b-c1efa2d5d901
TargetAppId=W:0000dc721578e9892996b31e2f81267055ca00000904!00006500913d65da05689cb74b4d97131f4693c35aac!wazuh-agent.exe
TargetAppVer=2023//08//04:14:16:59!20e670!wazuh-agent.exe
BootId=4294967295
ServiceSplit=11599872
TargetAsId=163
IsFatal=1
EtwNonCollectReason=1
Response.BucketId=fc259e22111f28034cda3d3157a2e66b
Response.BucketTable=1
Response.LegacyBucketId=2079041460117890667
Response.type=4
Sig[0].Name=Nome da Aplicação
Sig[0].Value=wazuh-agent.exe
Sig[1].Name=Versão da Aplicação
Sig[1].Value=0.0.0.0
Sig[2].Name=Carimbo de Data//Hora da Aplicação
Sig[2].Value=64cd085b
Sig[3].Name=Nome do Módulo com Falhas
Sig[3].Value=RPCRT4.dll
Sig[4].Name=Versão do Módulo com Falhas
Sig[4].Value=10.0.19041.3208
Sig[5].Name=Carimbo de Data//Hora do Módulo com Falhas
Sig[5].Value=fa2a686f
Sig[6].Name=Código de exceção
Sig[6].Value=c0000005
Sig[7].Name=Desvio de Exceção
Sig[7].Value=000270db
DynamicSig[1].Name=Versão do SO
DynamicSig[1].Value=10.0.19045.2.0.0.256.48
DynamicSig[2].Name=ID de Região
DynamicSig[2].Value=2070
DynamicSig[22].Name=Informações Adicionais 1
DynamicSig[22].Value=2beb
DynamicSig[23].Name=Informações Adicionais 2
DynamicSig[23].Value=2beba6fb4680d73a8c78ca7c24ccdb46
DynamicSig[24].Name=Informações Adicionais 3
DynamicSig[24].Value=0f5f
DynamicSig[25].Name=Informações Adicionais 4
DynamicSig[25].Value=0f5f347ec91b2c234974086c81e5c6fb
UI[2]=C:\Program Files (x86)\ossec-agent\wazuh-agent.exe
UI[5]=Fechar
UI[8]=wazuh-agent.exe deixou de funcionar e foi fechado
UI[9]=Um problema fez parar o funcionamento correto da aplicação. O Windows notificá-lo-á se houver uma solução disponível.
UI[10]=&Fechar
LoadedModule[0]=C:\Program Files (x86)\ossec-agent\wazuh-agent.exe
LoadedModule[1]=C:\Windows\SYSTEM32\ntdll.dll
LoadedModule[2]=C:\Windows\System32\KERNEL32.DLL
LoadedModule[3]=C:\Windows\System32\KERNELBASE.dll
LoadedModule[4]=C:\Windows\System32\ADVAPI32.dll
LoadedModule[5]=C:\Windows\System32\msvcrt.dll
LoadedModule[6]=C:\Windows\System32\sechost.dll
LoadedModule[7]=C:\Windows\System32\RPCRT4.dll
LoadedModule[8]=C:\Windows\System32\CRYPT32.dll
LoadedModule[9]=C:\Windows\System32\ucrtbase.dll
LoadedModule[10]=C:\Windows\System32\PSAPI.DLL
LoadedModule[11]=C:\Windows\System32\SHLWAPI.dll
LoadedModule[12]=C:\Program Files (x86)\ossec-agent\libwinpthread-1.dll
LoadedModule[13]=C:\Windows\System32\USER32.dll
LoadedModule[14]=C:\Windows\System32\win32u.dll
LoadedModule[15]=C:\Windows\System32\GDI32.dll
LoadedModule[16]=C:\Windows\System32\gdi32full.dll
LoadedModule[17]=C:\Windows\System32\msvcp_win.dll
LoadedModule[18]=C:\Windows\System32\WINTRUST.dll
LoadedModule[19]=C:\Windows\System32\WS2_32.dll
LoadedModule[20]=C:\Windows\SYSTEM32\wevtapi.dll
LoadedModule[21]=C:\Program Files (x86)\ossec-agent\libwazuhext.dll
LoadedModule[22]=C:\Windows\SYSTEM32\WSOCK32.DLL
LoadedModule[23]=C:\Windows\SYSTEM32\MSASN1.dll
LoadedModule[24]=C:\Windows\SYSTEM32\CRYPTSP.dll
LoadedModule[25]=C:\Windows\system32\rsaenh.dll
LoadedModule[26]=C:\Windows\System32\bcrypt.dll
LoadedModule[27]=C:\Windows\SYSTEM32\CRYPTBASE.dll
LoadedModule[28]=C:\Windows\System32\bcryptPrimitives.dll
LoadedModule[29]=C:\Windows\System32\imagehlp.dll
LoadedModule[30]=C:\Windows\SYSTEM32\gpapi.dll
LoadedModule[31]=C:\Windows\System32\cryptnet.dll
LoadedModule[32]=C:\Windows\SYSTEM32\profapi.dll
LoadedModule[33]=C:\Program Files (x86)\ossec-agent\sysinfo.dll
LoadedModule[34]=C:\Program Files (x86)\ossec-agent\libgcc_s_dw2-1.dll
LoadedModule[35]=C:\Windows\SYSTEM32\IPHLPAPI.DLL
LoadedModule[36]=C:\Program Files (x86)\ossec-agent\libstdc++-6.dll
LoadedModule[37]=C:\Windows\SYSTEM32\WINHTTP.dll
LoadedModule[38]=C:\Windows\SYSTEM32\SspiCli.dll
LoadedModule[39]=C:\Windows\system32\mswsock.dll
LoadedModule[40]=C:\Windows\SYSTEM32\WINNSI.DLL
LoadedModule[41]=C:\Windows\System32\NSI.dll
LoadedModule[42]=C:\Program Files (x86)\ossec-agent\syscollector.dll
LoadedModule[43]=C:\Program Files (x86)\ossec-agent\rsync.dll
LoadedModule[44]=C:\Program Files (x86)\ossec-agent\dbsync.dll
LoadedModule[45]=C:\Windows\SYSTEM32\dhcpcsvc6.DLL
LoadedModule[46]=C:\Windows\system32\napinsp.dll
LoadedModule[47]=C:\Windows\SYSTEM32\ntmarta.dll
LoadedModule[48]=C:\Windows\SYSTEM32\dhcpcsvc.DLL
LoadedModule[49]=C:\Windows\system32\pnrpnsp.dll
LoadedModule[50]=C:\Windows\SYSTEM32\DNSAPI.dll
LoadedModule[51]=C:\Windows\system32\wshbth.dll
LoadedModule[52]=C:\Windows\system32\NLAapi.dll
LoadedModule[53]=C:\Windows\System32\winrnr.dll
State[0].Key=Transport.DoneStage1
State[0].Value=1
OsInfo[0].Key=vermaj
OsInfo[0].Value=10
OsInfo[1].Key=vermin
OsInfo[1].Value=0
OsInfo[2].Key=verbld
OsInfo[2].Value=19045
OsInfo[3].Key=ubr
OsInfo[3].Value=3208
OsInfo[4].Key=versp
OsInfo[4].Value=0
OsInfo[5].Key=arch
OsInfo[5].Value=9
OsInfo[6].Key=lcid
OsInfo[6].Value=2070
OsInfo[7].Key=geoid
OsInfo[7].Value=193
OsInfo[8].Key=sku
OsInfo[8].Value=48
OsInfo[9].Key=domain
OsInfo[9].Value=1
OsInfo[10].Key=prodsuite
OsInfo[10].Value=256
OsInfo[11].Key=ntprodtype
OsInfo[11].Value=1
OsInfo[12].Key=platid
OsInfo[12].Value=10
OsInfo[13].Key=sr
OsInfo[13].Value=0
OsInfo[14].Key=tmsi
OsInfo[14].Value=222284781
OsInfo[15].Key=osinsty
OsInfo[15].Value=1
OsInfo[16].Key=iever
OsInfo[16].Value=11.789.19041.0-11.0.1000
OsInfo[17].Key=portos
OsInfo[17].Value=0
OsInfo[18].Key=ram
OsInfo[18].Value=7926
OsInfo[19].Key=svolsz
OsInfo[19].Value=237
OsInfo[20].Key=wimbt
OsInfo[20].Value=0
OsInfo[21].Key=blddt
OsInfo[21].Value=191206
OsInfo[22].Key=bldtm
OsInfo[22].Value=1406
OsInfo[23].Key=bldbrch
OsInfo[23].Value=vb_release
OsInfo[24].Key=bldchk
OsInfo[24].Value=0
OsInfo[25].Key=wpvermaj
OsInfo[25].Value=0
OsInfo[26].Key=wpvermin
OsInfo[26].Value=0
OsInfo[27].Key=wpbuildmaj
OsInfo[27].Value=0
OsInfo[28].Key=wpbuildmin
OsInfo[28].Value=0
OsInfo[29].Key=osver
OsInfo[29].Value=10.0.19041.3208.amd64fre.vb_release.191206-1406
OsInfo[30].Key=buildflightid
OsInfo[31].Key=edition
OsInfo[31].Value=Professional
OsInfo[32].Key=ring
OsInfo[33].Key=expid
OsInfo[34].Key=fconid
OsInfo[35].Key=containerid
OsInfo[36].Key=containertype
OsInfo[37].Key=edu
OsInfo[37].Value=0
FriendlyEventName=Deixou de funcionar
ConsentKey=APPCRASH
AppName=wazuh-agent.exe
AppPath=C:\Program Files (x86)\ossec-agent\wazuh-agent.exe
NsPartner=windows
NsGroup=windows8
ApplicationIdentity=CF433907EA7BD1CAD555BDAF5C6E4EB5
MetadataHash=-1099065602

Community User Thread

• https://wazuh.slack.com/archives/C0A933R8E/p1692803888515089

Details

2023/08/23 08:58:14 wazuh-agent: INFO: Starting new log after rotation.
2023/08/23 08:58:14 wazuh-agent: INFO: Agent is now online. Process unlocked, continuing...
2023/08/23 08:58:14 rootcheck: INFO: Starting rootcheck scan.
2023/08/23 08:58:14 wazuh-agent: INFO: Agent is now online. Process unlocked, continuing...
2023/08/23 08:58:16 wazuh-agent: INFO: Agent is now online. Process unlocked, continuing...
2023/08/23 08:58:19 rootcheck: INFO: Ending rootcheck scan.
2023/08/23 08:58:33 wazuh-agent: INFO: (6009): File integrity monitoring scan ended.
2023/08/23 09:19:58 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2023/08/23 09:20:31 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2023/08/23 10:18:36 wazuh-agent: ERROR: Connection socket: Uma ligação existente foi forçada a fechar pelo anfitrião remoto. (10054)
2023/08/23 10:18:36 wazuh-agent: ERROR: (1137): Lost connection with manager. Setting lock.
2023/08/23 10:18:36 wazuh-agent: INFO: Closing connection to server ([148.69.129.196]:1124/tcp).
2023/08/23 10:18:36 wazuh-agent: INFO: Trying to connect to server ([148.69.129.196]:1124/tcp).
2023/08/23 10:18:36 wazuh-agent: ERROR: (1216): Unable to connect to '[148.69.129.196]:1124/tcp': 'Uma operação de socket foi tentada numa rede inacessível.'.
2023/08/23 10:19:26 wazuh-agent: INFO: Received exit signal. Starting exit process.
2023/08/23 10:19:26 wazuh-agent: INFO: Set pending exit signal.
2023/08/23 10:19:26 wazuh-agent: INFO: Exit completed successfully.
2023/08/23 10:19:36 wazuh-agent: INFO: Received exit signal. Starting exit process.
2023/08/23 10:19:36 wazuh-agent: INFO: Set pending exit signal.
2023/08/23 10:19:36 wazuh-agent: INFO: Exit completed successfully.
2023/08/23 10:19:52 wazuh-agent: INFO: Using notify time: 10 and max time to reconnect: 60
2023/08/23 10:19:52 wazuh-agent: INFO: (1410): Reading authentication keys file.
2023/08/23 10:19:52 wazuh-agent: INFO: Started (pid: 9064).
2023/08/23 10:19:52 wazuh-agent: INFO: Using AES as encryption method.
2023/08/23 10:19:52 wazuh-agent: INFO: Trying to connect to server ([148.69.129.196]:1124/tcp).
2023/08/23 10:19:52 rootcheck: INFO: Started (pid: 9064).
2023/08/23 10:19:52 wazuh-agent: INFO: (6002): Monitoring registry entry: 'HKEY_LOCAL_MACHINE\Software\Classes\batfile', with options 'size | permissions | owner | group | mtime | hash_md5 | hash_sha1 | hash_sha256'
2023/08/23 10:19:52 wazuh-agent: INFO: (6002): Monitoring registry entry: 'HKEY_LOCAL_MACHINE\Software\Classes\cmdfile', with options 'size | permissions | owner | group | mtime | hash_md5 | hash_sha1 | hash_sha256'
[LOTS of Registry monitoring entries]
2023/08/23 10:19:52 wazuh-agent: INFO: (6206): Ignore 'registry' entry 'HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\ADOVMPPackage\Final'
2023/08/23 10:19:52 wazuh-agent: INFO: (6207): Ignore 'registry' sregex '\Enum$'
2023/08/23 10:19:52 wazuh-agent: INFO: Started (pid: 9064).
2023/08/23 10:19:52 wazuh-agent: INFO: (4102): Connected to the server ([148.69.129.196]:1124/tcp).
2023/08/23 10:19:52 wazuh-modulesd:agent-upgrade: INFO: (8153): Module Agent Upgrade started.
2023/08/23 10:19:52 sca: INFO: Module started.
2023/08/23 10:19:52 wazuh-modulesd:osquery: INFO: Module disabled. Exiting...
2023/08/23 10:19:52 wazuh-modulesd:ciscat: INFO: Module disabled. Exiting...
2023/08/23 10:19:52 sca: INFO: Loaded policy 'C:\Program Files (x86)\ossec-agent\ruleset\sca\cis_win10_enterprise.yml'
2023/08/23 10:19:52 wazuh-agent: INFO: Windows version is 6.0 or newer. (Microsoft Windows 10 Pro [Ver: 10.0.19045.3208] - Wazuh v4.5.0).
2023/08/23 10:19:52 sca: INFO: Starting Security Configuration Assessment scan.
2023/08/23 10:19:52 wazuh-agent: INFO: (1951): Analyzing event log: 'Security'.
2023/08/23 10:19:52 wazuh-agent: INFO: (1950): Analyzing file: 'active-response\active-responses.log'.
2023/08/23 10:19:52 wazuh-agent: INFO: (1951): Analyzing event log: 'Microsoft-Windows-PowerShell/Operational'.
2023/08/23 10:19:52 wazuh-agent: INFO: (1951): Analyzing event log: 'Microsoft-Windows-Sysmon/Operational'.
2023/08/23 10:19:52 wazuh-agent: INFO: (6000): Starting daemon...
2023/08/23 10:19:52 wazuh-agent: INFO: (6010): File integrity monitoring scan frequency: 43200 seconds
2023/08/23 10:19:52 wazuh-agent: INFO: (6008): File integrity monitoring scan started.
2023/08/23 10:19:52 sca: INFO: Starting evaluation of policy: 'C:\Program Files (x86)\ossec-agent\ruleset\sca\cis_win10_enterprise.yml'
2023/08/23 10:19:52 rootcheck: INFO: Starting rootcheck scan.
2023/08/23 10:19:52 wazuh-modulesd:syscollector: INFO: Module started.
2023/08/23 10:19:52 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2023/08/23 10:19:52 wazuh-agent: INFO: Started (pid: 9064).
2023/08/23 10:19:52 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2023/08/23 10:44:24 wazuh-agent: INFO: Received exit signal. Starting exit process.
2023/08/23 10:44:24 wazuh-agent: INFO: Set pending exit signal.
2023/08/23 10:44:24 wazuh-agent: INFO: Exit completed successfully.
2023/08/23 10:44:41 wazuh-agent: INFO: Using notify time: 10 and max time to reconnect: 60
2023/08/23 10:44:41 wazuh-agent: INFO: (1410): Reading authentication keys file.
2023/08/23 10:44:41 wazuh-agent: INFO: Started (pid: 12136).
2023/08/23 10:44:41 wazuh-agent: INFO: Using AES as encryption method.
2023/08/23 10:44:41 wazuh-agent: INFO: Trying to connect to server ([148.69.129.196]:1124/tcp).
2023/08/23 10:44:41 wazuh-agent: INFO: (4102): Connected to the server ([148.69.129.196]:1124/tcp).
2023/08/23 10:44:41 rootcheck: INFO: Started (pid: 12136).

• https://wazuh.slack.com/archives/C0A933R8E/p1692010088024729

Details

Event Type: APPCRASH (an application crash event)
Application Name: wazuh-agent.exe
Application Version: 0.0.0.0
Faulting Module: RPCRT4.dll (Remote Procedure Call Runtime Library)
Faulting Module Version: 10.0.17763.4644
Exception Code: c0000005 (Access Violation)
Fault Offset: 0002823b
Error Report Location: Multiple temporary files and a report archive have been created in the C:\ProgramData\Microsoft\Windows\WER\ directory.
This event confirms the details of the previous error and adds information about the Windows Error Reporting process. It seems that an error report was generated and saved to the specified paths.

2023/08/02 00:00:10 wazuh-agent: INFO: Starting new log after rotation.
2023/08/02 00:59:46 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2023/08/02 00:59:52 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2023/08/02 01:59:53 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2023/08/02 01:59:59 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2023/08/02 03:00:00 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2023/08/02 03:00:05 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2023/08/02 04:00:06 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2023/08/02 04:00:12 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2023/08/02 05:00:13 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2023/08/02 05:00:18 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2023/08/02 06:00:19 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2023/08/02 06:00:25 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2023/08/02 07:00:26 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2023/08/02 07:00:32 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2023/08/02 08:00:33 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2023/08/02 08:00:40 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2023/08/02 09:00:41 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2023/08/02 09:00:46 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2023/08/02 10:00:47 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2023/08/02 10:00:52 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2023/08/02 11:00:53 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2023/08/02 11:00:59 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2023/08/02 11:36:36 sca: INFO: Starting Security Configuration Assessment scan.
2023/08/02 11:36:36 sca: INFO: Starting evaluation of policy: 'C:\Program Files (x86)\ossec-agent\ruleset\sca\cis_win2019.yml'
2023/08/02 11:36:39 sca: INFO: Evaluation finished for policy 'C:\Program Files (x86)\ossec-agent\ruleset\sca\cis_win2019.yml'
2023/08/02 11:36:39 sca: INFO: Security Configuration Assessment scan finished. Duration: 3 seconds.
2023/08/02 11:38:18 rootcheck: INFO: Starting rootcheck scan.
2023/08/02 11:38:18 rootcheck: ERROR: No winmalware file: './shared/win_malware_rcl.txt'
2023/08/02 11:38:18 rootcheck: ERROR: No winapps file: './shared/win_applications_rcl.txt'
2023/08/02 11:38:23 rootcheck: INFO: Ending rootcheck scan.
2023/08/02 11:44:15 wazuh-agent: INFO: (6008): File integrity monitoring scan started.
2023/08/02 11:44:37 wazuh-agent: INFO: (6009): File integrity monitoring scan ended.
2023/08/02 12:01:00 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2023/08/02 12:01:06 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2023/08/02 13:01:07 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2023/08/02 13:01:12 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2023/08/02 13:38:26 wazuh-agent: WARNING: The eventlog service is down. Unable to collect logs from 'Microsoft-Windows-Sysmon/Operational' channel.
2023/08/02 13:38:26 wazuh-agent: WARNING: The eventlog service is down. Unable to collect logs from 'Microsoft-Windows-PowerShell/Operational' channel.
2023/08/02 13:38:26 wazuh-agent: WARNING: The eventlog service is down. Unable to collect logs from 'Security' channel.
2023/08/02 15:58:19 wazuh-agent: INFO: Trust verification of a module failed by using the signature method. The signature of file 'C:\Program Files (x86)\ossec-agent\libwazuhext.dll' is expired.
2023/08/02 15:58:20 wazuh-agent: INFO: Trust verification of a module failed by using the hash method. CryptCATAdminEnumCatalogFromHash failed with error 1168 for 'C:\Program Files (x86)\ossec-agent\libwazuhext.dll': Element not found.
2023/08/02 15:58:20 wazuh-agent: WARNING: The file 'C:\Program Files (x86)\ossec-agent\libwazuhext.dll' is not signed or its signature is invalid.
2023/08/02 15:58:35 wazuh-agent: INFO: Using notify time: 10 and max time to reconnect: 60
2023/08/02 15:58:35 wazuh-agent: INFO: (1410): Reading authentication keys file.
2023/08/02 15:58:35 wazuh-agent: INFO: Started (pid: 3172).
2023/08/02 15:58:35 wazuh-agent: INFO: Using AES as encryption method.
2023/08/02 15:58:35 wazuh-agent: INFO: Trying to connect to server ([[Redacted].129.196]:1124/tcp).
2023/08/02 15:58:35 rootcheck: INFO: Started (pid: 3172).
2023/08/02 15:58:35 wazuh-agent: INFO: (6002): Monitoring registry entry: 'HKEY_LOCAL_MACHINE\Software\Classes\batfile', with options 'size | permissions | owner | group | mtime | hash_md5 | hash_sha1 | hash_sha256'
2023/08/02 15:58:35 wazuh-agent: INFO: (6207): Ignore 'registry' sregex '\Enum$'
2023/08/02 15:58:35 wazuh-agent: INFO: Started (pid: 3172).
2023/08/02 15:58:35 wazuh-agent: INFO: (4102): Connected to the server ([[Redacted].129.196]:1124/tcp).
2023/08/02 15:58:35 wazuh-agent: INFO: Windows version is 6.0 or newer. (Microsoft Windows Server 2019 Standard [Ver: 10.0.17763.4645] - Wazuh v4.4.3).
2023/08/02 15:58:35 wazuh-agent: INFO: (1951): Analyzing event log: 'Security'.
2023/08/02 15:58:35 wazuh-agent: INFO: (1950): Analyzing file: 'active-response\active-responses.log'.
2023/08/02 15:58:35 wazuh-agent: INFO: (1951): Analyzing event log: 'Microsoft-Windows-PowerShell/Operational'.
2023/08/02 15:58:36 sca: INFO: Module started.
2023/08/02 15:58:36 sca: INFO: Loaded policy 'C:\Program Files (x86)\ossec-agent\ruleset\sca\cis_win2019.yml'
2023/08/02 15:58:36 sca: INFO: Starting Security Configuration Assessment scan.
2023/08/02 15:58:36 wazuh-agent: INFO: (1951): Analyzing event log: 'Microsoft-Windows-Sysmon/Operational'.
2023/08/02 15:58:36 wazuh-agent: INFO: (6000): Starting daemon...
2023/08/02 15:58:36 wazuh-agent: INFO: (6010): File integrity monitoring scan frequency: 43200 seconds
2023/08/02 15:58:36 wazuh-agent: INFO: (6008): File integrity monitoring scan started.
2023/08/02 15:58:37 sca: INFO: Starting evaluation of policy: 'C:\Program Files (x86)\ossec-agent\ruleset\sca\cis_win2019.yml'
2023/08/02 15:58:37 wazuh-modulesd:syscollector: INFO: Module started.
2023/08/02 15:58:37 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2023/08/02 15:58:37 rootcheck: INFO: Starting rootcheck scan.
2023/08/02 15:58:37 wazuh-agent: INFO: Started (pid: 3172).
2023/08/02 15:58:37 wazuh-modulesd:agent-upgrade: INFO: (8153): Module Agent Upgrade started.
2023/08/02 15:58:37 wazuh-modulesd:osquery: INFO: Module disabled. Exiting...
2023/08/02 15:58:37 wazuh-modulesd:ciscat: INFO: Module disabled. Exiting...
2023/08/02 15:58:37 rootcheck: ERROR: No winmalware file: './shared/win_malware_rcl.txt'
2023/08/02 15:58:37 rootcheck: ERROR: No winapps file: './shared/win_applications_rcl.txt'
2023/08/02 15:58:37 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2023/08/14 10:34:57 wazuh-agent: INFO: Trust verification of a module failed by using the signature method. The signature of file 'C:\Program Files (x86)\ossec-agent\libwazuhext.dll' is expired.
2023/08/14 10:34:57 wazuh-agent: INFO: Trust verification of a module failed by using the hash method. CryptCATAdminEnumCatalogFromHash failed with error 1168 for 'C:\Program Files (x86)\ossec-agent\libwazuhext.dll': Element not found.
2023/08/14 10:34:57 wazuh-agent: WARNING: The file 'C:\Program Files (x86)\ossec-agent\libwazuhext.dll' is not signed or its signature is invalid.
2023/08/14 10:35:13 wazuh-agent: INFO: Using notify time: 10 and max time to reconnect: 60
2023/08/14 10:35:13 wazuh-agent: INFO: (1410): Reading authentication keys file.
2023/08/14 10:35:13 wazuh-agent: INFO: Started (pid: 7392).
2023/08/14 10:35:13 wazuh-agent: INFO: Using AES as encryption method.
2023/08/14 10:35:13 wazuh-agent: INFO: Trying to connect to server ([[Redacted].129.196]:1124/tcp).
2023/08/14 10:35:13 rootcheck: INFO: Started (pid: 7392).
2023/08/14 10:35:13 wazuh-agent: INFO: (6002): Monitoring registry entry: 'HKEY_LOCAL_MACHINE\Software\Classes\batfile', with options 'size | permissions | owner | group | mtime | hash_md5 | hash_sha1 | hash_sha256'
2023/08/14 10:35:13 wazuh-agent: INFO: (6207): Ignore 'registry' sregex '\Enum$'
2023/08/14 10:35:13 wazuh-agent: INFO: Started (pid: 7392).
2023/08/14 10:35:13 wazuh-modulesd:agent-upgrade: INFO: (8153): Module Agent Upgrade started.
2023/08/14 10:35:13 sca: INFO: Module started.
2023/08/14 10:35:13 sca: INFO: Loaded policy 'C:\Program Files (x86)\ossec-agent\ruleset\sca\cis_win2019.yml'
2023/08/14 10:35:13 sca: INFO: Starting Security Configuration Assessment scan.
2023/08/14 10:35:13 wazuh-agent: INFO: Windows version is 6.0 or newer. (Microsoft Windows Server 2019 Standard [Ver: 10.0.17763.4645] - Wazuh v4.4.3).
2023/08/14 10:35:13 sca: INFO: Starting evaluation of policy: 'C:\Program Files (x86)\ossec-agent\ruleset\sca\cis_win2019.yml'
2023/08/14 10:35:13 wazuh-agent: INFO: (1951): Analyzing event log: 'Security'.
2023/08/14 10:35:13 wazuh-agent: INFO: (1950): Analyzing file: 'active-response\active-responses.log'.
2023/08/14 10:35:13 wazuh-agent: INFO: (1951): Analyzing event log: 'Microsoft-Windows-PowerShell/Operational'.
2023/08/14 10:35:13 wazuh-agent: INFO: (1951): Analyzing event log: 'Microsoft-Windows-Sysmon/Operational'.
2023/08/14 10:35:13 wazuh-agent: INFO: (6000): Starting daemon...
2023/08/14 10:35:13 wazuh-agent: INFO: (6010): File integrity monitoring scan frequency: 43200 seconds
2023/08/14 10:35:13 wazuh-agent: INFO: (6008): File integrity monitoring scan started.
2023/08/14 10:35:13 wazuh-modulesd:osquery: INFO: Module disabled. Exiting...
2023/08/14 10:35:13 wazuh-modulesd:ciscat: INFO: Module disabled. Exiting...
2023/08/14 10:35:13 wazuh-modulesd:syscollector: INFO: Module started.
2023/08/14 10:35:13 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2023/08/14 10:35:13 rootcheck: INFO: Starting rootcheck scan.
2023/08/14 10:35:13 rootcheck: ERROR: No winmalware file: './shared/win_malware_rcl.txt'
2023/08/14 10:35:13 rootcheck: ERROR: No winapps file: './shared/win_applications_rcl.txt'
2023/08/14 10:35:13 wazuh-agent: INFO: Started (pid: 7392).
2023/08/14 10:35:14 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2023/08/14 10:35:16 sca: INFO: Evaluation finished for policy 'C:\Program Files (x86)\ossec-agent\ruleset\sca\cis_win2019.yml'
2023/08/14 10:35:16 sca: INFO: Security Configuration Assessment scan finished. Duration: 3 seconds.
2023/08/14 10:35:19 rootcheck: INFO: Ending rootcheck scan.
2023/08/14 10:35:39 wazuh-agent: INFO: (6009): File integrity monitoring scan ended.
2023/08/14 10:35:39 wazuh-agent: INFO: (6012): Real-time file integrity monitoring started.

Issue related:

• Winagent crash in ntdll.dll when restarting due to dll load. #18122

[pereyra-m]

Work in progress

The issue is under revision:

• No similar errors were found for another applications
• A Windows 2019 VM was tested but the error couldn't be reproduced. The same VM is being updated to force a change in the DLL version
• Another Windows 2019 VM is under testing
• The rpcrt4.dll load couldn't be directly traced to any section of the Wazuh agent code, it's being called from an independent thread (this behavior is as old as v4.0.0)
• The Eventchannel feature still is the most probable related feature
• The RPCViewer tool couldn't give more clues about the crash
• If the Windows OS is having an internal failure, maybe installing all the updates of the system might solve the problem

[pereyra-m]

Work in progress

The issue couldn't be reproduced.
Two different environments were tested, and one of them was updated to change the version of rpcrt4.dll

Details

Window Server 2019 VM

Window Server 2019 VM Updated

Window Server 2019 in EC2

There is a Wazuh feature that ends up calling CryptRetrieveObjectByUrlWithTimeoutThreadProc according to the backtrace reported by another user. But it isn't clear how (maybe Eventchannel or the .dll signature verification). If this is the case, the only workaround possible is to disable the capabilities one by one until the one that is causing the failure is found.

The question was posted in two Window forums:

• https://answers.microsoft.com/en-us/windowserver/forum/server_experience-servertop_other/application-crash-in-rpcrt4dll/c23140e8-549e-43af-91dd-47c5eaa52f6b
• https://learn.microsoft.com/en-us/answers/questions/1355360/application-crash-in-rpcrt4-dll

We could be facing a similar situation to these old issues, where an internal Windows problem was fixed by a KB:

• https://support.microsoft.com/en-us/topic/fix-error-in-rpcrt4-dll-causes-intermittent-crashes-when-you-start-a-windows-embedded-compact-2013-based-device-678b07d4-06e8-a75f-4434-af00ce62706c
• https://support.microsoft.com/en-us/topic/fix-error-in-rpcrt4-dll-leads-to-intermittent-crashes-when-you-start-a-windows-embedded-compact-7-based-device-cc56d304-bd98-d1db-c48d-492dc31507d6
• https://support.microsoft.com/en-us/topic/fix-a-memory-leak-occurs-in-rpcrt4-dll-during-suspend-and-resume-operations-when-a-windows-embedded-compact-7-based-device-communicates-with-a-remote-server-52e23525-f851-b1b1-e780-d5f880e12fb0

[Brain2000]

I found several systems of mine were no longer running the agent. After checking, I had the same crash and upon searching for it, found this page.

[Brain2000]

This probably has nothing to do with the crash, but I found three items in the event viewer a few minutes before wazuh-agent crashed, referring to DFSR replication:
10:49:41PM

DFSRs (3556,P,98) \\.\C:\System Volume Information\DFSR\database_1D3_54D7_8AF6_1040\dfsr.db: The database engine (10.00.17763.0000) is starting a new instance (0).

Also, it did not crash right away, it took at least a week. And it only crashed on 2 out of 8 IDENTICAL systems.
It very well could be a race condition.

[pereyra-m]

Hi @Brain2000 !
Thank you for the information.

Could you provide more details?
Ideally, the OS of the agents, the Wazuh version you are using and the ossec.log/ossec.conf files from the agents.
It's useful to know if the agents keep crashing right away after a restart or not.

And it only crashed on 2 out of 8 IDENTICAL systems.

This issue is hard to reproduce indeed.

I can also provide instructions for enabling the core dumps if you have direct access to those agents. A backtrace will allow us to determine if we are in the presence of the same issue or not.

Regards.

[Brain2000]

@pereyra-m yes, if you send me how to enable core dumps, and I'll add it to all the servers running the wazuh-agent. Now, it may take me a few weeks to get one to crash, so please be patient. But when it does, I'll have a core dump I can upload.

The OS is Windows 2019 [10.0.17763.4737]
Wazuh-agent v4.4.3

I may have found something after looking at the ossec.log, I found that it is looking for cis_win2012r2.yml. Well, when I installed the wazuh-agent, the OS was Windows 2012R2. However, I upgraded it to Windows 2019 last month! Regardless of the crash, those yml files should ALL be installed and it should autodetect which one to use.

Is it possible the wrong yml file could cause the crash? It seems doubtful since it was random and took several weeks.

ossec.log:

2023/09/05 00:00:10 wazuh-agent: INFO: Starting new log after rotation.
2023/09/05 00:57:43 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2023/09/05 00:57:47 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2023/09/05 01:57:48 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2023/09/05 01:57:53 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2023/09/05 02:57:54 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2023/09/05 02:57:59 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2023/09/05 03:58:00 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2023/09/05 03:58:05 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2023/09/05 04:49:08 sca: INFO: Starting Security Configuration Assessment scan.
2023/09/05 04:49:08 sca: INFO: Skipping policy 'C:\Program Files (x86)\ossec-agent\ruleset\sca\cis_win2012r2.yml': 'Check that the Windows platform is Windows Server 2012 R2'
2023/09/05 04:49:08 sca: INFO: Security Configuration Assessment scan finished. Duration: 0 seconds.
2023/09/05 04:49:50 rootcheck: INFO: Starting rootcheck scan.
2023/09/05 04:49:55 rootcheck: INFO: Ending rootcheck scan.
2023/09/05 04:51:52 wazuh-agent: INFO: (6008): File integrity monitoring scan started.
2023/09/05 04:52:16 wazuh-agent: INFO: (6009): File integrity monitoring scan ended.
2023/09/05 04:58:06 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2023/09/05 04:58:10 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2023/09/05 05:58:11 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2023/09/05 05:58:16 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2023/09/05 06:58:17 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2023/09/05 06:58:21 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2023/09/05 07:58:22 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2023/09/05 07:58:27 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2023/09/05 08:58:28 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2023/09/05 08:58:37 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2023/09/05 09:58:38 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2023/09/05 09:58:43 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2023/09/05 10:58:44 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2023/09/05 10:58:50 wazuh-modulesd:syscollector: INFO: Evaluation finished.

ossec.conf:

<!--
  Wazuh - Agent - Default configuration for Windows
  More info at: https://documentation.wazuh.com
  Mailing list: https://groups.google.com/forum/#!forum/wazuh
-->

<ossec_config>

  <client>
    <server>
      <address>192.168.0.57</address>
      <port>1514</port>
      <protocol>tcp</protocol>
    </server>
    <config-profile>windows, windows2012R2, windows-server, windows-server-2012R2</config-profile>
    <crypto_method>aes</crypto_method>
    <notify_time>10</notify_time>
    <time-reconnect>60</time-reconnect>
    <auto_restart>yes</auto_restart>
    <enrollment>
        <enabled>yes</enabled>
        <manager_address>192.168.0.57</manager_address>
        <agent_name>MSTSERVER13</agent_name>
        <groups>default</groups>
    </enrollment>
  </client>


  <!-- Agent buffer options -->
  <client_buffer>
    <disabled>no</disabled>
    <queue_size>5000</queue_size>
    <events_per_second>500</events_per_second>
  </client_buffer>

  <!-- Log analysis -->
  <localfile>
    <location>Application</location>
    <log_format>eventchannel</log_format>
  </localfile>

  <localfile>
    <location>Security</location>
    <log_format>eventchannel</log_format>
    <query>Event/System[EventID != 5145 and EventID != 5156 and EventID != 5447 and
      EventID != 4656 and EventID != 4658 and EventID != 4663 and EventID != 4660 and
      EventID != 4670 and EventID != 4690 and EventID != 4703 and EventID != 4907 and
      EventID != 5152 and EventID != 5157]</query>
  </localfile>

  <localfile>
    <location>System</location>
    <log_format>eventchannel</log_format>
  </localfile>

  <localfile>
    <location>active-response\active-responses.log</location>
    <log_format>syslog</log_format>
  </localfile>

  <!-- Policy monitoring -->
  <rootcheck>
    <disabled>no</disabled>
    <windows_apps>./shared/win_applications_rcl.txt</windows_apps>
    <windows_malware>./shared/win_malware_rcl.txt</windows_malware>
  </rootcheck>

  <!-- Security Configuration Assessment -->
  <sca>
    <enabled>yes</enabled>
    <scan_on_start>yes</scan_on_start>
    <interval>12h</interval>
    <skip_nfs>yes</skip_nfs>
  </sca>

  <!-- File integrity monitoring -->
  <syscheck>

    <disabled>no</disabled>

    <!-- Frequency that syscheck is executed default every 12 hours -->
    <frequency>43200</frequency>

    <!-- Default files to be monitored. -->
    <directories recursion_level="0" restrict="regedit.exe$|system.ini$|win.ini$">%WINDIR%</directories>

    <directories recursion_level="0" restrict="at.exe$|attrib.exe$|cacls.exe$|cmd.exe$|eventcreate.exe$|ftp.exe$|lsass.exe$|net.exe$|net1.exe$|netsh.exe$|reg.exe$|regedt32.exe|regsvr32.exe|runas.exe|sc.exe|schtasks.exe|sethc.exe|subst.exe$">%WINDIR%\SysNative</directories>
    <directories recursion_level="0">%WINDIR%\SysNative\drivers\etc</directories>
    <directories recursion_level="0" restrict="WMIC.exe$">%WINDIR%\SysNative\wbem</directories>
    <directories recursion_level="0" restrict="powershell.exe$">%WINDIR%\SysNative\WindowsPowerShell\v1.0</directories>
    <directories recursion_level="0" restrict="winrm.vbs$">%WINDIR%\SysNative</directories>

    <!-- 32-bit programs. -->
    <directories recursion_level="0" restrict="at.exe$|attrib.exe$|cacls.exe$|cmd.exe$|eventcreate.exe$|ftp.exe$|lsass.exe$|net.exe$|net1.exe$|netsh.exe$|reg.exe$|regedit.exe$|regedt32.exe$|regsvr32.exe$|runas.exe$|sc.exe$|schtasks.exe$|sethc.exe$|subst.exe$">%WINDIR%\System32</directories>
    <directories recursion_level="0">%WINDIR%\System32\drivers\etc</directories>
    <directories recursion_level="0" restrict="WMIC.exe$">%WINDIR%\System32\wbem</directories>
    <directories recursion_level="0" restrict="powershell.exe$">%WINDIR%\System32\WindowsPowerShell\v1.0</directories>
    <directories recursion_level="0" restrict="winrm.vbs$">%WINDIR%\System32</directories>

    <directories realtime="yes">%PROGRAMDATA%\Microsoft\Windows\Start Menu\Programs\Startup</directories>

    <ignore>%PROGRAMDATA%\Microsoft\Windows\Start Menu\Programs\Startup\desktop.ini</ignore>

    <ignore type="sregex">.log$|.htm$|.jpg$|.png$|.chm$|.pnf$|.evtx$</ignore>

    <!-- Windows registry entries to monitor. -->
    <windows_registry>HKEY_LOCAL_MACHINE\Software\Classes\batfile</windows_registry>
    <windows_registry>HKEY_LOCAL_MACHINE\Software\Classes\cmdfile</windows_registry>
    <windows_registry>HKEY_LOCAL_MACHINE\Software\Classes\comfile</windows_registry>
    <windows_registry>HKEY_LOCAL_MACHINE\Software\Classes\exefile</windows_registry>
    <windows_registry>HKEY_LOCAL_MACHINE\Software\Classes\piffile</windows_registry>
    <windows_registry>HKEY_LOCAL_MACHINE\Software\Classes\AllFilesystemObjects</windows_registry>
    <windows_registry>HKEY_LOCAL_MACHINE\Software\Classes\Directory</windows_registry>
    <windows_registry>HKEY_LOCAL_MACHINE\Software\Classes\Folder</windows_registry>
    <windows_registry arch="both">HKEY_LOCAL_MACHINE\Software\Classes\Protocols</windows_registry>
    <windows_registry arch="both">HKEY_LOCAL_MACHINE\Software\Policies</windows_registry>
    <windows_registry>HKEY_LOCAL_MACHINE\Security</windows_registry>
    <windows_registry arch="both">HKEY_LOCAL_MACHINE\Software\Microsoft\Internet Explorer</windows_registry>

    <windows_registry>HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services</windows_registry>
    <windows_registry>HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\KnownDLLs</windows_registry>
    <windows_registry>HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\SecurePipeServers\winreg</windows_registry>

    <windows_registry arch="both">HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Run</windows_registry>
    <windows_registry arch="both">HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\RunOnce</windows_registry>
    <windows_registry>HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\RunOnceEx</windows_registry>
    <windows_registry arch="both">HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\URL</windows_registry>
    <windows_registry arch="both">HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Policies</windows_registry>
    <windows_registry arch="both">HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Windows</windows_registry>
    <windows_registry arch="both">HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Winlogon</windows_registry>

    <windows_registry arch="both">HKEY_LOCAL_MACHINE\Software\Microsoft\Active Setup\Installed Components</windows_registry>

    <!-- Windows registry entries to ignore. -->
    <registry_ignore>HKEY_LOCAL_MACHINE\Security\Policy\Secrets</registry_ignore>
    <registry_ignore>HKEY_LOCAL_MACHINE\Security\SAM\Domains\Account\Users</registry_ignore>
    <registry_ignore type="sregex">\Enum$</registry_ignore>
    <registry_ignore>HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\MpsSvc\Parameters\AppCs</registry_ignore>
    <registry_ignore>HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\MpsSvc\Parameters\PortKeywords\DHCP</registry_ignore>
    <registry_ignore>HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\MpsSvc\Parameters\PortKeywords\IPTLSIn</registry_ignore>
    <registry_ignore>HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\MpsSvc\Parameters\PortKeywords\IPTLSOut</registry_ignore>
    <registry_ignore>HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\MpsSvc\Parameters\PortKeywords\RPC-EPMap</registry_ignore>
    <registry_ignore>HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\MpsSvc\Parameters\PortKeywords\Teredo</registry_ignore>
    <registry_ignore>HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\PolicyAgent\Parameters\Cache</registry_ignore>
    <registry_ignore>HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\RunOnceEx</registry_ignore>
    <registry_ignore>HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\ADOVMPPackage\Final</registry_ignore>

    <!-- Frequency for ACL checking (seconds) -->
    <windows_audit_interval>60</windows_audit_interval>

    <!-- Nice value for Syscheck module -->
    <process_priority>10</process_priority>

    <!-- Maximum output throughput -->
    <max_eps>100</max_eps>

    <!-- Database synchronization settings -->
    <synchronization>
      <enabled>yes</enabled>
      <interval>5m</interval>
      <max_interval>1h</max_interval>
      <max_eps>10</max_eps>
    </synchronization>
  </syscheck>

  <!-- System inventory -->
  <wodle name="syscollector">
    <disabled>no</disabled>
    <interval>1h</interval>
    <scan_on_start>yes</scan_on_start>
    <hardware>yes</hardware>
    <os>yes</os>
    <network>yes</network>
    <packages>yes</packages>
    <ports all="no">yes</ports>
    <processes>yes</processes>

    <!-- Database synchronization settings -->
    <synchronization>
      <max_eps>10</max_eps>
    </synchronization>
  </wodle>

  <!-- CIS policies evaluation -->
  <wodle name="cis-cat">
    <disabled>yes</disabled>
    <timeout>1800</timeout>
    <interval>1d</interval>
    <scan-on-start>yes</scan-on-start>

    <java_path>\\server\jre\bin\java.exe</java_path>
    <ciscat_path>C:\cis-cat</ciscat_path>
  </wodle>

  <!-- Osquery integration -->
  <wodle name="osquery">
    <disabled>yes</disabled>
    <run_daemon>yes</run_daemon>
    <bin_path>C:\Program Files\osquery\osqueryd</bin_path>
    <log_path>C:\Program Files\osquery\log\osqueryd.results.log</log_path>
    <config_path>C:\Program Files\osquery\osquery.conf</config_path>
    <add_labels>yes</add_labels>
  </wodle>

  <!-- Active response -->
  <active-response>
    <disabled>no</disabled>
    <ca_store>wpk_root.pem</ca_store>
    <ca_verification>yes</ca_verification>
  </active-response>

  <!-- Choose between plain or json format (or both) for internal logs -->
  <logging>
    <log_format>plain</log_format>
  </logging>

</ossec_config>

<!-- END of Default Configuration. -->

[pereyra-m]

Thank you @Brain2000 !

I don't see any problem in the shared logs.
To enable the crash dumps, you should run this registry file
enable_wazuh_dumps.zip

Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps\wazuh-agent.exe]
"DumpFolder"="c:\\CrashDumps"
"DumpCount"=dword:0000000a
"DumpType"=dword:00000002

They will be stored in c:\\CrashDumps

Is it possible the wrong yml file could cause the crash? It seems doubtful since it was random and took several weeks.

It doesn't seem probable, but the dump will tell us the exact point where the agent is crashing.

Regards.

[Brain2000]

Got it, will set this up today... will return when one crashes in a week or two

[Dwordcito]

@Brain2000 any news?

[Brain2000]

@Dwordcito It may take a few weeks.

[joseraeiro]

@pereyra-m I have an update. I have a crash dump related to this issue! I tried to upload it here, but it didn't accept the extension. I have uploaded it here instead.

https://files.fm/u/87pewfyty

[pereyra-m]

Thank you @joseraeiro !

Let me upload the dump file here so it doesn't get deleted

wazuh-agent.exe.3208.zip

The analysis of the dump shows a similar trace to the ones we've seen around the rpcrt4.dll library.

Details

*******************************************************************************
*                                                                             *
*                        Exception Analysis                                   *
*                                                                             *
*******************************************************************************

*** WARNING: Check Image - Checksum mismatch - Dump: 0xbece4, File: 0xbed3f - c:\symbolcache\rpcrt4.dll\41CE1768bd000\rpcrt4.dll

KEY_VALUES_STRING: 1

    Key  : AV.Dereference
    Value: NullClassPtr

    Key  : AV.Fault
    Value: Read

    Key  : Analysis.CPU.mSec
    Value: 1156

    Key  : Analysis.Elapsed.mSec
    Value: 6228

    Key  : Analysis.IO.Other.Mb
    Value: 8

    Key  : Analysis.IO.Read.Mb
    Value: 22

    Key  : Analysis.IO.Write.Mb
    Value: 38

    Key  : Analysis.Init.CPU.mSec
    Value: 3405

    Key  : Analysis.Init.Elapsed.mSec
    Value: 715682

    Key  : Analysis.Memory.CommitPeak.Mb
    Value: 76

    Key  : Failure.Bucket
    Value: NULL_CLASS_PTR_READ_c0000005_winnsi.dll!RpcNsiRegisterChangeNotification

    Key  : Failure.Hash
    Value: {fa4fe29a-56a5-9456-16d4-773eb6835a43}

    Key  : Timeline.OS.Boot.DeltaSec
    Value: 61

    Key  : Timeline.Process.Start.DeltaSec
    Value: 24

    Key  : WER.OS.Branch
    Value: rs5_release

    Key  : WER.OS.Version
    Value: 10.0.17763.1


FILE_IN_CAB:  wazuh-agent.exe.3208.dmp

NTGLOBALFLAG:  0

PROCESS_BAM_CURRENT_THROTTLED: 0

PROCESS_BAM_PREVIOUS_THROTTLED: 0

APPLICATION_VERIFIER_FLAGS:  0

CONTEXT:  (.ecxr)
eax=00081318 ebx=ffffffff ecx=0000003c edx=00000001 esi=00081344 edi=00081318
eip=74b5823b esp=020ff108 ebp=020ff124 iopl=0         nv up ei ng nz na pe nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010286
rpcrt4!NdrGetBuffer+0x3b:
74b5823b 817b04efcdab89  cmp     dword ptr [ebx+4],89ABCDEFh ds:002b:00000003=????????
Resetting default scope

EXCEPTION_RECORD:  (.exr -1)
ExceptionAddress: 74b5823b (rpcrt4!NdrGetBuffer+0x0000003b)
   ExceptionCode: c0000005 (Access violation)
  ExceptionFlags: 00000000
NumberParameters: 2
   Parameter[0]: 00000000
   Parameter[1]: 00000003
Attempt to read from address 00000003

PROCESS_NAME:  wazuh-agent.exe

READ_ADDRESS:  00000003 

ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%p referenced memory at 0x%p. The memory could not be %s.

EXCEPTION_CODE_STR:  c0000005

EXCEPTION_PARAMETER1:  00000000

EXCEPTION_PARAMETER2:  00000003

STACK_TEXT:  
020ff124 74b35bde     00081344 0000003c ffffffff rpcrt4!NdrGetBuffer+0x3b
020ff544 6f592845     6f591008 6f591340 020ff560 rpcrt4!NdrAsyncClientCall+0x1ce
020ff558 6f591ac7     020ff5cc ffffffff 00000000 winnsi!RpcNsiRegisterChangeNotification+0x23
020ff62c 6f591969     ffffffff 020ff63c 00000000 winnsi!NsiRpcRegisterChangeNotificationEx+0x147
020ff664 71837d0b     ffffffff 71831f20 00000007 winnsi!NsiRpcRegisterChangeNotification+0x49
020ff698 71837c6e     7183cc50 00000000 000fd448 IPHLPAPI!InternalRegisterChangeNotification+0x7b
020ff6b0 6a76453b     00000000 6a7646d0 000fd448 IPHLPAPI!NotifyIpInterfaceChange+0x6e
020ff6f8 6a76426e     00000000 00000000 020ff71c winhttp!NetworkChangeMonitor::Startup+0x79
020ff720 6a764ead     00000000 00000000 000bba30 winhttp!StartGlobalNetworkChangeMonitor+0x4e
020ff744 6a765421     6a7deb58 00000000 00000000 winhttp!WxRegisterForNetworkChangeNotification+0x35
020ff770 6a76608f     00000000 6a765f60 000bba30 winhttp!InitializeNetworkChangeMonitor+0x64
020ff810 6a765fd7     020ffb30 6a765f60 000bba30 winhttp!INTERNET_SESSION_HANDLE_OBJECT::LoadAutomaticProxyResolvers+0x90
020ff834 6a74ec5b     020ffb30 00000000 717d1b20 winhttp!INTERNET_SESSION_HANDLE_OBJECT::SetProxySettings+0x77
020ffafc 6a74aaad     020ffb30 0000000c 00000000 winhttp!WinHttpSetOptionInternal+0x8ab
020ffb68 717d6128     717d1b20 00000004 00000000 winhttp!WinHttpOpen+0x3cd
020ffb90 717d7130     0008bb48 020ffbc8 00202004 cryptnet!InetGetBindings+0x1a
020ffbcc 717d65e8     001326d8 00000006 00202004 cryptnet!CInetSynchronousRetriever::RetrieveObjectByUrl+0x160
020ffc08 717d5df9     001326d8 00000006 00202004 cryptnet!InetRetrieveEncodedObject+0x58
020ffc6c 717d5c40     001326d8 00000006 00202004 cryptnet!CObjectRetrievalManager::RetrieveObjectByUrl+0x9f
020ffce8 754005c9     00132618 754005b0 020ffd54 cryptnet!CryptRetrieveObjectByUrlWithTimeoutThreadProc+0x80
020ffcf8 774d777d     00132618 e958fd97 00000000 kernel32!BaseThreadInitThunk+0x19
020ffd54 774d774d     ffffffff 774f6603 00000000 ntdll!__RtlUserThreadStart+0x2f
020ffd64 00000000     717d5bc0 00132618 00000000 ntdll!_RtlUserThreadStart+0x1b


STACK_COMMAND:  ~6s; .ecxr ; kb

SYMBOL_NAME:  winnsi!RpcNsiRegisterChangeNotification+23

MODULE_NAME: winnsi

IMAGE_NAME:  winnsi.dll

FAILURE_BUCKET_ID:  NULL_CLASS_PTR_READ_c0000005_winnsi.dll!RpcNsiRegisterChangeNotification

OS_VERSION:  10.0.17763.1

BUILDLAB_STR:  rs5_release

OSPLATFORM_TYPE:  x86

OSNAME:  Windows 10

IMAGE_VERSION:  10.0.17763.1

FAILURE_ID_HASH:  {fa4fe29a-56a5-9456-16d4-773eb6835a43}

Followup:     MachineOwner
---------

Can you confirm the agent version?

[joseraeiro]

Yes, it's the version 4.5.0.

Thank you very much for all your help so far. This is a critical issue, I have a covering of barely 60% in a client's network. All the other agents are crashing.

I'll try to provide more dumps whenever the client sends them my way.

[Brain2000]

This is good news. Mine have not crashed again yet, but I'm checking daily. If/when they do, I'll double check that it's the same stack trace.

[dfoux]

Good morning @pereyra-m .

After trying to analyze the dump provided by @joseraeiro I've assumed that, when the Wazuh agent tries to launch itself in a Windows environment there is a checksum mismatch causing an Access violation exception at the address "74b5823b" of the rpcrt4.dll library.

We have already considered restarting the Wazuh agent service after crashing automatically but we were advised not to do it until we get further feedback from you.

Do you have any updates on this or has your team ever dealt with a similar issue before?

We really need to have this issue sorted out asap as we have many agents down for the last month and a half.

[pereyra-m]

Hello again everyone, I apologize for the late response.

We've been analyzing this issue for a while now, but we still haven't found the exact cause. It wasn't possible to reproduce the problem in our testing environments.
These are the steps I can recommend right now:

• Make sure the Windows hosts have all the corresponding OS patches and fixes, so we can discard a bug in the rpcrt4.dll library
• The same behavior was found in another client's machine and it was solved by upgrading the agent. Please upgrade Wazuh to the last stable version
• There is an important fix for the Windows agent that will be released soon in v4.5.3 (The dataprovider isn't properly managing heap memory during hardware scan on Windows - Implementation #18773). Until this version is available, you can temporarily disable the hardware scan of syscollector (https://documentation.wazuh.com/current/user-manual/reference/ossec-conf/wodle-syscollector.html#hardware). This way we can know if this is causing the crash in these environments.

Please post an update after applying the steps I suggested above.
Regards.

[EdwardsCP]

@pereyra-m I have some agents in my environment that are also crashing, faulting module RPCRT4.dll.

This is occurring for us on ossec agent versions 4.5.0 and 4.5.3, but only effecting a small percentage of total agents (probably less than 10%). I just upgraded some to 4.5.3 today to try to resolve.

I collected a dump from one.
Windows Server 2022 21H2 (OS Build 20348.1668)
Agent version 4.5.3

Crash Dump Exception Analysis

*******************************************************************************
*                                                                             *
*                        Exception Analysis                                   *
*                                                                             *
*******************************************************************************

*** WARNING: Check Image - Checksum mismatch - Dump: 0xcb77e, File: 0xbf79e - C:\ProgramData\Dbg\sym\rpcrt4.dll\48C33478bc000\rpcrt4.dll

KEY_VALUES_STRING: 1

    Key  : AV.Dereference
    Value: NullClassPtr

    Key  : AV.Fault
    Value: Read

    Key  : Analysis.CPU.mSec
    Value: 890

    Key  : Analysis.Elapsed.mSec
    Value: 36960

    Key  : Analysis.IO.Other.Mb
    Value: 15

    Key  : Analysis.IO.Read.Mb
    Value: 6

    Key  : Analysis.IO.Write.Mb
    Value: 27

    Key  : Analysis.Init.CPU.mSec
    Value: 343

    Key  : Analysis.Init.Elapsed.mSec
    Value: 62761

    Key  : Analysis.Memory.CommitPeak.Mb
    Value: 73

    Key  : Failure.Bucket
    Value: NULL_CLASS_PTR_READ_c0000005_winnsi.dll!RpcNsiRegisterChangeNotification

    Key  : Failure.Hash
    Value: {fa4fe29a-56a5-9456-16d4-773eb6835a43}

    Key  : Timeline.OS.Boot.DeltaSec
    Value: 26

    Key  : Timeline.Process.Start.DeltaSec
    Value: 18

    Key  : WER.OS.Branch
    Value: fe_release

    Key  : WER.OS.Version
    Value: 10.0.20348.1

    Key  : WER.Process.Version
    Value: 4.5.3.0


FILE_IN_CAB:  wazuh-agent.exe.3624.dmp

NTGLOBALFLAG:  0

PROCESS_BAM_CURRENT_THROTTLED: 0

PROCESS_BAM_PREVIOUS_THROTTLED: 0

APPLICATION_VERIFIER_FLAGS:  0

CONTEXT:  (.ecxr)
eax=00b9c180 ebx=00b9c180 ecx=0000003c edx=ffffffff esi=00b9c1ac edi=00b9c168
eip=755a5fde esp=02a0f2a0 ebp=02a0f2cc iopl=0         nv up ei ng nz na pe nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010286
rpcrt4!NdrGetBuffer+0x3e:
755a5fde 817a04efcdab89  cmp     dword ptr [edx+4],89ABCDEFh ds:002b:00000003=????????
Resetting default scope

EXCEPTION_RECORD:  (.exr -1)
ExceptionAddress: 755a5fde (rpcrt4!NdrGetBuffer+0x0000003e)
   ExceptionCode: c0000005 (Access violation)
  ExceptionFlags: 00000000
NumberParameters: 2
   Parameter[0]: 00000000
   Parameter[1]: 00000003
Attempt to read from address 00000003

PROCESS_NAME:  wazuh-agent.exe

READ_ADDRESS:  00000003 

ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%p referenced memory at 0x%p. The memory could not be %s.

EXCEPTION_CODE_STR:  c0000005

EXCEPTION_PARAMETER1:  00000000

EXCEPTION_PARAMETER2:  00000003

STACK_TEXT:  
02a0f2cc 75586a92     00b9c1ac 0000003c ffffffff rpcrt4!NdrGetBuffer+0x3e
02a0f6e8 73752b63     73751008 73751378 02a0f704 rpcrt4!NdrAsyncClientCall+0x202
02a0f6fc 73751c2b     02a0f770 ffffffff 00000000 winnsi!RpcNsiRegisterChangeNotification+0x23
02a0f7d0 73751ad9     ffffffff 02a0f7e0 00000000 winnsi!NsiRpcRegisterChangeNotificationEx+0x13b
02a0f808 745031bb     ffffffff 74501980 00000007 winnsi!NsiRpcRegisterChangeNotification+0x49
02a0f83c 74503110     745099c0 00000000 73bda090 IPHLPAPI!InternalRegisterChangeNotification+0x7b
02a0f854 73ba8f7f     00000000 73bdc120 00b71430 IPHLPAPI!NotifyIpInterfaceChange+0x60
02a0f8a0 73ba9a3b     00000001 00000000 75d49ff2 winhttp!NetworkChangeMonitor::Startup+0x6b
02a0f8c8 73ba8547     00000001 00000000 00bc3660 winhttp!StartGlobalNetworkChangeMonitor+0x58
02a0f8ec 73baa9e1     73c51d20 00000000 00000000 winhttp!WxRegisterForNetworkChangeNotification+0x35
02a0f918 73baade3     00bc3660 73bed7f0 00bc3710 winhttp!InitializeNetworkChangeMonitor+0xb5
02a0f9c0 73bed851     00bc3660 73bed7f0 02a0fa54 winhttp!INTERNET_SESSION_HANDLE_OBJECT::LoadAutomaticProxyResolvers+0x93
02a0f9e0 73bb6f03     02a0fa54 00000000 00bc3660 winhttp!INTERNET_SESSION_HANDLE_OBJECT::SetProxySettings+0x61
02a0fa1c 73ba5a17     02a0fa54 0000000c 00000000 winhttp!WinHttpSetOptionInternal+0x7d3
02a0fa78 747462a6     74741ab0 00000004 00000000 winhttp!WinHttpOpen+0x1f7
02a0faa0 74747350     00b57960 02a0fad8 00000001 cryptnet!InetGetBindings+0x1a
02a0fadc 747467a8     00bb0c28 00000002 00202005 cryptnet!CInetSynchronousRetriever::RetrieveObjectByUrl+0x160
02a0fb18 74745f41     00bb0c28 00000002 00202005 cryptnet!InetRetrieveEncodedObject+0x58
02a0fb7c 74745da0     00bb0c28 00000002 00202005 cryptnet!CObjectRetrievalManager::RetrieveObjectByUrl+0x95
02a0fbfc 76186a19     00bb0b68 76186a00 02a0fc64 cryptnet!CryptRetrieveObjectByUrlWithTimeoutThreadProc+0x80
02a0fc0c 77217c9d     00bb0b68 424d60e3 00000000 kernel32!BaseThreadInitThunk+0x19
02a0fc64 77217c6b     ffffffff 77246d79 00000000 ntdll!__RtlUserThreadStart+0x2b
02a0fc74 00000000     74745d20 00bb0b68 00000000 ntdll!_RtlUserThreadStart+0x1b


STACK_COMMAND:  ~9s; .ecxr ; kb

SYMBOL_NAME:  winnsi!RpcNsiRegisterChangeNotification+23

MODULE_NAME: winnsi

IMAGE_NAME:  winnsi.dll

FAILURE_BUCKET_ID:  NULL_CLASS_PTR_READ_c0000005_winnsi.dll!RpcNsiRegisterChangeNotification

OS_VERSION:  10.0.20348.1

BUILDLAB_STR:  fe_release

OSPLATFORM_TYPE:  x86

OSNAME:  Windows 10

IMAGE_VERSION:  10.0.20348.1

FAILURE_ID_HASH:  {fa4fe29a-56a5-9456-16d4-773eb6835a43}

Followup:     MachineOwner

Edit: Also, it's crashing on 4.5.3 with the syscollector wodle disabled.

OSSEC.log w/ syscollector disabled

2023/10/18 11:36:26 wazuh-agent: INFO: Using notify time: 10 and max time to reconnect: 60
2023/10/18 11:36:26 wazuh-agent: INFO: (1410): Reading authentication keys file.
2023/10/18 11:36:26 wazuh-agent: INFO: Started (pid: 3824).
2023/10/18 11:36:26 wazuh-agent: INFO: Using AES as encryption method.
2023/10/18 11:36:26 wazuh-agent: INFO: Trying to connect to server ([REDACTED]:1514/tcp).
2023/10/18 11:36:26 wazuh-agent: INFO: (4102): Connected to the server ([REDACTED]:1514/tcp).
2023/10/18 11:36:26 rootcheck: INFO: Started (pid: 3824).
2023/10/18 11:36:26 wazuh-agent: INFO: (6002): Monitoring registry entry: 'HKEY_LOCAL_MACHINE\Software\Classes\batfile', with options 'size | permissions | owner | group | mtime | hash_md5 | hash_sha1 | hash_sha256'
2023/10/18 11:36:26 wazuh-agent: INFO: (6002): Monitoring registry entry: 'HKEY_LOCAL_MACHINE\Software\Classes\cmdfile', with options 'size | permissions | owner | group | mtime | hash_md5 | hash_sha1 | hash_sha256'
2023/10/18 11:36:26 wazuh-agent: INFO: (6002): Monitoring registry entry: 'HKEY_LOCAL_MACHINE\Software\Classes\comfile', with options 'size | permissions | owner | group | mtime | hash_md5 | hash_sha1 | hash_sha256'
2023/10/18 11:36:26 wazuh-agent: INFO: (6002): Monitoring registry entry: 'HKEY_LOCAL_MACHINE\Software\Classes\exefile', with options 'size | permissions | owner | group | mtime | hash_md5 | hash_sha1 | hash_sha256'
2023/10/18 11:36:26 wazuh-agent: INFO: (6002): Monitoring registry entry: 'HKEY_LOCAL_MACHINE\Software\Classes\piffile', with options 'size | permissions | owner | group | mtime | hash_md5 | hash_sha1 | hash_sha256'
2023/10/18 11:36:26 wazuh-agent: INFO: (6002): Monitoring registry entry: 'HKEY_LOCAL_MACHINE\Software\Classes\AllFilesystemObjects', with options 'size | permissions | owner | group | mtime | hash_md5 | hash_sha1 | hash_sha256'
2023/10/18 11:36:26 wazuh-agent: INFO: (6002): Monitoring registry entry: 'HKEY_LOCAL_MACHINE\Software\Classes\Directory', with options 'size | permissions | owner | group | mtime | hash_md5 | hash_sha1 | hash_sha256'
2023/10/18 11:36:26 wazuh-agent: INFO: (6002): Monitoring registry entry: 'HKEY_LOCAL_MACHINE\Software\Classes\Folder', with options 'size | permissions | owner | group | mtime | hash_md5 | hash_sha1 | hash_sha256'
2023/10/18 11:36:26 wazuh-agent: INFO: (6002): Monitoring registry entry: 'HKEY_LOCAL_MACHINE\Software\Classes\Protocols [x64]', with options 'size | permissions | owner | group | mtime | hash_md5 | hash_sha1 | hash_sha256'
2023/10/18 11:36:26 wazuh-agent: INFO: (6002): Monitoring registry entry: 'HKEY_LOCAL_MACHINE\Software\Classes\Protocols', with options 'size | permissions | owner | group | mtime | hash_md5 | hash_sha1 | hash_sha256'
2023/10/18 11:36:26 wazuh-agent: INFO: (6002): Monitoring registry entry: 'HKEY_LOCAL_MACHINE\Software\Policies [x64]', with options 'size | permissions | owner | group | mtime | hash_md5 | hash_sha1 | hash_sha256'
2023/10/18 11:36:26 wazuh-agent: INFO: (6002): Monitoring registry entry: 'HKEY_LOCAL_MACHINE\Software\Policies', with options 'size | permissions | owner | group | mtime | hash_md5 | hash_sha1 | hash_sha256'
2023/10/18 11:36:26 wazuh-agent: INFO: (6002): Monitoring registry entry: 'HKEY_LOCAL_MACHINE\Security', with options 'size | permissions | owner | group | mtime | hash_md5 | hash_sha1 | hash_sha256'
2023/10/18 11:36:26 wazuh-agent: INFO: (6002): Monitoring registry entry: 'HKEY_LOCAL_MACHINE\Software\Microsoft\Internet Explorer [x64]', with options 'size | permissions | owner | group | mtime | hash_md5 | hash_sha1 | hash_sha256'
2023/10/18 11:36:26 wazuh-agent: INFO: (6002): Monitoring registry entry: 'HKEY_LOCAL_MACHINE\Software\Microsoft\Internet Explorer', with options 'size | permissions | owner | group | mtime | hash_md5 | hash_sha1 | hash_sha256'
2023/10/18 11:36:26 wazuh-agent: INFO: (6002): Monitoring registry entry: 'HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services', with options 'size | permissions | owner | group | mtime | hash_md5 | hash_sha1 | hash_sha256'
2023/10/18 11:36:26 wazuh-agent: INFO: (6002): Monitoring registry entry: 'HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\KnownDLLs', with options 'size | permissions | owner | group | mtime | hash_md5 | hash_sha1 | hash_sha256'
2023/10/18 11:36:26 wazuh-agent: INFO: (6002): Monitoring registry entry: 'HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\SecurePipeServers\winreg', with options 'size | permissions | owner | group | mtime | hash_md5 | hash_sha1 | hash_sha256'
2023/10/18 11:36:26 wazuh-agent: INFO: (6002): Monitoring registry entry: 'HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Run [x64]', with options 'size | permissions | owner | group | mtime | hash_md5 | hash_sha1 | hash_sha256'
2023/10/18 11:36:26 wazuh-agent: INFO: (6002): Monitoring registry entry: 'HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Run', with options 'size | permissions | owner | group | mtime | hash_md5 | hash_sha1 | hash_sha256'
2023/10/18 11:36:26 wazuh-agent: INFO: (6002): Monitoring registry entry: 'HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\RunOnce [x64]', with options 'size | permissions | owner | group | mtime | hash_md5 | hash_sha1 | hash_sha256'
2023/10/18 11:36:26 wazuh-agent: INFO: (6002): Monitoring registry entry: 'HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\RunOnce', with options 'size | permissions | owner | group | mtime | hash_md5 | hash_sha1 | hash_sha256'
2023/10/18 11:36:26 wazuh-agent: INFO: (6002): Monitoring registry entry: 'HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\RunOnceEx', with options 'size | permissions | owner | group | mtime | hash_md5 | hash_sha1 | hash_sha256'
2023/10/18 11:36:26 wazuh-agent: INFO: Windows version is 6.0 or newer. (Microsoft Windows Server 2022 Standard [Ver: 10.0.20348.1668] - Wazuh v4.5.3).
2023/10/18 11:36:26 wazuh-agent: INFO: (6002): Monitoring registry entry: 'HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\URL [x64]', with options 'size | permissions | owner | group | mtime | hash_md5 | hash_sha1 | hash_sha256'
2023/10/18 11:36:26 wazuh-agent: INFO: (6002): Monitoring registry entry: 'HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\URL', with options 'size | permissions | owner | group | mtime | hash_md5 | hash_sha1 | hash_sha256'
2023/10/18 11:36:26 wazuh-agent: INFO: (6002): Monitoring registry entry: 'HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Policies [x64]', with options 'size | permissions | owner | group | mtime | hash_md5 | hash_sha1 | hash_sha256'
2023/10/18 11:36:26 wazuh-agent: INFO: (6002): Monitoring registry entry: 'HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Policies', with options 'size | permissions | owner | group | mtime | hash_md5 | hash_sha1 | hash_sha256'
2023/10/18 11:36:26 wazuh-agent: INFO: (6002): Monitoring registry entry: 'HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Windows [x64]', with options 'size | permissions | owner | group | mtime | hash_md5 | hash_sha1 | hash_sha256'
2023/10/18 11:36:26 wazuh-agent: INFO: (6002): Monitoring registry entry: 'HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Windows', with options 'size | permissions | owner | group | mtime | hash_md5 | hash_sha1 | hash_sha256'
2023/10/18 11:36:26 wazuh-agent: INFO: (6002): Monitoring registry entry: 'HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Winlogon [x64]', with options 'size | permissions | owner | group | mtime | hash_md5 | hash_sha1 | hash_sha256'
2023/10/18 11:36:26 wazuh-agent: INFO: (6002): Monitoring registry entry: 'HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Winlogon', with options 'size | permissions | owner | group | mtime | hash_md5 | hash_sha1 | hash_sha256'
2023/10/18 11:36:26 wazuh-agent: INFO: (6002): Monitoring registry entry: 'HKEY_LOCAL_MACHINE\Software\Microsoft\Active Setup\Installed Components [x64]', with options 'size | permissions | owner | group | mtime | hash_md5 | hash_sha1 | hash_sha256'
2023/10/18 11:36:26 wazuh-agent: INFO: (6002): Monitoring registry entry: 'HKEY_LOCAL_MACHINE\Software\Microsoft\Active Setup\Installed Components', with options 'size | permissions | owner | group | mtime | hash_md5 | hash_sha1 | hash_sha256'
2023/10/18 11:36:26 wazuh-agent: INFO: (6003): Monitoring path: 'c:\programdata\microsoft\windows\start menu\programs\startup', with options 'size | permissions | owner | group | mtime | inode | hash_md5 | hash_sha1 | hash_sha256 | attributes | realtime'.
2023/10/18 11:36:26 wazuh-agent: INFO: (6003): Monitoring path: 'c:\windows', with options 'size | permissions | owner | group | mtime | inode | hash_md5 | hash_sha1 | hash_sha256 | attributes | scheduled'.
2023/10/18 11:36:26 wazuh-agent: INFO: (6003): Monitoring path: 'c:\windows\sysnative', with options 'size | permissions | owner | group | mtime | inode | hash_md5 | hash_sha1 | hash_sha256 | attributes | scheduled'.
2023/10/18 11:36:26 wazuh-agent: INFO: (6003): Monitoring path: 'c:\windows\sysnative\drivers\etc', with options 'size | permissions | owner | group | mtime | inode | hash_md5 | hash_sha1 | hash_sha256 | attributes | scheduled'.
2023/10/18 11:36:26 wazuh-agent: INFO: (6003): Monitoring path: 'c:\windows\sysnative\wbem', with options 'size | permissions | owner | group | mtime | inode | hash_md5 | hash_sha1 | hash_sha256 | attributes | scheduled'.
2023/10/18 11:36:26 wazuh-agent: INFO: (6003): Monitoring path: 'c:\windows\sysnative\windowspowershell\v1.0', with options 'size | permissions | owner | group | mtime | inode | hash_md5 | hash_sha1 | hash_sha256 | attributes | scheduled'.
2023/10/18 11:36:26 wazuh-agent: INFO: (6003): Monitoring path: 'c:\windows\system32', with options 'size | permissions | owner | group | mtime | inode | hash_md5 | hash_sha1 | hash_sha256 | attributes | scheduled'.
2023/10/18 11:36:26 wazuh-agent: INFO: (6003): Monitoring path: 'c:\windows\system32\drivers\etc', with options 'size | permissions | owner | group | mtime | inode | hash_md5 | hash_sha1 | hash_sha256 | attributes | scheduled'.
2023/10/18 11:36:26 wazuh-agent: INFO: (6003): Monitoring path: 'c:\windows\system32\wbem', with options 'size | permissions | owner | group | mtime | inode | hash_md5 | hash_sha1 | hash_sha256 | attributes | scheduled'.
2023/10/18 11:36:26 wazuh-agent: INFO: (6003): Monitoring path: 'c:\windows\system32\windowspowershell\v1.0', with options 'size | permissions | owner | group | mtime | inode | hash_md5 | hash_sha1 | hash_sha256 | attributes | scheduled'.
2023/10/18 11:36:26 wazuh-agent: INFO: (6206): Ignore 'file' entry 'c:\programdata\microsoft\windows\start menu\programs\startup\desktop.ini'
2023/10/18 11:36:26 wazuh-agent: INFO: (1951): Analyzing event log: 'Application'.
2023/10/18 11:36:26 wazuh-agent: INFO: (6207): Ignore 'file' sregex '.log$|.htm$|.jpg$|.png$|.chm$|.pnf$|.evtx$'
2023/10/18 11:36:26 wazuh-agent: INFO: (6206): Ignore 'registry' entry 'HKEY_LOCAL_MACHINE\Security\Policy\Secrets'
2023/10/18 11:36:26 wazuh-agent: INFO: (6206): Ignore 'registry' entry 'HKEY_LOCAL_MACHINE\Security\SAM\Domains\Account\Users'
2023/10/18 11:36:26 wazuh-agent: INFO: (6206): Ignore 'registry' entry 'HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\MpsSvc\Parameters\AppCs'
2023/10/18 11:36:26 wazuh-agent: INFO: (1951): Analyzing event log: 'Security'.
2023/10/18 11:36:26 wazuh-agent: INFO: (6206): Ignore 'registry' entry 'HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\MpsSvc\Parameters\PortKeywords\DHCP'
2023/10/18 11:36:26 wazuh-agent: INFO: (6206): Ignore 'registry' entry 'HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\MpsSvc\Parameters\PortKeywords\IPTLSIn'
2023/10/18 11:36:26 wazuh-agent: INFO: (6206): Ignore 'registry' entry 'HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\MpsSvc\Parameters\PortKeywords\IPTLSOut'
2023/10/18 11:36:26 wazuh-agent: INFO: (6206): Ignore 'registry' entry 'HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\MpsSvc\Parameters\PortKeywords\RPC-EPMap'
2023/10/18 11:36:26 sca: INFO: Module started.
2023/10/18 11:36:26 wazuh-modulesd:osquery: INFO: Module disabled. Exiting...
2023/10/18 11:36:26 wazuh-agent: INFO: (6206): Ignore 'registry' entry 'HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\MpsSvc\Parameters\PortKeywords\Teredo'
2023/10/18 11:36:26 wazuh-agent: INFO: (6206): Ignore 'registry' entry 'HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\PolicyAgent\Parameters\Cache'
2023/10/18 11:36:26 wazuh-modulesd:syscollector: INFO: Module disabled. Exiting...
2023/10/18 11:36:26 wazuh-agent: INFO: (1951): Analyzing event log: 'System'.
2023/10/18 11:36:26 wazuh-modulesd:ciscat: INFO: Module disabled. Exiting...
2023/10/18 11:36:26 wazuh-modulesd:agent-upgrade: INFO: (8153): Module Agent Upgrade started.
2023/10/18 11:36:26 wazuh-agent: INFO: (6206): Ignore 'registry' entry 'HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\RunOnceEx'
2023/10/18 11:36:26 sca: INFO: Loaded policy 'c:\Program Files (x86)\ossec-agent\ruleset\sca\cis_win2022.yml'
2023/10/18 11:36:26 wazuh-agent: INFO: (1950): Analyzing file: 'active-response\active-responses.log'.
2023/10/18 11:36:26 wazuh-agent: INFO: (6206): Ignore 'registry' entry 'HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\ADOVMPPackage\Final'
2023/10/18 11:36:26 sca: INFO: Starting Security Configuration Assessment scan.
2023/10/18 11:36:26 wazuh-agent: INFO: (6207): Ignore 'registry' sregex '\Enum$'
2023/10/18 11:36:26 wazuh-agent: INFO: Started (pid: 3824).
2023/10/18 11:36:26 wazuh-agent: INFO: (6000): Starting daemon...
2023/10/18 11:36:26 wazuh-agent: INFO: (6010): File integrity monitoring scan frequency: 43200 seconds
2023/10/18 11:36:26 wazuh-agent: INFO: (6008): File integrity monitoring scan started.
2023/10/18 11:36:26 rootcheck: INFO: Starting rootcheck scan.
2023/10/18 11:36:26 sca: INFO: Starting evaluation of policy: 'c:\Program Files (x86)\ossec-agent\ruleset\sca\cis_win2022.yml'
2023/10/18 11:36:26 wazuh-agent: INFO: Started (pid: 3824).

wazuh-agent.exe.3624.zip

[pereyra-m]

Hello @EdwardsCP !

Thank you for all the information provided.
The backtrace you've shown is the same we are analyzing.

Considering you are able to reproduce the issue, would it be possible to run some tests?

• First, please disable sca (https://documentation.wazuh.com/current/user-manual/reference/ossec-conf/sca.html#enabled), syscheck (https://documentation.wazuh.com/current/user-manual/reference/ossec-conf/syscheck.html#disabled) and rootcheck (https://documentation.wazuh.com/current/user-manual/reference/ossec-conf/rootcheck.html#disabled). This way we can discard these modules.
• If the crash still happens, I'll share here a custom .msi package that is based in v4.5.3 with IMAGE_TRUST_CHECKS=0 (https://documentation.wazuh.com/current/development/makefile.html). Consider that this installer isn't signed and it has the signature verification feature disabled. It should be installed briefly in an affected environment to verify if still crashes and it shouldn't be used in production
  wazuh-agent-4.5.3-2.zip

I'll be waiting for the results of the proposed tests.
Regards.

[EdwardsCP]

@pereyra-m,
Same behavior with sca, syscheck, and rootcheck disabled.

I've tried to reproduce the crash with the 4.5.3-2 installer you provided, but so far have not been successful. The server I'm using to test has most consistently had the service crash when it starts automatically after the system boots. It will usually start without crashing if I manually start the service at some other point in time. I've rebooted it twice with 4.5.3-2 installed, and the service started automatically and continued to run both times.

[EdwardsCP]

@pereyra-m ,
I rebooted the server I'm testing with a 3rd time, and the service started automatically and didn't crash.
Then I reinstalled the production copy of 4.5.3-1. After rebooting the server, the service started and crashed.

[pereyra-m]

Hello again @EdwardsCP !

Those tests seem really solid, thank you very much.
We'll be working on the signature verification feature.

In the meanwhile, do you have some extra details that could help us to reproduce the crash?
Are the affected hosts configured in some special way related to certificate chains, network access/connectivity, etc?
You've mentioned that only a small percentage of your agents are affected, have they something in common?

Regards.

[EdwardsCP]

In the meanwhile, do you have some extra details that could help us to reproduce the crash? Are the affected hosts configured in some special way related to certificate chains, network access/connectivity, etc? You've mentioned that only a small percentage of your agents are affected, have they something in common?

I can't identify any reason for some of our hosts behaving this way and others not. In addition to seeing it on some Server 2022 hosts, I'm also seeing it on Windows 11 22H2.

We have about 60 Win11 hosts that are about as close to identical as you can get in a production environment. They are identical make/model hardware, all imaged with the same OS/Drivers/Apps/etc deployment about 5-6 months ago, have identical AD Group Policies applied, same Endpoint Security software (with identical policies), and are managed by the same patch management system. On any given day, if I check for hosts that are online but disconnected from Wazuh (service crashed), we generally have about 2/3 of the total hosts online (40/60) and roughly 10% (4 or 5) of them are disconnected from Wazuh because the service crashed.

[pereyra-m]

Hi @EdwardsCP .

I understand that those hosts are almost identical then, thank you.
It hasn't been possible to reproduce the issue because all the users report a similar situation, there isn't an evident difference in the failing hosts.

I've been working on some reliability improvements around the signature verification feature, and it'll be really helpful for us if you could try another test package. If the tests are successful, we'll be able to release an official patch as soon as possible.

wazuh-agent-4.5.3-3.zip

Please, install it on the same machine you used for the last test.

Regards.

[EdwardsCP]

@pereyra-m, no crashes for my test server on 4.5.3-3 after 2 reboots. On the first reboot, sca, syscheck, and rootcheck were all still disabled. On the second reboot, I reenabled those modules.

[pereyra-m]

Thank you for all your help @EdwardsCP !! You've greatly contributed to improve Wazuh!

We'll be posting here any update.

[Dwordcito]

Thanks @EdwardsCP , merged in 4.7.0

[EdwardsCP]

@pereyra-m and/or @Dwordcito ,
Any thoughts on deploying the 4.5.3-3 agent in a production environment until 4.7 is released? It seems like it might be a good solution to bridge the gap and prevent our agents from crashing, but want to make sure there isn't any significant concern.

[dfoux]

When do you guys predict to release the version 4.7.0?

[pereyra-m]

Hi @EdwardsCP and @dfoux !

We are working to speed up the release process and it may be available next month.
In the meantime, you always can compile the agent by sources, here you have the required steps

https://documentation.wazuh.com/current/deployment-options/wazuh-from-sources/wazuh-agent/index.html

You could take the last stable branch and apply these changes, or simply compile the package without the signature verification feature until v4.7.0 is released.
The second option requires to add a flag in the step 4 like this (see https://documentation.wazuh.com/current/development/makefile.html#available-flags):

make deps TARGET=winagent
make TARGET=winagent  IMAGE_TRUST_CHECKS=0

Regards.

[joseraeiro]

Hello @pereyra-m !

I see that Wazuh 4.7.0 was released but see no mention of this bug having been fixed:

https://documentation.wazuh.com/current/release-notes/release-4-7-0.html

Could you please confirm that this issue is solved on this version?

[pereyra-m]

Hello @joseraeiro !

There is an entry under Resolved Issues named Improved reliability of the signature verification mechanism.
The fix has been released in this version.