Race condition when adding multiple agents quickly

[rossengeorgiev]

Trying to add multiple agents at the same time often return the same agent id for multiple requests. Generate multiple entries in clients.keys with the same agent id. Also, some of the shared secret will have a matching end half.

This presents a problem when attempting to automate key management with tools like salt, puppet, etc.

Steps to reproduce

1. Setup a freshly installed wazuh-manager and wazuh-api with no auth/ssl configured on ubuntu 16.04
2. Then try and add 10 agents at same time.

root@master:~$ for i in `seq 1 10`; do { curl -X PUT "http://127.0.0.1:55000/agents/myagentname$i?pretty" & } 2>/dev/null; done; wait 2>/dev/null
{
   "error": 0,
   "data": "001"
}
{
   "error": 0,
   "data": "002"
}
{
   "error": 0,
   "data": "003"
}
{
   "error": 0,
   "data": "004"
}
{
   "error": 0,
   "data": "004"
}
{
   "error": 0,
   "data": "005"
}
{
   "error": 0,
   "data": "006"
}
{
   "error": 0,
   "data": "006"
}
{
   "error": 0,
   "data": "007"
}
{
   "error": 0,
   "data": "007"
}
root@master:~$ cat /var/ossec/etc/client.keys
001 myagentname10 any cdbef82797197a5d13c53ceed24a8a48878f2a249288aa861ee4244d91306a03
002 myagentname8 any 58d981c501e84239bd88d242e5b33a5da88ca47804744af52d36742b61004271
003 myagentname5 any 130398127b6dc1a28fb4910b95790c7b003b7166d32c1373fe54992beb4dd7d5
004 myagentname9 any 43ffb2da13374aeb50de63ae8db62818556dcd93bf99b475c9cfbbeabb99ac88
004 myagentname6 any b9bc5b05d3c6ba38ae45fc2a7505dac1556dcd93bf99b475c9cfbbeabb99ac88
005 myagentname4 any 66ed6bf4f2f598ae7f00cab43678b16aa2e5f2fc76e433e20be359f5cd7433b8
006 myagentname1 any db5a0c4ebfd17514fe91cff2ac5a6ec31fb019b96bbadf4c799916475532f53a
006 myagentname2 any 8606bf6a7ecd8df140e6bb4af80363e11fb019b96bbadf4c799916475532f53a
007 myagentname7 any 2f55396540dbe5ec8c36005b57c4c235a139f693dee30e894bf452f4b8e90dc7
007 myagentname3 any d1244742f481652e67f579c4559beb5da139f693dee30e894bf452f4b8e90dc7

Version

root@master:~$ curl  http://127.0.0.1:55000/manager/info?pretty
{
   "error": 0,
   "data": {
      "installation_date": "Tue Sep 19 12:03:54 UTC 2017",
      "version": "v2.1.1",
      "openssl_support": "yes",
      "max_agents": "8000",
      "ruleset_version": "v2.1.0",
      "path": "/var/ossec",
      "tz_name": "UTC",
      "type": "server",
      "tz_offset": "+0000"
   }
}```

[mgmacias95]

Hi @rossengeorgiev,

This is a known issue commented in wazuh/wazuh#176.

The reason why this is happening is because the API call to add agents without using authd is not prepared for massive calls. We're working on this issue but I recommend you to use authd while we fix it.

When authd is activated, the API uses it. The steps to enable authd are explained in the Wazuh docs.

Best regards,
Marta

[rossengeorgiev]

Hi, authd is not ideal for managing keys, especially when looking into creating automated and dynamic deployments. A working API that has a granular access control would be extremely useful. This issue should really be high priority bug and blocking for v3,0. Current documentation should also be updated to mention that there is race condition.

[jesuslinares]

Hi @rossengeorgiev ,

we will fix the race condition in the API, but for now, we recommend to use authd.

If you enable authd, each request to the API related to register/remove agent will be "forwarded" to authd. Does it work for you?.

especially when looking into creating automated and dynamic deployments

What is your use case?.

Thanks.

[rossengeorgiev]

Hi @jesuslinares, thanks for clarifying. I didn't realize wazuh-api could make use of authd. I will look into giving that a try. My use case is large redundant deployment using saltstack. It will be monitoring a large number of hosts. My goal is to automate as many of the tasks as possible.

[rossengeorgiev]

Enabling authd seems to work.

[rossengeorgiev]

@jesuslinares any chance there could be conf switch to force the use of authd only? I fear that if authd dies for whatever reason it might turn into a mess.

[jesuslinares]

You are totally right. We are going to include that option in the next version.

For now, you can change the code.

• Go to /var/ossec/api/framework
• Remove the if-else section and use only "authd" functions.
  • https://github.com/wazuh/wazuh-api/blob/stable/framework/wazuh/agent.py#L260
  • https://github.com/wazuh/wazuh-api/blob/stable/framework/wazuh/agent.py#L391

Thanks so much.

[mgmacias95]

Hello @rossengeorgiev,

As you suggested, a new parameter has been added in API configuration to force the use of authd (see #43 and wazuh/wazuh#305). This will be added in Wazuh 3.1.

We're not closing this issue since the original bug is not yet fixed.

Best regards,
Marta

[mgmacias95]

Hello again,

This bug has been fixed in wazuh/wazuh#306.

Best regards,
Marta