Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Wazuh api requests cause file descriptors overflow #23367

Open
Radcriminal opened this issue May 9, 2024 · 0 comments
Open

Wazuh api requests cause file descriptors overflow #23367

Radcriminal opened this issue May 9, 2024 · 0 comments

Comments

@Radcriminal
Copy link

Radcriminal commented May 9, 2024

Wazuh version Component Install type Install method Platform
4.7.3-40714 Apid Manager Packages Ubuntu jammy

Consecutive requests to wazuh-api may violate the OS file descriptor limit.

/var/ossec/logs/api.log

2024/05/09 15:11:23 INFO: admin <REMOVED> "GET /sca/10955/checks/sshd_misconfiguration" with parameters {"result": "failed"} and body {} done in 0.047s: 200
2024/05/09 15:11:23 INFO: admin <REMOVED> "GET /sca/10957/checks/sshd_misconfiguration" with parameters {"result": "failed"} and body {} done in 0.031s: 200
2024/05/09 15:11:23 INFO: admin <REMOVED> "GET /sca/10958/checks/sshd_misconfiguration" with parameters {"result": "failed"} and body {} done in 0.045s: 200
2024/05/09 15:11:23 INFO: admin <REMOVED> "GET /sca/11130/checks/sshd_misconfiguration" with parameters {"result": "failed"} and body {} done in 0.040s: 200
2024/05/09 15:11:23 INFO: admin <REMOVED> "GET /sca/10959/checks/sshd_misconfiguration" with parameters {"result": "failed"} and body {} done in 0.052s: 200
2024/05/09 15:11:23 ERROR: Error handling request
Traceback (most recent call last):
  File "/var/ossec/framework/python/lib/python3.9/site-packages/connexion/apis/aiohttp_api.py", line 50, in problems_middleware
    response = yield from handler(request)
  File "/var/ossec/framework/python/lib/python3.9/site-packages/aiohttp/web_middlewares.py", line 105, in impl
    return await handler(request)
  File "/var/ossec/framework/python/lib/python3.9/site-packages/api-4.7.3-py3.9.egg/api/middlewares.py", line 207, in response_postprocessing
    return await handler(request)
  File "/var/ossec/framework/python/lib/python3.9/site-packages/api-4.7.3-py3.9.egg/api/middlewares.py", line 183, in security_middleware
    return await handler(request)
  File "/var/ossec/framework/python/lib/python3.9/site-packages/api-4.7.3-py3.9.egg/api/middlewares.py", line 115, in request_logging
    return await handler(request)
  File "/var/ossec/framework/python/lib/python3.9/site-packages/api-4.7.3-py3.9.egg/api/middlewares.py", line 45, in set_secure_headers
    resp = await handler(request)
  File "/var/ossec/framework/python/lib/python3.9/site-packages/aiohttp_cache/middleware.py", line 59, in cache_middleware
    return await handler(request)
  File "/var/ossec/framework/python/lib/python3.9/site-packages/connexion/decorators/coroutine_wrappers.py", line 21, in wrapper
    connexion_response = function(connexion_request)
  File "/var/ossec/framework/python/lib/python3.9/site-packages/connexion/decorators/security.py", line 322, in wrapper
    token_info = get_authorization_info(auth_funcs, request, required_scopes)
  File "/var/ossec/framework/python/lib/python3.9/site-packages/connexion/decorators/security.py", line 127, in get_authorization_info
    token_info = func(request, required_scopes)
  File "/var/ossec/framework/python/lib/python3.9/site-packages/connexion/decorators/security.py", line 302, in wrapper
    return verify_authorization_token(request, bearer_info_func)
  File "/var/ossec/framework/python/lib/python3.9/site-packages/connexion/decorators/security.py", line 175, in verify_authorization_token
    token_info = token_info_func(token)
  File "/var/ossec/framework/python/lib/python3.9/site-packages/api-4.7.3-py3.9.egg/api/authentication.py", line 275, in decode_token
    data = raise_if_exc(pool.submit(asyncio.run, dapi.distribute_function()).result()).to_dict()
  File "/var/ossec/framework/python/lib/python3.9/concurrent/futures/_base.py", line 446, in result
    return self.__get_result()
  File "/var/ossec/framework/python/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
  File "/var/ossec/framework/python/lib/python3.9/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/var/ossec/framework/python/lib/python3.9/asyncio/runners.py", line 39, in run
    loop = events.new_event_loop()
  File "/var/ossec/framework/python/lib/python3.9/asyncio/events.py", line 761, in new_event_loop
    return get_event_loop_policy().new_event_loop()
  File "/var/ossec/framework/python/lib/python3.9/asyncio/events.py", line 659, in new_event_loop
    return self._loop_factory()
  File "/var/ossec/framework/python/lib/python3.9/site-packages/uvloop/__init__.py", line 41, in _loop_factory
    return new_event_loop()
  File "/var/ossec/framework/python/lib/python3.9/site-packages/uvloop/__init__.py", line 20, in new_event_loop
    return Loop()
  File "uvloop/loop.pyx", line 165, in uvloop.loop.Loop.__cinit__
OSError: [Errno 24] Too many open files '/var/ossec/api/configuration/security/private_key.pem'

Seems some thread does not catch up closing fd's and at some point process breaches limit of 1024 fd's (default value) and crash.
You can monitor how many open fd's are using this command: watch -n 1 "ls /proc/${pid_of_wazuh-apid.py}/fd/ | wc -l"

One solution is to throttle api requests to let wazuh close fd's and another is to increase the os fd's limits, but anyway, it looks like a bug.

@Radcriminal Radcriminal changed the title Wazuh api requests causes file descriptors overflow Wazuh api requests cause file descriptors overflow May 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant