New `Agent comms API` endpoint client

[TomasTurina]

Description

As detailed in wazuh/wazuh#22677, Wazuh's current communication setup is complex and needs to be refactored.

We want to replace the current wazuh-agentd service in charge of communicating with the server with a new agent. Additionally, the agent-auth tool will also be replaced by this agent.

The new agent must be able to perform the following tasks:

• Registration:
  • Whoever installs an agent has to register it on the server only once.
  • The UUID v7 is chosen by the agent and will identify it.
  • For the request, it will use its UUID.
  • The response will include the agent credentials that need to be stored.
  • The agent will use the Server management API (it needs the login token).
• Authentication and authorization:
  • To send/receive events, the agent needs a token.
  • For the request, it will use its UUID and credentials.
  • When it expires it must request another one and so on.
  • Proposal: OAuth 2.0 with JWTs.
  • The agent will use the Agent comms API.
• Communication agent → server:
  • Asynchronous and event driven.
  • Open connection, pass token, send events, close connection.
  • Bulking, batching, and buffering.
  • Proposal: HTTPS, API REST.
  • The agent will use the Agent comms API.
• Communication server → agent:
  • Connection-oriented.
  • The connection is the agent's initiative and must be kept established (pass token).
  • The agent would pull to see if it has anything (long polling GET).
  • Proposal: HTTPS, API REST.
  • The agent will use the Agent comms API.

Additionally, these are the API endpoints that the agent will use to communicate with the server:

• /login
  Authenticate (request token).
• /events/stateless
  Send events.
• /events/stateful
  The same as the previous one, but it requires persistent data.
• /commands
  In opposite direction, request made by agent to manager.

The focus of this issue will be on the following tasks:

• Initial client design, including technological and library research.
• PoC implementation for agent.

Implementation restrictions

• Work with C++ (at least 17) and CMake.
• Look for a library that has everything: libcurl, boost, gRPC, etc.
• Everything as standard as possible.
• Collaborate with the server team to align on communication protocols and API integration.

Plan

• Conduct initial research on system design and relevant technologies.
• Explore client-side designs and technologies for seamless interaction with the Agent Comms API.
• Describe the behavior of the new agent, create UML designs.
• Define new building system and new repository structure: Proposal.
• Work on a Proof of Concept implementation to validate communication flows between agents and servers.
  • Initially, work with mocks.
  • Establish connection, run endpoints.
  • Upload it to a /poc folder in the new repository.

POC working branch: https://github.com/wazuh/wazuh-agent/tree/1-spike-new-agent-comms-api-endpoint-client

[jr0me]

Initial libraries investigation

There are several C++ libraries that support our requirements. The ones described in this small list are widely used in the industry and tested. Note that no single library covers all needs: those that support high-level protocols like HTTP and HTTP/2 often lack native support for UDP, while libraries handling low-level protocols may not provide comprehensive support for higher-level abstractions.

According to wazuh/wazuh#23395 the server team is also considering a number of libraries, we should encourage sharing the same technology across both Agent and Server.

Libraries for Network Connections:

ZeroMQ (ØMQ):
ZeroMQ (ØMQ) is a high-performance asynchronous messaging library designed for distributed and concurrent applications. It supports various messaging patterns like publish/subscribe, request/reply, and client/server over multiple transports, including TCP, UDP, in-process, inter-process, multicast, and WebSocket (does not natively support HTTP/HTTP2). Highly suitable for building scalable and resilient distributed systems. The libzmq library is licensed under the Mozilla Public License 2.0.

https://zeromq.org/get-started/

gRPC:
gRPC is a high-performance open-source RPC framework that facilitates seamless communication between client and server applications, making it ideal for distributed systems. It uses HTTP/2 and Protocol Buffers for efficient serialization (does not support raw TCP/UDP). gRPC supports multiple languages, allowing interoperability between different systems. It is designed for low-latency and scalable communication, with features like load balancing, tracing, and authentication. It can also be extended to support various data formats like JSON, Protobuf, and others. It's license is Apache 2.

https://grpc.io/

Boost.Asio / Boost.Beast:
Boost.Asio provides a comprehensive suite for asynchronous network programming, supporting TCP, UDP, and serial communication. Boost.Beast extends Boost.Asio with HTTP and WebSocket capabilities. These libraries are highly performant and integrate well with other Boost libraries, offering extensive functionality for complex networking tasks.

https://www.boost.org/users/license.html
https://www.boost.org/doc/libs/1_84_0/doc/html/boost_asio.html
https://www.boost.org/doc/libs/1_85_0/libs/beast/doc/html/index.html

ASIO (Standalone):
The standalone ASIO library offers asynchronous I/O capabilities without the Boost dependency. It supports TCP, UDP, and serial port communications, and is suitable for applications requiring high performance (requires additional libraries for HTTP/HTTP2). ASIO is lightweight and provides efficient network communication primitives.

https://www.boost.org/LICENSE_1_0.txt
https://think-async.com/Asio/

CURL and http-request:

CURL supports many protocols like HTTP and HTTPS, with built-in SSL/TLS for secure communication. Given that we already have a CURL wrapper http-request, continuing to use CURL with it allows leveraging existing infrastructure and avoiding new dependencies, simplifying maintenance.

While it lacks native WebSocket support, CURL's wide support makes it suitable for HTTP-based event sending and command endpoints. Compared to other options, aside from the increased complexity of adding new dependencies ZeroMQ offers high-performance messaging but limited protocol support; gRPC supports HTTP/2 and real-time communication but is more complex given the need for proto buffer definitions. Continuing with CURL leverages existing code and avoids new dependencies.

Note that wazuh/wazuh-http-request repository requires files from wazuh/wazuh. Currently, it doesn't work as a standalone project and some minor changes need to be made for it to be used as a library.

https://curl.se/libcurl/
https://github.com/wazuh/wazuh-http-request

Libraries for JWTs:

jwt-cpp:
jwt-cpp is a lightweight library for creating and verifying JSON Web Tokens (JWT). It supports various algorithms for signing and verifying tokens, making it suitable for implementing token-based authentication in C++ applications. jwt-cpp is easy to integrate and provides a straightforward API for handling JWTs. Licensed under MIT.

https://thalhammer.github.io/jwt-cpp/

[TomasTurina]

New agent

• Unique binary.
• CLI to run and manage it:
  • Agent registration.
  • Enable or disable modules.
  • Check the status of the modules.
  • …
• Internal agent API:
  • Choose between TCP socket or HTTP.
  • Everything must be done by API.
• Communication with the new server:
  • Through the API that the server exposes.
  • Define protocols: HTTP,…
  • Define message format: JSON,…
  • Agent -> manager:
    • Event driven, send and close.
    • Dispatcher and communicator.
    • Bulk of messages:
      • By time or quantity.
      • Compress messages (msgpack).
      • Reuse connections (optional).
    • Batching:
      • Pool of threads.
      • Send message if a thread is available.
    • Buffering:
      • Don't discard messages.
      • Push-back.
      • Rocksdb,...
  • Manager -> agent:
    • Agent starts the connection, manager accepts it.
    • Agent requests data from the manager.
      • Long polling GET waiting for available data.
      • Only one connection.
      • Reopen connection if it times out.
  • Implement statistics in memory.
• Integrate the modules we have to the new agent.
• Use common interfaces even if the implementation is different for each OS.
  • Wrappers: libcurl,...
  • jemalloc
  • ...
• Thinking in the future:
  • There must be a local interface to communicate with the user.
  • Incorporate processing in the agent.
• Research other products:
  • Vector.
  • Datadog.
  • Osquery.
  • …

New repository

• Select development environment, build tools and dependency management:
  • Work in C++ (at least 17).
  • Build with cmake.
  • For dependencies use vcpkg or cmake package manager, seeking to reduce future work.
• Select common libraries:
  • Logging.
  • Testing.
  • Benchmarking.
  • Concurrency.
  • …
• Scaffolding:
  • README.
  • Directory structure.
  • Be able to separate portable code from non-portable code.
  • Legacy code?
  • …
• First develop the protocol, then bring the rest of the modules and adapt them.

[jr0me]

Update

New repo structure

Initial draft considerations:

• We shouldn't mix folder that include code with others that include configuration, or scripts files for example.
• Separate owned code from external third-party dependencies
• When possible, each module folder should generate at most one binary program.
• Each module, or folder, contains its own tests.

Additionally, we should aim to identify functionalities that could be replaced by an external library to offload the maintenance of these components. Furthermore, we need to establish a strategy for common code shared between the agent and server to avoid duplication.

wazuh-agent/
├── CMakeLists.txt
├── vcpkg.json
├── dependencies/ -> may not be necessary
├── modules/
│   ├── agent/
│   │   ├── CMakeLists.txt
│   │   ├── include/
│   │   │   └── [header files...] (include error_messages)
│   │   ├── src/
│   │   │   └── [source files...]
│   │   └── tests/
│   │       ├── CMakeLists.txt
│   │       └── [test files...]
│   ├── active_response/ (former os_execd + active_response)
│   ├── agent_upgrade/
│   ├── aws/
│   ├── azure/
│   ├── docker/
│   ├── gcp/
│   ├── github/
│   ├── ms_graph/
│   ├── office365/
│   ├── logcollector/
│   ├── rootcheck/
│   ├── fim/ (former syscheck)
│   ├── inventory/ (former syscollector)
│   ├── sca/
│   └── [additional modules...]
├── common/
│   ├── data_provider/
│   ├── dbsync/
│   ├── rsync/ -> may not be necessary
│   ├── os_crypto/ -> may not be necessary
│   ├── os_net/ -> may not be necessary
│   ├── os_regex/ -> may not be necessary
│   ├── os_xml/ -> may not be necessary
│   ├── os_zlib/ -> may not be necessary
│   ├── shared/ (include headers)
│   ├── utils/ (from shared_modules)
│   ├── commonDefs.h (from shared_modules/common)
│   └── [additional modules...]
└── build/
    └── [build output...]

For folders that is not code related to the Agent:

wazuh-agent/
├── ci/
├── etc/
│   ├── config/
│   ├── selinux/
│   └── ruleset/
│       ├── sca/
│       └── rootcheck/
├── packages/
├── tools/
└── installers/
    ├── unix/ (former init folder, including upgrade.sh and install.sh)
    └── win32/

Missing from this initial draft are other files such as .gitignore, .clang-format, .clang-tidy, etc.

Complete structure:

Opción 1:

wazuh-agent/
├── CMakeLists.txt
├── vcpkg.json
├── ci/
├── common/
├── dependencies/ -> may not be necessary
├── etc/
├── installers/
├── modules/
├── packages/
└── tools/

Opción 2:

wazuh-agent/
├── ci/
├── etc/
├── installers/
├── packages/
├── src/
│   ├── CMakeLists.txt
│   ├── vcpkg.json
│   ├── common/
│   ├── dependencies/ -> may not be necessary
│   └── modules/
└── tools/

[sdvendramini]

Communications server -> agent

/commands endpoint

Note

This is a work in progress

• POC
  • I started to investigate about SSE connections, to know how I could mock the server and establish the connection through a GET command from the client.

Update: 03/06/2024

• I did the first server-->agent (SSE) communication test using a server based on this example and a client made entirely with libcurl.
• checked out the http-request repo to see if it could be used for the PoC. Looks like it might need some changes.
• I started running some tests with http-request but didn't have success yet.

Update: 04/06/2024

• I was working completing the communicacion ussing http-request with SSE communication.
• The aproach implementing SSE communication was changed for HTTP Long Polling.

Update: 05/06/2024

• I was working in the PoC implementing /commands endpoint using HTTP Long Polling.

Update: 06/06/2024

• I was working in the PoC implementing /commands endpoint using HTTP Long Polling. A timeout response mechanism was implemented to simulate real behavior. Now I'm working on the storage of the commands.

Update: 07/06/2024

• The structure of storage for commands is ready. And a thread was created to dispatch the received commands. It is necessary to separate the commands logic to a new class. And make sure the rocksDB wrapper works for events as well.

Update: 10/06/2024

• The logic of the endpoint /commands was separated to be in a new module. Parameterized the DB name for rocksDB.

Update: 11/06/2024

• Request /agents added to register new agents.

Update: 13/06/2024

• Today I started working with the server created by pyserver team. Our client requires some changes to use the endpoints correctly.