Add support to TCP and UDP protocols #224

ThibaudCartegnie · 2021-06-15T08:28:35Z

✨ Add support to TCP and UDP protocols #224

‼️ Once all the checklist is done you have to:

stash merge this pull request
delete the corresponding branch
close the associated issue

📃 Type of change

Breaking change: fix or feature that would cause existing functionality to not work as expected.

✒️ Description

This PR adds the support for udp and tcp besides Unix sockets.

There are 2 major changes for the protocol :

The Darwin Header no longer contains a Certitude by default and it is now defined as packed, lowering its size from 56 bytes to 48 bytes.
The Manager configuration file holds a 'network' field to specify how the filter receives data (Unix Socket, TCP or UDP)

This draft is pushed for early comments.

🎯 Test Environments

Ubuntu (20.04)

Redis (5.0.7)
Boost (1.71.0)
g++ (9.3)
CMake (3.16.3)
Python (3.8.5)
Valgrind (3.15.0)

HardenedBSD (12.2)

Redis (6.0.14)
Boost (1.72.0)
clang (10.0.1)
CMake (3.20.4)
Python (3.8.10)
Valgrind(N/A)

✔️ Checklist:

My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
(If new filter) I have added corresponding page to the documentation
(If other changes) I have made corresponding changes to the documentation
I have added tests that prove my fix is effective or that my feature works
New and existing unit tests pass locally with my changes

🙋 I certify on my honor that all the information provided is true, and I've done all I can to deliver a high quality code

…to 2 classes

…plied on the dga filter for now

If more than 1 certitude is sent back to client AND a body is set, the body is copied on the memory location of the certitudes It is fixed by using the correct offset when copying the body data. The other function using similar logic has been modified to keep it coherent.

UnixSessions are spawn by UnixServer and TcpSessions by TcpServer Included a modification when passing arguments from sessions to task (move instead of ref)

- Used to pass needed data from Session to Task

(some work is still need for the tasks)

State accessible by const getters only State mutable with only two methods : add Certitude and mutateBody

Added a few methods for DarwinPacket

Added ParseBody in ATask::run with error handling

Fixed issues with threadpool instance and size issues in the ASession

Fixed issues in the serialization Added support for ipv6 in Tcp sessions

Adapted python manager to handle tcp sockets

The packet was illformed near the certitudes NextFilterConnector: Working WIP

Abstract/Tcp/Unix, fixed configuration for nextFilter

- DarwinPacket copied only the first 15 bytes of event id - Destructor of AServer wasn't virtual leading to memory leak when UnixServer was destroyed

Service.update did not update nettwork field

Added retry attempts, modified tests accordingly

Proprietary filters needed additional methods for accessing private fields of a packet

Added possibility to run all tests in TCP mode

Assured working with ipv4 and ipv6 addresses Fixed a few log lines in the redis test file

Fixed a few problems and cleaned some code Added tests for tcp and udp Added possibility to run all test in unix or tcp

Removed unused methods

Specification of move semantic for DarwinPacket

Fixed a fex things accordingly

EvtIdToString passed from Session to DarwinPacket

Also cleaned ASession a bit

added packed attribute to darwin header Removed certitude list default size Modified parsing and serializing accordingly

Most case were unused variables or poorly chosen types

Fixed issue with positional flags between test/manager and filters

CMakeLists.txt

samples/base/AGenerator.hpp

samples/base/ATask.cpp

toolkit/Network.cpp

samples/fbuffer/Connectors/AConnector.cpp

samples/ftest/TestTask.cpp

samples/fyara/YaraTask.cpp

samples/base/network/ASession.hpp

- errors in comments - missing parts of tests (and logging missing) - removed useless variable in NextFilterConnector

frikilax · 2022-01-10T17:33:40Z

Yeah, I'll try being lighter next time when suggesting changes: applying/validating them is a pain on github 😅

frikilax · 2022-01-10T17:38:02Z

Outside my remaining comments (which aren't really important, except the one on RedisManager.cpp), everything seems great!

@HugoSoszynski thoughts on the PR and remaining comments I made (especially this one)?

Removed TODO : No handling of tcp case needed in that part, nothing to be changed

Thibaud Cartegnie and others added 30 commits June 9, 2021 17:51

separated unix socket logic from generic logic in the server class in…

a4ea276

…to 2 classes

Separated logic between tasks and Sessions, the separation is only ap…

534ba48

…plied on the dga filter for now

Separated Sessions into UnixSession and TcpSession

0149aee

UnixSessions are spawn by UnixServer and TcpSessions by TcpServer Included a modification when passing arguments from sessions to task (move instead of ref)

WIP : First draft of DarwinPacket struct

60f0efc

- Used to pass needed data from Session to Task

Finished integration of DarwinPacket in session

551209b

(some work is still need for the tasks)

Modified DarwinPacket to be more OOP-like

3c0be86

State accessible by const getters only State mutable with only two methods : add Certitude and mutateBody

Re-wired some things in the tasks

e8426bf

Added a few methods for DarwinPacket

Put back structs in c header

6072b7f

Added ParseBody in ATask::run with error handling

Adapted all compiled filters

f9b8b0f

Fixed issues with threadpool instance and size issues in the ASession

Added parsing of tcp/udp

de6f710

Fixed issues in the serialization Added support for ipv6 in Tcp sessions

Adapted TcpServer and Config for ip addresses

d957e01

Adapted python manager to handle tcp sockets

DarwinPacket: fixed issues with the serialization

711dc9d

The packet was illformed near the certitudes NextFilterConnector: Working WIP

Separated NextFilterConnector in :

3a46fe0

Abstract/Tcp/Unix, fixed configuration for nextFilter

Moved all network related objects to network folder

070660b

Fixed Set of asio buffer in ANextFilterConnector

3be3f48

Fixed memory issues:

2436f5d

- DarwinPacket copied only the first 15 bytes of event id - Destructor of AServer wasn't virtual leading to memory leak when UnixServer was destroyed

Fixed problem with the manager

80955db

Service.update did not update nettwork field

Fixed manager (failed previous commit)

55dfa07

RedisManager: modified Rate Limit connection logic

98a3039

Added retry attempts, modified tests accordingly

Adapted ATask, Asession and DarwinPacket

36e2ae0

Proprietary filters needed additional methods for accessing private fields of a packet

Adapted tests to TCP :

b8cbf63

Added possibility to run all tests in TCP mode

Added tests for TCP connections

00b207d

Assured working with ipv4 and ipv6 addresses Fixed a few log lines in the redis test file

Added UDP protocol to darwin

5139e2a

Fixed a few problems and cleaned some code Added tests for tcp and udp Added possibility to run all test in unix or tcp

Merge branch 'dev' into support_tcp_udp

79e7036

fixed few issues with the manager

b8d24db

Added comments documentation

9fabcd9

Removed unused methods

Fixed method names

47e58b7

Few modification of NextFilter handling

3daf60c

Specification of move semantic for DarwinPacket

Few fixes when passing information to next filters

06f9cbf

ThibaudCartegnie added 9 commits July 26, 2021 17:04

Added a tests for multi filters using the manager

8694a44

Fixed a fex things accordingly

Fixed issue with UUIDs

7b61fdd

EvtIdToString passed from Session to DarwinPacket

Fixed problems with next filters, cleaned tests

aa1b53c

Also cleaned ASession a bit

Modified config of next filter for easier configuration

aa05f12

Modified protocol.h

b088029

added packed attribute to darwin header Removed certitude list default size Modified parsing and serializing accordingly

Fixed clang/gcc warnings

13f4ab3

Most case were unused variables or poorly chosen types

Fixed compilation issues on hardenedBSD

b3478ed

Fixed issue with positional flags between test/manager and filters

Fixed last issue with UDP NextFilterConnector

15ef861

Added Thread Pool Cpp dependency

016d80f

ThibaudCartegnie marked this pull request as ready for review July 30, 2021 12:47

frikilax reviewed Dec 28, 2021

View reviewed changes

Fixed github TBE comments

2206178

- errors in comments - missing parts of tests (and logging missing) - removed useless variable in NextFilterConnector

ThibaudCartegnie mentioned this pull request Jan 21, 2022

Rework of the Python Filter (Depends on PR Support UDP TCP #224) #232

Draft

8 tasks

ThibaudCartegnie added 4 commits January 28, 2022 16:46

Fixed last few issues with comments

fd00830

Merge branch 'dev' into support_tcp_udp

2aab973

Adapted dga for new usage of darwin api

95cea44

Adapted dga for darwin api (forgot file)

1755c38

frikilax previously approved these changes Feb 1, 2022

View reviewed changes

HugoSoszynski previously approved these changes Mar 28, 2022

View reviewed changes

MANAGER:: Removed todo

c6352dd

Removed TODO : No handling of tcp case needed in that part, nothing to be changed

ThibaudCartegnie dismissed stale reviews from HugoSoszynski and frikilax via c6352dd April 7, 2022 09:07

frikilax approved these changes Apr 22, 2022

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support to TCP and UDP protocols #224

Add support to TCP and UDP protocols #224

ThibaudCartegnie commented Jun 15, 2021 •

edited

Loading

frikilax commented Jan 10, 2022

frikilax commented Jan 10, 2022

Add support to TCP and UDP protocols #224

Are you sure you want to change the base?

Add support to TCP and UDP protocols #224

Conversation

ThibaudCartegnie commented Jun 15, 2021 • edited Loading

✨ Add support to TCP and UDP protocols #224

📃 Type of change

✒️ Description

🎯 Test Environments

Ubuntu (20.04)

HardenedBSD (12.2)

✔️ Checklist:

frikilax commented Jan 10, 2022

frikilax commented Jan 10, 2022

ThibaudCartegnie commented Jun 15, 2021 •

edited

Loading