Migrate agent-group files to wazuh-db #10771
Assignees
Labels
Comments
davidjiglesias
added
module/agent
17 tasks
This was referenced Nov 24, 2021
Implement a new column in the
agent table of global.db to save the hash of the agents' groups
#10951
Closed
19 tasks
This was referenced Nov 25, 2021
15 tasks
14 tasks
5 tasks
13 tasks
1 task
8 tasks
Closed
13 tasks
Closed
5 tasks
4 tasks
4 tasks
2 tasks
Closed
2 tasks
6 tasks
3 tasks
davidjiglesias
pushed a commit
that referenced
this issue
May 9, 2022
* Implement sync_agent_groups_get command * Improve json groups response * Code review fixes * Create set-agent-group command * Fix set-agent-groups command * Add DOXYGEN * Add set-agent-groups helper method * Code review fixes * Fix upgrade mechanism * Code review changes * Moving the client.keys synchronization mechanism with wazuh-db from the synchronization module to authd in the master nodes * Fixing agent compilation errors * Fixing comments from PR * Fixing error generated during the database creation * Creating parsing methods for global backup mechanism * Implemented the global_create_backup method * Created wdb_global_get_backup command * Implementing wdb_global_restore_backup method * Reading new config for Wazuh-DB * Deleting old backups * Creating Wazuh-DB backup thread * Creating backup/db folder * Deleting more than one old backup if necessary * Closing global database after a backup restore command * Removing pre-restore backup from most recent snapshots list * Fixing scan-build warnings * Always setting wdb configuration block default values * Adding result message for regular backups * Adding the wdb configuration block to ossec.conf * Minor changes after review * Improving array of configuration nodes implementation * Improving the delete mechanism of old backups * Restoring eval_bool definition * Restoring get_time_interval definition place * Minor fixes * Fixes after review * Fixing Windows agent compilation * Replaced function that updates group tables info * Code review changes * Inserting groups files in database * Making modulesd synchronize client.keys only at startup for master node * After review improvements * Updating the backup creation method * Finalizing all statements to run VACUUM * Fixing scan-build warnings * Adding a getconfig command for Wazuh-DB * Fixing comment and Valgrind warnings * Number of groups per agent reduced * Truncate groups file * Adapt groups size truncation to the upgrade mechanism * Removing deprecated folders from installation scripts * Restoring agents database creation to the database synchronization module to avoid errors in authd * Fixing wazuh-db output buffer usage * Separating actions taken by master and worker nodes * Fixing comments from pull request * Fixing UT compilation * Creating pre and post upgrade global DB snapshots * Print out backup filename * Code review changes * Added tag to create snapshots with custom names during upgrade * Removed log debug 2 during upgrade * Code review changes * Added log messages, hide wazuh-db configuration * Added error log while doing backup by request and by interval * Code review changes * Fixing UT compilation * Testing wazuh-db settings parser * Deprecating the ability to set agent groups from share_download 'files.yml' * Remove group_source logic from the set-agent-groups command * Add error message when set-agent-groups has any fail. * Remove group_source from database module call * Adding the global group hash methods and cache * Improvements after review * Simplify the group_hash from the sql schemas * Adapt sync_agent_groups_get for new synchronization * sync_agent_groups_get new response format * Adapt wdb_get_global_group_hash usage * Added get-groups-integrity command * Code review changes * Code review changes * Removed duplicate function declaration after rebase * several tests for get_agent_group * PR corrections * Adding unit tests to wazuh-db 'get-config' command * Removing unused header * Review feedback - use cJSON_ArrayForeach macro * Deprecated unused helper, UTs and GROUPS_DIR monitoring * Removed unused and unnecessary functions * Removed unused functions * Code review fix * Deprecate wdb update agent group logic * Removed orphan wrappers * Unit test - Fixing <lookfor_agent_group> unit tests that now use wazuh-db instead share-download * Adding new mechanisms for global database error management during the upgrade. * Fixing comments from PR * Added wdb_integrity tests for group hash * Added checks suggestion * Code review changes * Unit tests - Adding unit tests to sha256_op and strarr_size utility functions * Unit tests - Fixing crypto unit tests to use mock functions to deal with files * Remove insert belongs command * Unify get_groups methods * Reuse wdb_global_select_agent_group method * Fix test_manager UT * Code review fixes * Eliminate the delete groups belong command * Add a mechanism to check if the group is empty * Implement the remove mode for set-agent-groups * Code review fixes * Unit test of get_groups_integrity * Remove duplicate entry * Code review changes * Fix lookfor_agent_group UT * Fix wdb_global UT * Fix wdb_global_helpers UT * Remove obsolete get_agent_group UT * Adding UT for parsing methods of the backup command * Adding UT for wdb_global_create_backup method * Updating wrapper for cJSON_Delete * Restoring cJSON_Delete wrapper changes * Adding more unit tests for wdb_global backup commands * Fixing compilation * Fixing comments from PR * Updating wdb structure initialization for test_wdb_parser * Adding more UT for wdb_global groups methods * Fixing issues after rebase * UT cases for get_agent_max_group_priority * Code review changes * Added UT cases for select groups belong * Fixing malformed cJSON object * Improving helper response testing * Unit tests - Adding tests for wdb * Unit tests - Review feedback changes * Removing duplicate definition of __wrap_wdb_exec_stmt_single_column wrapper * Add wdb_remove_group_db UT * Add wdb_global_assign_agent_group and wdb_global_unassign_agent_group UT * Code review fixes * UT for set-agent-groups parser * Code review changes * Adding unit test cases for functions related to the upgrade of the global database * Fixing comments from pull request * Renaming wdb_global_update_all_agents_groups_hash to wdb_global_adjust_v4 * Adding unit test coverage for function wdb_check_backup_enabled * Fixing issue reported by scan-build * fixing old tests and adding new for agent_groups * PR corrections * PR changes pt 2 * adding setup and teardown fucntions * Add set_agent_group and set_agent_group_context UT * Removing incorrect comment blocks * Code review changes * Added UTs for sync-agent-groups-get * Fixing UTs * Code review changes * Fix create_backup UT mem leak * Code review changes * Unit tests - Unit test for wdb_global_set_agent_groups_sync_status * Unit tests - Review feedback changes * Adding unit testing coverage for functions wm_sync_group_file and wm_sync_legacy_groups_files * Fixing comments from pull request * Added some UT cases for global_sync_agent_groups_get * UT cases for global_sync_agent_groups_get * Added one more UT case and merge 2 cases into one * Required changes after rebase * Code review changes * Coverity - Fixing coverity warnings for epic #10771 * get_groups_integrity UT fixes * fixing Unit Tests for wdb_global_helpers * moving errno reseting to teardown * adding accesing file test for coverage completion and PR corrections * Code review changes * Fixing strings used in the output of the settings request to the wazuh-db socket * Refactor `wdb_upgrade_global` * Bugfix - changing EACESS to ENOENT * Bugfix - Updating UT that used the EACESS errno * Fix segfault after global db restoration * Fixing backup creation on startup * Adding new get-group-agents command * Adding UT for get-group-agents command * Rework of UT with wdb_exec_stmt_sized wrappers * Fixing compilation after rebase * Fixing UT compilation * Improving redaction after review * Default group is assigned to an agent if all groups are removed * Fix SEGFAULT * Added warn for non existent group * Checking whether the group name contains comma or not, fix SEGFAULT * Fixed UT and added new test cases * Code review changes * Coding style improvements * Updating wdb_global_select_groups with wdb_exec_stmt_sized * Adding sqlite error message for wdb_global_select_group_belong * Removing unused statements * Improving return values * Adding new parameter for wdb_global_sync_agent_groups_get * Updating UT for wdb_global_sync_agent_groups_get * Fixing comments from review * Validate groups to be assigned + UTs * Fix wrong ok response * Code review changes * Code review changes * Adding mode check in set_agent_groups * Fixing group_sync_status value for cluster sync * Updating UT with w_is_single_node() wrapper * Consider the cluster as disabled if the corresponding tag isn't found * Added checks for dot and double dot directories * Code review changes * Fix err response on empty group hash information * Adding checks for alternative parameters in sync-agent-groups-get wazuh-db command * Fixing scan-build issues * Fixing comments from pull request * Fixes after rebase * Updating UT and fixing memory leak * Add all API/Cluster/Framework changes related to agent-groups (#12608) * Change Agent-groups get tag to Local agent-group (#12648) * Fix 'agent_groups' not working for single node (#12685) * Fix 'agent_groups' not working for single node * Add corresponding unit test * Fix typos and change 'type' for 'request_type' * Fixing compilation * Change log location and message (#12729) * Fixing UT after rebase * Add master local agent-groups delay (#12739) * Update AIT mapping (#12741) * Fixing groups synchronization between etc/shared folder and wazuh-db to allow the addition of groups with a name starting with . or .. to the group table in the global.db database * Discarding only hidden files in the inotify synchronization logic for groups. Directory events are handled to allow groups creation. * Fix error when displaying the time in the agent-info sync task log * Preventing printing NULL in path for inotify * Converting an error message to debug * Fix group related framework code when using restricted agent names (#12759) * Check if given group is a directory when looking for agent groups * Raise new exception when trying to create restricted groups * Add new unit tests * Add new API integration tests * [Agent-groups] Change the logger tag when processing agent-groups DB (#12766) * Add new tags for loggers when processing the whole database * Add/Fix unittests * Change worker docstrings * Remove hash sending when entire groups DB is sent (#12799) * Add agent-group task to sync_tasks dict (#12791) * Add 'Finished in' log in agent sync tasks (#12792) * Fix invalid time in agent-groups send task (#12836) * Fix negative values for agent-groups send times * Fix worker unittests * Change log order to avoid race condition (#12876) * Adding sanity check for negative values in alternative parameters of sync-agent-groups-get command * Adding unit test cases to cover the check of negative values in last_id and agent_registration_delta parameters * Fix typo (#13062) * Fix condition to assign default group when guess_agent_group option is enabled. * Restoring UT folder after rebase * Fix invalid tmp path on API integration tests environment Co-authored-by: palaciosjeremias <jeremias.palacios@wazuh.com> Co-authored-by: pereyra-m <matias.pereyra@wazuh.com> Co-authored-by: MiguelazoDS Cazajous-Miguel <miguel.cazajous@wazuh.com> Co-authored-by: Santiago Belluzzo <santiago.belluzzo@wazuh.com> Co-authored-by: LuchoD <lucio.donda@wazuh.com> Co-authored-by: LucioDonda <donda.lucio@gmail.com> Co-authored-by: Adrián Jesús Peña Rodríguez <adrian.p.rodriguez@wazuh.com> Co-authored-by: yanazaeva <67015786+yanazaeva@users.noreply.github.com> Co-authored-by: Víctor Fernández Poyatos <victor.f.poyatos@wazuh.com> Co-authored-by: José Luis López Sánchez <joseluis.lopez@wazuh.com> Co-authored-by: Nicolas Gomez Palacios <nicogp2@gmail.com>
FrancoRivero
pushed a commit
that referenced
this issue
May 13, 2022
* Implement sync_agent_groups_get command * Improve json groups response * Code review fixes * Create set-agent-group command * Fix set-agent-groups command * Add DOXYGEN * Add set-agent-groups helper method * Code review fixes * Fix upgrade mechanism * Code review changes * Moving the client.keys synchronization mechanism with wazuh-db from the synchronization module to authd in the master nodes * Fixing agent compilation errors * Fixing comments from PR * Fixing error generated during the database creation * Creating parsing methods for global backup mechanism * Implemented the global_create_backup method * Created wdb_global_get_backup command * Implementing wdb_global_restore_backup method * Reading new config for Wazuh-DB * Deleting old backups * Creating Wazuh-DB backup thread * Creating backup/db folder * Deleting more than one old backup if necessary * Closing global database after a backup restore command * Removing pre-restore backup from most recent snapshots list * Fixing scan-build warnings * Always setting wdb configuration block default values * Adding result message for regular backups * Adding the wdb configuration block to ossec.conf * Minor changes after review * Improving array of configuration nodes implementation * Improving the delete mechanism of old backups * Restoring eval_bool definition * Restoring get_time_interval definition place * Minor fixes * Fixes after review * Fixing Windows agent compilation * Replaced function that updates group tables info * Code review changes * Inserting groups files in database * Making modulesd synchronize client.keys only at startup for master node * After review improvements * Updating the backup creation method * Finalizing all statements to run VACUUM * Fixing scan-build warnings * Adding a getconfig command for Wazuh-DB * Fixing comment and Valgrind warnings * Number of groups per agent reduced * Truncate groups file * Adapt groups size truncation to the upgrade mechanism * Removing deprecated folders from installation scripts * Restoring agents database creation to the database synchronization module to avoid errors in authd * Fixing wazuh-db output buffer usage * Separating actions taken by master and worker nodes * Fixing comments from pull request * Fixing UT compilation * Creating pre and post upgrade global DB snapshots * Print out backup filename * Code review changes * Added tag to create snapshots with custom names during upgrade * Removed log debug 2 during upgrade * Code review changes * Added log messages, hide wazuh-db configuration * Added error log while doing backup by request and by interval * Code review changes * Fixing UT compilation * Testing wazuh-db settings parser * Deprecating the ability to set agent groups from share_download 'files.yml' * Remove group_source logic from the set-agent-groups command * Add error message when set-agent-groups has any fail. * Remove group_source from database module call * Adding the global group hash methods and cache * Improvements after review * Simplify the group_hash from the sql schemas * Adapt sync_agent_groups_get for new synchronization * sync_agent_groups_get new response format * Adapt wdb_get_global_group_hash usage * Added get-groups-integrity command * Code review changes * Code review changes * Removed duplicate function declaration after rebase * several tests for get_agent_group * PR corrections * Adding unit tests to wazuh-db 'get-config' command * Removing unused header * Review feedback - use cJSON_ArrayForeach macro * Deprecated unused helper, UTs and GROUPS_DIR monitoring * Removed unused and unnecessary functions * Removed unused functions * Code review fix * Deprecate wdb update agent group logic * Removed orphan wrappers * Unit test - Fixing <lookfor_agent_group> unit tests that now use wazuh-db instead share-download * Adding new mechanisms for global database error management during the upgrade. * Fixing comments from PR * Added wdb_integrity tests for group hash * Added checks suggestion * Code review changes * Unit tests - Adding unit tests to sha256_op and strarr_size utility functions * Unit tests - Fixing crypto unit tests to use mock functions to deal with files * Remove insert belongs command * Unify get_groups methods * Reuse wdb_global_select_agent_group method * Fix test_manager UT * Code review fixes * Eliminate the delete groups belong command * Add a mechanism to check if the group is empty * Implement the remove mode for set-agent-groups * Code review fixes * Unit test of get_groups_integrity * Remove duplicate entry * Code review changes * Fix lookfor_agent_group UT * Fix wdb_global UT * Fix wdb_global_helpers UT * Remove obsolete get_agent_group UT * Adding UT for parsing methods of the backup command * Adding UT for wdb_global_create_backup method * Updating wrapper for cJSON_Delete * Restoring cJSON_Delete wrapper changes * Adding more unit tests for wdb_global backup commands * Fixing compilation * Fixing comments from PR * Updating wdb structure initialization for test_wdb_parser * Adding more UT for wdb_global groups methods * Fixing issues after rebase * UT cases for get_agent_max_group_priority * Code review changes * Added UT cases for select groups belong * Fixing malformed cJSON object * Improving helper response testing * Unit tests - Adding tests for wdb * Unit tests - Review feedback changes * Removing duplicate definition of __wrap_wdb_exec_stmt_single_column wrapper * Add wdb_remove_group_db UT * Add wdb_global_assign_agent_group and wdb_global_unassign_agent_group UT * Code review fixes * UT for set-agent-groups parser * Code review changes * Adding unit test cases for functions related to the upgrade of the global database * Fixing comments from pull request * Renaming wdb_global_update_all_agents_groups_hash to wdb_global_adjust_v4 * Adding unit test coverage for function wdb_check_backup_enabled * Fixing issue reported by scan-build * fixing old tests and adding new for agent_groups * PR corrections * PR changes pt 2 * adding setup and teardown fucntions * Add set_agent_group and set_agent_group_context UT * Removing incorrect comment blocks * Code review changes * Added UTs for sync-agent-groups-get * Fixing UTs * Code review changes * Fix create_backup UT mem leak * Code review changes * Unit tests - Unit test for wdb_global_set_agent_groups_sync_status * Unit tests - Review feedback changes * Adding unit testing coverage for functions wm_sync_group_file and wm_sync_legacy_groups_files * Fixing comments from pull request * Added some UT cases for global_sync_agent_groups_get * UT cases for global_sync_agent_groups_get * Added one more UT case and merge 2 cases into one * Required changes after rebase * Code review changes * Coverity - Fixing coverity warnings for epic #10771 * get_groups_integrity UT fixes * fixing Unit Tests for wdb_global_helpers * moving errno reseting to teardown * adding accesing file test for coverage completion and PR corrections * Code review changes * Fixing strings used in the output of the settings request to the wazuh-db socket * Refactor `wdb_upgrade_global` * Bugfix - changing EACESS to ENOENT * Bugfix - Updating UT that used the EACESS errno * Fix segfault after global db restoration * Fixing backup creation on startup * Adding new get-group-agents command * Adding UT for get-group-agents command * Rework of UT with wdb_exec_stmt_sized wrappers * Fixing compilation after rebase * Fixing UT compilation * Improving redaction after review * Default group is assigned to an agent if all groups are removed * Fix SEGFAULT * Added warn for non existent group * Checking whether the group name contains comma or not, fix SEGFAULT * Fixed UT and added new test cases * Code review changes * Coding style improvements * Updating wdb_global_select_groups with wdb_exec_stmt_sized * Adding sqlite error message for wdb_global_select_group_belong * Removing unused statements * Improving return values * Adding new parameter for wdb_global_sync_agent_groups_get * Updating UT for wdb_global_sync_agent_groups_get * Fixing comments from review * Validate groups to be assigned + UTs * Fix wrong ok response * Code review changes * Code review changes * Adding mode check in set_agent_groups * Fixing group_sync_status value for cluster sync * Updating UT with w_is_single_node() wrapper * Consider the cluster as disabled if the corresponding tag isn't found * Added checks for dot and double dot directories * Code review changes * Fix err response on empty group hash information * Adding checks for alternative parameters in sync-agent-groups-get wazuh-db command * Fixing scan-build issues * Fixing comments from pull request * Fixes after rebase * Updating UT and fixing memory leak * Add all API/Cluster/Framework changes related to agent-groups (#12608) * Change Agent-groups get tag to Local agent-group (#12648) * Fix 'agent_groups' not working for single node (#12685) * Fix 'agent_groups' not working for single node * Add corresponding unit test * Fix typos and change 'type' for 'request_type' * Fixing compilation * Change log location and message (#12729) * Fixing UT after rebase * Add master local agent-groups delay (#12739) * Update AIT mapping (#12741) * Fixing groups synchronization between etc/shared folder and wazuh-db to allow the addition of groups with a name starting with . or .. to the group table in the global.db database * Discarding only hidden files in the inotify synchronization logic for groups. Directory events are handled to allow groups creation. * Fix error when displaying the time in the agent-info sync task log * Preventing printing NULL in path for inotify * Converting an error message to debug * Fix group related framework code when using restricted agent names (#12759) * Check if given group is a directory when looking for agent groups * Raise new exception when trying to create restricted groups * Add new unit tests * Add new API integration tests * [Agent-groups] Change the logger tag when processing agent-groups DB (#12766) * Add new tags for loggers when processing the whole database * Add/Fix unittests * Change worker docstrings * Remove hash sending when entire groups DB is sent (#12799) * Add agent-group task to sync_tasks dict (#12791) * Add 'Finished in' log in agent sync tasks (#12792) * Fix invalid time in agent-groups send task (#12836) * Fix negative values for agent-groups send times * Fix worker unittests * Change log order to avoid race condition (#12876) * Adding sanity check for negative values in alternative parameters of sync-agent-groups-get command * Adding unit test cases to cover the check of negative values in last_id and agent_registration_delta parameters * Fix typo (#13062) * Fix condition to assign default group when guess_agent_group option is enabled. * Restoring UT folder after rebase * Fix invalid tmp path on API integration tests environment Co-authored-by: palaciosjeremias <jeremias.palacios@wazuh.com> Co-authored-by: pereyra-m <matias.pereyra@wazuh.com> Co-authored-by: MiguelazoDS Cazajous-Miguel <miguel.cazajous@wazuh.com> Co-authored-by: Santiago Belluzzo <santiago.belluzzo@wazuh.com> Co-authored-by: LuchoD <lucio.donda@wazuh.com> Co-authored-by: LucioDonda <donda.lucio@gmail.com> Co-authored-by: Adrián Jesús Peña Rodríguez <adrian.p.rodriguez@wazuh.com> Co-authored-by: yanazaeva <67015786+yanazaeva@users.noreply.github.com> Co-authored-by: Víctor Fernández Poyatos <victor.f.poyatos@wazuh.com> Co-authored-by: José Luis López Sánchez <joseluis.lopez@wazuh.com> Co-authored-by: Nicolas Gomez Palacios <nicogp2@gmail.com>
This was referenced May 16, 2022
|
I'm closing this because all the issues related to the epic have been completed. |
3 tasks
This was referenced Jun 14, 2022
2 tasks
3 tasks
2 tasks
2 tasks
2 tasks
2 tasks
4 tasks
4 tasks
This was referenced Feb 14, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
We have to study and propose a solution to migrate current agent-group files information to wazuh-db.
The current solution using files does not scale well and we are already reaching the limits for cluster environments with a certain size.
Currently, the agent groups information is stored in files located in
/WAZUH_PATH/queue/agent-groups. Each agent possesses its own agent-group file, which is a plain text file containing the groups the agent belongs to, separated by a comma. The name of the file is the agent's ID.In a cluster environment, the master node is in charge of synchronizing these files among all worker nodes. The synchronization in this case occurs in both ways. When an agent connects to a worker node, the agent-group file is created locally, and then sent to the master node as part of the
Integrity synctask. Later on, the master compares the integrity of its files with those of the different worker nodes and send all missing or changed files to them, ensuring the integrity of the environment.When the cluster environment is big (20+ worker nodes, 50k+ agents), the task to calculate the integrity of the files in the master node plus the N integrity synchronizations for the N worker nodes is taking a lot of time, occupying the cluster process almost completely, and causing huge delays.
These delays end up producing problems in other modules, such as API distributed requests timeouts, agents not connecting at the right time because the client.keys file is not synchronized and update in the worker they are connecting to, etc.
By migrating the information and usage of the agent group files information completely to wazuh-db, we can greatly reduce the master load. These files containing agent group information do not change frequently, and so, calculating their content, compressing them in zip files and sending them to all the worker when necessary is very inefficient. With the information stored in the database, we can synchronize the information between databases only when it changes, and without using thousand of files and compress methods.
Tasks:
Branches
WORKING_BRANCH: https://github.com/wazuh/wazuh/tree/dev-10771-agent-groups-files-to-wazuh-db
WORKING_CLUSTER_BRANCH: https://github.com/wazuh/wazuh/tree/feature/10771-cluster-api-agent-groups
WORKING_BRANCH_DOCUMENTATION: https://github.com/wazuh/wazuh-documentation/tree/dev-10771-agent-groups-files-to-wazuh-db
TARGET_BRANCH:
masterThe text was updated successfully, but these errors were encountered: