DTT1 - Bugs - Fix Provision log references

[rauldpm]

Description

Related: #5125

We need to investigate and fix or improve the provision module logging from the Workflow engine as logs are not being identified as PROVISION but as ANSIBLE

[2024-04-08 14:05:13,956] [INFO] ALLOCATOR: Track file generated at /tmp/dtt1-poc/agent-linux-amazon-2-amd64/track.yaml
[2024-04-08 14:05:14,044] [INFO] [1018721] [ThreadPoolExecutor-0_0] [workflow_engine]: [allocate-agent-linux-amazon-2-amd64] Finished task in 78.38 seconds.
[2024-04-08 14:05:14,050] [INFO] [1018721] [ThreadPoolExecutor-0_0] [workflow_engine]: [provision-manager-linux-ubuntu-22.04-amd64] Starting task.
[2024-04-08 14:05:14,625] [DEBUG] ANSIBLE: Using inventory: {'all': {'hosts': {'192.168.57.2': {'ansible_port': 22, 'ansible_user': 'vagrant', 'ansible_ssh_private_key_file': '/tmp/wazuh-qa/VAGRANT-2954646C-69AE-4690-BDEC-42955E8416FC/instance_key'}}}}
[2024-04-08 14:05:14,626] [DEBUG] ANSIBLE: Running playbook: {'hosts': '192.168.57.2', 'become': True, 'gather_facts': True, 'tasks': [{'name': 'Capture ansible_os_family', 'set_fact': {'ansible_os_family': "{{ ansible_facts['distribution_file_variety'] }}", 'cacheable': 'yes'}}]}
[2024-04-08 14:05:17,697] [DEBUG] ANSIBLE: Playbook {'hosts': '192.168.57.2', 'become': True, 'gather_facts': True, 'tasks': [{'name': 'Capture ansible_os_family', 'set_fact': {'ansible_os_family': "{{ ansible_facts['distribution_file_variety'] }}", 'cacheable': 'yes'}}]} finished with status {'skipped': {}, 'ok': {'192.168.57.2': 2}, 'dark': {}, 'failures': {}, 'ignored': {}, 'rescued': {}, 'processed': {'192.168.57.2': 1}, 'changed': {}}
[2024-04-08 14:05:17,698] [DEBUG] ANSIBLE: Templates found: ['download.j2', 'install.j2']

[pro-akim]

Update

After some tests I could make the provision log be present

[2024-04-10 15:56:14] [DEBUG] [251900] [ThreadPoolExecutor-0_0] [workflow_engine]: Running task "provision-install-linux-ubuntu-18.04-amd64" with arguments: ['modules/provision/main.py', '--inventory=/tmp/dtt1-poc/agent-linux-ubuntu-18.04-amd64/inventory.yaml', "--dependencies={'manager': '/tmp/dtt1-poc/manager-linux-ubuntu-22.04-amd64/inventory.yaml'}", "--install={'component': 'wazuh-agent', 'type': 'package', 'version': '4.7.3', 'live': True}"]
[2024-04-10 15:56:52] [DEBUG] [251900] [ThreadPoolExecutor-0_0] [workflow_engine]: Finished task "provision-install-linux-ubuntu-18.04-amd64" execution with result:
[2024-04-10 15:56:15] [DEBUG] PROVISION: Setting dependencies: {'manager': '192.168.57.2'} for wazuh-agent component.
[2024-04-10 15:56:15] [INFO] PROVISION: Initiating provisionment.
[2024-04-10 15:56:15] [DEBUG] PROVISION: Running action install for components: [ComponentInfo(component='wazuh-agent', type='package', version='4.7.3', dependencies={'manager': '192.168.57.2'}, live=True)]
[2024-04-10 15:56:15] [INFO] PROVISION: Provisioning "wazuh-agent"...
[2024-04-10 15:56:15] [DEBUG] ANSIBLE: Using inventory: {'all': {'hosts': {'192.168.57.3': {'ansible_port': 22, 'ansible_user': 'vagrant', 'ansible_ssh_private_key_file': '/tmp/wazuh-qa/VAGRANT-92357979-83E8-4D94-8B87-9C32473A6CB8/instance_key'}}}}
[2024-04-10 15:56:15] [DEBUG] ANSIBLE: Running playbook: {'hosts': '192.168.57.3', 'become': True, 'gather_facts': True, 'tasks': [{'name': 'Capture ansible_os_family', 'set_fact': {'ansible_os_family': "{{ ansible_facts['distribution_file_variety'] }}", 'cacheable': 'yes'}}]} 

However, the behavior is not clear because when the PROVISION log is present, the ALLOCATOR module log is absent.

[pro-akim]

Update

Using the class Logger from /modules/generic/logger

(Here is possible to check the print of logger from provisioning where it is an object with the correct parameters)

Allocation and testing works.
Provision is linked to the method but for some unknown reason the log is not added to the log file.

Further research is required

[pro-akim]

Update

Even adding its own method is not printing information in the log file

While using the same method out of the provision module

[pro-akim]

Update

Changing the method in the testing module

However, when I change the name of provision to testing:

It seems that the logging module has some unknown conflict

[pro-akim]

Update

Testing a different library

Still it is not working

[pro-akim]

Update

Changing the methods this log could be generated in the log file

[2024-04-12 11:02:20,103] [INFO] PROVISION-MODULE: Starting provisioning

in

[2024-04-12 11:02:19] [INFO] ALLOCATION-MODULE: Track file generated at /tmp/dtt1-poc/manager-linux-centos-7-amd64/track.yaml

[2024-04-12 11:02:19,723] [INFO] [671192] [ThreadPoolExecutor-0_0] [workflow_engine]: [allocate-manager-linux-centos-7-amd64] Finished task in 48.35 seconds.
[2024-04-12 11:02:19,739] [INFO] [671192] [ThreadPoolExecutor-0_0] [workflow_engine]: [provision-manager-linux-centos-7-amd64] Starting task.
[2024-04-12 11:02:19,745] [DEBUG] [671192] [ThreadPoolExecutor-0_0] [workflow_engine]: Running task "provision-manager-linux-centos-7-amd64" with arguments: ['modules/provision/main.py', '--inventory=/tmp/dtt1-poc/manager-linux-centos-7-amd64/inventory.yaml', "--install={'component': 'wazuh-manager', 'type': 'assistant', 'version': '4.7.3', 'live': True}"]
[2024-04-12 11:02:20,103] [INFO] PROVISION-MODULE: Starting provisioning
[2024-04-12 11:02:20,108] [DEBUG] ANSIBLE: Using inventory: {'all': {'hosts': {'192.168.57.2': {'ansible_port': 22, 'ansible_user': 'vagrant', 'ansible_ssh_private_key_file': '/tmp/wazuh-qa/VAGRANT-E856A74B-46BE-432D-BBA1-5DE5BC6B9E5A/instance_key'}}}}

The ANSIBLE name will be kept in the log since it is a record of a method shared by several modules

[pro-akim]

Update

Moving to #5210 which has higher priority

[pro-akim]

Update

Tests were performed, however, the root of the issue was not found yet.

[mhamra]

UPDATE

• The loggers are created through the logging.config.dictConfig() method. The config.yaml file must declare the loggers before using the logging.getLogger function.

  After changing the config.yaml, the logger started to work.

• The Ansible class created another logger in its constructor. The Ansible constructor now receives a logger instance that the Ansible methods will use.

RESULT

Here's an example of a workflow run after modifying the logger instantiation: workflow.log

[mhamra]

UPDATE

I've removed the deployability/modules/testinghelpers/logger classes and replaced the logger imports with from modules.generic.logger import logger.

After those changes, the testing module generates no Testing log lines, and the Provision and Testing modules generate no ANSIBLE log lines.

The log generated by a workflow run is the following:

workflow.log

[fcaffieri]

We need to implement the same changes made to the modules within the allocation module to ensure that logging continues to work correctly.

[fcaffieri]

Once the allocation issue has been clarified, the module does not need changes. LGTM

[rauldpm]

LGTM