Implement debug symbol generation during the build process for macOS.

[Dwordcito]

Description

Related issue
#2886

Currently, maxOS packages generated do not include debug symbols, which can hinder debugging efforts when issues arise. This issue aims to implement the necessary steps to generate packages with debug symbols for macOS packages.

Proposed Solution

1. Update Packaging Scripts:
   • Modify the scripts responsible for generating macOS installer packages to incorporate the inclusion of debug symbols.
   • Implement any necessary logic to ensure that debug symbols are correctly packaged and included.

Research guidelines

Research on how we are currently compiling for macOS should be done.
A good understanding of the compiling process (which is different on macOS than on Linux) will be necessary.
A fast search indicated that for macOS debug symbols are always created separated from the binary, and then must be joined by using the dsymutil tool to bring them together. This might simplify this issue if found to be true.

• Enhanced Debugging: Including debug symbols in the packages facilitates easier debugging and troubleshooting of issues encountered by users.
• Improved Developer Experience: Developers benefit from having access to debug symbols when investigating issues reported by users.

Definition of Done

• The macOS packaging process is updated to incorporate the inclusion of debug symbols.

[Leoquicenoz]

Update

After the investigation I can be determined that dsymutil is used to create and manipulate debugging symbol files, so some modifications are made to the Makefile code.

[Leoquicenoz]

Update

After the modifications made in the code, we are getting the debug symbols separated from the binaries in .dSYM files.

src % ls symbols 
agent-auth.dSYM            firewalld-drop.dSYM        ipfw.dSYM                  npf.dSYM                   route-null.dSYM            wazuh-logcollector.dSYM
default-firewall-drop.dSYM host-deny.dSYM             kaspersky.dSYM             pf.dSYM                    wazuh-agentd.dSYM          wazuh-modulesd.dSYM
disable-account.dSYM       ip-customblock.dSYM        manage_agents.dSYM         restart-wazuh.dSYM         wazuh-execd.dSYM           wazuh-slack.dSYM

But in the compilation I am getting this output:

warning: no debug symbols in executable (-arch x86_64)
warning: no debug symbols in executable (-arch x86_64)
warning: no debug symbols in executable (-arch x86_64)
warning: no debug symbols in executable (-arch x86_64)
warning: no debug symbols in executable (-arch x86_64)
warning: no debug symbols in executable (-arch x86_64)
warning: no debug symbols in executable (-arch x86_64)
warning: no debug symbols in executable (-arch x86_64)
warning: no debug symbols in executable (-arch x86_64)
warning: no debug symbols in executable (-arch x86_64)
warning: no debug symbols in executable (-arch x86_64)
warning: no debug symbols in executable (-arch x86_64)
warning: no debug symbols in executable (-arch x86_64)
warning: no debug symbols in executable (-arch x86_64)
warning: no debug symbols in executable (-arch x86_64)
warning: no debug symbols in executable (-arch x86_64)
warning: no debug symbols in executable (-arch x86_64)
warning: no debug symbols in executable (-arch x86_64)

Initially I thought it was because in the branch I was testing it was not generating the debug symbols, but when testing it with the branch 21734-add-debug-symbols-macos, I also get this same message. I was verifying in this branch before adding the changes and when compiling, the resulting binaries do have the debug symbols as you can verify here:

gdb wazuh-modulesd 

Reading symbols from wazuh-modulesd...

And after having separated the debug symbols from the binaries, you get this output with gdb

gdb wazuh-modulesd 

Reading symbols from wazuh-modulesd...
(No debugging symbols found in wazuh-modulesd)

[Leoquicenoz]

ETA is delayed because some checks failed in the PR and must be investigated and resolved, so there was not enough time for the review.

[Leoquicenoz]

Update

After some tests, I was able to determine that with the changes that had been made previously, the .dSYM files were being generated after the second compilation, since the executable files were not found in the first compilation, so the code had to be modified so that they could be found in the first compilation.

This modification adds the following message at the end of the compilation:

/Library/Developer/CommandLineTools/usr/bin/make debug_symbols_mac
grep: /etc/os-release: No such file or directory
Makefile:2659: warning: overriding commands for target `win32/ui_resource.o'
Makefile:2599: warning: ignoring old commands for target `win32/ui_resource.o'
Makefile:2662: warning: overriding commands for target `win32/auth_resource.o' Makefile:2602: warning: overriding commands for target `win32/ui_resource.o'
Makefile:2602: warning: ignoring old commands for target `win32/auth_resource.o' Makefile:2599: warning: overriding commands for target `win32/ui_resource.o'

Done building agent

src % ls symbols 
agent-auth.dSYM            firewalld-drop.dSYM        kaspersky.dSYM             npf.dSYM                   shared_modules             wazuh-logcollector.dSYM
data_provider              host-deny.dSYM             libwazuhext.dylib.dSYM     pf.dSYM                    syscheckd                  wazuh-modulesd.dSYM
default-firewall-drop.dSYM ip-customblock.dSYM        libwazuhshared.dylib.dSYM  restart-wazuh.dSYM         wazuh-agentd.dSYM          wazuh-slack.dSYM
disable-account.dSYM       ipfw.dSYM                  manage_agents.dSYM         route-null.dSYM            wazuh-execd.dSYM           wazuh_modules

Testing the compilation on Linux, the following message is generated:

make[1]: Leaving directory '/wazuh/src'
make debug_symbols_mac
make[1]: Entering directory '/wazuh/src'
Makefile:2659: warning: overriding recipe for target 'win32/ui_resource.o'
Makefile:2599: warning: ignoring old recipe for target 'win32/ui_resource.o'
Makefile:2662: warning: overriding recipe for target 'win32/auth_resource.o'
Makefile:2602: warning: ignoring old recipe for target 'win32/auth_resource.o'
make[1]: Nothing to be done for 'debug_symbols_mac'.
make[1]: Leaving directory 'wazuh/src'

Done building agent

At the moment I am still looking for the reason why the tests fail in the PR.

[Leoquicenoz]

Update

In the PR several checks have been failing, after analyzing it a lot and trying several things, I created a branch in the wazuh/qa-integration-framework repo so it could match with the development branch, in local it worked and started to pass the tests that fail in the PR checks, but in github they keep failing even if I run the re-run jobs.

Local test execution

• Agent compilation and installation:

General settings:
    TARGET:             agent
    V:                  
    DEBUG:              
    DEBUGAD             
    INSTALLDIR:         /var/ossec
    DATABASE:           
    ONEWAY:             no
    CLEANFULL:          no
    RESOURCES_URL:      https://packages.wazuh.com/deps/25
    EXTERNAL_SRC_ONLY:  
    HTTP_REQUEST_BRANCH:dev-14153-vulndet-refactor
User settings:
    WAZUH_GROUP:        wazuh
    WAZUH_USER:         wazuh
USE settings:
    USE_ZEROMQ:         no
    USE_GEOIP:          no
    USE_PRELUDE:        no
    USE_INOTIFY:        no
    USE_BIG_ENDIAN:     no
    USE_SELINUX:        no
    USE_AUDIT:          yes
    DISABLE_SYSC:       no
    DISABLE_CISCAT:     no
    IMAGE_TRUST_CHECKS: 1
    CA_NAME:            DigiCert Assured ID Root CA
Mysql settings:
    includes:           
    libs:               
Pgsql settings:
    includes:           
    libs:               
Defines:
    -DOSSECHIDS -DUSER="wazuh" -DGROUPGLOBAL="wazuh" -DLinux -DINOTIFY_ENABLED -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DIMAGE_TRUST_CHECKS=1 -DCA_NAME='DigiCert Assured ID Root CA' -DENABLE_SYSC -DENABLE_CISCAT -DENABLE_AUDIT -DCLIENT
Compiler:
    CFLAGS            -pthread -Iexternal/libdb/build_unix/ -Iexternal/pacman/lib/libalpm/ -Iexternal/libarchive/libarchive -Wl,--start-group -Iexternal/audit-userspace/lib -g -DNDEBUG -O2 -DOSSECHIDS -DUSER="wazuh" -DGROUPGLOBAL="wazuh" -DLinux -DINOTIFY_ENABLED -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DIMAGE_TRUST_CHECKS=1 -DCA_NAME='DigiCert Assured ID Root CA' -DENABLE_SYSC -DENABLE_CISCAT -DENABLE_AUDIT -DCLIENT -pipe -Wall -Wextra -std=gnu99 -I./ -I./headers/ -Iexternal/openssl/include -Iexternal/cJSON/ -Iexternal/libyaml/include -Iexternal/curl/include -Iexternal/msgpack/include -Iexternal/bzip2/ -Ishared_modules/common -Ishared_modules/dbsync/include -Ishared_modules/rsync/include -Iwazuh_modules/syscollector/include  -Idata_provider/include  -Iexternal/libpcre2/include -Iexternal/rpm//builddir/output/include -Isyscheckd/include -Ishared_modules/router/include -Ishared_modules/content_manager/include -Iwazuh_modules/vulnerability_scanner/include -I./shared_modules/ 
    LDFLAGS           '-Wl,-rpath,/../lib' -pthread -lrt -ldl -O2 -Lshared_modules/dbsync/build/lib -Lshared_modules/rsync/build/lib  -Lwazuh_modules/syscollector/build/lib -Ldata_provider/build/lib -Lsyscheckd/build/lib
    LIBS              -lrt -ldl -lm 
    CC                gcc
    MAKE              make
make[1]: Leaving directory '/home/leo/Desktop/debug_symbols/Temp/wazuh/src'
make debug_symbols_mac
make[1]: Entering directory '/home/leo/Desktop/debug_symbols/Temp/wazuh/src'
Makefile:2659: warning: overriding recipe for target 'win32/ui_resource.o'
Makefile:2599: warning: ignoring old recipe for target 'win32/ui_resource.o'
Makefile:2662: warning: overriding recipe for target 'win32/auth_resource.o'
Makefile:2602: warning: ignoring old recipe for target 'win32/auth_resource.o'
make[1]: Nothing to be done for 'debug_symbols_mac'.
make[1]: Leaving directory '/home/leo/Desktop/debug_symbols/Temp/wazuh/src'

Done building agent

Wait for success...
success
Removing old SCA policies...
Installing SCA policies...


Created symlink /etc/systemd/system/multi-user.target.wants/wazuh-agent.service → /etc/systemd/system/wazuh-agent.service.

 - Configuration finished properly.

 - To start Wazuh:
      /var/ossec/bin/wazuh-control start

 - To stop Wazuh:
      /var/ossec/bin/wazuh-control stop

 - The configuration can be viewed or modified at /var/ossec/etc/ossec.conf


   Thanks for using Wazuh.
   Please don't hesitate to contact us if you need help or find
   any bugs.

   Use our public Mailing List at:
          https://groups.google.com/forum/#!forum/wazuh

   More information can be found at:
          - http://www.wazuh.com

    ---  Press ENTER to finish (maybe more information below). ---


 - More information at: 
   https://documentation.wazuh.com/

• test_agentd

 python3 -m pytest -vvs ./test_agentd
=========================================================================================== test session starts ============================================================================================
platform linux -- Python 3.10.12, pytest-7.1.2, pluggy-1.4.0 -- /usr/bin/python3
cachedir: .pytest_cache
metadata: {'Python': '3.10.12', 'Platform': 'Linux-6.5.0-26-generic-x86_64-with-glibc2.35', 'Packages': {'pytest': '7.1.2', 'pluggy': '1.4.0'}, 'Plugins': {'html': '3.1.1', 'metadata': '3.1.1'}}
rootdir: /home/leo/Desktop/debug_symbols/Temp/wazuh/tests/integration, configfile: pytest.ini
plugins: html-3.1.1, metadata-3.1.1
collected 37 items                                                                                                                                                                                         

test_agentd/test_agentd_multi_server/test_agentd_multi_server.py::test_agentd_multi_server[refuse_remoted_accept_enrollment] PASSED
test_agentd/test_agentd_multi_server/test_agentd_multi_server.py::test_agentd_multi_server[only_enrollment_and_only_remoted_0] PASSED
test_agentd/test_agentd_multi_server/test_agentd_multi_server.py::test_agentd_multi_server[only_enrollment_and_only_remoted_1] PASSED
test_agentd/test_agentd_multi_server/test_agentd_multi_server.py::test_agentd_multi_server[server_down_fallback_tcp_0] PASSED
test_agentd/test_agentd_multi_server/test_agentd_multi_server.py::test_agentd_multi_server[server_down_fallback_tcp_1] PASSED
test_agentd/test_agentd_multi_server/test_agentd_multi_server.py::test_agentd_multi_server[only_one_server_available] PASSED
test_agentd/test_agentd_multi_server/test_agentd_multi_server.py::test_agentd_multi_server[all_servers_unavailable] PASSED
test_agentd/test_agentd_multi_server/test_agentd_multi_server.py::test_agentd_multi_server[server_down_fallback_udp_0] PASSED
test_agentd/test_agentd_multi_server/test_agentd_multi_server.py::test_agentd_multi_server[server_down_fallback_udp_1] PASSED
test_agentd/test_agentd_parametrized_reconnections/test_agentd_parametrized_reconnections.py::test_agentd_parametrized_reconnections[Test_case_0] PASSED
test_agentd/test_agentd_parametrized_reconnections/test_agentd_parametrized_reconnections.py::test_agentd_parametrized_reconnections[Test_case_1] PASSED
test_agentd/test_agentd_parametrized_reconnections/test_agentd_parametrized_reconnections.py::test_agentd_parametrized_reconnections[Test_case_2] PASSED
test_agentd/test_agentd_parametrized_reconnections/test_agentd_parametrized_reconnections.py::test_agentd_parametrized_reconnections[Test_case_3] PASSED
test_agentd/test_agentd_parametrized_reconnections/test_agentd_parametrized_reconnections.py::test_agentd_parametrized_reconnections[Test_case_4] PASSED
test_agentd/test_agentd_parametrized_reconnections/test_agentd_parametrized_reconnections.py::test_agentd_parametrized_reconnections[Test_case_5] PASSED
test_agentd/test_agentd_parametrized_reconnections/test_agentd_parametrized_reconnections.py::test_agentd_parametrized_reconnections[Test_case_6] PASSED
test_agentd/test_agentd_parametrized_reconnections/test_agentd_parametrized_reconnections.py::test_agentd_parametrized_reconnections[Test_case_7] PASSED
test_agentd/test_agentd_parametrized_reconnections/test_agentd_parametrized_reconnections.py::test_agentd_parametrized_reconnections[Test_case_8] PASSED
test_agentd/test_agentd_reconnection/test_agentd_connection_retries_pre_enrollment.py::test_agentd_connection_retries_pre_enrollment[TCP protocol] PASSED
test_agentd/test_agentd_reconnection/test_agentd_connection_retries_pre_enrollment.py::test_agentd_connection_retries_pre_enrollment[UDP protocol] PASSED
test_agentd/test_agentd_reconnection/test_agentd_initial_enrollment_retries.py::test_agentd_initial_enrollment_retries[TCP protocol] PASSED
test_agentd/test_agentd_reconnection/test_agentd_initial_enrollment_retries.py::test_agentd_initial_enrollment_retries[UDP protocol] PASSED
test_agentd/test_agentd_reconnection/test_agentd_reconection_enrollment_no_keys.py::test_agentd_reconection_enrollment_no_keys[TCP protocol] PASSED
test_agentd/test_agentd_reconnection/test_agentd_reconection_enrollment_no_keys.py::test_agentd_reconection_enrollment_no_keys[UDP protocol] PASSED
test_agentd/test_agentd_reconnection/test_agentd_reconection_enrollment_no_keys_file.py::test_agentd_reconection_enrollment_no_keys_file[TCP protocol] PASSED
test_agentd/test_agentd_reconnection/test_agentd_reconection_enrollment_no_keys_file.py::test_agentd_reconection_enrollment_no_keys_file[UDP protocol] PASSED
test_agentd/test_agentd_reconnection/test_agentd_reconection_enrollment_with_keys.py::test_agentd_reconection_enrollment_with_keys[TCP protocol] PASSED
test_agentd/test_agentd_reconnection/test_agentd_reconection_enrollment_with_keys.py::test_agentd_reconection_enrollment_with_keys[UDP protocol] ----------------------------------------
Exception occurred during processing of request from ('127.0.0.1', 44624)
Traceback (most recent call last):
  File "/usr/lib/python3.10/socketserver.py", line 683, in process_request_thread
    self.finish_request(request, client_address)
  File "/usr/lib/python3.10/socketserver.py", line 360, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python3.10/socketserver.py", line 747, in __init__
    self.handle()
  File "/usr/local/lib/python3.10/dist-packages/wazuh_testing/tools/mitm.py", line 234, in handle
    self.request[1].sendto(response, self.client_address)
OSError: [Errno 9] Bad file descriptor
----------------------------------------
PASSED
test_agentd/test_agentd_state/test_agentd_state.py::test_agentd_state[No_connection_with_remoted] PASSED
test_agentd/test_agentd_state/test_agentd_state.py::test_agentd_state[Successful_connection_with_remoted] PASSED
test_agentd/test_agentd_state/test_agentd_state.py::test_agentd_state[Only_remote_request_available] PASSED
test_agentd/test_agentd_state_config/test_agentd_state_config.py::test_agentd_state_config[Negative_state_interval] Job for wazuh-agent.service failed because the control process exited with error code.
See "systemctl status wazuh-agent.service" and "journalctl -xeu wazuh-agent.service" for details.
PASSED
test_agentd/test_agentd_state_config/test_agentd_state_config.py::test_agentd_state_config[Undefined_state_interval_value] PASSED
test_agentd/test_agentd_state_config/test_agentd_state_config.py::test_agentd_state_config[State_interval_option_dont_exist] Job for wazuh-agent.service failed because the control process exited with error code.
See "systemctl status wazuh-agent.service" and "journalctl -xeu wazuh-agent.service" for details.
PASSED
test_agentd/test_agentd_state_config/test_agentd_state_config.py::test_agentd_state_config[Zero_state_interval] PASSED
test_agentd/test_agentd_state_config/test_agentd_state_config.py::test_agentd_state_config[Too_big_state_interval] Job for wazuh-agent.service failed because the control process exited with error code.
See "systemctl status wazuh-agent.service" and "journalctl -xeu wazuh-agent.service" for details.
PASSED
test_agentd/test_agentd_state_config/test_agentd_state_config.py::test_agentd_state_config[Default_state_interval] PASSED

============================================================================================= warnings summary =============================================================================================
test_agentd/test_agentd_multi_server/test_agentd_multi_server.py: 5 warnings
test_agentd/test_agentd_reconnection/test_agentd_initial_enrollment_retries.py: 2 warnings
test_agentd/test_agentd_reconnection/test_agentd_reconection_enrollment_no_keys.py: 2 warnings
test_agentd/test_agentd_reconnection/test_agentd_reconection_enrollment_no_keys_file.py: 2 warnings
test_agentd/test_agentd_reconnection/test_agentd_reconection_enrollment_with_keys.py: 2 warnings
  /usr/local/lib/python3.10/dist-packages/wazuh_testing/tools/certificate_controller.py:89: DeprecationWarning: X509Extension support in pyOpenSSL is deprecated. You should use the APIs in cryptography.
    crypto.X509Extension(b'basicConstraints', True, b'CA:TRUE, pathlen:0'),

test_agentd/test_agentd_multi_server/test_agentd_multi_server.py: 5 warnings
test_agentd/test_agentd_reconnection/test_agentd_initial_enrollment_retries.py: 2 warnings
test_agentd/test_agentd_reconnection/test_agentd_reconection_enrollment_no_keys.py: 2 warnings
test_agentd/test_agentd_reconnection/test_agentd_reconection_enrollment_no_keys_file.py: 2 warnings
test_agentd/test_agentd_reconnection/test_agentd_reconection_enrollment_with_keys.py: 2 warnings
  /usr/local/lib/python3.10/dist-packages/wazuh_testing/tools/certificate_controller.py:90: DeprecationWarning: X509Extension support in pyOpenSSL is deprecated. You should use the APIs in cryptography.
    crypto.X509Extension(b'subjectKeyIdentifier', False, b'hash', subject=cert),

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=============================================================================== 37 passed, 26 warnings in 2035.61s (0:33:55) ===============================================================================

[Leoquicenoz]

Update

What happened was that as the last commit had been set before creating the branch in the wazuh/qa-integration-framework repo, that's why it was not taking that new branch and the checks remained the same, to make a new commit (without significant changes) went from 21 checks that were failing to 4, with a rebase was able to solve the check that failed test_execd, but still other tests continue to fail.

[Leoquicenoz]

ETA is delayed due to changes made to the base branch, from base branch 4.9.0 to 22080-migrate-and-adapt-the-wazuh-agent-subsystem, which caused some checks to fail.

[Leoquicenoz]

Update

Changes made in the wazuh/packages repository are moved to wazuh/wazuh