Skip to content

Conversation

@kgiusti
Copy link
Contributor

@kgiusti kgiusti commented Nov 1, 2019

To run the AddressSanitizer, define RUNTIME_CHECK=asan via CMake:

cmake .. -DRUNTIME_CHECK=asan

This patch includes minor code tweaks to issues found when running
with the sanitizer enabled.

@kgiusti kgiusti requested a review from jirkadanek November 1, 2019 19:03
@codecov-io
Copy link

codecov-io commented Nov 1, 2019

Codecov Report

Merging #608 into master will increase coverage by 0.03%.
The diff coverage is 72.72%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #608      +/-   ##
==========================================
+ Coverage   86.44%   86.47%   +0.03%     
==========================================
  Files          91       91              
  Lines       20622    20618       -4     
==========================================
+ Hits        17826    17830       +4     
+ Misses       2796     2788       -8
Impacted Files Coverage Δ
src/alloc_pool.c 96.01% <ø> (ø) ⬆️
router/src/main.c 70% <72.72%> (-0.59%) ⬇️
src/message.c 91.08% <0%> (+0.09%) ⬆️
src/router_node.c 93.21% <0%> (+0.11%) ⬆️
src/iterator.c 89.34% <0%> (+0.16%) ⬆️
src/router_core/connections.c 94.19% <0%> (+0.22%) ⬆️
src/router_core/transfer.c 94.02% <0%> (+0.74%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f03308c...1c2917d. Read the comment docs.

@jirkadanek
Copy link

I didn't notice the GH mention before. I read through the PR diff. I'll now try compiling and running with GCC and Clang.

@jirkadanek
Copy link

When compiling and running tests with gcc, with libwebsocket available, I got this crash in test 9 and in other tests which don't readily print these messages, and in test 19 which prints it. After that I killed ctest.

When compiled without libwebsocket, I get clean result, except for system_tests_console. That probably should not run, when I don't have libwebsockets?

9: -----------------------------------------------------
9: Suppressions used:
9:   count      bytes template
9:       1         48 qdr_core_subscribe
9:     530     855269 *libpython*
9:       1       1208 run_unit_tests.c
9: -----------------------------------------------------

The libwebsocket fail

9: 2019-11-04 15:20:33.327766 +0100 AGENT (debug) Add entity: RouterEntity(allowResumableLinkRoute=True, allowUnsettledMulticast=False, area=0, defaultDistribution=balanced, helloIntervalSeconds=1, helloMaxAgeSeconds=3, hostName=nixos, id=QDR, mode=standalone, raIntervalFluxSeconds=4, raIntervalSeconds=30, remoteLsMaxAgeSeconds=60, saslConfigName=qdrouterd, type=org.apache.qpid.dispatch.router, workerThreads=4)
9: ASAN:DEADLYSIGNAL
9: =================================================================
9: ==18538==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x7f61fce828eb bp 0x7ffcc66b7530 sp 0x7ffcc66b6ca0 T0)
9: ==18538==The signal is caused by a READ memory access.
9: ==18538==Hint: address points to the zero page.
9:     #0 0x7f61fce828ea  (/nix/store/mvvw8d463lnbhz9nif13xfjk9n793p6r-gcc-7.4.0-lib/lib/libasan.so.4+0x5a8ea)
9:     #1 0x7f61fb0ec46c in lh_insert (/nix/store/z8bn38sz1z1vm31j8qhkgzhw1frpyc5x-openssl-1.0.2t/lib/libcrypto.so.1.0.0+0x12f46c)
9:     #2 0x7f61fb032b8a in OBJ_NAME_add (/nix/store/z8bn38sz1z1vm31j8qhkgzhw1frpyc5x-openssl-1.0.2t/lib/libcrypto.so.1.0.0+0x75b8a)
9:     #3 0x7f61faeab3da in ossl_init_ssl_base_ossl_ (/nix/store/x8gkxxkrw46q3sxhpfrcvc2hlgvb144w-openssl-1.1.1c/lib/libssl.so.1.1+0x393da)
9:     #4 0x7f61fc6b69c8 in __pthread_once_slow (/nix/store/iykxb0bmfjmi7s53kfg6pjbfpd8jmza6-glibc-2.27/lib/libpthread.so.0+0xf9c8)
9:     #5 0x7f61fad84b78 in CRYPTO_THREAD_run_once (/nix/store/x8gkxxkrw46q3sxhpfrcvc2hlgvb144w-openssl-1.1.1c/lib/libcrypto.so.1.1+0x20db78)
9:     #6 0x7f61faeab5d6 in OPENSSL_init_ssl (/nix/store/x8gkxxkrw46q3sxhpfrcvc2hlgvb144w-openssl-1.1.1c/lib/libssl.so.1.1+0x395d6)
9:     #7 0x7f61fc2fea6f in lws_context_init_ssl_library (/nix/store/z3pys76m2hy4f8xr00yzsh98irwywsqg-libwebsockets-3.2.0/lib/libwebsockets.so.15+0x2ca6f)
9:     #8 0x7f61fc2e4467 in lws_create_context (/nix/store/z3pys76m2hy4f8xr00yzsh98irwywsqg-libwebsockets-3.2.0/lib/libwebsockets.so.15+0x12467)
9:     #9 0x7f61fcb06723 in qd_http_server ../src/http-libwebsockets.c:836
9:     #10 0x7f61fcaedee0 in qd_server ../src/server.c:1244
9:     #11 0x7f61fc9c718d in qd_dispatch_prepare ../src/dispatch.c:320
9:     #12 0x7f61f800e05d in ffi_call_unix64 (/nix/store/8pivgdaciz2add2wv7ff13gbz0rijs3f-libffi-3.2.1/lib/libffi.so.6+0x805d)
9:     #13 0x7f61f800d022 in ffi_call (/nix/store/8pivgdaciz2add2wv7ff13gbz0rijs3f-libffi-3.2.1/lib/libffi.so.6+0x7022)
9:     #14 0x7f61f7e2a93c in _ctypes_callproc (/nix/store/yglc8kgzmdaskyngv90y1mw96fl00pjy-python3-3.7.4/lib/python3.7/lib-dynload/_ctypes.cpython-37m-x86_64-linux-gnu.so+0x1293c)
9:     #15 0x7f61f7e2144b in PyCFuncPtr_call (/nix/store/yglc8kgzmdaskyngv90y1mw96fl00pjy-python3-3.7.4/lib/python3.7/lib-dynload/_ctypes.cpython-37m-x86_64-linux-gnu.so+0x944b)
9:     #16 0x7f61fc3ce7d2 in _PyObject_FastCallKeywords (/nix/store/yglc8kgzmdaskyngv90y1mw96fl00pjy-python3-3.7.4/lib/libpython3.7m.so.1.0+0x9f7d2)
9:     #17 0x7f61fc3a49a7 in _PyEval_EvalFrameDefault (/nix/store/yglc8kgzmdaskyngv90y1mw96fl00pjy-python3-3.7.4/lib/libpython3.7m.so.1.0+0x759a7)
9:     #18 0x7f61fc4bc3fd in _PyEval_EvalCodeWithName (/nix/store/yglc8kgzmdaskyngv90y1mw96fl00pjy-python3-3.7.4/lib/libpython3.7m.so.1.0+0x18d3fd)
9:     #19 0x7f61fc3ce02e in _PyFunction_FastCallDict (/nix/store/yglc8kgzmdaskyngv90y1mw96fl00pjy-python3-3.7.4/lib/libpython3.7m.so.1.0+0x9f02e)
9:     #20 0x7f61fc3cfcb6 in PyObject_CallFunction (/nix/store/yglc8kgzmdaskyngv90y1mw96fl00pjy-python3-3.7.4/lib/libpython3.7m.so.1.0+0xa0cb6)
9:     #21 0x7f61fc9c581a in qd_dispatch_load_config ../src/dispatch.c:120
9:     #22 0x40494a in main ../tests/run_unit_tests.c:52
9:     #23 0x7f61fb61cb8d in __libc_start_main (/nix/store/iykxb0bmfjmi7s53kfg6pjbfpd8jmza6-glibc-2.27/lib/libc.so.6+0x22b8d)
9:     #24 0x404cd9 in _start (/home/jdanek/repos/qpid/qpid-dispatch/build/tests/unit_tests+0x404cd9)
9: 
9: AddressSanitizer can not provide additional info.
9: SUMMARY: AddressSanitizer: SEGV (/nix/store/mvvw8d463lnbhz9nif13xfjk9n793p6r-gcc-7.4.0-lib/lib/libasan.so.4+0x5a8ea) 
9: ==18538==ABORTING
 9/63 Test  #9: unit_tests ........................................***Failed    0.30 sec

Console test fail

54: Test command: /nix/store/yglc8kgzmdaskyngv90y1mw96fl00pjy-python3-3.7.4/bin/python "/home/jdanek/repos/qpid/qpid-dispatch/build/tests/run.py" "-m" "unittest" "-v" "system_tests_console"
54: Test timeout computed to be: 360
54: ERROR
54: /nix/store/yglc8kgzmdaskyngv90y1mw96fl00pjy-python3-3.7.4/lib/python3.7/subprocess.py:858: ResourceWarning: subprocess 6685 is still running
54:   ResourceWarning, source=self)
54: ResourceWarning: Enable tracemalloc to get the object allocation traceback
54: 
54: ======================================================================
54: ERROR: setUpClass (system_tests_console.ConsoleTest)
54: ----------------------------------------------------------------------
54: Traceback (most recent call last):
54:   File "/home/jdanek/repos/qpid/qpid-dispatch/tests/system_test.py", line 206, in wait_port
54:     retry_exception(connect, exception_test=check, **retry_kwargs)
54:   File "/home/jdanek/repos/qpid/qpid-dispatch/tests/system_test.py", line 156, in retry_exception
54:     return function()
54:   File "/home/jdanek/repos/qpid/qpid-dispatch/tests/system_test.py", line 201, in connect
54:     s.connect((host, port))
54: ConnectionRefusedError: [Errno 111] Connection refused
54: 
54: During handling of the above exception, another exception occurred:
54: 
54: Traceback (most recent call last):
54:   File "/home/jdanek/repos/qpid/qpid-dispatch/tests/system_tests_console.py", line 91, in setUpClass
54:     cls.router = cls.tester.qdrouterd('test-router', config)
54:   File "/home/jdanek/repos/qpid/qpid-dispatch/tests/system_test.py", line 628, in qdrouterd
54:     return self.cleanup(Qdrouterd(*args, **kwargs))
54:   File "/home/jdanek/repos/qpid/qpid-dispatch/tests/system_test.py", line 417, in __init__
54:     self.wait_ready()
54:   File "/home/jdanek/repos/qpid/qpid-dispatch/tests/system_test.py", line 549, in wait_ready
54:     self.wait_ports(**retry_kwargs)
54:   File "/home/jdanek/repos/qpid/qpid-dispatch/tests/system_test.py", line 534, in wait_ports
54:     wait_ports(self.ports_family, **retry_kwargs)
54:   File "/home/jdanek/repos/qpid/qpid-dispatch/tests/system_test.py", line 214, in wait_ports
54:     wait_port(port=port, protocol_family=protocol_family, **retry_kwargs)
54:   File "/home/jdanek/repos/qpid/qpid-dispatch/tests/system_test.py", line 208, in wait_port
54:     raise Exception("wait_port timeout on port %s: %s" % (port, e))
54: Exception: wait_port timeout on port 20734: [Errno 111] Connection refused
54: 
54: ----------------------------------------------------------------------
54: Ran 0 tests in 60.010s
54: 
54: FAILED (errors=1)
54/62 Test #54: system_tests_console ..............................***Failed   61.04 sec

@jirkadanek
Copy link

With Clang, dispatch failed to link. I think this is because of -Wl,-z,defs flag. google/sanitizers#380. I'll check if that is added by CMake because I am doing something wrong, or if CMake txts need updating for Clang.

FAILED: src/libqpid-dispatch.so 
: && /nix/store/k12qdjjkz6fnkdpqcg016nq8ikj0ixq2-clang-wrapper-8.0.1/bin/clang -fPIC -g -fno-omit-frame-pointer -fsanitize=address,undefined -O2 -g -DNDEBUG  -Wl,-z,defs -shared -Wl,-soname,libqpid-dispatch.so -o src/libqpid-dispatch.so src/CMakeFiles/qpid-dispatch.dir/alloc_pool.c.o src/CMakeFiles/qpid-dispatch.dir/amqp.c.o src/CMakeFiles/qpid-dispatch.dir/bitmask.c.o src/CMakeFiles/qpid-dispatch.dir/buffer.c.o src/CMakeFiles/qpid-dispatch.dir/error.c.o src/CMakeFiles/qpid-dispatch.dir/compose.c.o src/CMakeFiles/qpid-dispatch.dir/connection_manager.c.o src/CMakeFiles/qpid-dispatch.dir/container.c.o src/CMakeFiles/qpid-dispatch.dir/discriminator.c.o src/CMakeFiles/qpid-dispatch.dir/dispatch.c.o src/CMakeFiles/qpid-dispatch.dir/entity.c.o src/CMakeFiles/qpid-dispatch.dir/entity_cache.c.o src/CMakeFiles/qpid-dispatch.dir/failoverlist.c.o src/CMakeFiles/qpid-dispatch.dir/hash.c.o src/CMakeFiles/qpid-dispatch.dir/iterator.c.o src/CMakeFiles/qpid-dispatch.dir/log.c.o src/CMakeFiles/qpid-dispatch.dir/message.c.o src/CMakeFiles/qpid-dispatch.dir/parse.c.o src/CMakeFiles/qpid-dispatch.dir/parse_tree.c.o src/CMakeFiles/qpid-dispatch.dir/policy.c.o src/CMakeFiles/qpid-dispatch.dir/proton_utils.c.o src/CMakeFiles/qpid-dispatch.dir/remote_sasl.c.o src/CMakeFiles/qpid-dispatch.dir/posix/threading.c.o src/CMakeFiles/qpid-dispatch.dir/python_embedded.c.o src/CMakeFiles/qpid-dispatch.dir/router_agent.c.o src/CMakeFiles/qpid-dispatch.dir/router_config.c.o src/CMakeFiles/qpid-dispatch.dir/address_lookup_utils.c.o src/CMakeFiles/qpid-dispatch.dir/router_core/agent.c.o src/CMakeFiles/qpid-dispatch.dir/router_core/agent_address.c.o src/CMakeFiles/qpid-dispatch.dir/router_core/agent_config_address.c.o src/CMakeFiles/qpid-dispatch.dir/router_core/agent_config_auto_link.c.o src/CMakeFiles/qpid-dispatch.dir/router_core/agent_connection.c.o src/CMakeFiles/qpid-dispatch.dir/router_core/agent_config_link_route.c.o src/CMakeFiles/qpid-dispatch.dir/router_core/agent_link.c.o src/CMakeFiles/qpid-dispatch.dir/router_core/agent_router.c.o src/CMakeFiles/qpid-dispatch.dir/router_core/agent_conn_link_route.c.o src/CMakeFiles/qpid-dispatch.dir/router_core/connections.c.o src/CMakeFiles/qpid-dispatch.dir/router_core/core_events.c.o src/CMakeFiles/qpid-dispatch.dir/router_core/core_link_endpoint.c.o src/CMakeFiles/qpid-dispatch.dir/router_core/core_client_api.c.o src/CMakeFiles/qpid-dispatch.dir/router_core/delivery.c.o src/CMakeFiles/qpid-dispatch.dir/router_core/error.c.o src/CMakeFiles/qpid-dispatch.dir/router_core/exchange_bindings.c.o src/CMakeFiles/qpid-dispatch.dir/router_core/forwarder.c.o src/CMakeFiles/qpid-dispatch.dir/router_core/route_control.c.o src/CMakeFiles/qpid-dispatch.dir/router_core/router_core.c.o src/CMakeFiles/qpid-dispatch.dir/router_core/router_core_thread.c.o src/CMakeFiles/qpid-dispatch.dir/router_core/route_tables.c.o src/CMakeFiles/qpid-dispatch.dir/router_core/management_agent.c.o src/CMakeFiles/qpid-dispatch.dir/router_core/terminus.c.o src/CMakeFiles/qpid-dispatch.dir/router_core/transfer.c.o src/CMakeFiles/qpid-dispatch.dir/router_core/core_timer.c.o src/CMakeFiles/qpid-dispatch.dir/router_core/modules/edge_router/module.c.o src/CMakeFiles/qpid-dispatch.dir/router_core/modules/edge_router/addr_proxy.c.o src/CMakeFiles/qpid-dispatch.dir/router_core/modules/edge_router/connection_manager.c.o src/CMakeFiles/qpid-dispatch.dir/router_core/modules/edge_router/link_route_proxy.c.o src/CMakeFiles/qpid-dispatch.dir/router_core/modules/edge_router/edge_mgmt.c.o src/CMakeFiles/qpid-dispatch.dir/router_core/modules/test_hooks/core_test_hooks.c.o src/CMakeFiles/qpid-dispatch.dir/router_core/modules/edge_addr_tracking/edge_addr_tracking.c.o src/CMakeFiles/qpid-dispatch.dir/router_core/modules/address_lookup_server/address_lookup_server.c.o src/CMakeFiles/qpid-dispatch.dir/router_core/modules/address_lookup_client/lookup_client.c.o src/CMakeFiles/qpid-dispatch.dir/router_core/modules/stuck_delivery_detection/delivery_tracker.c.o src/CMakeFiles/qpid-dispatch.dir/router_node.c.o src/CMakeFiles/qpid-dispatch.dir/router_pynode.c.o src/CMakeFiles/qpid-dispatch.dir/schema_enum.c.o src/CMakeFiles/qpid-dispatch.dir/server.c.o src/CMakeFiles/qpid-dispatch.dir/timer.c.o src/CMakeFiles/qpid-dispatch.dir/trace_mask.c.o src/CMakeFiles/qpid-dispatch.dir/python_utils.c.o src/CMakeFiles/qpid-dispatch.dir/http-none.c.o  -Wl,-rpath,/home/jdanek/repos/qpid/qpid-proton/build/install/lib64: /home/jdanek/repos/qpid/qpid-proton/build/install/lib64/libqpid-proton-core.so /home/jdanek/repos/qpid/qpid-proton/build/install/lib64/libqpid-proton-proactor.so -lpthread -lrt -ldl -lpython2.7 && :
/nix/store/mgdjnsrkqgmxqjaqaxgqyqm7fwyi96fk-binutils-2.31.1/bin/ld: warning: size of symbol `qdr_address_columns' changed from 176 in src/CMakeFiles/qpid-dispatch.dir/router_core/agent.c.o to 224 in src/CMakeFiles/qpid-dispatch.dir/router_core/agent_address.c.o
/nix/store/mgdjnsrkqgmxqjaqaxgqyqm7fwyi96fk-binutils-2.31.1/bin/ld: warning: size of symbol `qdr_config_address_columns' changed from 96 in src/CMakeFiles/qpid-dispatch.dir/router_core/agent.c.o to 128 in src/CMakeFiles/qpid-dispatch.dir/router_core/agent_config_address.c.o
/nix/store/mgdjnsrkqgmxqjaqaxgqyqm7fwyi96fk-binutils-2.31.1/bin/ld: warning: size of symbol `qdr_config_auto_link_columns' changed from 136 in src/CMakeFiles/qpid-dispatch.dir/router_core/agent.c.o to 192 in src/CMakeFiles/qpid-dispatch.dir/router_core/agent_config_auto_link.c.o
/nix/store/mgdjnsrkqgmxqjaqaxgqyqm7fwyi96fk-binutils-2.31.1/bin/ld: warning: size of symbol `qdr_connection_columns' changed from 176 in src/CMakeFiles/qpid-dispatch.dir/router_core/agent.c.o to 224 in src/CMakeFiles/qpid-dispatch.dir/router_core/agent_connection.c.o
/nix/store/mgdjnsrkqgmxqjaqaxgqyqm7fwyi96fk-binutils-2.31.1/bin/ld: warning: size of symbol `qdr_config_link_route_columns' changed from 112 in src/CMakeFiles/qpid-dispatch.dir/router_core/agent.c.o to 160 in src/CMakeFiles/qpid-dispatch.dir/router_core/agent_config_link_route.c.o
/nix/store/mgdjnsrkqgmxqjaqaxgqyqm7fwyi96fk-binutils-2.31.1/bin/ld: warning: size of symbol `qdr_link_columns' changed from 240 in src/CMakeFiles/qpid-dispatch.dir/router_core/agent.c.o to 288 in src/CMakeFiles/qpid-dispatch.dir/router_core/agent_link.c.o
/nix/store/mgdjnsrkqgmxqjaqaxgqyqm7fwyi96fk-binutils-2.31.1/bin/ld: warning: size of symbol `qdr_router_columns' changed from 248 in src/CMakeFiles/qpid-dispatch.dir/router_core/agent.c.o to 288 in src/CMakeFiles/qpid-dispatch.dir/router_core/agent_router.c.o
/nix/store/mgdjnsrkqgmxqjaqaxgqyqm7fwyi96fk-binutils-2.31.1/bin/ld: src/CMakeFiles/qpid-dispatch.dir/alloc_pool.c.o: in function `qd_alloc':
/home/jdanek/repos/qpid/qpid-dispatch/build/../src/alloc_pool.c:250: undefined reference to `__asan_option_detect_stack_use_after_return'
/nix/store/mgdjnsrkqgmxqjaqaxgqyqm7fwyi96fk-binutils-2.31.1/bin/ld: /home/jdanek/repos/qpid/qpid-dispatch/build/../src/alloc_pool.c:250: undefined reference to `__asan_stack_malloc_0'
/nix/store/mgdjnsrkqgmxqjaqaxgqyqm7fwyi96fk-binutils-2.31.1/bin/ld: src/CMakeFiles/qpid-dispatch.dir/alloc_pool.c.o: in function `qd_alloc_init':
/home/jdanek/repos/qpid/qpid-dispatch/build/../src/alloc_pool.c:231: undefined reference to `__asan_memset'
/nix/store/mgdjnsrkqgmxqjaqaxgqyqm7fwyi96fk-binutils-2.31.1/bin/ld: src/CMakeFiles/qpid-dispatch.dir/alloc_pool.c.o: in function `qd_alloc':
/home/jdanek/repos/qpid/qpid-dispatch/build/../src/alloc_pool.c:314: undefined reference to `__ubsan_handle_type_mismatch_v1'
/nix/store/mgdjnsrkqgmxqjaqaxgqyqm7fwyi96fk-binutils-2.31.1/bin/ld: /home/jdanek/repos/qpid/qpid-dispatch/build/../src/alloc_pool.c:314: undefined reference to `__ubsan_handle_type_mismatch_v1'
/nix/store/mgdjnsrkqgmxqjaqaxgqyqm7fwyi96fk-binutils-2.31.1/bin/ld: /home/jdanek/repos/qpid/qpid-dispatch/build/../src/alloc_pool.c:314: undefined reference to `__ubsan_handle_type_mismatch_v1'

@jirkadanek
Copy link

Ok, it is the tsan change. http://clang.llvm.org/docs/AddressSanitizer.html mentions that the shared asan library will not link when -Wl,-z,defs is used.

Before

if (NOT (USE_SANITIZERS OR USE_TSAN))
set(CATCH_UNDEFINED "-Wl,-z,defs")
endif ()

Now

if (APPLE)
set(QPID_DISPATCH_LIB_LINK_FLAGS "-Wl,-flat_namespace ${QPID_DISPATCH_LIB_LINK_FLAGS}")
else(APPLE)
set(QPID_DISPATCH_LIB_LINK_FLAGS "-Wl,-z,defs ${QPID_DISPATCH_LIB_LINK_FLAGS}")
endif(APPLE)

@kgiusti
Copy link
Contributor Author

kgiusti commented Nov 4, 2019

When compiling and running tests with gcc, with libwebsocket available, I got this crash in test 9 and in other tests which don't readily print these messages, and in test 19 which prints it. After that I killed ctest.

When compiled without libwebsocket, I get clean result, except for system_tests_console. That probably should not run, when I don't have libwebsockets?

9: -----------------------------------------------------
9: Suppressions used:
9:   count      bytes template
9:       1         48 qdr_core_subscribe
9:     530     855269 *libpython*
9:       1       1208 run_unit_tests.c
9: -----------------------------------------------------

The libwebsocket fail

9: 2019-11-04 15:20:33.327766 +0100 AGENT (debug) Add entity: RouterEntity(allowResumableLinkRoute=True, allowUnsettledMulticast=False, area=0, defaultDistribution=balanced, helloIntervalSeconds=1, helloMaxAgeSeconds=3, hostName=nixos, id=QDR, mode=standalone, raIntervalFluxSeconds=4, raIntervalSeconds=30, remoteLsMaxAgeSeconds=60, saslConfigName=qdrouterd, type=org.apache.qpid.dispatch.router, workerThreads=4)
9: ASAN:DEADLYSIGNAL
9: =================================================================
9: ==18538==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x7f61fce828eb bp 0x7ffcc66b7530 sp 0x7ffcc66b6ca0 T0)
9: ==18538==The signal is caused by a READ memory access.
9: ==18538==Hint: address points to the zero page.
9:     #0 0x7f61fce828ea  (/nix/store/mvvw8d463lnbhz9nif13xfjk9n793p6r-gcc-7.4.0-lib/lib/libasan.so.4+0x5a8ea)
9:     #1 0x7f61fb0ec46c in lh_insert (/nix/store/z8bn38sz1z1vm31j8qhkgzhw1frpyc5x-openssl-1.0.2t/lib/libcrypto.so.1.0.0+0x12f46c)
9:     #2 0x7f61fb032b8a in OBJ_NAME_add (/nix/store/z8bn38sz1z1vm31j8qhkgzhw1frpyc5x-openssl-1.0.2t/lib/libcrypto.so.1.0.0+0x75b8a)
9:     #3 0x7f61faeab3da in ossl_init_ssl_base_ossl_ (/nix/store/x8gkxxkrw46q3sxhpfrcvc2hlgvb144w-openssl-1.1.1c/lib/libssl.so.1.1+0x393da)
9:     #4 0x7f61fc6b69c8 in __pthread_once_slow (/nix/store/iykxb0bmfjmi7s53kfg6pjbfpd8jmza6-glibc-2.27/lib/libpthread.so.0+0xf9c8)
9:     #5 0x7f61fad84b78 in CRYPTO_THREAD_run_once (/nix/store/x8gkxxkrw46q3sxhpfrcvc2hlgvb144w-openssl-1.1.1c/lib/libcrypto.so.1.1+0x20db78)
9:     #6 0x7f61faeab5d6 in OPENSSL_init_ssl (/nix/store/x8gkxxkrw46q3sxhpfrcvc2hlgvb144w-openssl-1.1.1c/lib/libssl.so.1.1+0x395d6)
9:     #7 0x7f61fc2fea6f in lws_context_init_ssl_library (/nix/store/z3pys76m2hy4f8xr00yzsh98irwywsqg-libwebsockets-3.2.0/lib/libwebsockets.so.15+0x2ca6f)
9:     #8 0x7f61fc2e4467 in lws_create_context (/nix/store/z3pys76m2hy4f8xr00yzsh98irwywsqg-libwebsockets-3.2.0/lib/libwebsockets.so.15+0x12467)
9:     #9 0x7f61fcb06723 in qd_http_server ../src/http-libwebsockets.c:836
9:     #10 0x7f61fcaedee0 in qd_server ../src/server.c:1244
9:     #11 0x7f61fc9c718d in qd_dispatch_prepare ../src/dispatch.c:320
9:     #12 0x7f61f800e05d in ffi_call_unix64 (/nix/store/8pivgdaciz2add2wv7ff13gbz0rijs3f-libffi-3.2.1/lib/libffi.so.6+0x805d)
9:     #13 0x7f61f800d022 in ffi_call (/nix/store/8pivgdaciz2add2wv7ff13gbz0rijs3f-libffi-3.2.1/lib/libffi.so.6+0x7022)
9:     #14 0x7f61f7e2a93c in _ctypes_callproc (/nix/store/yglc8kgzmdaskyngv90y1mw96fl00pjy-python3-3.7.4/lib/python3.7/lib-dynload/_ctypes.cpython-37m-x86_64-linux-gnu.so+0x1293c)
9:     #15 0x7f61f7e2144b in PyCFuncPtr_call (/nix/store/yglc8kgzmdaskyngv90y1mw96fl00pjy-python3-3.7.4/lib/python3.7/lib-dynload/_ctypes.cpython-37m-x86_64-linux-gnu.so+0x944b)
9:     #16 0x7f61fc3ce7d2 in _PyObject_FastCallKeywords (/nix/store/yglc8kgzmdaskyngv90y1mw96fl00pjy-python3-3.7.4/lib/libpython3.7m.so.1.0+0x9f7d2)
9:     #17 0x7f61fc3a49a7 in _PyEval_EvalFrameDefault (/nix/store/yglc8kgzmdaskyngv90y1mw96fl00pjy-python3-3.7.4/lib/libpython3.7m.so.1.0+0x759a7)
9:     #18 0x7f61fc4bc3fd in _PyEval_EvalCodeWithName (/nix/store/yglc8kgzmdaskyngv90y1mw96fl00pjy-python3-3.7.4/lib/libpython3.7m.so.1.0+0x18d3fd)
9:     #19 0x7f61fc3ce02e in _PyFunction_FastCallDict (/nix/store/yglc8kgzmdaskyngv90y1mw96fl00pjy-python3-3.7.4/lib/libpython3.7m.so.1.0+0x9f02e)
9:     #20 0x7f61fc3cfcb6 in PyObject_CallFunction (/nix/store/yglc8kgzmdaskyngv90y1mw96fl00pjy-python3-3.7.4/lib/libpython3.7m.so.1.0+0xa0cb6)
9:     #21 0x7f61fc9c581a in qd_dispatch_load_config ../src/dispatch.c:120
9:     #22 0x40494a in main ../tests/run_unit_tests.c:52
9:     #23 0x7f61fb61cb8d in __libc_start_main (/nix/store/iykxb0bmfjmi7s53kfg6pjbfpd8jmza6-glibc-2.27/lib/libc.so.6+0x22b8d)
9:     #24 0x404cd9 in _start (/home/jdanek/repos/qpid/qpid-dispatch/build/tests/unit_tests+0x404cd9)
9: 
9: AddressSanitizer can not provide additional info.
9: SUMMARY: AddressSanitizer: SEGV (/nix/store/mvvw8d463lnbhz9nif13xfjk9n793p6r-gcc-7.4.0-lib/lib/libasan.so.4+0x5a8ea) 
9: ==18538==ABORTING
 9/63 Test  #9: unit_tests ........................................***Failed    0.30 sec

Console test fail

54: Test command: /nix/store/yglc8kgzmdaskyngv90y1mw96fl00pjy-python3-3.7.4/bin/python "/home/jdanek/repos/qpid/qpid-dispatch/build/tests/run.py" "-m" "unittest" "-v" "system_tests_console"
54: Test timeout computed to be: 360
54: ERROR
54: /nix/store/yglc8kgzmdaskyngv90y1mw96fl00pjy-python3-3.7.4/lib/python3.7/subprocess.py:858: ResourceWarning: subprocess 6685 is still running
54:   ResourceWarning, source=self)
54: ResourceWarning: Enable tracemalloc to get the object allocation traceback
54: 
54: ======================================================================
54: ERROR: setUpClass (system_tests_console.ConsoleTest)
54: ----------------------------------------------------------------------
54: Traceback (most recent call last):
54:   File "/home/jdanek/repos/qpid/qpid-dispatch/tests/system_test.py", line 206, in wait_port
54:     retry_exception(connect, exception_test=check, **retry_kwargs)
54:   File "/home/jdanek/repos/qpid/qpid-dispatch/tests/system_test.py", line 156, in retry_exception
54:     return function()
54:   File "/home/jdanek/repos/qpid/qpid-dispatch/tests/system_test.py", line 201, in connect
54:     s.connect((host, port))
54: ConnectionRefusedError: [Errno 111] Connection refused
54: 
54: During handling of the above exception, another exception occurred:
54: 
54: Traceback (most recent call last):
54:   File "/home/jdanek/repos/qpid/qpid-dispatch/tests/system_tests_console.py", line 91, in setUpClass
54:     cls.router = cls.tester.qdrouterd('test-router', config)
54:   File "/home/jdanek/repos/qpid/qpid-dispatch/tests/system_test.py", line 628, in qdrouterd
54:     return self.cleanup(Qdrouterd(*args, **kwargs))
54:   File "/home/jdanek/repos/qpid/qpid-dispatch/tests/system_test.py", line 417, in __init__
54:     self.wait_ready()
54:   File "/home/jdanek/repos/qpid/qpid-dispatch/tests/system_test.py", line 549, in wait_ready
54:     self.wait_ports(**retry_kwargs)
54:   File "/home/jdanek/repos/qpid/qpid-dispatch/tests/system_test.py", line 534, in wait_ports
54:     wait_ports(self.ports_family, **retry_kwargs)
54:   File "/home/jdanek/repos/qpid/qpid-dispatch/tests/system_test.py", line 214, in wait_ports
54:     wait_port(port=port, protocol_family=protocol_family, **retry_kwargs)
54:   File "/home/jdanek/repos/qpid/qpid-dispatch/tests/system_test.py", line 208, in wait_port
54:     raise Exception("wait_port timeout on port %s: %s" % (port, e))
54: Exception: wait_port timeout on port 20734: [Errno 111] Connection refused
54: 
54: ----------------------------------------------------------------------
54: Ran 0 tests in 60.010s
54: 
54: FAILED (errors=1)
54/62 Test #54: system_tests_console ..............................***Failed   61.04 sec

The libwebsockets issue is known - there's some sort of illegal memory bug in certain versions of libwebsockets:

https://issues.apache.org/jira/browse/DISPATCH-1258

@kgiusti
Copy link
Contributor Author

kgiusti commented Nov 4, 2019

@jdanekrh - thanks for the review. I've run the patch on Ubuntu 18.04 and 16.04 and have just pushed an update to the suppression files. I've also enabled ASAN checking on one of the travis builds and am testing that now.

kgiusti added a commit to kgiusti/dispatch that referenced this pull request Nov 4, 2019
To run the AddressSanitizer, define RUNTIME_CHECK=asan via CMake:

  cmake .. -DRUNTIME_CHECK=asan

This patch updates the .travis.yml file to enable address and
undefined behavior checking.  Also included are minor code tweaks to
issues found when running with the sanitizer enabled.

This closes apache#608
@kgiusti
Copy link
Contributor Author

kgiusti commented Nov 4, 2019

Ok, this patch now passes travis so it's ready for review.

README Outdated
This option turns on extra run time memory verification, including
leak checks.

Applicable only to GCC versions >= 7.4 and Clang versions >= 6.0.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Applicable only to GCC versions >= 7.4 and Clang versions >= 6.0.
Applicable only to GCC versions >= 5.4 and Clang versions >= 6.0.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch will update thanks.

getcwd_error = 1;
break;
if (errno != ERANGE) {
// Hard failure - can't recover from this
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change does not seem particularly related to this PR. Or is it? If not, can we make this change in a different commit ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is related - the address sanitizer threw a fit on that section of the code - detected a possible buffer overflow. That change is the fix.

To run the AddressSanitizer, define RUNTIME_CHECK=asan via CMake:

  cmake .. -DRUNTIME_CHECK=asan

This patch updates the .travis.yml file to enable address and
undefined behavior checking.  Also included are minor code tweaks to
issues found when running with the sanitizer enabled.

This closes apache#608
@asfgit asfgit closed this in 1fafe27 Nov 5, 2019
ChugR pushed a commit to ChugR/qpid-dispatch that referenced this pull request Oct 13, 2022
Delete the pseudo egress dispatch connection in the same manner as a
true tcp connection rather than on a timer thread.

Use exact data lengths to verify the TCP counter values.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants