publish message cause segment fault

[kubbo]

Problem Description

eCAL::CPublisher send(msg) coredump, the stack is：

How to reproduce

I cannot consistently reproduce this issue, but it occasionally occurs when multiple threads within the same process call the same publisher to send messages to the same topic

How did you get eCAL?

Ubuntu PPA (apt-get)

Environment

eCAL version： 5.12.1
Operating System： Linux fc22cf261eba 5.10.104-tegra #1 SMP PREEMPT Wed Aug 10 20:17:07 PDT 2022 aarch64 aarch64 aarch64 GNU/Linux
cmake version 3.22.1

eCAL System Information

; --------------------------------------------------
; NETWORK SETTINGS
; --------------------------------------------------
; network_enabled           = true / false          true  = all eCAL components communicate over network boundaries
;                                                   false = local host only communication
;
; multicast_config_version  = v1 / v2               UDP configuration version (Since eCAL 5.12.)
;                                                   v1: default behavior
;                                                   v2: new behavior, comes with a bit more intuitive handling regarding masking of the groups
; multicast_group           = 239.0.0.1             UDP multicast group base
;                                                   All registration and logging is sent on this address
; multicast_mask            = 0.0.0.1-0.0.0.255     v1: Mask maximum number of dynamic multicast group
;                             255.0.0.0-255.255.255.255 v2: masks are now considered like routes masking
;
; multicast_port             = 14000 + x            UDP multicast port number (eCAL will use at least the 2 following port
;                                                   numbers too, so please modify in steps of 10 (e.g. 1010, 1020 ...)
;
; multicast_ttl              = 0 + x                UDP ttl value, also known as hop limit, is used in determining 
;                                                   the intermediate routers being traversed towards the destination
;
; multicast_sndbuf           = 1024 * x             UDP send buffer in bytes
;  
; multicast_rcvbuf           = 1024 * x             UDP receive buffer in bytes
;
; multicast_join_all_if      = false                Linux specific setting to enable joining multicast groups on all network interfacs
;                                                   independent of their link state. Enabling this makes sure that eCAL processes
;                                                   receive data if they are started before network devices are up and running.
;  
; bandwidth_max_udp          = -1                   UDP bandwidth limit for eCAL udp layer (-1 == unlimited)
;  
; inproc_rec_enabled         = true                 Enable to receive on eCAL inner process layer
; shm_rec_enabled            = true                 Enable to receive on eCAL shared memory layer
; udp_mc_rec_enabled         = true                 Enable to receive on eCAL udp multicast layer
;
; npcap_enabled              = false                Enable to receive UDP traffic with the Npcap based receiver
;
; tcp_pubsub_num_executor_reader  = 4               Tcp_pubsub reader amount of threads that shall execute workload
; tcp_pubsub_num_executor_writer  = 4               Tcp_pubsub writer amount of threads that shall execute workload
; tcp_pubsub_max_reconnections    = 5               Tcp_pubsub reconnection attemps the session will try to reconnect in 
;                                                   case of an issue (a negative value means infinite reconnection attemps)
;
; host_group_name            =                      Common host group name that enables interprocess mechanisms across 
;                                                   (virtual) host borders (e.g, Docker); by default equivalent to local host name
; --------------------------------------------------

[network]
network_enabled           = false
multicast_config_version  = v1
multicast_group           = 239.0.0.1
multicast_mask            = 0.0.0.15
multicast_port            = 14000
multicast_ttl             = 2
multicast_sndbuf          = 5242880
multicast_rcvbuf          = 5242880

multicast_join_all_if     = false

bandwidth_max_udp         = -1

inproc_rec_enabled        = true
shm_rec_enabled           = true
tcp_rec_enabled           = true
udp_mc_rec_enabled        = true

npcap_enabled             = false

tcp_pubsub_num_executor_reader = 4
tcp_pubsub_num_executor_writer = 4
tcp_pubsub_max_reconnections   = 5

host_group_name           =

; --------------------------------------------------
; COMMON SETTINGS
; --------------------------------------------------
; registration_timeout    = 60000                   Timeout for topic registration in ms (internal)
; registration_refresh    = 100                    Topic registration refresh cylce (has to be smaller then registration timeout !)

; --------------------------------------------------
[common]
registration_timeout      = 60000
registration_refresh      = 100

; --------------------------------------------------
; TIME SETTINGS
; --------------------------------------------------
; timesync_module_rt      = "ecaltime-localtime"    Time synchronisation interface name (dynamic library)
;                                                   The name will be extended with platform suffix (32|64), debug suffix (d) and platform extension (.dll|.so)
;
;                                                   Available modules are:
;                                                     - ecaltime-localtime    local system time without synchronization        
;                                                     - ecaltime-linuxptp     For PTP / gPTP synchronization over ethernet on Linux
;                                                                             (device configuration in ecaltime.ini)
;                                                     - ecaltime-simtime      Simulation time as published by the eCAL Player.
; --------------------------------------------------
[time]
timesync_module_rt        = "ecaltime-localtime"

; ---------------------------------------------
; PROCESS SETTINGS
; ---------------------------------------------
;
; terminal_emulator = /usr/bin/x-terminal-emulator -e    command for starting applications with an external terminal emulator. If empty, the command will be ignored. Ignored on Windows.
;                                                            e.g.  /usr/bin/x-terminal-emulator -e
;                                                                  /usr/bin/gnome-terminal -x
;                                                                  /usr/bin/xterm -e
;
; ---------------------------------------------
[process]
terminal_emulator         = 

; --------------------------------------------------
; PUBLISHER SETTINGS
; --------------------------------------------------
; use_inproc              = 0, 1, 2                 Use inner process transport layer (0 = off, 1 = on, 2 = auto, default = 0)
; use_shm                 = 0, 1, 2                 Use shared memory transport layer (0 = off, 1 = on, 2 = auto, default = 2)
; use_tcp                 = 0, 1, 2                 Use tcp transport layer           (0 = off, 1 = on, 2 = auto, default = 0)
; use_udp_mc              = 0, 1, 2                 Use udp multicast transport layer (0 = off, 1 = on, 2 = auto, default = 2)
;
; memfile_minsize         = x * 4096 kB             Default memory file size for new publisher
;
; memfile_reserve         = 50 .. x %               Dynamic file size reserve before recreating memory file if topic size changes
;
; memfile_ack_timeout     = 0 .. x ms               Publisher timeout for ack event from subscriber that memory file content is processed
;
; memfile_buffer_count    = 1 .. x                  Number of parallel used memory file buffers for 1:n publish/subscribe ipc connections (default = 1)
; memfile_zero_copy       = 0, 1                    Allow matching subscriber to access memory file without copying its content in advance (blocking mode)
;
; share_ttype             = 0, 1                    Share topic type via registration layer
; share_tdesc             = 0, 1                    Share topic description via registration layer (switch off to disable reflection)
; --------------------------------------------------
[publisher]
use_inproc                = 0
use_shm                   = 2
use_tcp                   = 0
use_udp_mc                = 2

memfile_minsize           = 4096
memfile_reserve           = 50
memfile_ack_timeout       = 0
memfile_buffer_count      = 1
memfile_zero_copy         = 0

share_ttype               = 1
share_tdesc               = 1

; --------------------------------------------------
; MONITORING SETTINGS
; --------------------------------------------------
; timeout                 = 1000 + (x * 1000)       Timeout for topic monitoring in ms
; filter_excl             =  __.*                   Topics blacklist as regular expression (will not be monitored)
; filter_incl             =                         Topics whitelist as regular expression (will be monitored only)
; filter_log_con          = warning, error          Log messages logged to console (all, info, warning, error, fatal, debug1, debug2, debug3, debug4)
; filter_log_file         =                         Log messages to logged into file system
; filter_log_udp          = warning, error, fatal   Log messages logged via udp network
; --------------------------------------------------
[monitoring]
timeout                   = 5000
filter_excl               = __.*
filter_incl               =
filter_log_con            = error, fatal, warning
filter_log_file           =
filter_log_udp            = info, warning, error, fatal

; --------------------------------------------------
; SYS SETTINGS
; --------------------------------------------------
; filter_excl             = App1,App2               Apps blacklist to be excluded when importing tasks from cloud
; --------------------------------------------------
[sys]
filter_excl               = ^eCALSysClient$|^eCALSysGUI$|^eCALSys$

; --------------------------------------------------
; EXPERIMENTAL SETTINGS
; --------------------------------------------------
; shm_monitoring_enabled      = false              Enable distribution of monitoring/registration information via shared memory
; shm_monitoring_domain       = ecal_monitoring    Domain name for shared memory based monitoring/registration
; shm_monitoring_queue_size   = 1024               Queue size of monitoring/registration events
; network_monitoring_disabled = false              Disable distribution of monitoring/registration information via network
;
; drop_out_of_order_messages  = false              Enable dropping of payload messages that arrive out of order
; --------------------------------------------------
[experimental]
shm_monitoring_enabled      = false
shm_monitoring_domain       = ecal_mon
shm_monitoring_queue_size   = 1024
network_monitoring_disabled = false

drop_out_of_order_messages  = false

[FlorianReimold]

I understood, that you call publisher.Send() in parallel from multiple threads without any synchronization.

• @rex-schilasky: Is that API supposed to be thread safe?
• @kubbo: Does it still crash, if you protect the publisher.Send() with an std::mutex? The sending would need to be synchronized anyways, either bei you or by eCAL.

Kind Regards
Florian

[rex-schilasky]

I have to check and reproduce this. Thank you @kubbo for reporting.