dpdkstat and dpdkevent runtime error message "Cannot init tail queues for objects?"

[rithikach]

DPDK version : dpdk-stable-16.11.1
g++ ver : Red Hat 4.8.5-

1. Linux 3.10.0-514.2.2.el7.x86_64
2. installed dpdk-stable-16.11.1 and collectd with dpdkstat,event
3. code compiled and executed steps provided in github "https://github.com/collectd/collectd/blob/master/docs/BUILD.dpdkstat.md"
4. seeing the below error message in log file on invoking the dpdkstat using collectd,

Coremask "0x02" MemoryChannels "4" ProcessType "secondary" FilePrefix "rte" SharedMemObj "dpdk_collectd_stats_0" EnabledPortMask 0xffff PortName "Eth1"

Apr 7 07:13:35 abcd collectd-dpdk[17283]: EAL: Probing VFIO support...
Apr 7 07:13:35 abcd collectd-dpdk[17283]: EAL: WARNING: Address Space Layout Randomization (ASLR) is enabled in the kernel.
Apr 7 07:13:35 abcd collectd-dpdk[17283]: EAL: This may cause issues with mapping memory into secondary processes
Apr 7 07:13:35 abcd collectd-dpdk[17283]: EAL: Cannot initialize tailq: RTE_DISTRIBUTOR
Apr 7 07:13:35 abcd collectd-dpdk[17283]: PANIC in rte_eal_init():
Apr 7 07:13:35 abcd collectd-dpdk[17283]: Cannot init tail queues for objects
Apr 7 07:13:35 abcd collectd-dpdk[17283]: 13: [/lib64/libc.so.6(clone+0x6d) [0x7f01282b073d]]
Apr 7 07:13:35 abcd collectd-dpdk[17283]: 12: [/lib64/libpthread.so.0(+0x7dc5) [0x7f0128785dc5]]
Apr 7 07:13:35 abcd collectd-dpdk[17283]: 11: [./collectd() [0x40d62b]]
Apr 7 07:13:35 abcd collectd-dpdk[17283]: 10: [/opt/collectd/lib/collectd/dpdkstat.so(+0x2aa3) [0x7f01279a9aa3]]
Apr 7 07:13:35 abcd collectd-dpdk[17283]: 9: [/opt/collectd/lib/collectd/dpdkstat.so(+0x4677) [0x7f01279ab677]]
Apr 7 07:13:35 abcd collectd-dpdk[17283]: 8: [/opt/collectd/lib/collectd/dpdkstat.so(+0x4487) [0x7f01279ab487]]
Apr 7 07:13:35 abcd collectd-dpdk[17283]: 7: [/opt/collectd/lib/collectd/dpdkstat.so(+0x3893) [0x7f01279aa893]]
Apr 7 07:13:35 abcd collectd-dpdk[17283]: 6: [/opt/collectd/lib/collectd/dpdkstat.so(+0x41e1) [0x7f01279ab1e1]]
Apr 7 07:13:35 abcd collectd-dpdk[17283]: 5: [/opt/collectd/lib/collectd/dpdkstat.so(+0x40dc) [0x7f01279ab0dc]]
Apr 7 07:13:35 abcd collectd-dpdk[17283]: 4: [/opt/collectd/lib/collectd/dpdkstat.so(+0x3d26) [0x7f01279aad26]]
Apr 7 07:13:35 abcd collectd-dpdk[17283]: 3: [/usr/lib/librte_eal.so.3.1(rte_eal_init+0xfed) [0x7f012588efed]]
Apr 7 07:13:35 abcd collectd-dpdk[17283]: 2: [/usr/lib/librte_eal.so.3.1(__rte_panic+0xc0) [0x7f012588db1a]]
Apr 7 07:13:35 abcd collectd-dpdk[17283]: 1: [/usr/lib/librte_eal.so.3.1(rte_dump_stack+0x1a) [0x7f012589608a]]

please let me know what could be the problem? is this environment issue ?

do we have any document that help me in understanding what situation we need to use dpdkstats and dpdkevent plugin need to use ?

[maryamtahhan]

Hi @rithikach
Can you please provide a few more details,

1. what your config blocks are
2. are you running your DPDK primary app as a root user? or a with root permissions?

Thanks in advance
Maryam

[maryamtahhan]

Also what drivers are you using?

[rithikach]

Ans 1 ) config:

Coremask "0x0f" MemoryChannels "4" ProcessType "secondary" FilePrefix "rte" SharedMemObj "dpdk_collectd_stats_0" EnabledPortMask 0xffff PortName "Eth1"

Ans 2 ) sudo user

Ans 3) Hypervisor : VMware

Error message :
Apr 7 07:13:35 abcd collectd-dpdk[17283]: EAL: WARNING: Address Space Layout Randomization (ASLR) is enabled in the kernel.
Apr 7 07:13:35 abcd collectd-dpdk[17283]: EAL: This may cause issues with mapping memory into secondary processes
Apr 7 07:13:35 abcd collectd-dpdk[17283]: EAL: Cannot initialize tailq: RTE_DISTRIBUTOR
Apr 7 07:13:35 abcd collectd-dpdk[17283]: PANIC in rte_eal_init():
Apr 7 07:13:35 abcd collectd-dpdk[17283]: Cannot init tail queues for objects

[maryamtahhan]

@rithikach are you using the same EAL parameters for dpdkevents also, is dpdkevents enabled?
I'm assuming from 3, you are using the DPDK VFs for a NIC that you have...

[maryamtahhan]

is the primary app you are trying to monitor using the same version of DPDK also?

[rithikach]

EAL parameters below ,

<Plugin "dpdkevents">

Coremask "0x0f"
MemoryChannels "4"
ProcessType "secondary"
FilePrefix "rte"

<Event "link_status">
SendEventsOnUpdate true
EnabledPortMask 0xffff
PortName "eth1"
SendNotification false

<Event "keep_alive">
SendEventsOnUpdate true
LCoreMask "0x0f"
KeepAliveShmName "/dpdk_keepalive_shm_name"
SendNotification false

Coremask "0x0f" MemoryChannels "4" ProcessType "secondary" FilePrefix "rte" SharedMemObj "dpdk_collectd_stats_0" EnabledPortMask 0xffff PortName "eth1"

yes VFs we are trying with VF's methods.

Primary app developed in DPDK.
and we are trying to monitor using the same DPDK version dpdk-stable-16.11.1.

[rithikach]

build/app/dpdk-procinfo -- -p 0xf --stats
DPDK provided procinfo works fine... on VFs

[rithikach]

is this issue with xstats function?

[maryamtahhan]

the issue isn't with the xstats function... it's used by dpdk-procinfo also...

I haven't seen the error before but I'm also struggling to debug because the thread is jumping between 2 plugins... we do internal testing of all port types that can be used with DPDK including the VFs and haven't seen the issue before.
do you have the same issue with both plugins - or just 1 ? do you have both plugins enabled?

[maryamtahhan]

Hi @rithikach
it looks like there's an issue with DPDK 16.11 --> in my environment it passes EAL initialization, but fails later on. so I'm filling a bug internally to address this.
In the mean time - could you please do me one last favour and try the same steps for DPDK 16.07 for the dpdkstat plugin only?

FYI: There's a bunch of dpdkevents bugfixes coming very soon.

[maryamtahhan]

@rithikach BTW - if you really need to monitor DPDK metrics with a newer version of DPDK you can use dpdk_procinfo which was updated in DPDK 17.05 to be able to publish stats to collectd through the exec plugin - if you need more info on this let me know.

[rithikach]

Problem with both plugins. seeing same error messages with previous DPDK 16.07 .

Error message :
Apr 7 07:13:35 abcd collectd-dpdk[17283]: EAL: WARNING: Address Space Layout Randomization (ASLR) is enabled in the kernel.
Apr 7 07:13:35 abcd collectd-dpdk[17283]: EAL: This may cause issues with mapping memory into secondary processes
Apr 7 07:13:35 abcd collectd-dpdk[17283]: EAL: Cannot initialize tailq: RTE_DISTRIBUTOR
Apr 7 07:13:35 abcd collectd-dpdk[17283]: PANIC in rte_eal_init():
Apr 7 07:13:35 abcd collectd-dpdk[17283]: Cannot init tail queues for objects

if you have documnet and good refernce in understanding dpdkstat, dpdkevent. can you share me?

[maryamtahhan]

we haven't been able to reproduce this behaviour
Can you please disable ASLR and try again:
echo 0 > /proc/sys/kernel/randomize_va_space
Thanks for your patience.
Maryam

[maryamtahhan]

Does your version of DPDK use this patch by any chance:
http://dpdk.org/ml/archives/dev/2016-September/047332.html

[maryamtahhan]

and have you tried using the new DPDK param where you explicitly pass the driver in:
#2505