Skip to content

Commit

Permalink
netdev-dpdk: Limit rate of DPDK logs.
Browse files Browse the repository at this point in the history
DPDK could produce huge amount of logs. For example, in case of
exhausting of a mempool in vhost-user port, following message will be
printed on each call to 'rte_vhost_dequeue_burst()':

    |ERR|VHOST_DATA: Failed to allocate memory for mbuf.

These messages are increasing ovs-vswitchd.log size extremely fast
making it unreadable and non-parsable by a common linux utils like
grep, less etc. Moreover continuously growing log could exhaust the
HDD space in a few hours breaking normal operation of the whole system.

To avoid such issues, DPDK log rate limited to 600 messages per minute.
This value is high, because we still want to see many big logs like
vhost-user configuration sequence. The debug messages are treated
separately to avoid looss of errors/warnings in case of intensive debug
enabled in DPDK.

Signed-off-by: Ilya Maximets <i.maximets@samsung.com>
Acked-by: Aaron Conole <aconole@redhat.com>
Signed-off-by: Ian Stokes <ian.stokes@intel.com>
  • Loading branch information
igsilya authored and istokes committed Mar 23, 2018
1 parent 86ffff0 commit 3c70cfd
Showing 1 changed file with 6 additions and 4 deletions.
10 changes: 6 additions & 4 deletions lib/dpdk.c
Expand Up @@ -270,20 +270,22 @@ static ssize_t
dpdk_log_write(void *c OVS_UNUSED, const char *buf, size_t size)
{
char *str = xmemdup0(buf, size);
static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(600, 600);
static struct vlog_rate_limit dbg_rl = VLOG_RATE_LIMIT_INIT(600, 600);

switch (rte_log_cur_msg_loglevel()) {
case RTE_LOG_DEBUG:
VLOG_DBG("%s", str);
VLOG_DBG_RL(&dbg_rl, "%s", str);
break;
case RTE_LOG_INFO:
case RTE_LOG_NOTICE:
VLOG_INFO("%s", str);
VLOG_INFO_RL(&rl, "%s", str);
break;
case RTE_LOG_WARNING:
VLOG_WARN("%s", str);
VLOG_WARN_RL(&rl, "%s", str);
break;
case RTE_LOG_ERR:
VLOG_ERR("%s", str);
VLOG_ERR_RL(&rl, "%s", str);
break;
case RTE_LOG_CRIT:
case RTE_LOG_ALERT:
Expand Down

0 comments on commit 3c70cfd

Please sign in to comment.