Crazy memory usage #2527

ghost · 2018-06-21T20:49:40Z

Could anyone here explain why the memory usage of the version 5 release is literally crazy? I'm exporting one full table to a virtual machine with 8GB of memory and the entire memory is filled by that from zebra, including a fair amount of swap as well.

Memory usage for a full table used to be below 4GB in older releases.

What's the point + reason for this increase? Makes any usage of frr useless and turns anyone who doesn't want to literally and unnecessarily waste memory use bird.

donaldsharp · 2018-06-21T20:50:57Z

Can we get a show memory and a show thread cpu, show ip route and show run?

ghost · 2018-06-21T20:59:17Z

Kernel: 4.9.0-6-amd64
Distribution: Debian Stretch
Memory: 8GB
CPU: 4 vCores

free:

              total        used        free      shared  buff/cache   available
Mem:        8179256     8020884      120052         744       38320        1868
Swap:       8385532     7010152     1375380

show thread cpu:

Thread statistics for zebra:

Showing statistics for pthread main
-----------------------------------
                      CPU (user+system): Real (wall-clock):
Active   Runtime(ms)   Invoked Avg uSec Max uSecs Avg uSec Max uSecs  Type  Thread
    0          0.000         4        0         0       11        28   T   if_zebra_speed_update
    1        120.000        15     8000     48000     7994     47408 R     zebra_accept
   15          0.000        22        0         0       14        29 R     zserv_read
    1          0.000         2        0         0       18        21 R     kernel_read
    0        268.000        28     9571     12000     9547     12217   T   work_queue_run
    0         20.000       199      100     20000      104     19396  W    zserv_write
    3          0.000         6        0         0       41        44   T   zebra_ptm_connect
    0        368.000        22    16727    144000    16776    142100    E  &zserv_process_messages
    1          0.000         3        0         0       27        32 R     vtysh_accept
    1          0.000        19        0         0       39       104 R     vtysh_read


Total thread statistics
-------------------------
                      CPU (user+system): Real (wall-clock):
Active   Runtime(ms)   Invoked Avg uSec Max uSecs Avg uSec Max uSecs  Type  Thread
   22        776.000       320     2425    144000     2433    142100 RWTEX TOTAL

Thread statistics for ripd:

Showing statistics for pthread main
-----------------------------------
                      CPU (user+system): Real (wall-clock):
Active   Runtime(ms)   Invoked Avg uSec Max uSecs Avg uSec Max uSecs  Type  Thread
    1          0.000        14        0         0       24        52 R     zclient_read
    1          0.000         3        0         0       30        44 R     vtysh_accept
    1          0.000        13        0         0       31        52 R     vtysh_read
    0          0.000         1        0         0      211       211    E  zclient_connect


Total thread statistics
-------------------------
                      CPU (user+system): Real (wall-clock):
Active   Runtime(ms)   Invoked Avg uSec Max uSecs Avg uSec Max uSecs  Type  Thread
    3          0.000        31        0         0       33       211 RWTEX TOTAL

Thread statistics for ripngd:

Showing statistics for pthread main
-----------------------------------
                      CPU (user+system): Real (wall-clock):
Active   Runtime(ms)   Invoked Avg uSec Max uSecs Avg uSec Max uSecs  Type  Thread
    1          0.000         3        0         0       26        34 R     vtysh_accept
    1          0.000        13        0         0       35        57 R     vtysh_read
    1          0.000        14        0         0       23        47 R     zclient_read
    0          0.000         1        0         0      195       195    E  zclient_connect


Total thread statistics
-------------------------
                      CPU (user+system): Real (wall-clock):
Active   Runtime(ms)   Invoked Avg uSec Max uSecs Avg uSec Max uSecs  Type  Thread
    3          0.000        31        0         0       34       195 RWTEX TOTAL

Thread statistics for ospfd:

Showing statistics for pthread main
-----------------------------------
                      CPU (user+system): Real (wall-clock):
Active   Runtime(ms)   Invoked Avg uSec Max uSecs Avg uSec Max uSecs  Type  Thread
    1          0.000         1        0         0       21        21   T   ospf_lsa_refresh_walker
    0          0.000         2        0         0       22        29  W    ospf_write
    1          0.000         3        0         0       28        36 R     vtysh_accept
    0          0.000         1        0         0      181       181    E  zclient_connect
    0          0.000         2        0         0       11        18   T   ospf_spf_calculate_timer
    0          0.000         1        0         0        1         1    E  ospf_default_originate_timer
    0          0.000         2        0         0       12        20    E  ospf_ism_event
    1          0.000        17        0         0       27        67 R     zclient_read
    1          4.000         2     2000      4000       60        84 R     ospf_read
    1          0.000        16        0         0     1011     15341 R     vtysh_read
    0          0.000         1        0         0        2         2   T   ospf_sr_update_schedule
    0          0.000         4        0         0      470       944    EX ospf_nsm_event
    1          0.000         2        0         0       18        30   T   (ospf_hello_timer)
    0          0.000         1        0         0       35        35   T   ospf_ase_calculate_timer
    0          0.000        26        0         0        0         1    E  ospf_external_lsa_originate_timer


Total thread statistics
-------------------------
                      CPU (user+system): Real (wall-clock):
Active   Runtime(ms)   Invoked Avg uSec Max uSecs Avg uSec Max uSecs  Type  Thread
    6          4.000        81       49      4000      236     15341 RWTEX TOTAL

Thread statistics for ospf6d:

Showing statistics for pthread main
-----------------------------------
                      CPU (user+system): Real (wall-clock):
Active   Runtime(ms)   Invoked Avg uSec Max uSecs Avg uSec Max uSecs  Type  Thread
    1          0.000        13        0         0       30        52 R     vtysh_read
    1          0.000        14        0         0       21        49 R     zclient_read
    1          0.000         3        0         0       23        36 R     vtysh_accept
    0          0.000         1        0         0      246       246    E  zclient_connect


Total thread statistics
-------------------------
                      CPU (user+system): Real (wall-clock):
Active   Runtime(ms)   Invoked Avg uSec Max uSecs Avg uSec Max uSecs  Type  Thread
    3          0.000        31        0         0       32       246 RWTEX TOTAL

Thread statistics for ldpd:

Showing statistics for pthread main
-----------------------------------
                      CPU (user+system): Real (wall-clock):
Active   Runtime(ms)   Invoked Avg uSec Max uSecs Avg uSec Max uSecs  Type  Thread
    1          0.000         3        0         0       22        30 R     vtysh_accept
    0          0.000        27        0         0       41       902  W    iev->handler_write
    2          0.000         1        0         0       14        14 R     iev_ldpe->handler_read
    0          0.000         1        0         0      190       190    E  zclient_connect
    1          0.000        10        0         0       37        60 R     vtysh_read
    2          0.000         3        0         0       12        13 R     iev_lde->handler_read
    1          0.000        21        0         0       11        31 R     zclient_read


Total thread statistics
-------------------------
                      CPU (user+system): Real (wall-clock):
Active   Runtime(ms)   Invoked Avg uSec Max uSecs Avg uSec Max uSecs  Type  Thread
    7          0.000        66        0         0       31       902 RWTEX TOTAL

Thread statistics for bgpd:

Showing statistics for pthread main
-----------------------------------
                      CPU (user+system): Real (wall-clock):
Active   Runtime(ms)   Invoked Avg uSec Max uSecs Avg uSec Max uSecs  Type  Thread
    1          0.000         3        0         0       35        65 R     vtysh_accept
    1          0.000        13        0         0      829     10165 R     vtysh_read
    0          0.000         1        0         0      181       181   T   (bgp_start_timer)
    2          0.000        18        0         0        7        19 R     zclient_read
    0          0.000         2        0         0       34        49    E  zclient_connect
    0          0.000         2        0         0       83        94    E  bgp_event


Showing statistics for pthread BGP I/O thread
---------------------------------------------
                      CPU (user+system): Real (wall-clock):
Active   Runtime(ms)   Invoked Avg uSec Max uSecs Avg uSec Max uSecs  Type  Thread


Showing statistics for pthread BGP Keepalives thread
----------------------------------------------------
                      CPU (user+system): Real (wall-clock):
Active   Runtime(ms)   Invoked Avg uSec Max uSecs Avg uSec Max uSecs  Type  Thread
No data to display yet.


Total thread statistics
-------------------------
                      CPU (user+system): Real (wall-clock):
Active   Runtime(ms)   Invoked Avg uSec Max uSecs Avg uSec Max uSecs  Type  Thread
    4          0.000        39        0         0      293     10165 RWTEX TOTAL

Thread statistics for isisd:

Showing statistics for pthread main
-----------------------------------
                      CPU (user+system): Real (wall-clock):
Active   Runtime(ms)   Invoked Avg uSec Max uSecs Avg uSec Max uSecs  Type  Thread
    1          0.000        13        0         0       33        58 R     vtysh_read
    0          0.000         1        0         0      178       178    E  zclient_connect
    1          0.000         3        0         0       25        35 R     vtysh_accept
    1          0.000        14        0         0       21        37 R     zclient_read


Total thread statistics
-------------------------
                      CPU (user+system): Real (wall-clock):
Active   Runtime(ms)   Invoked Avg uSec Max uSecs Avg uSec Max uSecs  Type  Thread
    3          0.000        31        0         0       31       178 RWTEX TOTAL

Thread statistics for pimd:

Showing statistics for pthread main
-----------------------------------
                      CPU (user+system): Real (wall-clock):
Active   Runtime(ms)   Invoked Avg uSec Max uSecs Avg uSec Max uSecs  Type  Thread
    1          0.000        13        0         0       34        57 R     vtysh_read
    1          0.000         3        0         0       28        33 R     vtysh_accept
    1          0.000        22        0         0        9        32 R     zclient_read
    1          0.000         1        0         0       20        20   T   wheel_timer_thread
    0          0.000         1        0         0       12        12    E  zclient_lookup_connect
    0          0.000         1        0         0      251       251    E  zclient_connect


Total thread statistics
-------------------------
                      CPU (user+system): Real (wall-clock):
Active   Runtime(ms)   Invoked Avg uSec Max uSecs Avg uSec Max uSecs  Type  Thread
    4          0.000        41        0         0       25       251 RWTEX TOTAL

Thread statistics for nhrpd:

Showing statistics for pthread main
-----------------------------------
                      CPU (user+system): Real (wall-clock):
Active   Runtime(ms)   Invoked Avg uSec Max uSecs Avg uSec Max uSecs  Type  Thread
    1          0.000        13        0         0       32        58 R     vtysh_read
    1          0.000         7        0         0       47        53   T   vici_reconnect
    1          0.000         1        0         0       18        18 R     netlink_route_recv
    1          0.000         3        0         0       25        38 R     vtysh_accept
    0          0.000         1        0         0        3         3   T   evmgr_reconnect
    0          0.000         1        0         0      205       205    E  zclient_connect
    1          0.000        21        0         0        9        32 R     zclient_read


Total thread statistics
-------------------------
                      CPU (user+system): Real (wall-clock):
Active   Runtime(ms)   Invoked Avg uSec Max uSecs Avg uSec Max uSecs  Type  Thread
    5          0.000        47        0         0       26       205 RWTEX TOTAL

Thread statistics for eigrpd:

Showing statistics for pthread main
-----------------------------------
                      CPU (user+system): Real (wall-clock):
Active   Runtime(ms)   Invoked Avg uSec Max uSecs Avg uSec Max uSecs  Type  Thread
    1          0.000        14        0         0        6        19 R     zclient_read
    1          0.000         3        0         0       28        36 R     vtysh_accept
    0          0.000         1        0         0      199       199    E  zclient_connect
    1          0.000        13        0         0       35        58 R     vtysh_read


Total thread statistics
-------------------------
                      CPU (user+system): Real (wall-clock):
Active   Runtime(ms)   Invoked Avg uSec Max uSecs Avg uSec Max uSecs  Type  Thread
    3          0.000        31        0         0       27       199 RWTEX TOTAL

Thread statistics for babeld:

Showing statistics for pthread main
-----------------------------------
                      CPU (user+system): Real (wall-clock):
Active   Runtime(ms)   Invoked Avg uSec Max uSecs Avg uSec Max uSecs  Type  Thread
    1          0.000        14        0         0       15        34 R     zclient_read
    1          0.000         3        0         0       26        37 R     vtysh_accept
    1          0.000        13        0         0       34        52 R     vtysh_read
    0          0.000         1        0         0      238       238    E  zclient_connect


Total thread statistics
-------------------------
                      CPU (user+system): Real (wall-clock):
Active   Runtime(ms)   Invoked Avg uSec Max uSecs Avg uSec Max uSecs  Type  Thread
    3          0.000        31        0         0       32       238 RWTEX TOTAL

Thread statistics for watchfrr:

Showing statistics for pthread main
-----------------------------------
                      CPU (user+system): Real (wall-clock):
Active   Runtime(ms)   Invoked Avg uSec Max uSecs Avg uSec Max uSecs  Type  Thread
   13          0.000        26        0         0       10        20 R     handle_read
   13          4.000        26      153      4000       33        87   T   wakeup_send_echo
    0          0.000        13        0         0      507       575   T   wakeup_init
    1          0.000         2        0         0       22        25 R     vtysh_accept
    0          0.000         6        0         0       27        38 R     vtysh_read


Total thread statistics
-------------------------
                      CPU (user+system): Real (wall-clock):
Active   Runtime(ms)   Invoked Avg uSec Max uSecs Avg uSec Max uSecs  Type  Thread
   27          4.000        73       54      4000      108       575 RWTEX TOTAL

Thread statistics for pbrd:

Showing statistics for pthread main
-----------------------------------
                      CPU (user+system): Real (wall-clock):
Active   Runtime(ms)   Invoked Avg uSec Max uSecs Avg uSec Max uSecs  Type  Thread
    1          0.000        13        0         0       33        49 R     vtysh_read
    1          0.000        14        0         0        8        51 R     zclient_read
    1          0.000         3        0         0       23        35 R     vtysh_accept
    0          0.000         1        0         0      194       194    E  zclient_connect


Total thread statistics
-------------------------
                      CPU (user+system): Real (wall-clock):
Active   Runtime(ms)   Invoked Avg uSec Max uSecs Avg uSec Max uSecs  Type  Thread
    3          0.000        31        0         0       26       194 RWTEX TOTAL

show memory:

Memory statistics for zebra:
System allocator statistics:
  Total heap allocated:  > 2GB
  Holding block headers: 16 MiB
  Used small blocks:     0 bytes
  Used ordinary blocks:  > 2GB
  Free small blocks:     1952 bytes
  Free ordinary blocks:  368 KiB
  Ordinary blocks:       7739
  Small blocks:          58
  Holding blocks:        1
(see system documentation for 'mallinfo' for meaning)
--- qmem libfrr ---
Buffer                        :         18      24
Buffer data                   :          1    4120
Host config                   :          2  (variably sized)
String vector                 :          3  (variably sized)
Command Tokens                :       3370      72
Command Token Text            :       2508  (variably sized)
Command Token Help            :       2508  (variably sized)
Command Argument              :          2  (variably sized)
Command Argument Name         :        620  (variably sized)
Graph                         :         24       8
Graph Node                    :       3963      32
Hash                          :        122  (variably sized)
Hash Bucket                   :    1159684      32
Hash Index                    :         61  (variably sized)
Hook entry                    :          9      48
Interface                     :          4     248
Connected                     :          7      40
Link List                     :         41      40
Link Node                     :        117      24
Logging                       :          1      72
Temporary memory              :         21  (variably sized)
Nexthop                       :    1264944     112
NetNS Context                 :          2  (variably sized)
NetNS Name                    :          1      18
Priority queue                :          1      32
Priority queue data           :          1     256
Prefix                        :          8      40
Privilege information         :          2  (variably sized)
Stream                        :    1689253      40
Stream data                   :    1689253  (variably sized)
Stream FIFO                   :         30      24
Route table                   :         19      48
Route node                    :    1159073  (variably sized)
Thread                        :         38  (variably sized)
Thread master                 :          3  (variably sized)
Thread stats                  :         11      64
Vector                        :       7981      16
Vector index                  :       7981  (variably sized)
VRF                           :          1     184
VRF bit-map                   :       1269  (variably sized)
VTY                           :          6  (variably sized)
Work queue                    :          2  (variably sized)
Work queue item               :          1      24
Work queue name string        :          1      22
--- qmem Label Manager ---
Label Manager Chunk           :          2      16
--- qmem zebra ---
ZEBRA VRF                     :          1     904
Route Entry                   :     632492      80
Static route                  :          1     192
RIB destination               :     632480      48
RIB table info                :          4      16
Nexthop tracking object       :          4     192
Zebra Name Space              :          1     312
--- qmem Table Manager ---

Memory statistics for ripd:
System allocator statistics:
  Total heap allocated:  3148 KiB
  Holding block headers: 0 bytes
  Used small blocks:     0 bytes
  Used ordinary blocks:  3038 KiB
  Free small blocks:     1488 bytes
  Free ordinary blocks:  110 KiB
  Ordinary blocks:       1
  Small blocks:          43
  Holding blocks:        0
(see system documentation for 'mallinfo' for meaning)
--- qmem libfrr ---
Buffer                        :          3      24
Buffer data                   :          1    4120
Host config                   :          2  (variably sized)
String vector                 :          3  (variably sized)
Command Tokens                :       1972      72
Command Token Text            :       1568  (variably sized)
Command Token Help            :       1568  (variably sized)
Command Argument              :          2  (variably sized)
Command Argument Name         :        318  (variably sized)
Graph                         :         18       8
Graph Node                    :       2383      32
Hash                          :         63  (variably sized)
Hash Bucket                   :        423      32
Hash Index                    :         32  (variably sized)
Hook entry                    :          2      48
Interface                     :          4     248
Connected                     :          7      40
Link List                     :         19      40
Link Node                     :         53      24
Logging                       :          1      72
Temporary memory              :          2  (variably sized)
Priority queue                :          1      32
Priority queue data           :          1     256
Prefix                        :          8      40
Privilege information         :          2  (variably sized)
Stream                        :          2      40
Stream data                   :          2   16384
Route table                   :          2      48
Thread                        :          7  (variably sized)
Thread master                 :          3  (variably sized)
Thread stats                  :          5      64
Vector                        :       4811      16
Vector index                  :       4811  (variably sized)
VRF                           :          1     184
VRF bit-map                   :         82  (variably sized)
VTY                           :          6  (variably sized)
Zclient                       :          1    2872
Redistribution instance IDs   :          3       2
--- qmem ripd ---
RIP interface                 :          4     136

Memory statistics for ripngd:
System allocator statistics:
  Total heap allocated:  3020 KiB
  Holding block headers: 0 bytes
  Used small blocks:     0 bytes
  Used ordinary blocks:  2914 KiB
  Free small blocks:     1472 bytes
  Free ordinary blocks:  106 KiB
  Ordinary blocks:       1
  Small blocks:          43
  Holding blocks:        0
(see system documentation for 'mallinfo' for meaning)
--- qmem libfrr ---
Buffer                        :          3      24
Buffer data                   :          1    4120
Host config                   :          2  (variably sized)
String vector                 :          3  (variably sized)
Command Tokens                :       1662      72
Command Token Text            :       1302  (variably sized)
Command Token Help            :       1302  (variably sized)
Command Argument              :          2  (variably sized)
Command Argument Name         :        257  (variably sized)
Graph                         :         16       8
Graph Node                    :       1998      32
Hash                          :         57  (variably sized)
Hash Bucket                   :        348      32
Hash Index                    :         29  (variably sized)
Hook entry                    :          2      48
Interface                     :          8  (variably sized)
Connected                     :          7      40
Link List                     :         18      40
Link Node                     :         50      24
Logging                       :          1      72
Temporary memory              :          2  (variably sized)
Priority queue                :          1      32
Priority queue data           :          1     256
Prefix                        :          8      40
Privilege information         :          2  (variably sized)
Stream                        :          2      40
Stream data                   :          2   16384
Route table                   :          1      48
Thread                        :          7  (variably sized)
Thread master                 :          3  (variably sized)
Thread stats                  :          5      64
Vector                        :       4037      16
Vector index                  :       4037  (variably sized)
VRF                           :          1     184
VRF bit-map                   :         82  (variably sized)
VTY                           :          6  (variably sized)
Zclient                       :          1    2872
Redistribution instance IDs   :          3       2
--- qmem ripngd ---

Memory statistics for ospfd:
System allocator statistics:
  Total heap allocated:  4568 KiB
  Holding block headers: 0 bytes
  Used small blocks:     0 bytes
  Used ordinary blocks:  4463 KiB
  Free small blocks:     1520 bytes
  Free ordinary blocks:  105 KiB
  Ordinary blocks:       2
  Small blocks:          43
  Holding blocks:        0
(see system documentation for 'mallinfo' for meaning)
--- qmem libfrr ---
Buffer                        :          3      24
Buffer data                   :          1    4120
Host config                   :          2  (variably sized)
String vector                 :          3  (variably sized)
Command Tokens                :       3393      72
Command Token Text            :       2429  (variably sized)
Command Token Help            :       2429  (variably sized)
Command Argument              :          2  (variably sized)
Command Argument Name         :        462  (variably sized)
Graph                         :         16       8
Graph Node                    :       3971      32
Hash                          :        348  (variably sized)
Hash Bucket                   :        679      32
Hash Index                    :        174  (variably sized)
Hook entry                    :          2      48
Interface                     :          4     248
Connected                     :          7      40
Link List                     :         68      40
Link Node                     :        237      24
Logging                       :          1      72
Temporary memory              :          2  (variably sized)
Priority queue                :          1      32
Priority queue data           :          1     256
Prefix                        :          8      40
Privilege information         :          2  (variably sized)
Stream                        :          4      40
Stream data                   :          4  (variably sized)
Route table                   :        147      48
Route node                    :         66      96
Thread                        :         37  (variably sized)
Thread master                 :          3  (variably sized)
Thread stats                  :         25      64
Vector                        :       7981      16
Vector index                  :       7981  (variably sized)
VRF                           :          1     184
VRF bit-map                   :         83  (variably sized)
VTY                           :          6  (variably sized)
Zclient                       :          1    2872
Redistribution instance IDs   :          3       2
--- qmem ospfd ---
OSPF top                      :          1    3624
OSPF area                     :          1     248
OSPF network                  :          1       8
OSPF interface                :          2     288
OSPF neighbor                 :          3    1472
OSPF route                    :         11      56
OSPF LSA                      :         13     112
OSPF LSA data                 :         13  (variably sized)
OSPF LSDB                     :          2     408
OSPF packet                   :          1      24
OSPF FIFO queue               :          1      24
OSPF path                     :         11      16
OSPF ext. info                :          3      48
OSPF if info                  :          4      32
OSPF if params                :          4     104
OSPF MPLS parameters          :          4     248
OSPF Extended parameters      :          4     136
OSPF opaque function table    :          3     112
OSPF External route table     :          1      16
OSPF Redistriute              :          1      32

Memory statistics for ospf6d:
System allocator statistics:
  Total heap allocated:  3372 KiB
  Holding block headers: 0 bytes
  Used small blocks:     0 bytes
  Used ordinary blocks:  3264 KiB
  Free small blocks:     1472 bytes
  Free ordinary blocks:  108 KiB
  Ordinary blocks:       1
  Small blocks:          43
  Holding blocks:        0
(see system documentation for 'mallinfo' for meaning)
--- qmem libfrr ---
Buffer                        :          3      24
Buffer data                   :          1    4120
Host config                   :          2  (variably sized)
String vector                 :          3  (variably sized)
Command Tokens                :       2670      72
Command Token Text            :       1992  (variably sized)
Command Token Help            :       1992  (variably sized)
Command Argument              :          2  (variably sized)
Command Argument Name         :        354  (variably sized)
Graph                         :         16       8
Graph Node                    :       3153      32
Hash                          :         52  (variably sized)
Hash Bucket                   :        496      32
Hash Index                    :         26  (variably sized)
Interface                     :          4     248
Connected                     :          7      40
Link List                     :         16      40
Link Node                     :         47      24
Logging                       :          1      72
Temporary memory              :          2  (variably sized)
Priority queue                :          1      32
Priority queue data           :          1     256
Prefix                        :          8      40
Privilege information         :          2  (variably sized)
Stream                        :          2      40
Stream data                   :          2   16384
Thread                        :          8  (variably sized)
Thread master                 :          3  (variably sized)
Thread stats                  :          6      64
Vector                        :       6346      16
Vector index                  :       6346  (variably sized)
VRF                           :          1     184
VRF bit-map                   :         82  (variably sized)
VTY                           :          6  (variably sized)
Zclient                       :          1    2872
Redistribution instance IDs   :          3       2
--- qmem ospf6d ---

Memory statistics for ldpd:
System allocator statistics:
  Total heap allocated:  3320 KiB
  Holding block headers: 0 bytes
  Used small blocks:     0 bytes
  Used ordinary blocks:  3192 KiB
  Free small blocks:     1504 bytes
  Free ordinary blocks:  128 KiB
  Ordinary blocks:       2
  Small blocks:          44
  Holding blocks:        0
(see system documentation for 'mallinfo' for meaning)
--- qmem libfrr ---
Buffer                        :          3      24
Buffer data                   :          1    4120
Host config                   :          2  (variably sized)
String vector                 :          3  (variably sized)
Command Tokens                :       1732      72
Command Token Text            :       1336  (variably sized)
Command Token Help            :       1336  (variably sized)
Command Argument              :          2  (variably sized)
Command Argument Name         :        350  (variably sized)
Graph                         :         18       8
Graph Node                    :       2064      32
Hash                          :         40  (variably sized)
Hash Bucket                   :        348      32
Hash Index                    :         20  (variably sized)
Interface                     :          4     248
Connected                     :          7      40
Link List                     :         16      40
Link Node                     :         35      24
Logging                       :          1      72
Temporary memory              :          2  (variably sized)
Priority queue                :          1      32
Priority queue data           :          1     256
Prefix                        :          8      40
Privilege information         :          2  (variably sized)
Stream                        :          2      40
Stream data                   :          2   16384
Thread                        :         11  (variably sized)
Thread master                 :          3  (variably sized)
Thread stats                  :          8      64
Vector                        :       4169      16
Vector index                  :       4169  (variably sized)
VRF                           :          1     184
VRF bit-map                   :         82  (variably sized)
VTY                           :          6  (variably sized)
Zclient                       :          1    2872
Redistribution instance IDs   :          3       2

Memory statistics for bgpd:
System allocator statistics:
  Total heap allocated:  542 MiB
  Holding block headers: 16 MiB
  Used small blocks:     0 bytes
  Used ordinary blocks:  536 MiB
  Free small blocks:     92 KiB
  Free ordinary blocks:  5867 KiB
  Ordinary blocks:       169
  Small blocks:          1170
  Holding blocks:        1
(see system documentation for 'mallinfo' for meaning)
--- qmem libfrr ---
Buffer                        :          4      24
Buffer data                   :        557    4120
Host config                   :          2  (variably sized)
String vector                 :          3  (variably sized)
Command Tokens                :       9102      72
Command Token Text            :       6779  (variably sized)
Command Token Help            :       6779  (variably sized)
Command Argument              :          2  (variably sized)
Command Argument Name         :       1426  (variably sized)
FRR POSIX Thread              :          4  (variably sized)
POSIX synchronization primitives:          4  (variably sized)
Graph                         :         35       8
Graph Node                    :      10864      32
Hash                          :        370  (variably sized)
Hash Bucket                   :    1475225      32
Hash Index                    :        186  (variably sized)
Interface                     :          4     248
Connected                     :          7      40
Link List                     :         38      40
Link Node                     :        220      24
Logging                       :          1      72
Temporary memory              :          5  (variably sized)
Nexthop                       :          3     112
Priority queue                :          3      32
Priority queue data           :          3     256
Prefix                        :          8      40
Privilege information         :          2  (variably sized)
Ring buffer                   :          4  (variably sized)
Skip List                     :          2      56
Skip Node                     :          4     160
Socket union                  :          3      28
Stream                        :         11      40
Stream data                   :         11  (variably sized)
Stream FIFO                   :          4      24
Route table                   :        106      48
Thread                        :         23  (variably sized)
Thread master                 :         11  (variably sized)
Thread stats                  :         23      64
Vector                        :      21805      16
Vector index                  :      21805  (variably sized)
VRF                           :          1     184
VRF bit-map                   :        161  (variably sized)
VTY                           :          6  (variably sized)
Work queue                    :          3     144
Work queue name string        :          3  (variably sized)
Zclient                       :          2    2872
Redistribution instance IDs   :          6       2
--- qmem rfapi ---
NVE Configuration             :          1    2560
RFAPI Generic                 :          1     296
RFAPI Import Table            :          1     208
--- qmem bgpd ---
BGP Label FIFO                :          1      48
BGP PBR Context               :          1      16
BGP instance                  :          2  (variably sized)
BGP listen socket details     :          2      48
BGP peer                      :          3   20688
BGP peer hostname             :          4  (variably sized)
BGP peer af                   :          1      80
BGP update group              :          1     104
BGP update subgroup           :          1     240
BGP packet                    :          1      56
BGP attribute                 :     107420     224
BGP aspath                    :      98124      40
BGP aspath seg                :      98299      24
BGP aspath segment data       :      98299  (variably sized)
BGP aspath str                :      98124  (variably sized)
BGP table                     :         93      40
BGP node                      :    1267816     152
BGP route                     :     693148     112
BGP ancillary route info      :     693148     200
BGP connected                 :          5       4
BGP synchronise               :         43      72
community                     :         24      40
community val                 :         24  (variably sized)
community str                 :         24  (variably sized)
extcommunity                  :          1      32
extcommunity val              :          1  (variably sized)
extcommunity str              :          1  (variably sized)
community-list handler        :          1      96
BGP transit attr              :          4      24
BGP transit val               :          4  (variably sized)
BGP nexthop                   :          3      72
BGP own address               :          3       8
Large Community               :         32      40
Large Community display string:         32  (variably sized)
Large Community value         :         32  (variably sized)

Memory statistics for isisd:
System allocator statistics:
  Total heap allocated:  3216 KiB
  Holding block headers: 0 bytes
  Used small blocks:     0 bytes
  Used ordinary blocks:  3111 KiB
  Free small blocks:     1504 bytes
  Free ordinary blocks:  105 KiB
  Ordinary blocks:       1
  Small blocks:          44
  Holding blocks:        0
(see system documentation for 'mallinfo' for meaning)
--- qmem libfrr ---
Buffer                        :          3      24
Buffer data                   :          1    4120
Host config                   :          2  (variably sized)
String vector                 :          3  (variably sized)
Command Tokens                :       2171      72
Command Token Text            :       1707  (variably sized)
Command Token Help            :       1707  (variably sized)
Command Argument              :          2  (variably sized)
Command Argument Name         :        283  (variably sized)
Graph                         :         16       8
Graph Node                    :       2642      32
Hash                          :         52  (variably sized)
Hash Bucket                   :        488      32
Hash Index                    :         26  (variably sized)
Hook entry                    :          2      48
Interface                     :          4     248
Connected                     :          7      40
Link List                     :         34      40
Link Node                     :         57      24
Logging                       :          1      72
Temporary memory              :          2  (variably sized)
Priority queue                :          1      32
Priority queue data           :          1     256
Prefix                        :         15      40
Privilege information         :          2  (variably sized)
Stream                        :          2      40
Stream data                   :          2   16384
Thread                        :          8  (variably sized)
Thread master                 :          3  (variably sized)
Thread stats                  :          6      64
Vector                        :       5323      16
Vector index                  :       5323  (variably sized)
VRF                           :          1     184
VRF bit-map                   :         82  (variably sized)
VTY                           :          6  (variably sized)
Zclient                       :          1    2872
Redistribution instance IDs   :          3       2
--- qmem isisd ---
ISIS                          :          1     160
ISIS circuit                  :          3     704
ISIS MPLS_TE parameters       :          3     144

Memory statistics for pimd:
System allocator statistics:
  Total heap allocated:  16 MiB
  Holding block headers: 0 bytes
  Used small blocks:     0 bytes
  Used ordinary blocks:  16 MiB
  Free small blocks:     1504 bytes
  Free ordinary blocks:  110 KiB
  Ordinary blocks:       1
  Small blocks:          44
  Holding blocks:        0
(see system documentation for 'mallinfo' for meaning)
--- qmem libfrr ---
Buffer                        :          4      24
Buffer data                   :          1    4120
Host config                   :          2  (variably sized)
String vector                 :          3  (variably sized)
Command Tokens                :       2654      72
Command Token Text            :       2046  (variably sized)
Command Token Help            :       2046  (variably sized)
Command Argument              :          2  (variably sized)
Command Argument Name         :        364  (variably sized)
Graph                         :         16       8
Graph Node                    :       3243      32
Hash                          :         64  (variably sized)
Hash Bucket                   :        605      32
Hash Index                    :         32  (variably sized)
Interface                     :          4     248
Connected                     :          7      40
Link List                     :        122      40
Link Node                     :         54      24
Logging                       :          1      72
Temporary memory              :          2  (variably sized)
Priority queue                :          1      32
Priority queue data           :          1     256
Prefix                        :          8      40
Privilege information         :          2  (variably sized)
Stream                        :          4      40
Stream data                   :          4   16384
Route table                   :          1      48
Route node                    :          1      96
Thread                        :          9  (variably sized)
Thread master                 :          3  (variably sized)
Thread stats                  :          8      64
Vector                        :       6525      16
Vector index                  :       6525  (variably sized)
VRF                           :          1     184
VRF bit-map                   :        108  (variably sized)
VTY                           :          6  (variably sized)
Timer Wheel                   :          1      72
Timer Wheel Slot List         :          1     800
Zclient                       :          2    2872
Redistribution instance IDs   :          3       2
--- qmem pimd ---
PIM RP info                   :          1     176
PIM global state              :          1     544
PIM SSM configuration         :          1       8

Memory statistics for nhrpd:
System allocator statistics:
  Total heap allocated:  28 MiB
  Holding block headers: 2052 KiB
  Used small blocks:     0 bytes
  Used ordinary blocks:  27 MiB
  Free small blocks:     1536 bytes
  Free ordinary blocks:  463 KiB
  Ordinary blocks:       2
  Small blocks:          45
  Holding blocks:        1
(see system documentation for 'mallinfo' for meaning)
--- qmem libfrr ---
Buffer                        :          3      24
Buffer data                   :          1    4120
Host config                   :          2  (variably sized)
String vector                 :          3  (variably sized)
Command Tokens                :       1157      72
Command Token Text            :        927  (variably sized)
Command Token Help            :        927  (variably sized)
Command Argument              :          2  (variably sized)
Command Argument Name         :        134  (variably sized)
Graph                         :         12       8
Graph Node                    :       1387      32
Hash                          :         42  (variably sized)
Hash Bucket                   :     134360      32
Hash Index                    :         21  (variably sized)
Hook entry                    :          2      48
Interface                     :          4     248
Connected                     :          7      40
Link List                     :         16      40
Link Node                     :         38      24
Logging                       :          1      72
Temporary memory              :          2  (variably sized)
Priority queue                :          1      32
Priority queue data           :          1     256
Prefix                        :          8      40
Privilege information         :          2  (variably sized)
Stream                        :          2      40
Stream data                   :          2   16384
Route table                   :          4      48
Route node                    :     134114      96
Thread                        :         10  (variably sized)
Thread master                 :          3  (variably sized)
Thread stats                  :          9      64
Vector                        :       2805      16
Vector index                  :       2805  (variably sized)
VRF                           :          1     184
VRF bit-map                   :         82  (variably sized)
VTY                           :          6  (variably sized)
Zclient                       :          1    2872
Redistribution instance IDs   :          3       2
--- qmem NHRP ---
NHRP interface                :          4     424
NHRP routing entry            :      69917      48

Memory statistics for eigrpd:
System allocator statistics:
  Total heap allocated:  3152 KiB
  Holding block headers: 0 bytes
  Used small blocks:     0 bytes
  Used ordinary blocks:  3044 KiB
  Free small blocks:     1472 bytes
  Free ordinary blocks:  108 KiB
  Ordinary blocks:       1
  Small blocks:          43
  Holding blocks:        0
(see system documentation for 'mallinfo' for meaning)
--- qmem libfrr ---
Buffer                        :          3      24
Buffer data                   :          1    4120
Host config                   :          2  (variably sized)
String vector                 :          3  (variably sized)
Command Tokens                :       2029      72
Command Token Text            :       1605  (variably sized)
Command Token Help            :       1605  (variably sized)
Command Argument              :          2  (variably sized)
Command Argument Name         :        319  (variably sized)
Graph                         :         18       8
Graph Node                    :       2417      32
Hash                          :         57  (variably sized)
Hash Bucket                   :        400      32
Hash Index                    :         29  (variably sized)
Hook entry                    :          1      48
Interface                     :          4     248
Connected                     :          7      40
Link List                     :         18      40
Link Node                     :         50      24
Logging                       :          1      72
Temporary memory              :          2  (variably sized)
Priority queue                :          1      32
Priority queue data           :          1     256
Prefix                        :          8      40
Privilege information         :          2  (variably sized)
Stream                        :          2      40
Stream data                   :          2   16384
Thread                        :          7  (variably sized)
Thread master                 :          3  (variably sized)
Thread stats                  :          5      64
Vector                        :       4877      16
Vector index                  :       4877  (variably sized)
VRF                           :          1     184
VRF bit-map                   :         82  (variably sized)
VTY                           :          6  (variably sized)
Zclient                       :          1    2872
Redistribution instance IDs   :          3       2
--- qmem eigrpd ---

Memory statistics for babeld:
% [BABELD] Unknown command: do show memory


Memory statistics for watchfrr:
System allocator statistics:
  Total heap allocated:  264 KiB
  Holding block headers: 0 bytes
  Used small blocks:     0 bytes
  Used ordinary blocks:  204 KiB
  Free small blocks:     1328 bytes
  Free ordinary blocks:  60 KiB
  Ordinary blocks:       1
  Small blocks:          38
  Holding blocks:        0
(see system documentation for 'mallinfo' for meaning)
--- qmem libfrr ---
Buffer                        :          1      24
Buffer data                   :          1    4120
Host config                   :          1       9
String vector                 :          3  (variably sized)
Command Tokens                :        343      72
Command Token Text            :        285  (variably sized)
Command Token Help            :        285  (variably sized)
Command Argument              :          2  (variably sized)
Command Argument Name         :         36  (variably sized)
Graph                         :          6       8
Graph Node                    :        436      32
Hash                          :         16  (variably sized)
Hash Bucket                   :        102      32
Hash Index                    :          8  (variably sized)
Link List                     :          5      40
Link Node                     :         11      24
Logging                       :          1      72
Temporary memory              :          2  (variably sized)
Priority queue                :          1      32
Priority queue data           :          1     256
Thread                        :         31  (variably sized)
Thread master                 :          3  (variably sized)
Thread stats                  :          7      64
Vector                        :        889      16
Vector index                  :        889  (variably sized)
VTY                           :          3  (variably sized)

Memory statistics for pbrd:
System allocator statistics:
  Total heap allocated:  2820 KiB
  Holding block headers: 0 bytes
  Used small blocks:     0 bytes
  Used ordinary blocks:  2713 KiB
  Free small blocks:     1472 bytes
  Free ordinary blocks:  107 KiB
  Ordinary blocks:       1
  Small blocks:          43
  Holding blocks:        0
(see system documentation for 'mallinfo' for meaning)
--- qmem libfrr ---
Buffer                        :          3      24
Buffer data                   :          1    4120
Host config                   :          2  (variably sized)
String vector                 :          3  (variably sized)
Command Tokens                :       1146      72
Command Token Text            :        898  (variably sized)
Command Token Help            :        898  (variably sized)
Command Argument              :          2  (variably sized)
Command Argument Name         :        169  (variably sized)
Graph                         :         14       8
Graph Node                    :       1378      32
Hash                          :         36  (variably sized)
Hash Bucket                   :        244      32
Hash Index                    :         18  (variably sized)
Interface                     :          4     248
Connected                     :          7      40
Link List                     :         16      40
Link Node                     :         35      24
Logging                       :          1      72
Temporary memory              :          2  (variably sized)
Priority queue                :          1      32
Priority queue data           :          1     256
Prefix                        :          8      40
Privilege information         :          2  (variably sized)
Stream                        :          2      40
Stream data                   :          2   16384
Thread                        :          7  (variably sized)
Thread master                 :          3  (variably sized)
Thread stats                  :          5      64
Vector                        :       2789      16
Vector index                  :       2789  (variably sized)
VRF                           :          1     184
VRF bit-map                   :         82  (variably sized)
VTY                           :          6  (variably sized)
Zclient                       :          1    2872
Redistribution instance IDs   :          3       2
--- qmem pbrd ---
PBR Interface                 :          4     100
`

relevant config:

`
router bgp xxx
 neighbor x.x.x.x remote-as internal
 neighbor x.x.x.x update-source x.x.x.y
!
router ospf
 redistribute connected
 network x/24 area 0
!

I won't be posting 700k lines (for show ip route) on here.

donaldsharp · 2018-06-21T21:01:46Z

Stream data : 1689253 (variably sized)```

I would suspect we have a leak in stream memory.  Investigating

ghost · 2018-06-21T21:03:56Z

Thanks. FRR always used slightly more memory than bird (around ~400MB for a full table with fib export on bird2, around ~ 1.5GB for that on frr) but suddenly allocating a steady 8GB can't be right.

donaldsharp · 2018-06-21T22:38:54Z

some more commands to run ss -aep | grep frr, show workqueues, does zebra have any error messages in it's logs?

donaldsharp · 2018-06-21T22:47:09Z

I have a recreate:
30580 frr 15 -5 10.956g 3.123g 3344 S 0.0 13.3 0:13.54 zebra
30587 frr 15 -5 709892 520416 4308 S 0.0 2.1 0:07.52 bgpd
30596 frr 15 -5 73956 8792 2700 S 0.0 0.0 0:03.33 pimd
30603 frr 15 -5 59308 4276 2468 S 0.0 0.0 0:00.00 pbrd

mwinter-osr · 2018-06-21T22:49:24Z

I've just set up a FRR 5.0 (RPKI package version for debian 9 as published on the github release page).
The issue does not show up in my setup. I have 1 full BGP feed receive and 1 full BGP feed to advertise (IPv4 only, no IPv6)
Memory after startup is similar to 4.0 RPKI version as before.

Does this happen immediatly? Can you post config and exact version (ie "show version" if built on your own, otherwise where you got the package)

This commit gathers data from the client->obuf_fifo and puts it all into the buffer for writing. This will address the fact that the multiple events created caused the memory of zebra to grow to unrealistic levels of usage when we are redistributing data to other protocols. Recreate Memory: robot# show memory Memory statistics for zebra: System allocator statistics: Total heap allocated: 1930 MiB Holding block headers: 16 MiB Used small blocks: 0 bytes Used ordinary blocks: 1911 MiB Free small blocks: 1968 bytes Free ordinary blocks: 19 MiB Ordinary blocks: 5210 Small blocks: 58 Holding blocks: 1 New Zebra Memory: Memory statistics for zebra: System allocator statistics: Total heap allocated: 478 MiB Holding block headers: 16 MiB Used small blocks: 0 bytes Used ordinary blocks: 415 MiB Free small blocks: 1968 bytes Free ordinary blocks: 63 MiB Ordinary blocks: 4909 Small blocks: 58 Holding blocks: 1 New show threads cpu for Zebra: robot# show thread cpu Thread statistics for zebra: Showing statistics for pthread main ----------------------------------- CPU (user+system): Real (wall-clock): Active Runtime(ms) Invoked Avg uSec Max uSecs Avg uSec Max uSecs Type Thread 0 6465.766 801 8072 54775 10810 225356 T work_queue_run 1 0.096 4 24 37 24 38 R vtysh_accept 0 8.690 533 16 54 154 6286 W zserv_flush_data 0 254.102 290 876 2224 971 6958 W zserv_write 0 1992.936 7854 253 115333 266 116288 E &zserv_process_messages 1 1.351 6 225 245 226 249 R zebra_accept 1 0.152 8 19 22 19 23 T zebra_ptm_connect 1 0.124 7 17 24 18 24 R kernel_read 1 121.460 122 995 107273 1021 108707 R vtysh_read 6 686.460 7854 87 150 93 6006 R zserv_read 0 0.040 1 40 40 39 39 T zebra_route_map_update_timer 0 0.412 6 68 170 499 1520 T if_zebra_speed_update Fixes: FRRouting#2527 Signed-off-by: Donald Sharp <sharpd@cumulusnetworks.com>

donaldsharp · 2018-06-22T11:56:50Z

@sybevnao -> If you have the ability to try this branch, I believe this will fix your issue.

mwinter-osr · 2018-06-22T14:13:37Z

@sybevnao If you have a chance to test the fix from Donald, then feel free to use the packages here:
https://ci1.netdef.org/browse/FRR-FRRPULLREQ-4153/artifact

These are the packages built based on Donald's fix. Feedback would be very welcome.

This commit gathers data from the client->obuf_fifo and puts it all into the buffer for writing. This will address the fact that the multiple events created caused the memory of zebra to grow to unrealistic levels of usage when we are redistributing data to other protocols. Recreate Memory: robot# show memory Memory statistics for zebra: System allocator statistics: Total heap allocated: 1930 MiB Holding block headers: 16 MiB Used small blocks: 0 bytes Used ordinary blocks: 1911 MiB Free small blocks: 1968 bytes Free ordinary blocks: 19 MiB Ordinary blocks: 5210 Small blocks: 58 Holding blocks: 1 New Zebra Memory: Memory statistics for zebra: System allocator statistics: Total heap allocated: 478 MiB Holding block headers: 16 MiB Used small blocks: 0 bytes Used ordinary blocks: 415 MiB Free small blocks: 1968 bytes Free ordinary blocks: 63 MiB Ordinary blocks: 4909 Small blocks: 58 Holding blocks: 1 New show threads cpu for Zebra: robot# show thread cpu Thread statistics for zebra: Showing statistics for pthread main ----------------------------------- CPU (user+system): Real (wall-clock): Active Runtime(ms) Invoked Avg uSec Max uSecs Avg uSec Max uSecs Type Thread 0 6465.766 801 8072 54775 10810 225356 T work_queue_run 1 0.096 4 24 37 24 38 R vtysh_accept 0 8.690 533 16 54 154 6286 W zserv_flush_data 0 254.102 290 876 2224 971 6958 W zserv_write 0 1992.936 7854 253 115333 266 116288 E &zserv_process_messages 1 1.351 6 225 245 226 249 R zebra_accept 1 0.152 8 19 22 19 23 T zebra_ptm_connect 1 0.124 7 17 24 18 24 R kernel_read 1 121.460 122 995 107273 1021 108707 R vtysh_read 6 686.460 7854 87 150 93 6006 R zserv_read 0 0.040 1 40 40 39 39 T zebra_route_map_update_timer 0 0.412 6 68 170 499 1520 T if_zebra_speed_update Fixes: FRRouting#2527 Signed-off-by: Donald Sharp <sharpd@cumulusnetworks.com>

ghost · 2018-06-22T18:19:56Z

I've just upgraded the frr package using the packages provided by @mwinter-osr, after successfully installing & restarting all services, massive decrease in memory usage can be observed:

Linux "free"

              total        used        free      shared  buff/cache   available
Mem:        8179256     1661624     6375204        8664      142428     6305112
Swap:       8385532           0     8385532

zebra

Memory statistics for zebra:
System allocator statistics:
  Total heap allocated:  777 MiB
  Holding block headers: 16 MiB
  Used small blocks:     0 bytes
  Used ordinary blocks:  508 MiB
  Free small blocks:     1968 bytes
  Free ordinary blocks:  269 MiB
  Ordinary blocks:       100229
  Small blocks:          58
  Holding blocks:        1

bgpd

Memory statistics for bgpd:
System allocator statistics:
  Total heap allocated:  564 MiB
  Holding block headers: 16 MiB
  Used small blocks:     0 bytes
  Used ordinary blocks:  534 MiB
  Free small blocks:     8120 KiB
  Free ordinary blocks:  30 MiB
  Ordinary blocks:       14430
  Small blocks:          103156
  Holding blocks:        1

BGP summary

IPv4 Unicast Summary:
BGP router identifier x, local AS number x vrf-id 0
BGP table version 693917
RIB entries 1266982, using 184 MiB of memory
Peers 1, using 21 KiB of memory

Neighbor        V         AS MsgRcvd MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd
x 4      x 137753       6        0    0    0 00:03:58       692745

Total number of neighbors 1

@donaldsharp indeed seems to have spot the culprit and current usage is more than reasonable.

Thanks to @donaldsharp and @mwinter-osr for your effort and the super-quick responses and follow-ups!

donaldsharp · 2018-06-22T18:43:40Z

@sybevnao -> If you are not using pim, then shutting it off would reduce some memory usage in zebra. ( See 6298b1f ) that I just filed.

ghost · 2018-06-22T18:57:03Z

@donaldsharp there indeed seems to be a bit of an improvement after setting pimd=no:

zebra

Total heap allocated:  597 MiB

Additionally, I've just taken a look at the CPU usage of the entire system now, which seems to outperform BIRD on a system with similar specifications now:

frr

%Cpu(s):  0.1 us,  0.2 sy,  0.0 ni, 99.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

bird

%Cpu(s): 14.7 us,  8.7 sy,  0.0 ni, 76.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.3 st

Good job. My hypothesis here is that frr deals better with occasional bgp updates.

donaldsharp · 2018-06-22T18:58:41Z

@sybevnao good to hear. I believe we still have some serious room for improvements ;)

This commit gathers data from the client->obuf_fifo and puts it all into the buffer for writing. This will address the fact that the multiple events created caused the memory of zebra to grow to unrealistic levels of usage when we are redistributing data to other protocols. Recreate Memory: robot# show memory Memory statistics for zebra: System allocator statistics: Total heap allocated: 1930 MiB Holding block headers: 16 MiB Used small blocks: 0 bytes Used ordinary blocks: 1911 MiB Free small blocks: 1968 bytes Free ordinary blocks: 19 MiB Ordinary blocks: 5210 Small blocks: 58 Holding blocks: 1 New Zebra Memory: Memory statistics for zebra: System allocator statistics: Total heap allocated: 478 MiB Holding block headers: 16 MiB Used small blocks: 0 bytes Used ordinary blocks: 415 MiB Free small blocks: 1968 bytes Free ordinary blocks: 63 MiB Ordinary blocks: 4909 Small blocks: 58 Holding blocks: 1 New show threads cpu for Zebra: robot# show thread cpu Thread statistics for zebra: Showing statistics for pthread main ----------------------------------- CPU (user+system): Real (wall-clock): Active Runtime(ms) Invoked Avg uSec Max uSecs Avg uSec Max uSecs Type Thread 0 6465.766 801 8072 54775 10810 225356 T work_queue_run 1 0.096 4 24 37 24 38 R vtysh_accept 0 8.690 533 16 54 154 6286 W zserv_flush_data 0 254.102 290 876 2224 971 6958 W zserv_write 0 1992.936 7854 253 115333 266 116288 E &zserv_process_messages 1 1.351 6 225 245 226 249 R zebra_accept 1 0.152 8 19 22 19 23 T zebra_ptm_connect 1 0.124 7 17 24 18 24 R kernel_read 1 121.460 122 995 107273 1021 108707 R vtysh_read 6 686.460 7854 87 150 93 6006 R zserv_read 0 0.040 1 40 40 39 39 T zebra_route_map_update_timer 0 0.412 6 68 170 499 1520 T if_zebra_speed_update PR=59228 Fixes: FRRouting#2527 Signed-off-by: Donald Sharp <sharpd@cumulusnetworks.com> (cherry picked from commit 3734642) Signed-off-by: Philippe Guibert <philippe.guibert@6wind.com>

donaldsharp mentioned this issue Jun 22, 2018

zebra: Gather up output data and push in buffer to send #2529

Merged

qlyoung added this to the 5.1 milestone Jun 26, 2018

qlyoung added the bug label Jun 26, 2018

donaldsharp closed this as completed Jul 4, 2018

ghost mentioned this issue Nov 4, 2018

Arbitarily high memory usage (again?) #3283

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Crazy memory usage #2527

Crazy memory usage #2527

ghost commented Jun 21, 2018

donaldsharp commented Jun 21, 2018

ghost commented Jun 21, 2018 •

edited by ghost

Loading

donaldsharp commented Jun 21, 2018

ghost commented Jun 21, 2018

donaldsharp commented Jun 21, 2018

donaldsharp commented Jun 21, 2018

mwinter-osr commented Jun 21, 2018

donaldsharp commented Jun 22, 2018 •

edited

Loading

mwinter-osr commented Jun 22, 2018

ghost commented Jun 22, 2018

donaldsharp commented Jun 22, 2018

ghost commented Jun 22, 2018

donaldsharp commented Jun 22, 2018

Crazy memory usage #2527

Crazy memory usage #2527

Comments

ghost commented Jun 21, 2018

donaldsharp commented Jun 21, 2018

ghost commented Jun 21, 2018 • edited by ghost Loading

donaldsharp commented Jun 21, 2018

ghost commented Jun 21, 2018

donaldsharp commented Jun 21, 2018

donaldsharp commented Jun 21, 2018

mwinter-osr commented Jun 21, 2018

donaldsharp commented Jun 22, 2018 • edited Loading

mwinter-osr commented Jun 22, 2018

ghost commented Jun 22, 2018

donaldsharp commented Jun 22, 2018

ghost commented Jun 22, 2018

donaldsharp commented Jun 22, 2018

ghost commented Jun 21, 2018 •

edited by ghost

Loading

donaldsharp commented Jun 22, 2018 •

edited

Loading