Performance issue.

[fatherfbi]

Hi Vincent,
First of all, thank you for such a wonderful product!
I have a number of questions regarding performance, first I will describe my installation, I have 10 routers with the following configuration.

set services flow-monitoring version9 template ipv4 flow-active-timeout 10
set services flow-monitoring version9 template ipv4 flow-inactive-timeout 10
set services flow-monitoring version9 template ipv4 template-refresh-rate packets 30
set services flow-monitoring version9 template ipv4 template-refresh-rate seconds 30
set services flow-monitoring version9 template ipv4 option-refresh-rate packets 30
set services flow-monitoring version9 template ipv4 option-refresh-rate seconds 30
set services flow-monitoring version9 template ipv4 ipv4-template
set services flow-monitoring version9 template ipv6 flow-active-timeout 10
set services flow-monitoring version9 template ipv6 flow-inactive-timeout 10
set services flow-monitoring version9 template ipv6 template-refresh-rate packets 30
set services flow-monitoring version9 template ipv6 template-refresh-rate seconds 30
set services flow-monitoring version9 template ipv6 option-refresh-rate packets 30
set services flow-monitoring version9 template ipv6 option-refresh-rate seconds 30
set services flow-monitoring version9 template ipv6 ipv6-template

set forwarding-options sampling instance AKVORADO input rate 1024
set forwarding-options sampling instance AKVORADO input max-packets-per-second 65535
set forwarding-options sampling instance AKVORADO family inet output flow-server 10.255.*.* port 2055
set forwarding-options sampling instance AKVORADO family inet output flow-server 10.255.*.* routing-instance mgmt
set forwarding-options sampling instance AKVORADO family inet output flow-server 10.255.*.* autonomous-system-type origin
set forwarding-options sampling instance AKVORADO family inet output flow-server 10.255.*.* source-address 10.255.*.*
set forwarding-options sampling instance AKVORADO family inet output flow-server 10.255.*.* version9 template ipv4
set forwarding-options sampling instance AKVORADO family inet output inline-jflow source-address 10.255.*.*
set forwarding-options sampling instance AKVORADO family inet6 output flow-server 10.255.*.* port 2055
set forwarding-options sampling instance AKVORADO family inet6 output flow-server 10.255.*.* routing-instance mgmt
set forwarding-options sampling instance AKVORADO family inet6 output flow-server 10.255.*.* autonomous-system-type origin
set forwarding-options sampling instance AKVORADO family inet6 output flow-server 10.255.*.* source-address 10.255.*.*
set forwarding-options sampling instance AKVORADO family inet6 output flow-server 10.255.*.* version9 template ipv6
set forwarding-options sampling instance AKVORADO family inet6 output inline-jflow source-address 10.255.*.*

All routers and interfaces are described in the static provider, this is almost 400 interfaces. The main configuration looks like this.
config/akvorado.yaml

kafka:
  topic: flows
  version: 3.3.1
  brokers:
    - kafka:9092
  topic-configuration:
    num-partitions: 8
    replication-factor: 1
    config-entries:
      # The retention policy in Kafka is mainly here to keep a buffer
      # for ClickHouse.
      segment.bytes: 1073741824
      retention.ms: 86400000 # 1 day
      cleanup.policy: delete
      compression.type: producer

geoip:
  optional: true
  # When running on Docker, these paths are inside the container. By default,
  # IPinfo databases are used. (https://ipinfo.io/)
  asn-database:
    - /usr/share/GeoIP/asn.mmdb
  geo-database:
    - /usr/share/GeoIP/country.mmdb
  # If you want to use MaxmindDB, check `.env`, `docker-compose-maxmind.yml` and
  # update these paths:
  #asn-database:
  # - /usr/share/GeoIP/GeoLite2-ASN.mmdb
  #geo-database:
  # - /usr/share/GeoIP/GeoLite2-Country.mmdb

clickhouse:
  orchestrator-url: http://akvorado-orchestrator:8080
  kafka:
    consumers: 10
  resolutions:
    - interval: 0
      ttl: 168h  # 7 days
    - interval: 1m
      ttl: 720h  # 1 months
    - interval: 5m
      ttl: 2160h # 3 months
    - interval: 1h
      ttl: 4320h # 6 months
  system-log-ttl: 0
  servers:
    - clickhouse:9000
  prometheus-endpoint: /metrics
  asns:
    ****: *****
    *****: *****
  networks:
    # You should customize this section with your networks. This
    # populates the Src/DstNetName/Role/Site/Region/Tenant fields.
    **.**.96.0/19:
      name: ipv4-customers
      role: customers
    **.**.52.0/22:
      name: ipv4-servers
      role: servers
  network-sources: []
    # amazon:
    #   url: https://ip-ranges.amazonaws.com/ip-ranges.json
    #   interval: 6h
    #   transform: |
    #     (.prefixes + .ipv6_prefixes)[] |
    #     { prefix: (.ip_prefix // .ipv6_prefix), tenant: "amazon", region: .region, role: .service|ascii_downcase }
    # gcp:
    #   url: https://www.gstatic.com/ipranges/cloud.json
    #   interval: 6h
    #   transform: |
    #     .prefixes[] |
    #     { prefix: (.ipv4Prefix // .ipv6Prefix), tenant: "google-cloud", region: .scope }

inlet: !include "inlet.yaml"
console: !include "console.yaml"

config/console.yaml

http:
  cache:
    type: redis
    server: redis:6379
default-visualize-options:
  graph-type: stacked
  start: 6 hours ago
  end: now
  filter: InIfBoundary = external
  dimensions:
      - ExporterName
  limit: 10
homepage-top-widgets:
  - src-as
  - dst-as
  - protocol
  - dst-port
  - etype

config/inlet.yaml

---
kafka:
  compression-codec: zstd
metadata:
  workers: 20
  providers:
    - type: static
      exporters:
        10.255.**.**:
          name: **-R2
          default:
            name: unknown
            description: Unknown interface
            speed: 100
          ifindexes:
            571:
              name: ae1.390
              description: UPL ***
              speed: 300000
<<<<<a lot of information>>>>>>
    - type: snmp
      communities:
        10.255.**.**: "@*********"
        10.255.**.**: "@*********"
        10.255.**.**: "@*********"
        10.255.**.**: "@*********"
        10.255.**.**: "@*********"
        10.255.**.**: "@*********"
        10.255.**.**: "@*********"
        10.255.**.**: "@*********"
        10.255.**.**: "@*********"
        10.255.**.**: "@*********"
flow:
  inputs:
    - type: udp
      decoder: netflow
      listen: :2055
      workers: 20
      receive-buffer: 10485760
    - type: udp
      decoder: sflow
      listen: :6343
      workers: 20
      receive-buffer: 10485760
routing:
  provider:
    type: bmp
    listen: 0.0.0.0:10179
    collect-asns: true
    collect-aspaths: true
    collect-communities: true
core:
  workers: 20
  exporter-classifiers:
    - ClassifySiteRegex(Exporter.Name, "^([^-]+)-", "$1")
    - Exporter.Name startsWith "**-" && ClassifyRegion("**")
    - Exporter.Name startsWith "**-" && ClassifyRegion("**")
    - Exporter.Name startsWith "**-" && ClassifyRegion("**")
    - Exporter.Name startsWith "**-" && ClassifyRegion("**")
    - Exporter.Name startsWith "**-" && ClassifyRegion("**")
    - Exporter.Name startsWith "**-" && ClassifyRegion("**")
    - Exporter.Name startsWith "**-" && ClassifyRegion("**")
    - Exporter.Name startsWith "**-" && ClassifyRegion("**")
    - Exporter.Name startsWith "**-" && ClassifyRegion("**")
  interface-classifiers:
    - |
      ClassifyConnectivityRegex(Interface.Description, "^(?i)(pni|ix|upl|ptv|ggc|mna):? ", "$1") &&
      ClassifyProviderRegex(Interface.Description, "^\\S+?\\s(\\S+)", "$1") &&
      ClassifyExternal()
    - ClassifyConnectivityRegex(Interface.Description, "^(?i)(dnl|ibgp|lt|mpls|ytv):? ", "$1") &&
      ClassifyInternal()
    - ClassifyInternal()root@akvorado-v1:/akvorado/config#

Result of work

VM characteristics

System Ubuntu 22
20 vCPU Xeon(R) Gold 6138 CPU @ 2.00GHz
32GB RAM
1TB SAS SSD

Everything works fine
CPU Usage

Disk IOPS

Docker stats

BUT!
If I remove static exporters from the configuration and start adding routers one by one, I noticed that with a large number of network interfaces, the inlet process cannot cope and statistics are not collected. Can you tell me the logic of the work so that I can somehow try to optimize my installation and eliminate the bottleneck?
Correct me if I'm wrong. When receiving a flow packet, we know the incoming and outgoing interfaces IfIndex, then to find out the interface's description and speed, the system queries it by snmp, after we know the interface name, the interface classifier is applied to the name and only after that the record gets into the database? Or maybe this is a docker problem and you need to switch to a without docker installation

[vincentbernat]

Why do you say the inlet cannot cope? Is its CPU usage increasing? It may take a bit of time to learn the interface names, but the drop in flow processing should be short. Have a look at the error counters to see if there is any permanent problem.

[fatherfbi]

Why do you say the inlet cannot cope? Is its CPU usage increasing? It may take a bit of time to learn the interface names, but the drop in flow processing should be short. Have a look at the error counters to see if there is any permanent problem.

Hi Vincent,
I'm preparing the test environment now, once I've done all the tests I'll write a report.

[fatherfbi]

Hi Vincent,
I am writing a report on my testing.

I found that the weak point is the regexp. This is regexp config that processed 50k Flows/s for me

  interface-classifiers:
    - |
      ClassifyConnectivityRegex(Interface.Description, "^(?i)(pni|ix|upl):? ", "$1") &&
      ClassifyExternal()
    - ClassifyInternal()

After I added a regular expression to classify providers.

  interface-classifiers:
    - |
      ClassifyConnectivityRegex(Interface.Description, "^(?i)(pni|ix|upl):? ", "$1") &&
      ClassifyProviderRegex(Interface.Description, "^\\S+?\\s(\\S+)", "$1") &&
      ClassifyExternal()
    - ClassifyInternal()

There has been a delay in the enrichment of the flow, this can be seen from the graph.

After I added the regular expression for the internal interfaces

  interface-classifiers:
    - |
      ClassifyConnectivityRegex(Interface.Description, "^(?i)(pni|ix|upl):?", "$1") &&
      ClassifyProviderRegex(Interface.Description, "^\\S+?\\s(\\S+)", "$1") &&
      ClassifyExternal()
    - ClassifyConnectivityRegex(Interface.Description, "^(?i)(ibgp|mpls|dnl|)-?", "$1") &&
      ClassifyInternal()
    - ClassifyInternal()

The data stopped being displayed in the system, and the processor load began to look like this.

After that I started to reduce the number of threads and the system recovered when it reached <15k Flows/s.

As a workaround, I had to use a static provider and describe all 25 routers and 300 interfaces in it.

[vincentbernat]

What would be useful is to have a CPU profile for each case to understand if there is something we can optimize: https://demo.akvorado.net/docs/troubleshooting#profiling.