Akvorado v2.0.4, SNMPv3 AuthPriv, metadata and sampling miss

[roodey3009]

Hello. I'm trying to run akvorado.
I've encountered an issue where I'm not receiving data in the web interface.
I went through the troubleshooting in the documentation and reached the stage:
curl -s http://127.0.0.1:8080/api/v0/outlet/metrics | grep 'akvorado_outlet_core.*errors'

*@kem-srv-netflow:/opt/akvorado/docker# curl -s http://127.0.0.1:8080/api/v0/outlet/metrics | grep 'akvorado_outlet_core.*errors'
# HELP akvorado_outlet_core_flows_errors_total Number of flows with errors.
# TYPE akvorado_outlet_core_flows_errors_total counter
akvorado_outlet_core_flows_errors_total{error="metadata cache miss",exporter="10.40.252.1"} 1.97548e+06
akvorado_outlet_core_flows_errors_total{error="sampling rate missing",exporter="10.40.252.1"} 1.963744e+06

*@kem-srv-netflow:/opt/akvorado/docker# curl -s http://127.0.0.1:8080/api/v0/outlet/metrics | grep 'akvorado_outlet_core.*errors'
# HELP akvorado_outlet_core_flows_errors_total Number of flows with errors.
# TYPE akvorado_outlet_core_flows_errors_total counter
akvorado_outlet_core_flows_errors_total{error="metadata cache miss",exporter="10.40.252.1"} 2.006668e+06
akvorado_outlet_core_flows_errors_total{error="sampling rate missing",exporter="10.40.252.1"} 1.994932e+06

It seems like everything is configured according to the documentation.
inlet.yaml

---
flow:
  inputs:
    # NetFlow port
    - type: udp
      decoder: netflow
      listen: :2055
      workers: 16
      # Before increasing this value, look for it in the troubleshooting section
      # of the documentation.
      receive-buffer: 212992
    # IPFIX port
    - type: udp
      decoder: netflow
      listen: :4739
      workers: 16
      receive-buffer: 212992
    # sFlow port
    - type: udp
      decoder: sflow
      listen: :6343
      workers: 16
      receive-buffer: 212992

outlet.yaml

---
metadata:
  providers:
    - type: snmp
      credentials:
        ::/0:
          user-name: myuser
          authentication-protocol: MD5
          authentication-passphrase: "mypass"
          privacy-protocol: AES
          privacy-passphrase: "mypass"

routing:
  provider:
    type: bmp
    # Before increasing this value, look for it in the troubleshooting section
    # of the documentation.
    receive-buffer: 212992
core:
  exporter-classifiers:
    # This is an example. This should be customized depending on how
    # your exporters are named.
    - ClassifySiteRegex(Exporter.Name, "^([^-]+)-", "$1")
    - ClassifyRegion("europe")
    - ClassifyTenant("acme")
    - ClassifyRole("edge")
  interface-classifiers:
    # This is an example. This must be customized depending on the
    # descriptions of your interfaces. In the following, we assume
    # external interfaces are named "Transit: Cogent" Or "IX:
    # FranceIX".
    - |
      ClassifyConnectivityRegex(Interface.Description, "^(?i)(transit|pni|ppni|ix):? ", "$1") &&
      ClassifyProviderRegex(Interface.Description, "^\\S+?\\s(\\S+)", "$1") &&
      ClassifyExternal()
    - ClassifyInternal()

I checked from this host using snmpwalk - the target host responds and returns data:

*@kem-srv-netflow:/opt/akvorado/config# snmpwalk -v3 -l authPriv -u myuser -a md5 -A mypass -x AES -X mypass 10.143.40.1
iso.3.6.1.2.1.1.1.0 = STRING: "Cisco IOS Software, ASR1000 Software (PPC_LINUX_IOSD-ADVENTERPRISEK9-M), Version 15.5(3)S9, RELEASE SOFTWARE (fc3)

And now I can't figure out what the problem might be?

[vincentbernat]

What do you get:

curl -s http://127.0.0.1:8080/api/v0/outlet/metrics | grep 'akvorado_outlet_metadata'

You can also check the logs (docker compose logs akvorado-outlet | grep snmp). If this is a timeout, try AES192, AES192-C, AES256 or AES256-C for the privacy algorithm instead of AES.

[roodey3009]

*@kem-srv-netflow:/opt/akvorado/config# curl -s http://127.0.0.1:8080/api/v0/outlet/metrics | grep 'akvorado_outlet_metadata'
# HELP akvorado_outlet_metadata_cache_expired_entries_total Number of cache entries expired.
# TYPE akvorado_outlet_metadata_cache_expired_entries_total counter
akvorado_outlet_metadata_cache_expired_entries_total 0
# HELP akvorado_outlet_metadata_cache_hits_total Number of lookups retrieved from cache.
# TYPE akvorado_outlet_metadata_cache_hits_total counter
akvorado_outlet_metadata_cache_hits_total 7.459458e+06
# HELP akvorado_outlet_metadata_cache_misses_total Number of lookup miss.
# TYPE akvorado_outlet_metadata_cache_misses_total counter
akvorado_outlet_metadata_cache_misses_total 0
# HELP akvorado_outlet_metadata_cache_refresh_runs_total Number of times the cache refresh was triggered.
# TYPE akvorado_outlet_metadata_cache_refresh_runs_total counter
akvorado_outlet_metadata_cache_refresh_runs_total 64
# HELP akvorado_outlet_metadata_cache_refreshes_total Number of entries refreshed in cache.
# TYPE akvorado_outlet_metadata_cache_refreshes_total counter
akvorado_outlet_metadata_cache_refreshes_total 4
# HELP akvorado_outlet_metadata_cache_size_entries Number of entries in cache.
# TYPE akvorado_outlet_metadata_cache_size_entries gauge
akvorado_outlet_metadata_cache_size_entries 2
# HELP akvorado_outlet_metadata_provider_errors_total Number of provider errors.
# TYPE akvorado_outlet_metadata_provider_errors_total counter
akvorado_outlet_metadata_provider_errors_total 0
# HELP akvorado_outlet_metadata_provider_requests_total Number of provider requests.
# TYPE akvorado_outlet_metadata_provider_requests_total counter
akvorado_outlet_metadata_provider_requests_total 4
# HELP akvorado_outlet_metadata_provider_snmp_poller_error_requests_total Number of failed requests.
# TYPE akvorado_outlet_metadata_provider_snmp_poller_error_requests_total counter
akvorado_outlet_metadata_provider_snmp_poller_error_requests_total{error="ifalias missing",exporter="10.40.252.1"} 4
akvorado_outlet_metadata_provider_snmp_poller_error_requests_total{error="ifdescr missing",exporter="10.40.252.1"} 4
akvorado_outlet_metadata_provider_snmp_poller_error_requests_total{error="ifname missing",exporter="10.40.252.1"} 4
akvorado_outlet_metadata_provider_snmp_poller_error_requests_total{error="ifspeed missing",exporter="10.40.252.1"} 4
# HELP akvorado_outlet_metadata_provider_snmp_poller_seconds Time to successfully poll for values.
# TYPE akvorado_outlet_metadata_provider_snmp_poller_seconds summary
akvorado_outlet_metadata_provider_snmp_poller_seconds{exporter="10.40.252.1",quantile="0.5"} 0.000995364
akvorado_outlet_metadata_provider_snmp_poller_seconds{exporter="10.40.252.1",quantile="0.9"} 0.001080845
akvorado_outlet_metadata_provider_snmp_poller_seconds{exporter="10.40.252.1",quantile="0.99"} 0.001080845
akvorado_outlet_metadata_provider_snmp_poller_seconds_sum{exporter="10.40.252.1"} 0.04676130199999999
akvorado_outlet_metadata_provider_snmp_poller_seconds_count{exporter="10.40.252.1"} 4

You can also check the logs (docker compose logs akvorado-outlet | grep snmp)
I have nothing on that command

*@kem-srv-netflow:/opt/akvorado# docker compose logs docker-akvorado-outlet
*@kem-srv-netflow:/opt/akvorado# 

*@kem-srv-netflow:/opt/akvorado# docker ps
CONTAINER ID   IMAGE                               COMMAND                  CREATED       STATUS                 PORTS                                                                                                                                             NAMES
95aad27be3c3   ghcr.io/akvorado/akvorado:2.0.4     "/usr/local/bin/akvo…"   4 hours ago   Up 4 hours (healthy)   8080/tcp, 0.0.0.0:10179->10179/tcp, [::]:10179->10179/tcp                                                                                         docker-akvorado-outlet-1
02cd50bac63b   ghcr.io/akvorado/akvorado:2.0.4     "/usr/local/bin/akvo…"   4 hours ago   Up 4 hours (healthy)   0.0.0.0:2055->2055/udp, [::]:2055->2055/udp, 0.0.0.0:4739->4739/udp, [::]:4739->4739/udp, 0.0.0.0:6343->6343/udp, [::]:6343->6343/udp, 8080/tcp   docker-akvorado-inlet-1
1c9c0145d4f4   ghcr.io/akvorado/akvorado:2.0.4     "/usr/local/bin/akvo…"   4 hours ago   Up 4 hours (healthy)   8080/tcp                                                                                                                                          docker-akvorado-console-1
503f644f12fe   ghcr.io/akvorado/akvorado:2.0.4     "/usr/local/bin/akvo…"   4 hours ago   Up 4 hours (healthy)   8080/tcp                                                                                                                                          docker-akvorado-orchestrator-1
c48a9be7cea5   ghcr.io/kafbat/kafka-ui:v1.4.2      "/bin/sh -c 'java --…"   4 hours ago   Up 4 hours             8080/tcp                                                                                                                                          docker-kafka-ui-1
e4147538dbb4   ghcr.io/akvorado/akvorado:2.0.4     "/usr/local/bin/akvo…"   4 hours ago   Up 4 hours (healthy)                                                                                                                                                     docker-akvorado-conntrack-fixer-1
d905552ccf70   traefik:v3.6                        "/entrypoint.sh trae…"   4 hours ago   Up 4 hours             80/tcp, 127.0.0.1:8080->8080/tcp, 0.0.0.0:8081->8081/tcp, [::]:8081->8081/tcp                                                                     docker-traefik-1
9c9eaba3a3fc   clickhouse/clickhouse-server:25.8   "/entrypoint.sh"         4 hours ago   Up 4 hours (healthy)   8123/tcp, 9000/tcp, 9009/tcp                                                                                                                      docker-clickhouse-1
64ff5fd383b2   apache/kafka:4.1.1                  "/__cacert_entrypoin…"   4 hours ago   Up 4 hours (healthy)   9092/tcp                                                                                                                                          docker-kafka-1
febcce60c7b6   valkey/valkey:7.2                   "docker-entrypoint.s…"   4 hours ago   Up 4 hours (healthy)   6379/tcp                                                                                                                                          docker-redis-1
*@kem-srv-netflow:/opt/akvorado#

[vincentbernat]

Either use docker logs docker-akvorado-outlet-1 or docker compose logs akvorado-outlet.

In docker/versions.yml, replace :2.0.4 with :main and then restart the outlet. The log message will say more clearly which SNMP index does not work.

Since you have an ASR 1000, you are also impacted by #89. You need to configure a default sampling rate to workaround this.