Targets are UP but no metrics

[MaksimBalticom]

Hello,
I have a list of devices in .yml file that I query with snmp_exporter. The problem is that I see these devies in Status - Target. All devices are UP. But there are no metrics for those devices. There should be many o them according to snmp.yml file.

What did you do?

Configured prometheus.yml to use different jobs (job per switch model). Configured 3 new modules in snmp.yml.

What did you expect to see?

Many new metrics for now models.

What did you see instead? Under which circumstances?

'no data' in Console tab

Environment

          total        used        free      shared  buff/cache   available

Mem: 7974664 715016 6247176 8548 1012472 6929064
Swap: 8257532 0 8257532

processor : 0,1,2,3
vendor_id : GenuineIntel
cpu family : 6
model : 42
model name : Intel Xeon E312xx (Sandy Bridge)
stepping : 1
microcode : 0x1
cpu MHz : 3092.972
cache size : 4096 KB
physical id : 0
siblings : 1
core id : 0
cpu cores : 1
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx rdtscp lm constant_tsc rep_good nopl eagerfpu pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm fsgsbase smep erms xsaveopt
bogomips : 6185.94
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management:

• System information:

Linux 3.10.0-514.26.1.el7.x86_64 x86_64

• Prometheus version:

prometheus, version 1.7.1 (branch: master, revision: 3afb3ff)
build user: root@0aa1b7fc430d
build date: 20170612-11:44:05
go version: go1.8.3

• Prometheus configuration file:

global:
  scrape_interval: 1m
  evaluation_interval: 1m
scrape_configs:
  - job_name: 'DGS3120A'
    file_sd_configs:
        - files :
          - DGS3120A.yml
    metrics_path: /snmp
    params:
      module: [GS221024]
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:9116  # SNMP exporter.
#  - job_name: 'DGS3120B'
#    file_sd_configs:
#        - files :
#          - DGS3120B.yml
#    metrics_path: /snmp
#    params:
#      module: [DGS3120B]
#    relabel_configs:
#      - source_labels: [__address__]
#        target_label: __param_target
#      - source_labels: [__param_target]
#        target_label: instance
#      - target_label: __address__
#        replacement: 127.0.0.1:9116  # SNMP exporter.
#  - job_name: 'GS22108'
#    scrape_interval: 5m
#    scrape_timeout: 5m
#    file_sd_configs:
#        - files :
#          - GS22108.yml
#    metrics_path: /snmp
#    params:
#      module: [GS22108]
#    relabel_configs:
#      - source_labels: [__address__]
#        target_label: __param_target
#      - source_labels: [__param_target]
#        target_label: instance
#      - target_label: __address__
#        replacement: 127.0.0.1:9116  # SNMP exporter.
#  - job_name: 'GS221024'
#    scrape_interval: 5m
#    scrape_timeout: 5m
#    file_sd_configs:
#        - files :
#          - GS221024.yml
#    metrics_path: /snmp
#    params:
#      module: [GS221024]
#    relabel_configs:
#      - source_labels: [__address__]
#        target_label: __param_target
#      - source_labels: [__param_target]
#        target_label: instance
#      - target_label: __address__
#        replacement: 127.0.0.1:9116  # SNMP exporter.
#  - job_name: 'GS22008'
#    scrape_interval: 5m
#    scrape_timeout: 5m
#    file_sd_configs:
#        - files :
#          - GS22008.yml
#    metrics_path: /snmp
#    params:
#      module: [GS22008]
#    relabel_configs:
#      - source_labels: [__address__]
#        target_label: __param_target
#      - source_labels: [__param_target]
#        target_label: instance
#      - target_label: __address__
#        replacement: 127.0.0.1:9116  # SNMP exporter.
  - job_name: 'GS220024'
    scrape_interval: 5m
    scrape_timeout: 5m
    file_sd_configs:
        - files :
          - GS220024.yml
    metrics_path: /snmp
    params:
      module: [GS220024]
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:9116  # SNMP exporter.
  - job_name: 'cisco'
    scrape_interval: 5m
    scrape_timeout: 5m
    file_sd_configs:
        - files :
          - cisco.yml
    metrics_path: /snmp
    params:
      module: [cisco]
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:9116  # SNMP exporter.
#  - job_name: 'snmp'
#    file_sd_configs:
#        - files :
#          - targets.yml
#    metrics_path: /snmp
#    params:
#      module: [apcups]
#    relabel_configs:
#      - source_labels: [__address__]
#        target_label: __param_target
#      - source_labels: [__param_target]
#        target_label: instance
#      - target_label: __address__
#        replacement: 127.0.0.1:9116  # SNMP exporter.
#  - job_name: 'telegraf'
#    scrape_interval: 1m
#    static_configs:
#      - targets: ['127.0.0.1:9126']

• Alertmanager configuration file:

insert configuration here (if relevant to the issue)

• Logs:

insert Prometheus and Alertmanager logs relevant to the issue here

[gouthamve]

Could you also share the prometheus logs?

[MaksimBalticom]

I have just restarted Prometheus and this is the log since restart:

time="2017-07-06T18:11:59+03:00" level=info msg="Starting prometheus (version=1.7.1, branch=master, revision=3afb3fffa3a29c3de865e1172fb740442e9d0133)" source="main.go:88"
time="2017-07-06T18:11:59+03:00" level=info msg="Build context (go=go1.8.3, user=root@0aa1b7fc430d, date=20170612-11:44:05)" source="main.go:89"
time="2017-07-06T18:11:59+03:00" level=info msg="Host details (Linux 3.10.0-514.26.1.el7.x86_64 #1 SMP Thu Jun 29 16:05:25 UTC 2017 x86_64 grafana.balticom.lv (none))" source="main.go:90"
time="2017-07-06T18:11:59+03:00" level=info msg="Loading configuration file prometheus.yml" source="main.go:252"
time="2017-07-06T18:11:59+03:00" level=info msg="Loading series map and head chunks..." source="storage.go:428"
time="2017-07-06T18:12:00+03:00" level=info msg="108602 series loaded." source="storage.go:439"
time="2017-07-06T18:12:00+03:00" level=info msg="Listening on :9090" source="web.go:259"
time="2017-07-06T18:12:00+03:00" level=info msg="Starting target manager..." source="targetmanager.go:63"
time="2017-07-06T18:17:00+03:00" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:633"
time="2017-07-06T18:17:01+03:00" level=info msg="Done checkpointing in-memory metrics and chunks in 1.529277813s." source="persistence.go:665"
time="2017-07-06T18:22:01+03:00" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:633"
time="2017-07-06T18:22:03+03:00" level=info msg="Done checkpointing in-memory metrics and chunks in 1.547338424s." source="persistence.go:665"

[MaksimBalticom]

I have solved it. The problem was that I had snmp OIDs in my snmp.yml file, but there were no walk OIDs. I added walk OIDs and now it is functioning.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.