Cannot connect to host from within Docker container

[geowa4]

What did you do?

To get up and running quickly, I built my own Docker image with the following Dockerfile.

FROM prom/prometheus
COPY prometheus.yml /etc/prometheus/prometheus.yml

The contents of prometheus.yml are listed below in the designated section.

I've run the container with docker run --name prom --rm -it --add-host 'app.local:10.10.2.103' -p 9090:9090 my_prometheus.

What did you expect to see?

When querying up in Prometheus with my browser (http://localhost:9090/graph), I expect to see a 1 for app.local:4000 as I do with checking Prometheus itself.

What did you see instead? Under which circumstances?

I see a 0 for the up metric even though the application is indeed up and has a working /metrics endpoint. To verify connectivity to the host, I run docker exec -it prom wget app.local:4000/metrics -O /dev/stdout. The yields the text I expect.

Environment

• System information:

$ uname -srm
Darwin 15.6.0 x86_64

• Prometheus version:

$ docker exec -it prom prometheus -version
prometheus, version 1.1.2 (branch: master, revision: 36fbdcc30fd13ad796381dc934742c559feeb1b5)
  build user:       root@a74d279a0d22
  build date:       20160908-13:12:43
  go version:       go1.6.3

• Alertmanager version:
  NA, but for good measure.

$ docker exec -it prom alertmanager -version
rpc error: code = 2 desc = oci runtime error: exec failed: exec: "alertmanager": executable file not found in $PATH

• Prometheus configuration file:

# my global config
global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.
  evaluation_interval: 15s # By default, scrape targets every 15 seconds.
  # scrape_timeout is set to the global default (10s).

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
      monitor: 'codelab-monitor'

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first.rules"
  # - "second.rules"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets:
          - 'localhost:9090'
  - job_name: 'hello'

    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets:
          - 'app.local:4000'

• Alertmanager configuration file:
  NA
• Logs:

INFO[0000] Starting prometheus (version=1.1.2, branch=master, revision=36fbdcc30fd13ad796381dc934742c559feeb1b5)  source=main.go:73
INFO[0000] Build context (go=go1.6.3, user=root@a74d279a0d22, date=20160908-13:12:43)  source=main.go:74
INFO[0000] Loading configuration file /etc/prometheus/prometheus.yml  source=main.go:221
INFO[0000] Loading series map and head chunks...         source=storage.go:358
INFO[0000] 0 series loaded.                              source=storage.go:363
WARN[0000] No AlertManagers configured, not dispatching any alerts  source=notifier.go:176
INFO[0000] Listening on :9090                            source=web.go:233
INFO[0000] Starting target manager...                    source=targetmanager.go:76
INFO[0050] Completed maintenance sweep through 4 in-memory fingerprints in 40.002076731s.  source=storage.go:1086
INFO[0300] Checkpointing in-memory metrics and chunks...  source=persistence.go:548
INFO[0300] Done checkpointing in-memory metrics and chunks in 11.06216ms.  source=persistence.go:572

[geowa4]

For reference, I figured I would also add the response from app.local:4000/metrics. It's a Phoenix application using libraries from https://github.com/deadtrickster.

# TYPE erlang_vm_ets_limit gauge
# HELP erlang_vm_ets_limit The maximum number of ETS tables allowed
erlang_vm_ets_limit 2053
# TYPE erlang_vm_logical_processors gauge
# HELP erlang_vm_logical_processors The detected number of logical processors configured in the system
erlang_vm_logical_processors 8
# TYPE erlang_vm_logical_processors_available gauge
# HELP erlang_vm_logical_processors_available The detected number of logical processors available to the Erlang runtime system
erlang_vm_logical_processors_available unknown
# TYPE erlang_vm_logical_processors_online gauge
# HELP erlang_vm_logical_processors_online The detected number of logical processors online on the system
erlang_vm_logical_processors_online 8
# TYPE erlang_vm_port_count gauge
# HELP erlang_vm_port_count The number of ports currently existing at the local node
erlang_vm_port_count 22
# TYPE erlang_vm_port_limit gauge
# HELP erlang_vm_port_limit The maximum number of simultaneously existing ports at the local node
erlang_vm_port_limit 65536
# TYPE erlang_vm_process_count gauge
# HELP erlang_vm_process_count The number of processes currently existing at the local node
erlang_vm_process_count 258
# TYPE erlang_vm_process_limit gauge
# HELP erlang_vm_process_limit The maximum number of simultaneously existing at the local node
erlang_vm_process_limit 262144
# TYPE erlang_vm_schedulers gauge
# HELP erlang_vm_schedulers The number of scheduler threads used by the emulator
erlang_vm_schedulers 8
# TYPE erlang_vm_schedulers_online gauge
# HELP erlang_vm_schedulers_online The number of schedulers online
erlang_vm_schedulers_online 8
# TYPE erlang_vm_smp_support untyped
# HELP erlang_vm_smp_support 1 if the emulator has been compiled with SMP support, otherwise 0
erlang_vm_smp_support 1
# TYPE erlang_vm_threads untyped
# HELP erlang_vm_threads 1 if the emulator has been compiled with thread support, otherwise 0
erlang_vm_threads 1
# TYPE erlang_vm_thread_pool_size gauge
# HELP erlang_vm_thread_pool_size the number of async threads in the async thread pool used for asynchronous driver calls
erlang_vm_thread_pool_size 10
# TYPE erlang_vm_time_correction untyped
# HELP erlang_vm_time_correction 1 if time correction is enabled, otherwise 0
erlang_vm_time_correction 1
# TYPE erlang_vm_statistics_context_switches counter
# HELP erlang_vm_statistics_context_switches Total number of context switches since the system started
erlang_vm_statistics_context_switches 148479
# TYPE erlang_vm_statistics_garbage_collection_number_of_gcs counter
# HELP erlang_vm_statistics_garbage_collection_number_of_gcs Garbage collection: number of GCs
erlang_vm_statistics_garbage_collection_number_of_gcs 14614
# TYPE erlang_vm_statistics_garbage_collection_words_reclaimed counter
# HELP erlang_vm_statistics_garbage_collection_words_reclaimed Garbage collection: words reclaimed
erlang_vm_statistics_garbage_collection_words_reclaimed 58983716
# TYPE erlang_vm_statistics_bytes_received_total counter
# HELP erlang_vm_statistics_bytes_received_total Total number of bytes received through ports
erlang_vm_statistics_bytes_received_total 28197931
# TYPE erlang_vm_statistics_bytes_output_total counter
# HELP erlang_vm_statistics_bytes_output_total Total number of bytes output to ports
erlang_vm_statistics_bytes_output_total 8378231
# TYPE erlang_vm_statistics_reductions_total counter
# HELP erlang_vm_statistics_reductions_total Total reductions
erlang_vm_statistics_reductions_total 22120432
# TYPE erlang_vm_statistics_run_queues_length_total counter
# HELP erlang_vm_statistics_run_queues_length_total Total length of the run-queues
erlang_vm_statistics_run_queues_length_total 0
# TYPE erlang_vm_statistics_runtime_milliseconds counter
# HELP erlang_vm_statistics_runtime_milliseconds The sum of the runtime for all threads in the Erlang runtime system. Can be greater than wall clock time
erlang_vm_statistics_runtime_milliseconds 4240
# TYPE erlang_vm_statistics_wallclock_time_milliseconds counter
# HELP erlang_vm_statistics_wallclock_time_milliseconds Information about wall clock. Same as erlang_vm_statistics_runtime_milliseconds except that real time is measured
erlang_vm_statistics_wallclock_time_milliseconds 694297
# TYPE erlang_vm_memory_bytes_total gauge
# HELP erlang_vm_memory_bytes_total The total amount of memory currently allocated. This is the same as the sum of the memory size for processes and system.
erlang_vm_memory_bytes_total{kind="system"} 45324608
erlang_vm_memory_bytes_total{kind="processes"} 8707664
# TYPE erlang_vm_memory_processes_bytes_total gauge
# HELP erlang_vm_memory_processes_bytes_total The total amount of memory currently allocated for the Erlang processes.
erlang_vm_memory_processes_bytes_total{usage="used"} 8706424
erlang_vm_memory_processes_bytes_total{usage="free"} 1240
# TYPE erlang_vm_memory_system_bytes_total gauge
# HELP erlang_vm_memory_system_bytes_total The total amount of memory currently allocated for the emulator that is not directly related to any Erlang process. Memory presented as processes is not included in this memory.
erlang_vm_memory_system_bytes_total{usage="atom"} 891849
erlang_vm_memory_system_bytes_total{usage="binary"} 2075096
erlang_vm_memory_system_bytes_total{usage="code"} 27114495
erlang_vm_memory_system_bytes_total{usage="ets"} 2593000
erlang_vm_memory_system_bytes_total{usage="other"} 12650168
# TYPE erlang_vm_memory_atom_bytes_total gauge
# HELP erlang_vm_memory_atom_bytes_total The total amount of memory currently allocated for atoms. This memory is part of the memory presented as system memory.
erlang_vm_memory_atom_bytes_total{usage="used"} 874668
erlang_vm_memory_atom_bytes_total{usage="free"} 17181
# TYPE erlang_vm_ets_tables gauge
# HELP erlang_vm_ets_tables Erlang VM ETS Tables count
erlang_vm_ets_tables 69
# TYPE erlang_vm_dets_tables gauge
# HELP erlang_vm_dets_tables Erlang VM DETS Tables count
erlang_vm_dets_tables 0
# TYPE telemetry_scrape_size_bytes summary
# HELP telemetry_scrape_size_bytes Scrape size, uncompressed
telemetry_scrape_size_bytes_count{registry="default",content_type="text/plain; version=0.0.4"} 191
telemetry_scrape_size_bytes_sum{registry="default",content_type="text/plain; version=0.0.4"} 1436694
# TYPE telemetry_scrape_duration_seconds summary
# HELP telemetry_scrape_duration_seconds Scrape duration
telemetry_scrape_duration_seconds_count{registry="default",content_type="text/plain; version=0.0.4"} 191
telemetry_scrape_duration_seconds_sum{registry="default",content_type="text/plain; version=0.0.4"} 0.65218632
# TYPE ecto_queue_duration_microseconds histogram
# HELP ecto_queue_duration_microseconds The time spent to check the connection out in microseconds.
# TYPE phoenix_controller_call_duration_microseconds histogram
# HELP phoenix_controller_call_duration_microseconds Whole controller pipeline execution time in microseconds.
# TYPE http_request_duration_microseconds histogram
# HELP http_request_duration_microseconds The HTTP request latencies in microseconds.
# TYPE ecto_decode_duration_microseconds histogram
# HELP ecto_decode_duration_microseconds The time spent decoding the result in microseconds.
# TYPE ecto_query_duration_microseconds histogram
# HELP ecto_query_duration_microseconds Total Ecto query time in microseconds.
# TYPE ecto_db_query_duration_microseconds histogram
# HELP ecto_db_query_duration_microseconds The time spent executing the DB query in microseconds.
# TYPE http_requests_total counter
# HELP http_requests_total Total number of HTTP requests made.

[tattsun]

I think that erlang_vm_logical_processors_available unknown is problem.
Prometheus can't parse unknown as value.
(You can get error information at http://localhost:9090/targets)

FYI: https://prometheus.io/docs/instrumenting/exposition_formats/

[juliusv]

Yeah, the /metrics output is invalid. The sample values cannot be unknown, it has to be a valid float value, including the possible valid special values NaN, Inf, -Inf. So it should read NaN instead.

[juliusv]

Btw., you can also go to /targets on your Prometheus server and you should see the exact scrape error for the target.

[geowa4]

Thanks for the /targets tip. When I go there, I get this error message for the endpoint http://app.local:4000/metrics: Get http://app.local:4000/metrics: dial tcp: lookup app.local on 127.0.0.11:53: no such host.

[tattsun]

Ah, it's wired. Could you tell me what is printed at Endpoint section in /targets?
If you configured correctly, you should get text http://app.local:4000/metrics there.

[geowa4]

Never mind, I had another issue going on to cause that one. The error is in fact text format parsing error in line 9: expected float as value, got "unknown". Thanks for the help everyone! I think I can close this an open an issue in my client library.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.