Coroot UI doesn't see pg-agent metrics

[doonydoo]

Hello,
Thank you for this wonderful and convinient instrument!
Unfotunately, I met some troubles with pg-agent.

After installation I've seen the relevant metrics in Prometheus, but my Coroot UI doesn't see them (though it registered the instance of pg-agent)

I started pg-agent with
docker run -d --name coroot-pg-agent -p <port>:80 --env DSN="postgresql://<user>:<password>@<ip>:5432/postgres?connect_timeout=1&statement_timeout=30000" ghcr.io/coroot/coroot-pg-agent

But got in logs of Coroot UI container:
couldn't find actual instance for "postgres", initial instance is "postgres_exporter@my.domain" (map[])
(yes, I have the role of postgres_exporter in PostgreSQL, but use another role for Coroot pg-agent).
What have I done wrong?

Also, I've got none of those metrics: pg_lock_awaiting_queries, pg_wal_receiver_status, pg_wal_replay_paused, pg_wal_receive_lsn, pg_wal_reply_lsn. May be, it was happened because of using PostgreSQL 11 version?

[def]

Your screenshot shows no connections between Prometheus and coroot-pg-agent (there should be an inbound connection to pg-agent). Are you sure that Prometheus has discovered the pg-agent's endpoint?
It seems that the Postgres metrics in your Prometheus are being gathered by pg_exporter, not coroot-pg-agent. Please check this out.

[doonydoo]

Thank you for the answer!
Yes, my Prometheus catches the metrics from pg-agent, I see it in it's interface

and also see graphics
But Coroot UI still doesn't get these metrics from Prometheus

[def]

To understand the case better, I need to see the following metrics with their labels, including the original IP addresses (Coroot maps Postgres metrics to the corresponding containers by matching IP:PORT pairs)

container_net_tcp_listen_info{container_id=~".*pg-agent.*"}
container_net_tcp_active_connections{container_id=~".*pg-agent.*"}
pg_up

[doonydoo]

container_net_tcp_listen_info{container_id="/docker/coroot-pg-agent", instance="10.100.10.111:9887", job="Node-agent_coroot", listen_addr="10.100.10.111:9886", machine_id="4150b6543c254de9bf577a", proxy="dockerd"} 1

container_net_tcp_listen_info{container_id="/docker/coroot-pg-agent", instance="10.100.10.111:9887", job="Node-agent_coroot", listen_addr="127.0.0.1:9886", machine_id="4150b6543c254de9bf577a", proxy="dockerd"} 1

container_net_tcp_listen_info{container_id="/docker/coroot-pg-agent", instance="10.100.10.111:9887", job="Node-agent_coroot", listen_addr="172.17.0.1:9886", machine_id="4150b6543c254de9bf577a", proxy="dockerd"} 1

container_net_tcp_listen_info{container_id="/docker/coroot-pg-agent", instance="10.100.10.111:9887", job="Node-agent_coroot", listen_addr="172.17.0.2:80", machine_id="4150b6543c254de9bf577a"} 1

container_net_tcp_listen_info{container_id="/docker/coroot-pg-agent", instance="10.100.10.111:9887", job="Node-agent_coroot", listen_addr="172.18.0.1:9886", machine_id="4150b6543c254de9bf577a", proxy="dockerd"} 1

container_net_tcp_listen_info{container_id="/docker/coroot-pg-agent", instance="10.100.10.111:9887", job="Node-agent_coroot", listen_addr="172.19.0.1:9886", machine_id="4150b6543c254de9bf577a", proxy="dockerd"} 1

container_net_tcp_listen_info{container_id="/docker/coroot-pg-agent", instance="10.100.10.111:9887", job="Node-agent_coroot", listen_addr="172.20.0.1:9886", machine_id="4150b6543c254de9bf577a", proxy="dockerd"} 1

container_net_tcp_listen_info{container_id="/docker/coroot-pg-agent", instance="10.100.10.111:9887", job="Node-agent_coroot", listen_addr="172.21.0.1:9886", machine_id="4150b6543c254de9bf577a", proxy="dockerd"} 1

---

container_net_tcp_active_connections{actual_destination="10.100.10.111:5432", container_id="/docker/coroot-pg-agent", destination="10.100.10.111:5432", instance="10.100.10.111:9887", job="Node-agent_coroot", machine_id="4150b6543c254de9bf577a"} 1

---

pg_up{instance="10.100.10.111:9187", job="Databases"} 1
pg_up{instance="10.100.10.111:9886", job="PG-agent_coroot"} 1

where port 9886 is used by pg-agent, port 9887 is used by node-agent, port 9187 is used by postgres_exporter

[def]

At this level, everything looks correct.

Can you please show:
container_net_tcp_listen_info{listen_addr=~".+:5432"}

What does the Postgres instance look like in the Coroot UI?

[doonydoo]

container_net_tcp_listen_info{container_id="/docker/postgres", instance="10.100.10.111:9887", job="Node-agent_coroot", listen_addr="10.100.10.111:5432", machine_id="4150b6543c254de9bf577a", proxy="dockerd"} 1
container_net_tcp_listen_info{container_id="/docker/postgres", instance="10.100.10.111:9887", job="Node-agent_coroot", listen_addr="127.0.0.1:5432", machine_id="4150b6543c254de9bf577a", proxy="dockerd"} 1
container_net_tcp_listen_info{container_id="/docker/postgres, instance="10.100.10.111:9887", job="Node-agent_coroot", listen_addr="172.17.0.1:5432", machine_id="4150b6543c254de9bf577a", proxy="dockerd"} 1
container_net_tcp_listen_info{container_id="/docker/postgres", instance="10.100.10.111:9887", job="Node-agent_coroot", listen_addr="172.18.0.1:5432", machine_id="4150b6543c254de9bf577a", proxy="dockerd"} 1
container_net_tcp_listen_info{container_id="/docker/postgres", instance="10.100.10.111:9887", job="Node-agent_coroot", listen_addr="172.19.0.1:5432", machine_id="4150b6543c254de9bf577a", proxy="dockerd"} 1
container_net_tcp_listen_info{container_id="/docker/postgres", instance="10.100.10.111:9887", job="Node-agent_coroot", listen_addr="172.20.0.1:5432", machine_id="4150b6543c254de9bf577a", proxy="dockerd"} 1
container_net_tcp_listen_info{container_id="/docker/postgres", instance="10.100.10.111:9887", job="Node-agent_coroot", listen_addr="172.20.0.28:5432", machine_id="4150b6543c254de9bf577a"} 1
container_net_tcp_listen_info{container_id="/docker/postgres", instance="10.100.10.111:9887", job="Node-agent_coroot", listen_addr="172.21.0.1:5432", machine_id="4150b6543c254de9bf577a", proxy="dockerd"} 1
container_net_tcp_listen_info{container_id="/system.slice/docker.service", instance="10.100.10.111:9887", job="Node-agent_coroot", listen_addr="10.100.10.111:5432", machine_id="4150b6543c254de9bf577a} 1
container_net_tcp_listen_info{container_id="/system.slice/docker.service", instance="10.100.10.111:9887", job="Node-agent_coroot", listen_addr="127.0.0.1:5432", machine_id="4150b6543c254de9bf577a} 1
container_net_tcp_listen_info{container_id="/system.slice/docker.service", instance="10.100.10.111:9887", job="Node-agent_coroot", listen_addr="172.17.0.1:5432", machine_id="4150b6543c254de9bf577a} 1
container_net_tcp_listen_info{container_id="/system.slice/docker.service", instance="10.100.10.111:9887", job="Node-agent_coroot", listen_addr="172.18.0.1:5432", machine_id="4150b6543c254de9bf577a} 1
container_net_tcp_listen_info{container_id="/system.slice/docker.service", instance="10.100.10.111:9887", job="Node-agent_coroot", listen_addr="172.19.0.1:5432", machine_id="4150b6543c254de9bf577a} 1
container_net_tcp_listen_info{container_id="/system.slice/docker.service", instance="10.100.10.111:9887", job="Node-agent_coroot", listen_addr="172.20.0.1:5432", machine_id="4150b6543c254de9bf577a} 1
container_net_tcp_listen_info{container_id="/system.slice/docker.service", instance="10.100.10.111:9887", job="Node-agent_coroot", listen_addr="172.21.0.1:5432", machine_id="4150b6543c254de9bf577a} 1
container_net_tcp_listen_info{container_id="/system.slice/docker.service", instance="10.100.10.111:9887", job="Node-agent_coroot", listen_addr="[::1]:5432", machine_id="4150b6543c254de9bf577a} 1

[def]

It looks like a bug in Coroot, will try to reproduce it

[def]

@doonydoo, can you please update Coroot to version 0.10.1 and verify if this issue has been fixed?

[doonydoo]

Thank you, Nikolay, now it works!

But still have no infoirmation from pg-agent about locks metrics: pg_lock_awaiting_queries, pg_wal_receiver_status, pg_wal_replay_paused, pg_wal_receive_lsn, pg_wal_reply_lsn (I use PostgreSQL 11 version).

Is it possible to get the graph like this in my UI?

[def]

The pg_lock_awaiting_queries metric is reported only if the agent has detected blocked queries, so it seems that there have not been any such situations yet in your case.

The pg_wal_receiver_status, pg_wal_replay_paused, pg_wal_receive_lsn, pg_wal_reply_lsn metrics are gathered only if the Postgres server is a replica. It it true for your case?
If the instance is the master, only pg_wal_current_lsn is gathered.

[doonydoo]

@def Thank you for your help and for the explanation!