Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Request to support openvswitch and ovs-db-server on separate hosts #9

Open
xavpaice opened this issue Nov 8, 2021 · 4 comments
Open

Request to support openvswitch and ovs-db-server on separate hosts #9

xavpaice opened this issue Nov 8, 2021 · 4 comments

Comments

@xavpaice
Copy link

xavpaice commented Nov 8, 2021

In the OpenStack environments deploy by Canonical's Juju charms, there is typically a cluster of 'ovn-central' hosts which run ovn-northd, ovn-ovsdb-server-nb and ovn-ovsdb-server-sb. On the compute hosts themselves, we run ovn-controller and openvswitch.

When I run ovn-exporter on the compute hosts the collector reports:

# ./ovn-exporter -database.northbound.file.data.path /var/lib/ovn/ovnnb_db.db -database.northbound.file.log.path /var/log/ovn/ovsdb-server-nb.log -database.northbound.file.pid.path /var/run/ovn/ovnnb_db.pid -database.northbound.socket.control unix:/var/run/ovn/ovnnb_db.ctl -database.northbound.socket.remote unix:/var/run/ovn/ovnnb_db.sock -database.southbound.file.data.path /var/lib/ovn/ovnsb_db.db -database.southbound.file.log.path /var/log/ovn/ovsdb-server-sb.log -database.southbound.file.pid.path /var/run/ovn/ovnsb_db.pid -database.southbound.socket.control unix:/var/run/ovn/ovnsb_db.ctl -database.southbound.socket.remote unix:/var/run/ovn/ovnsb_db.sock -service.ovn.northd.file.log.path /var/log/ovn/ovn-northd.log -service.ovn.northd.file.pid.path /var/run/ovn/ovn-northd.pid -log.level debug
INFO[0000] Starting ovn-exporter (version=1.0.3, branch=main, revision=v1.0.3-2-g3882334)  source="main.go:122"
INFO[0000] Build context (go=go1.17.3, user=xav, date=2021-11-08)  source="main.go:123"
DEBU[0000] unknown: NewExporter() calls Connect()        source="ovn_exporter.go:301"
ERRO[0000] ovn-exporter failed to finalize exporter calls properly: [failed connecting to OVN_Northbound via unix:/var/run/ovn/ovnnb_db.sock: dial unix /var/run/ovn/ovnnb_db.sock: connect: no such file or directory failed connecting to OVN_Southbound via unix:/var/run/ovn/ovnsb_db.sock: dial unix /var/run/ovn/ovnsb_db.sock: connect: no such file or directory]  source="main.go:167"
INFO[0000] OVS system-id: unknown                        source="main.go:170"
INFO[0000] Listening on :9476                            source="main.go:185"
DEBU[0014] unknown: GatherMetrics() called               source="ovn_exporter.go:410"
DEBU[0014] unknown: GatherMetrics() locked               source="ovn_exporter.go:415"
ERRO[0014] Open_vSwitch: system-id is not 32 characters in length, but 37  source="ovn_exporter.go:428"
DEBU[0014] unknown: GatherMetrics() calls GetProcessInfo(ovsdb-server)  source="ovn_exporter.go:447"
DEBU[0014] unknown: GatherMetrics() completed GetProcessInfo(ovsdb-server)  source="ovn_exporter.go:462"
DEBU[0014] unknown: GatherMetrics() calls GetProcessInfo(ovsdb-server-southbound)  source="ovn_exporter.go:447"
ERRO[0014] ovsdb-server-southbound: pid-open /var/run/ovn/ovnsb_db.pid: no such file or directory  source="ovn_exporter.go:449"
DEBU[0014] unknown: GatherMetrics() completed GetProcessInfo(ovsdb-server-southbound)  source="ovn_exporter.go:462"
DEBU[0014] unknown: GatherMetrics() calls GetProcessInfo(ovsdb-server-southbound-monitoring)  source="ovn_exporter.go:447"
DEBU[0014] unknown: GatherMetrics() completed GetProcessInfo(ovsdb-server-southbound-monitoring)  source="ovn_exporter.go:462"
DEBU[0014] unknown: GatherMetrics() calls GetProcessInfo(ovsdb-server-northbound)  source="ovn_exporter.go:447"
ERRO[0014] ovsdb-server-northbound: pid-open /var/run/ovn/ovnnb_db.pid: no such file or directory  source="ovn_exporter.go:449"
DEBU[0014] unknown: GatherMetrics() completed GetProcessInfo(ovsdb-server-northbound)  source="ovn_exporter.go:462"
DEBU[0014] unknown: GatherMetrics() calls GetProcessInfo(ovsdb-server-northbound-monitoring)  source="ovn_exporter.go:447"
DEBU[0014] unknown: GatherMetrics() completed GetProcessInfo(ovsdb-server-northbound-monitoring)  source="ovn_exporter.go:462"
DEBU[0014] unknown: GatherMetrics() calls GetProcessInfo(ovn-northd)  source="ovn_exporter.go:447"
ERRO[0014] ovn-northd: pid-open /var/run/ovn/ovn-northd.pid: no such file or directory  source="ovn_exporter.go:449"
DEBU[0014] unknown: GatherMetrics() completed GetProcessInfo(ovn-northd)  source="ovn_exporter.go:462"
DEBU[0014] unknown: GatherMetrics() calls GetProcessInfo(ovn-northd-monitoring)  source="ovn_exporter.go:447"
DEBU[0014] unknown: GatherMetrics() completed GetProcessInfo(ovn-northd-monitoring)  source="ovn_exporter.go:462"
DEBU[0014] unknown: GatherMetrics() calls GetProcessInfo(ovs-vswitchd)  source="ovn_exporter.go:447"
DEBU[0014] unknown: GatherMetrics() completed GetProcessInfo(ovs-vswitchd)  source="ovn_exporter.go:462"
DEBU[0014] unknown: GatherMetrics() calls GetLogFileInfo(ovsdb-server)  source="ovn_exporter.go:473"
DEBU[0014] unknown: GatherMetrics() completed GetLogFileInfo(ovsdb-server)  source="ovn_exporter.go:480"
DEBU[0014] unknown: GatherMetrics() calls GetLogFileEventStats(ovsdb-server)  source="ovn_exporter.go:489"
DEBU[0014] unknown: GatherMetrics() completed GetLogFileEventStats(ovsdb-server)  source="ovn_exporter.go:496"
DEBU[0014] unknown: GatherMetrics() calls GetLogFileInfo(ovsdb-server-southbound)  source="ovn_exporter.go:473"
ERRO[0014] ovsdb-server-southbound: log-file-stat /var/log/ovn/ovsdb-server-sb.log: no such file or directory  source="ovn_exporter.go:476"
DEBU[0014] unknown: GatherMetrics() calls GetLogFileInfo(ovsdb-server-northbound)  source="ovn_exporter.go:473"
ERRO[0014] ovsdb-server-northbound: log-file-stat /var/log/ovn/ovsdb-server-nb.log: no such file or directory  source="ovn_exporter.go:476"
DEBU[0014] unknown: GatherMetrics() calls GetLogFileInfo(ovn-northd)  source="ovn_exporter.go:473"
ERRO[0014] ovn-northd: log-file-stat /var/log/ovn/ovn-northd.log: no such file or directory  source="ovn_exporter.go:476"
DEBU[0014] unknown: GatherMetrics() calls GetLogFileInfo(ovs-vswitchd)  source="ovn_exporter.go:473"
DEBU[0014] unknown: GatherMetrics() completed GetLogFileInfo(ovs-vswitchd)  source="ovn_exporter.go:480"
DEBU[0014] unknown: GatherMetrics() calls GetLogFileEventStats(ovs-vswitchd)  source="ovn_exporter.go:489"
DEBU[0014] unknown: GatherMetrics() completed GetLogFileEventStats(ovs-vswitchd)  source="ovn_exporter.go:496"
DEBU[0014] unknown: GatherMetrics() calls GetChassis()   source="ovn_exporter.go:512"
ERRO[0014] OVN_Southbound: OVN_Southbound: 'Chassis' table error: 'transact' method, query: 'SELECT _uuid, name, encaps FROM Chassis' failed: client unavailable  source="ovn_exporter.go:514"
DEBU[0014] unknown: GatherMetrics() completed GetChassis()  source="ovn_exporter.go:530"
DEBU[0014] unknown: GatherMetrics() calls GetLogicalSwitches()  source="ovn_exporter.go:532"
ERRO[0014] OVN_Southbound: OVN_Northbound: 'Logical_Switch' table error: 'transact' method, query: 'SELECT _uuid, external_ids, name, ports FROM Logical_Switch' failed: client unavailable  source="ovn_exporter.go:535"
DEBU[0014] unknown: GatherMetrics() completed GetLogicalSwitches()  source="ovn_exporter.go:589"
DEBU[0014] unknown: GatherMetrics() calls GetLogicalSwitchPorts()  source="ovn_exporter.go:591"
ERRO[0014] OVN_Southbound: OVN_Northbound: 'Logical_Switch_Port' table error: 'transact' method, query: 'SELECT _uuid, addresses, external_ids, name, up FROM Logical_Switch_Port' failed: client unavailable  source="ovn_exporter.go:594"
DEBU[0014] unknown: GatherMetrics() completed GetLogicalSwitchPorts()  source="ovn_exporter.go:641"
DEBU[0014] unknown: GatherMetrics() calls AppListCommands(ovsdb-server)  source="ovn_exporter.go:653"
DEBU[0014] unknown: GatherMetrics() completed AppListCommands(ovsdb-server)  source="ovn_exporter.go:659"
DEBU[0014] unknown: GatherMetrics() calls GetAppCoverageMetrics(ovsdb-server)  source="ovn_exporter.go:661"
DEBU[0014] unknown: GatherMetrics() completed GetAppCoverageMetrics(ovsdb-server)  source="ovn_exporter.go:692"
DEBU[0014] unknown: GatherMetrics() calls GetAppMemoryMetrics(ovsdb-server)  source="ovn_exporter.go:695"
DEBU[0014] unknown: GatherMetrics() completed GetAppMemoryMetrics(ovsdb-server)  source="ovn_exporter.go:711"
DEBU[0014] unknown: GatherMetrics() calls AppListCommands(ovsdb-server-southbound)  source="ovn_exporter.go:653"
ERRO[0014] ovsdb-server-southbound: failed 'list-commands' from ovsdb-server-southbound: dial unix /var/run/ovn/ovnsb_db.ctl: connect: no such file or directory  source="ovn_exporter.go:655"
DEBU[0014] unknown: GatherMetrics() completed AppListCommands(ovsdb-server-southbound)  source="ovn_exporter.go:657"
DEBU[0014] unknown: GatherMetrics() calls AppListCommands(ovsdb-server-northbound)  source="ovn_exporter.go:653"
ERRO[0014] ovsdb-server-northbound: failed 'list-commands' from ovsdb-server-northbound: dial unix /var/run/ovn/ovnnb_db.ctl: connect: no such file or directory  source="ovn_exporter.go:655"
DEBU[0014] unknown: GatherMetrics() completed AppListCommands(ovsdb-server-northbound)  source="ovn_exporter.go:657"
DEBU[0014] unknown: GatherMetrics() calls IsDefaultPortUp(ovsdb-server-southbound)  source="ovn_exporter.go:936"
DEBU[0014] unknown: GatherMetrics() completed IsDefaultPortUp(ovsdb-server-southbound)  source="ovn_exporter.go:950"
DEBU[0014] unknown: GatherMetrics() calls IsSslPortUp(ovsdb-server-southbound)  source="ovn_exporter.go:951"
DEBU[0014] unknown: GatherMetrics() completed IsSslPortUp(ovsdb-server-southbound)  source="ovn_exporter.go:965"
DEBU[0014] unknown: GatherMetrics() calls IsDefaultPortUp(ovsdb-server-northbound)  source="ovn_exporter.go:936"
DEBU[0014] unknown: GatherMetrics() completed IsDefaultPortUp(ovsdb-server-northbound)  source="ovn_exporter.go:950"
DEBU[0014] unknown: GatherMetrics() calls IsSslPortUp(ovsdb-server-northbound)  source="ovn_exporter.go:951"
DEBU[0014] unknown: GatherMetrics() completed IsSslPortUp(ovsdb-server-northbound)  source="ovn_exporter.go:965"
DEBU[0014] unknown: GatherMetrics() returns              source="ovn_exporter.go:1017"
DEBU[0014] unknown: Collect() calls RLock()              source="ovn_exporter.go:369"
DEBU[0014] unknown: Collect() sends 61 metrics to a shared channel  source="ovn_exporter.go:401"

Running on the 'ovn-central' hosts:

# ./ovn-exporter -database.northbound.file.data.path /var/lib/ovn/ovnnb_db.db -database.northbound.file.log.path /var/log/ovn/ovsdb-server-nb.log -database.northbound.file.pid.path /var/run/ovn/ovnnb_db.pid -database.northbound.socket.control unix:/var/run/ovn/ovnnb_db.ctl -database.northbound.socket.remote unix:/var/run/ovn/ovnnb_db.sock -database.southbound.file.data.path /var/lib/ovn/ovnsb_db.db -database.southbound.file.log.path /var/log/ovn/ovsdb-server-sb.log -database.southbound.file.pid.path /var/run/ovn/ovnsb_db.pid -database.southbound.socket.control unix:/var/run/ovn/ovnsb_db.ctl -database.southbound.socket.remote unix:/var/run/ovn/ovnsb_db.sock -service.ovn.northd.file.log.path /var/log/ovn/ovn-northd.log -service.ovn.northd.file.pid.path /var/run/ovn/ovn-northd.pid -log.level debug
INFO[0000] Starting ovn-exporter (version=1.0.3, branch=main, revision=v1.0.3-2-g3882334)  source="main.go:122"
INFO[0000] Build context (go=go1.17.3, user=xav, date=2021-11-08)  source="main.go:123"
DEBU[0000] unknown: NewExporter() calls Connect()        source="ovn_exporter.go:301"
ERRO[0000] ovn-exporter failed to finalize exporter calls properly: [failed connecting to Open_vSwitch via unix:/var/run/openvswitch/db.sock: dial unix /var/run/openvswitch/db.sock: connect: no such file or directory]  source="main.go:167"
INFO[0000] OVS system-id: unknown                        source="main.go:170"
INFO[0000] Listening on :9476                            source="main.go:185"
DEBU[0006] unknown: GatherMetrics() called               source="ovn_exporter.go:410"
DEBU[0006] unknown: GatherMetrics() locked               source="ovn_exporter.go:415"
ERRO[0006] Open_vSwitch: open /etc/openvswitch/system-id.conf: no such file or directory  source="ovn_exporter.go:428"
DEBU[0006] unknown: GatherMetrics() calls GetProcessInfo(ovsdb-server)  source="ovn_exporter.go:447"
ERRO[0006] ovsdb-server: pid-open /var/run/openvswitch/ovsdb-server.pid: no such file or directory  source="ovn_exporter.go:449"
DEBU[0006] unknown: GatherMetrics() completed GetProcessInfo(ovsdb-server)  source="ovn_exporter.go:462"
DEBU[0006] unknown: GatherMetrics() calls GetProcessInfo(ovsdb-server-southbound)  source="ovn_exporter.go:447"
DEBU[0006] unknown: GatherMetrics() completed GetProcessInfo(ovsdb-server-southbound)  source="ovn_exporter.go:462"
DEBU[0006] unknown: GatherMetrics() calls GetProcessInfo(ovsdb-server-southbound-monitoring)  source="ovn_exporter.go:447"
DEBU[0006] unknown: GatherMetrics() completed GetProcessInfo(ovsdb-server-southbound-monitoring)  source="ovn_exporter.go:462"
DEBU[0006] unknown: GatherMetrics() calls GetProcessInfo(ovsdb-server-northbound)  source="ovn_exporter.go:447"
DEBU[0006] unknown: GatherMetrics() completed GetProcessInfo(ovsdb-server-northbound)  source="ovn_exporter.go:462"
DEBU[0006] unknown: GatherMetrics() calls GetProcessInfo(ovsdb-server-northbound-monitoring)  source="ovn_exporter.go:447"
DEBU[0006] unknown: GatherMetrics() completed GetProcessInfo(ovsdb-server-northbound-monitoring)  source="ovn_exporter.go:462"
DEBU[0006] unknown: GatherMetrics() calls GetProcessInfo(ovn-northd)  source="ovn_exporter.go:447"
DEBU[0006] unknown: GatherMetrics() completed GetProcessInfo(ovn-northd)  source="ovn_exporter.go:462"
DEBU[0006] unknown: GatherMetrics() calls GetProcessInfo(ovn-northd-monitoring)  source="ovn_exporter.go:447"
DEBU[0006] unknown: GatherMetrics() completed GetProcessInfo(ovn-northd-monitoring)  source="ovn_exporter.go:462"
DEBU[0006] unknown: GatherMetrics() calls GetProcessInfo(ovs-vswitchd)  source="ovn_exporter.go:447"
ERRO[0006] ovs-vswitchd: pid-open /var/run/openvswitch/ovs-vswitchd.pid: no such file or directory  source="ovn_exporter.go:449"
DEBU[0006] unknown: GatherMetrics() completed GetProcessInfo(ovs-vswitchd)  source="ovn_exporter.go:462"
DEBU[0006] unknown: GatherMetrics() calls GetLogFileInfo(ovsdb-server)  source="ovn_exporter.go:473"
ERRO[0006] ovsdb-server: log-file-stat /var/log/openvswitch/ovsdb-server.log: no such file or directory  source="ovn_exporter.go:476"
DEBU[0006] unknown: GatherMetrics() calls GetLogFileInfo(ovsdb-server-southbound)  source="ovn_exporter.go:473"
DEBU[0006] unknown: GatherMetrics() completed GetLogFileInfo(ovsdb-server-southbound)  source="ovn_exporter.go:480"
DEBU[0006] unknown: GatherMetrics() calls GetLogFileEventStats(ovsdb-server-southbound)  source="ovn_exporter.go:489"
DEBU[0006] unknown: GatherMetrics() completed GetLogFileEventStats(ovsdb-server-southbound)  source="ovn_exporter.go:496"
DEBU[0006] unknown: GatherMetrics() calls GetLogFileInfo(ovsdb-server-northbound)  source="ovn_exporter.go:473"
DEBU[0006] unknown: GatherMetrics() completed GetLogFileInfo(ovsdb-server-northbound)  source="ovn_exporter.go:480"
DEBU[0006] unknown: GatherMetrics() calls GetLogFileEventStats(ovsdb-server-northbound)  source="ovn_exporter.go:489"
DEBU[0006] unknown: GatherMetrics() completed GetLogFileEventStats(ovsdb-server-northbound)  source="ovn_exporter.go:496"
DEBU[0006] unknown: GatherMetrics() calls GetLogFileInfo(ovn-northd)  source="ovn_exporter.go:473"
DEBU[0006] unknown: GatherMetrics() completed GetLogFileInfo(ovn-northd)  source="ovn_exporter.go:480"
DEBU[0006] unknown: GatherMetrics() calls GetLogFileEventStats(ovn-northd)  source="ovn_exporter.go:489"
DEBU[0006] unknown: GatherMetrics() completed GetLogFileEventStats(ovn-northd)  source="ovn_exporter.go:496"
DEBU[0006] unknown: GatherMetrics() calls GetLogFileInfo(ovs-vswitchd)  source="ovn_exporter.go:473"
ERRO[0006] ovs-vswitchd: log-file-stat /var/log/openvswitch/ovs-vswitchd.log: no such file or directory  source="ovn_exporter.go:476"
DEBU[0006] unknown: GatherMetrics() calls GetChassis()   source="ovn_exporter.go:512"
DEBU[0006] unknown: GatherMetrics() completed GetChassis()  source="ovn_exporter.go:530"
DEBU[0006] unknown: GatherMetrics() calls GetLogicalSwitches()  source="ovn_exporter.go:532"
ERRO[0006] OVN_Southbound: OVN_Northbound: no switch found  source="ovn_exporter.go:535"
DEBU[0006] unknown: GatherMetrics() completed GetLogicalSwitches()  source="ovn_exporter.go:589"
DEBU[0006] unknown: GatherMetrics() calls GetLogicalSwitchPorts()  source="ovn_exporter.go:591"
ERRO[0006] OVN_Southbound: OVN_Northbound: no logical switch port found  source="ovn_exporter.go:594"
DEBU[0006] unknown: GatherMetrics() completed GetLogicalSwitchPorts()  source="ovn_exporter.go:641"
DEBU[0006] unknown: GatherMetrics() calls AppListCommands(ovsdb-server)  source="ovn_exporter.go:653"
ERRO[0006] ovsdb-server: failed 'list-commands' from ovsdb-server: dial unix /var/run/openvswitch/ovsdb-server.0.ctl: connect: no such file or directory  source="ovn_exporter.go:655"
DEBU[0006] unknown: GatherMetrics() completed AppListCommands(ovsdb-server)  source="ovn_exporter.go:657"
DEBU[0006] unknown: GatherMetrics() calls AppListCommands(ovsdb-server-southbound)  source="ovn_exporter.go:653"
DEBU[0006] unknown: GatherMetrics() completed AppListCommands(ovsdb-server-southbound)  source="ovn_exporter.go:659"
DEBU[0006] unknown: GatherMetrics() calls GetAppCoverageMetrics(ovsdb-server-southbound)  source="ovn_exporter.go:661"
DEBU[0006] unknown: GatherMetrics() completed GetAppCoverageMetrics(ovsdb-server-southbound)  source="ovn_exporter.go:692"
DEBU[0006] unknown: GatherMetrics() calls GetAppMemoryMetrics(ovsdb-server-southbound)  source="ovn_exporter.go:695"
DEBU[0006] unknown: GatherMetrics() completed GetAppMemoryMetrics(ovsdb-server-southbound)  source="ovn_exporter.go:711"
DEBU[0006] unknown: GatherMetrics() calls GetAppClusteringInfo(ovsdb-server-southbound)  source="ovn_exporter.go:714"
DEBU[0006] unknown: GatherMetrics() completed GetAppClusteringInfo(ovsdb-server-southbound)  source="ovn_exporter.go:915"
DEBU[0006] unknown: GatherMetrics() calls AppListCommands(ovsdb-server-northbound)  source="ovn_exporter.go:653"
DEBU[0006] unknown: GatherMetrics() completed AppListCommands(ovsdb-server-northbound)  source="ovn_exporter.go:659"
DEBU[0006] unknown: GatherMetrics() calls GetAppCoverageMetrics(ovsdb-server-northbound)  source="ovn_exporter.go:661"
DEBU[0006] unknown: GatherMetrics() completed GetAppCoverageMetrics(ovsdb-server-northbound)  source="ovn_exporter.go:692"
DEBU[0006] unknown: GatherMetrics() calls GetAppMemoryMetrics(ovsdb-server-northbound)  source="ovn_exporter.go:695"
DEBU[0006] unknown: GatherMetrics() completed GetAppMemoryMetrics(ovsdb-server-northbound)  source="ovn_exporter.go:711"
DEBU[0006] unknown: GatherMetrics() calls GetAppClusteringInfo(ovsdb-server-northbound)  source="ovn_exporter.go:714"
DEBU[0006] unknown: GatherMetrics() completed GetAppClusteringInfo(ovsdb-server-northbound)  source="ovn_exporter.go:915"
DEBU[0006] unknown: GatherMetrics() calls IsDefaultPortUp(ovsdb-server-southbound)  source="ovn_exporter.go:936"
DEBU[0006] unknown: GatherMetrics() completed IsDefaultPortUp(ovsdb-server-southbound)  source="ovn_exporter.go:950"
DEBU[0006] unknown: GatherMetrics() calls IsSslPortUp(ovsdb-server-southbound)  source="ovn_exporter.go:951"
DEBU[0006] unknown: GatherMetrics() completed IsSslPortUp(ovsdb-server-southbound)  source="ovn_exporter.go:965"
DEBU[0006] unknown: GatherMetrics() calls IsRaftPortUp(ovsdb-server-southbound)  source="ovn_exporter.go:968"
DEBU[0006] unknown: GatherMetrics() completed IsRaftPortUp(ovsdb-server-southbound)  source="ovn_exporter.go:982"
DEBU[0006] unknown: GatherMetrics() calls IsDefaultPortUp(ovsdb-server-northbound)  source="ovn_exporter.go:936"
DEBU[0006] unknown: GatherMetrics() completed IsDefaultPortUp(ovsdb-server-northbound)  source="ovn_exporter.go:950"
DEBU[0006] unknown: GatherMetrics() calls IsSslPortUp(ovsdb-server-northbound)  source="ovn_exporter.go:951"
DEBU[0006] unknown: GatherMetrics() completed IsSslPortUp(ovsdb-server-northbound)  source="ovn_exporter.go:965"
DEBU[0006] unknown: GatherMetrics() calls IsRaftPortUp(ovsdb-server-northbound)  source="ovn_exporter.go:968"
DEBU[0006] unknown: GatherMetrics() completed IsRaftPortUp(ovsdb-server-northbound)  source="ovn_exporter.go:982"
DEBU[0006] unknown: GatherMetrics() returns              source="ovn_exporter.go:1017"
DEBU[0006] unknown: Collect() calls RLock()              source="ovn_exporter.go:369"
DEBU[0006] unknown: Collect() sends 168 metrics to a shared channel  source="ovn_exporter.go:401"

Metrics are available on each host, but with errors in the log that may need to be cleared.

Note also that system_id="unknown" on both hosts. On the ovn-central there's no OpenVswitch, but on the compute host the log reports Open_vSwitch: system-id is not 32 characters in length, but 37 source="ovn_exporter.go:428"

FYI, some paths used differ from default (set via CLI flags):

-rw-r----- 1 root root 14602 Nov  7 22:16 /var/lib/ovn/ovnnb_db.db
-rw-r----- 1 root root 23127 Nov  7 22:16 /var/lib/ovn/ovnsb_db.db
-rw-r----- 1 root root 35244 Nov  7 22:16 /var/log/ovn/ovn-northd.log
-rw-r----- 1 root root 18171 Nov  8 01:01 /var/log/ovn/ovsdb-server-nb.log
-rw-r----- 1 root root 18335 Nov  8 01:01 /var/log/ovn/ovsdb-server-sb.log
-rw-r--r-- 1 root root     5 Nov  7 22:16 /var/run/ovn/ovn-northd.pid
srwxr-x--- 1 root root     0 Nov  7 22:16 /var/run/ovn/ovnnb_db.ctl
-rw-r--r-- 1 root root     5 Nov  7 22:16 /var/run/ovn/ovnnb_db.pid
srwxr-x--- 1 root root     0 Nov  7 22:16 /var/run/ovn/ovnnb_db.sock
srwxr-x--- 1 root root     0 Nov  7 22:16 /var/run/ovn/ovnsb_db.ctl
-rw-r--r-- 1 root root     5 Nov  7 22:16 /var/run/ovn/ovnsb_db.pid
srwxr-x--- 1 root root     0 Nov  7 22:16 /var/run/ovn/ovnsb_db.sock

Paths on a compute host other than the above which don't exist, are default.
@greenpau
Copy link
Owner

In the OpenStack environments deploy by Canonical's Juju charms, there is typically a cluster of 'ovn-central' hosts which run ovn-northd, ovn-ovsdb-server-nb and ovn-ovsdb-server-sb. On the compute hosts themselves, we run ovn-controller and openvswitch.

@xavpaice , there is a need to selectively disable the collection of certain things, with flag.BoolVar and --disable-XYZ switch via cmd/ovn_exporter/main.go.

Add a map with features to the options passed to Exporter.

ovn_exporter/pkg/ovn_exporter/ovn_exporter.go

Lines 278 to 280 in 3882334

type Options struct {
Timeout int
}

Add the same map in the Exporter itself.

ovn_exporter/pkg/ovn_exporter/ovn_exporter.go

Lines 267 to 276 in 3882334

type Exporter struct {
sync.RWMutex
Client *ovsdb.OvnClient
timeout int
pollInterval int64
errors int64
errorsLocker sync.RWMutex
nextCollectionTicker int64
metrics []prometheus.Metric
}

Then, modify the below function to disable certain collections.

ovn_exporter/pkg/ovn_exporter/ovn_exporter.go

Lines 409 to 410 in 3882334

func (e *Exporter) GatherMetrics() {
log.Debugf("%s: GatherMetrics() called", e.Client.System.ID)

@fprzewozny
Copy link

Hi, any update here?

@artificial-intelligence
Copy link

artificial-intelligence commented Apr 25, 2024 •
edited
Loading

I guess I'll work on this now, as this is a road blocker for me.

@artificial-intelligence
Copy link

while implementing this I noticed that https://github.com/greenpau/ovsdb/blob/ba2fa6bc5e69ee5511f72847d060e6709036ae0f/ovn.go#L113 connects to all DBs (south- and northbound) and ovs as well and returns an error if it's not successful.

This is used in turn in

ovn_exporter/pkg/ovn_exporter/ovn_exporter.go

Line 321 in 79cb601

client := ovsdb.NewOvnClient()

to establish a new client connection.

I have implemented this as far as possible in ovn-exporter but I think we need to decouple the logic in ovsdb itself as well to make it really work.

artificial-intelligence added a commit to artificial-intelligence/ovn_exporter that referenced this issue May 9, 2024
@artificial-intelligence
feature: add toogles to disable various metrics
87775b0 
this is only partly working because ovsdb.Client
currently is tightly coupled, making connections
to NB, SB and ovs everytime.

Related: greenpau#9

Signed-off-by: Sven Kieske <kieske@osism.tech>
artificial-intelligence added a commit to artificial-intelligence/ovn_exporter that referenced this issue May 9, 2024
@artificial-intelligence
feature: add toggles to disable various metrics
2b2788a 
this is only partly working because ovsdb.Client
currently is tightly coupled, making connections
to NB, SB and ovs everytime.

Related: greenpau#9

Signed-off-by: Sven Kieske <kieske@osism.tech>
artificial-intelligence added a commit to artificial-intelligence/ovn_exporter that referenced this issue May 9, 2024
@artificial-intelligence
feature: add toggles to disable various metrics
8f39bac 
this is only partly working because ovsdb.Client
currently is tightly coupled, making connections
to NB, SB and ovs everytime.

Related: greenpau#9

Signed-off-by: Sven Kieske <kieske@osism.tech>
Open
11 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@greenpau @xavpaice @artificial-intelligence @fprzewozny