Initial metrics support based on Prometheus (#1571)

* Initial metrics support based on Prometheus

This commit introduces initial support for metrics in open5gs.

The metrics code is added as libogsmetrics (lib/metrics/), with a well
defined opaque API to manage different types of metrics, allowing for
different implementations for different technologies to scrap the
metrics (placed as lib/metrics/<impl>/. The implementation is right now
selected at build time, in order to be able to opt-out the related dependencies
for users not interested in the features. 2 implementations are already
provided in this commit to start with:
* void: Default implementation. Empty stubs, acts as a NOOP.
* prometheus: open5gs processes become Prometheus servers, offering
  states through an http server to the Prometheus scrappers. Relies on
  libprom (prometheus-client-ci [1] project) to track the metrics and format
  them during export, and libmicrohttpd to make the export possible through
  HTTP.

[1] https://github.com/digitalocean/prometheus-client-c

The prometheus-client-c is not well maintained nowadays in upstream, and
furthermore it uses a quite peculiar mixture of build systems (autolib
on the main dir, cmake for libprom in a subdir). This makes it difficult
to have it widely available in distros, and difficult to find it if it
is installed in the system. Hence, the best is to include it as a
meson subproject like we already do for freeDiameter. An open5gs fork is
requried in order to have an extra patch adding a top-level
CMakeList.txt in order to be able to includ eit from open5gs's meson
build. Furthermore, this allows adding bugfixes to the subproject if any
are found in the future.

* [SMF] Initial metrics support

* [SMF] Add metrics at gtp_node level

* docs: Add tutorial documenting metrics with Prometheus

.gitignore

@@ -1,6 +1,7 @@
 # This directory is fetched during first build and is present in this directory
 subprojects/freeDiameter
 subprojects/libtins
+subprojects/prometheus-client-c
 subprojects/usrsctp
 
 webui/.next

configs/open5gs/smf.yaml.in

@@ -563,3 +563,17 @@ max:
 #    handover:
 #        duration: 500
 time:
+
+#
+# metrics:
+#
+#  <Metrics Server>
+#
+#  o Metrics Server(http://<any address>:9090)
+#    metrics:
+#      addr: 0.0.0.0
+#      port: 9090
+#
+metrics:
+    addr: 0.0.0.0
+    port: 9090

docs/_docs/tutorial/04-metrics-prometheus.md

@@ -0,0 +1,134 @@
+---
+title: Metrics with Prometheus
+---
+
+#### 0. Introduction
+
+This tutorial explains how to export open5gs metrics to Prometheus, which can in
+turn be used to visualize or export them to other systems such as Grafana or
+StatsD.
+
+When this method is used, any open5gs program exporting metrics becomes a
+Prometheus server, which is basically an HTTP server serving Prometheus data to
+the Prometheus scrapper.
+
+Note: Only open5gs-smfd supports exporting metrics so far, though other may
+hopefully follow soon.
+
+#### 1. Enable Prometheus support during build
+
+Open5GS programs use a generic internal API available in libogsmetrics. This
+library implements the API based on configuration passed during open5gs build
+time. By default, the library will be built using the `void` implementation,
+which is basically a NO-OP implementation.
+
+In order to use the Prometheus, the `prometheus` metrics implementation needs to
+be selected at build time:
+
+```
+meson configure -Dmetrics_impl=prometheus build
+```
+
+This will enable building the implementation under lib/metrics/prometheus/,
+which uses:
+
+* prometheus-client-c project (libprom): To generate the Prometheus expected
+  output format of the metrics
+* libmicrohttpd: To server the content generated by libprom as an HTTP server
+
+The `prometheus-client-c` project is not currently well maintained, and uses a
+weird mixture of build systems, which makes it difficult to make it available in
+most Linux distributions. As a result, a fork of the project is available under
+Open5GS GitHub namespace, with an extra patch applied making it possible to
+include it as a subproject, which will be fetched and built automatically when
+building the prometheus libmetrics implementation.
+
+#### 2. Configuring for runtime
+
+By default the created Prometheus HTTP server will be listening on `0.0.0.0`
+port `9090`.
+This can be configured under the following config file options:
+
+```
+#
+# metrics:
+#
+#  <Metrics Server>
+#
+#  o Metrics Server(http://<any address>:9090)
+#    metrics:
+#      addr: 0.0.0.0
+#      port: 9090
+#
+metrics:
+    addr: 0.0.0.0
+    port: 9090
+```
+
+Note: You may want to change the default IP address or port if you are running
+the Prometheus scrapper in the same host, since it will also spawn its own
+Prometheus server also in port 9090, which will collide.
+
+#### 3. Manual visualization
+
+Simply open the web browser at the following URL (changing IP address and port
+as configured in previous section):
+```
+http://127.0.0.1:9090/metrics
+```
+
+Note: URL `metrics/` (with a slash at the end) will not work.
+
+You should see some output similar to this one below:
+```
+# HELP ues_active Active User Equipments
+# TYPE ues_active gauge
+ues_active 2
+
+# HELP process_max_fds Maximum number of open file descriptors.
+# TYPE process_max_fds gauge
+process_max_fds 1024
+
+# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
+# TYPE process_virtual_memory_max_bytes gauge
+process_virtual_memory_max_bytes -1
+
+# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
+# TYPE process_cpu_seconds_total gauge
+process_cpu_seconds_total 0
+
+# HELP process_virtual_memory_bytes Virtual memory size in bytes.
+# TYPE process_virtual_memory_bytes gauge
+process_virtual_memory_bytes 3156643840
+
+# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
+# TYPE process_start_time_seconds gauge
+process_start_time_seconds 402433
+
+# HELP process_open_fds Number of open file descriptors.
+# TYPE process_open_fds gauge
+process_open_fds 23
+```
+
+#### 3. Integration with Prometheus scrapper
+
+Sample Prometheus scrapper configuration (`~/prometheus.yml`):
+```
+global:
+  scrape_interval: 10s
+
+scrape_configs:
+  - job_name: open5gs-smfd
+    static_configs:
+      - targets: ["192.168.1.140:9091"]
+```
+
+Where `192.168.1.140:9091` is the IP address and port where `open5gs-smfd` is
+serving its metrics, as configured in above sections.
+
+The Prometheus scrapper can be easily started from a docker container:
+```
+docker run -p 9090:9090 -v  /prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus
+```
+
+Then open your browser to be able to visualize the data: `http://localhost:9090/graph`

docs/_pages/docs.md

@@ -13,6 +13,7 @@ head_inline: "<style> ul { padding-bottom: 1em; } </style>"
   - [Your First LTE](tutorial/01-your-first-lte)
   - [VoLTE Setup with Kamailio IMS and Open5GS](tutorial/02-VoLTE-setup)
   - [Dockerized VoLTE Setup](tutorial/03-VoLTE-dockerized)
+  - [Metrics with Prometheus](tutorial/04-metrics-prometheus)
 
 - Troubleshooting
   - [Simple Issues](troubleshoot/01-simple-issues)
@@ -26,10 +27,10 @@ head_inline: "<style> ul { padding-bottom: 1em; } </style>"
   - [MacOSX(Intel)](platform/06-macosx-intel)
   - [FreeBSD](platform/07-freebsd)
   - [Alpine](platform/08-alpine)
-  
+
 - Hardware Specific Notes
   - [eNodeBs/gNodeBs tested on Open5GS](hardware/01-genodebs)
-  
+
 - @infinitydon
   - [Open5GS on Amazon Elastic Kubernetes Service](https://aws.amazon.com/blogs/opensource/open-source-mobile-core-network-implementation-on-amazon-elastic-kubernetes-service/)
   - [Kubernetes Open5GS Deployment](https://dev.to/infinitydon/virtual-4g-simulation-using-kubernetes-and-gns3-3b7k?fbclid=IwAR1p99h13a-mCfejanbBQe0H0-jp5grXkn5mWf1WrTHf47UtegB2-UHGGZQ)

lib/app/ogs-context.c

@@ -226,9 +226,12 @@ static void app_context_prepare(void)
      */
     self.time.handover.duration = ogs_time_from_msec(300);
 
+    /* Size of internal metrics pool (amount of ogs_metrics_spec_t) */
+    self.metrics.max_specs = 512;
+
     regenerate_all_timer_duration();
 }
- 
+
 static int app_context_validation(void)
 {
     if (self.parameter.no_ipv4 == 1 &&

lib/app/ogs-context.h

@@ -175,6 +175,10 @@ typedef struct ogs_app_context_s {
         } handover;
 
     } time;
+
+    struct metrics {
+        uint64_t max_specs;
+    } metrics;
 } ogs_app_context_t;
 
 int ogs_app_context_init(void);

lib/gtp/context.h

@@ -65,6 +65,7 @@ typedef struct ogs_gtp_context_s {
  * PGW gateway. Some of members may not be used by the specific type of node */
 typedef struct ogs_gtp_node_s {
     ogs_lnode_t     node;           /* A node of list_t */
+    void            *data_ptr;      /* Can be used by app */
 
     ogs_sockaddr_t  *sa_list;       /* Socket Address List Candidate */
 

lib/meson.build

@@ -21,6 +21,7 @@ subdir('core')
 subdir('ipfw')
 subdir('crypt')
 subdir('app')
+subdir('metrics')
 subdir('sctp')
 subdir('tun')
 subdir('dbi')

lib/metrics/context.c

@@ -0,0 +1,22 @@
+/*
+ * Copyright (C) 2022 by sysmocom - s.f.m.c. GmbH <info@sysmocom.de>
+ *
+ * This file is part of Open5GS.
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+#include "ogs-metrics.h"
+
+int __ogs_metrics_domain;

lib/metrics/context.h

@@ -0,0 +1,72 @@
+/*
+ * Copyright (C) 2022 by sysmocom - s.f.m.c. GmbH <info@sysmocom.de>
+ *
+ * This file is part of Open5GS.
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+#if !defined(OGS_METRICS_INSIDE) && !defined(OGS_METRICS_COMPILATION)
+#error "This header cannot be included directly."
+#endif
+
+#ifndef OGS_METRICS_CONTEXT_H
+#define OGS_METRICS_CONTEXT_H
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+typedef enum ogs_metrics_metric_type_s  {
+    OGS_METRICS_METRIC_TYPE_COUNTER,
+    OGS_METRICS_METRIC_TYPE_GAUGE,
+} ogs_metrics_metric_type_t;
+
+typedef struct ogs_metrics_context_s ogs_metrics_context_t;
+void ogs_metrics_context_init(void);
+void ogs_metrics_context_open(ogs_metrics_context_t *ctx);
+void ogs_metrics_context_close(ogs_metrics_context_t *ctx);
+void ogs_metrics_context_final(void);
+ogs_metrics_context_t *ogs_metrics_self(void);
+int ogs_metrics_context_parse_config(void);
+
+typedef struct ogs_metrics_spec_s ogs_metrics_spec_t;
+ogs_metrics_spec_t *ogs_metrics_spec_new(
+        ogs_metrics_context_t *ctx, ogs_metrics_metric_type_t type,
+        const char *name, const char *description,
+        int initial_val, unsigned int num_labels, const char **labels);
+void ogs_metrics_spec_free(ogs_metrics_spec_t *spec);
+
+typedef struct ogs_metrics_inst_s ogs_metrics_inst_t;
+ogs_metrics_inst_t *ogs_metrics_inst_new(
+        ogs_metrics_spec_t *spec,
+        unsigned int num_labels, const char **label_values);
+void ogs_metrics_inst_free(ogs_metrics_inst_t *inst);
+void ogs_metrics_inst_set(ogs_metrics_inst_t *inst, int val);
+void ogs_metrics_inst_reset(ogs_metrics_inst_t *inst);
+void ogs_metrics_inst_add(ogs_metrics_inst_t *inst, int val);
+static inline void ogs_metrics_inst_inc(ogs_metrics_inst_t *inst)
+{
+    ogs_metrics_inst_add(inst, 1);
+}
+static inline void ogs_metrics_inst_dec(ogs_metrics_inst_t *inst)
+{
+    ogs_metrics_inst_add(inst, -1);
+}
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* OGS_METRICS_CONTEXT_H */

lib/metrics/meson.build

@@ -0,0 +1,66 @@
+# Copyright (C) 2022 by sysmocom - s.f.m.c. GmbH <info@sysmocom.de>
+
+# This file is part of Open5GS.
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+libmetrics_file_list = '''
+    ogs-metrics.h
+    context.h
+    context.c
+'''
+libmetrics_dependencies = [libcore_dep, libapp_dep]
+
+
+metrics_impl_optval = get_option('metrics_impl')
+
+if metrics_impl_optval == 'prometheus'
+    # Note: This requires meson >= 0.51.0:
+    # 0.47.0: {'check arg in run_command'}
+    # 0.50.0: {'CMake Module'}
+    # 0.51.0: {'subproject'}
+
+    libmicrohttpd_dep = dependency('libmicrohttpd', version: '>=0.9.40')
+
+    cmake = import('cmake')
+    prometheus_client_c_proj = cmake.subproject('prometheus-client-c')
+    # generated cmake subproject seems to include
+    # open5gs/subprojects/prometheus-client-c/__CMake_build as include, which
+    # doesn't exist and fail:
+    missing_include_dir = join_paths(meson.current_source_dir(), '../../subprojects/prometheus-client-c/__CMake_build')
+    run_command('mkdir', '-p', missing_include_dir, check: true)
+    libprom_dep = prometheus_client_c_proj.dependency('prom')
+
+    libmetrics_dependencies = libmetrics_dependencies + [libprom_dep, libmicrohttpd_dep]
+    libmetrics_file_list = libmetrics_file_list + ' prometheus/context.c'
+else
+   libmetrics_file_list = libmetrics_file_list + ' void/context.c'
+endif
+
+libmetrics_sources = files(libmetrics_file_list.split())
+
+libmetrics_inc = include_directories('.')
+
+libmetrics = library('ogsmetrics',
+    sources : libmetrics_sources,
+    version : libogslib_version,
+    c_args : '-DOGS_METRICS_COMPILATION',
+    include_directories : [libmetrics_inc, libinc],
+    dependencies : libmetrics_dependencies,
+    install : true)
+
+libmetrics_dep = declare_dependency(
+    link_with : libmetrics,
+    include_directories : [libmetrics_inc, libinc],
+    dependencies : libmetrics_dependencies)