From d66bcb9409c762d3cf171c1593d97b8509748251 Mon Sep 17 00:00:00 2001 From: HereThereBeDragons Date: Fri, 20 Jan 2023 17:53:35 +0100 Subject: [PATCH] telemetry aggregator doc --- apx-parameters.rst | 7 +++++ cpt-configure.rst | 5 +-- cpt-telemetry.rst | 78 ++++++++++++++++++++++++++++++++++++++++++++++ part-advanced.rst | 1 + 4 files changed, 89 insertions(+), 2 deletions(-) create mode 100644 cpt-telemetry.rst diff --git a/apx-parameters.rst b/apx-parameters.rst index e5c53ee..0a81115 100644 --- a/apx-parameters.rst +++ b/apx-parameters.rst @@ -62,6 +62,11 @@ CVMFS_INITIAL_GENERATION Initial inode generation. Used for testing. CVMFS_INSTRUMENT_FUSE | When set to *true* gather performance statistics about the FUSE callbacks. | The results are displayed with `cvmfs_talk internal affairs`. CVMFS_NFS_INTERLEAVED_INODES In NFS mode, use only inodes of the form :math:`an+b`, specified as "b%a". +CVMFS_INFLUX_EXTRA_FIELDS Static fields always attached to the (absolute) output of the InfluxDB Telemetry Aggregator +CVMFS_INFLUX_EXTRA_TAGS Static tags always attached to the (absolute + delta) output of the InfluxDB Telemetry Aggregator +CVMFS_INFLUX_HOST Host name or IP address of the receiver of the InfluxDB Telemetry Aggregator +CVMFS_INFLUX_METRIC_NAME Name of the measurement of the InfluxDB Telemetry Aggregator +CVMFS_INFLUX_PORT Port of the host (receiver) of the InfluxDB Telemetry Aggregator CVMFS_IPFAMILY_PREFER Which IP protocol to prefer when connecting to proxies. Can be either 4 or 6. CVMFS_KCACHE_TIMEOUT Timeout in seconds for path names and file attributes in the kernel file system buffers. CVMFS_KEYS_DIR | Directory containing \*.pub files used as repository signing keys. @@ -113,6 +118,8 @@ CVMFS_SYSLOG_LEVEL | If set to 1 or 2, sets the syslog level for Ce | LOG_DEBUG or LOG_INFO respectively. CVMFS_SYSTEMD_NOKILL | If set to *yes*, modify the command line to ``@vmfs2 ...`` in order to | act as a systemd lowlevel storage manager. +CVMFS_TELEMETRY_RATE Rate in seconds for Telemetry Aggregator to send the telemetry. Minimum send rate >= 5 sec. +CVMFS_TELEMETRY_SEND ``ON`` to activate Telemetry Aggregator. CVMFS_TIMEOUT Timeout in seconds for HTTP requests with a proxy server. CVMFS_TIMEOUT_DIRECT Timeout in seconds for HTTP requests without a proxy server. CVMFS_TRACEFILE If set, enables the tracer and trace file system calls to the given file. diff --git a/cpt-configure.rst b/cpt-configure.rst index 12a8273..84d5663 100644 --- a/cpt-configure.rst +++ b/cpt-configure.rst @@ -1195,8 +1195,9 @@ the client experiences cache thrashing. cvmfs_talk internal affairs -prints the internal status information and performance counters. It can -be helpful for performance engineering. +prints the internal status information and performance counters. +It can be helpful for performance engineering. +They can also be exported in regular intervals (see :ref:`cpt_telemetry`). :: diff --git a/cpt-telemetry.rst b/cpt-telemetry.rst new file mode 100644 index 0000000..045d1fa --- /dev/null +++ b/cpt-telemetry.rst @@ -0,0 +1,78 @@ +.. _cpt_telemetry: + +Client Telemetry Aggregators +============================ + +It is possible to configure the client to send in regular intervals the performance counters listed by ``cvmfs_talk internal affairs``. +By default, an aggregator is available that exposes the counters in InfluxDB data format. +It can easily be replaced by any other aggregator in a form of a source code plugin. + +Independent of the aggregator following 2 client parameters must be set: +:: + + CVMFS_TELEMETRY_SEND=ON + CVMFS_TELEMETRY_RATE= # minimum send rate >= 5 sec + + +Influx Telemetry Aggregator +--------------------------- + +The Influx Telemetry Aggregator sends per timestamp two versions of the counters: +their absolute values and the delta between two timestamps to a socket. +For this, the measurement name given by ``CVMFS_INFLUX_METRIC_NAME`` is extended with either ``_absolute`` or ``_delta``. + +Mandatory client parameters for the Influx Telemetry Aggregator are + +:: + + CVMFS_INFLUX_HOST=localhost # IP address + CVMFS_INFLUX_PORT=8092 # Port + CVMFS_INFLUX_METRIC_NAME= # "Table" name + +And optional parameters are + +:: + + CVMFS_INFLUX_EXTRA_TAGS="some_tag=42,some_tag2=27" # always included + CVMFS_INFLUX_EXTRA_FIELDS="somefield=3" # not included in delta + +The general layout of the data send is + +:: + + # for absolute + CVMFS_INFLUX_METRIC_NAME_absolute,repo=@fqrn,CVMFS_INFLUX_EXTRA_TAGS countername=value,...,CVMFS_INFLUX_EXTRA_FIELDS timestamp + + # for delta (no CVMFS_INFLUX_EXTRA_FIELDS) + CVMFS_INFLUX_METRIC_NAME_delta,repo=@fqrn,CVMFS_INFLUX_EXTRA_TAGS countername=value_new - value_old,... timestamp + + + +.. warning:: + In the output, counters are only included if they have been used at least once (value != 0). + And for the very first measurement no delta values are available. + +Writing Your Own Aggregator +--------------------------- + +The ``TelemetryAggregator`` base class consists of a loop that for each time step +snapshots the counters (saved to ``counters_``), and calls ``PushMetrics()``. +``PushMetrics()`` needs to be overwritten by your own aggregator to perform all manipulations +needed for the counters and the sending/storing of the counters. + +To write your own aggregator you need the following parts: + +* Your aggregator must inherit from ``TelemetryAggregator`` +* Your aggregator's constructor must take care of additional client parameters needed. + In case your object is incorrectly constructed, ``is_zombie_`` **MUST** be set to ``true``. +* Your aggregator must overwrite ``PushMetrics()`` +* Create a new value for your aggregator in enum ``TelemetrySelector`` +* Add your aggregator inside the ``Create()`` of ``TelemetryAggregator`` using the newly created value of ``TelemetrySelector`` +* Change in ``mountpoint.cc`` the ``TelemetrySelector`` used in ``perf::TelemetryAggregator::Create`` + +.. note:: + + Please feel free to contribute your aggregator to the CVMFS project, so we can expand the number of + available aggregators to all users. + + diff --git a/part-advanced.rst b/part-advanced.rst index 770a60a..d776877 100644 --- a/part-advanced.rst +++ b/part-advanced.rst @@ -6,6 +6,7 @@ Advanced Topics :maxdepth: 2 cpt-plugins + cpt-telemetry cpt-tracer cpt-enter cpt-hpc