Telemetry framework + aggregator for influx #3098

HereThereBeDragons · 2022-11-09T22:29:19Z

Provides:

Framework for source code replacement of telemetry aggregator
telemetry aggregator sending influx data format over socket/udp

Missing

Verification that influx data format is correct
Comments in code
I guess this should have tests?
Documentation how to create your own telemetry aggregator ( telemetry aggregator doc doc-cvmfs#195 )

Client options to enable:

CVMFS_TELEMETRY_SEND=ON  //could be later changed to name of aggregator if multiple are available
CVMFS_TELEMETRY_RATE=5  //send rate in seconds.. is the minimum time interval in between to sends

TelemetryAggregatorInflux:

sends data in influx format
sends additional to statistics counter also: timestamp and reponame
sends data as absolute values and delta (seems like it is a bit harder to create the delta on the consuming site)

Parrameters required:

CVMFS_INFLUX_HOST=localhost
CVMFS_INFLUX_PORT=8092
CVMFS_INFLUX_METRIC_NAME=cvmfs_client_counters
CVMFS_INFLUX_EXTRA_TAGS=my_custom_tag=24,my_custom_tag2=12  //optional
CVMFS_INFLUX_EXTRA_FIELDS=os_version=8  //optional

HereThereBeDragons · 2022-11-09T22:29:56Z

@jblomer maybe you can have a first look if that is more or less what you thought of

jblomer

Great work, that fits in very well!

Things can be simplified a bit here and there, see comments.

And, as you mention, tests, comments, docs should come with it.

cvmfs/catalog_mgr.h

cvmfs/catalog_mgr_impl.h

cvmfs/cvmfs.cc

cvmfs/mountpoint.h

cvmfs/mountpoint.cc

cvmfs/telemetry_aggregator_influx.cc

cernvm-bot · 2022-12-14T16:00:26Z

linter finished with errors:

cvmfs/mountpoint.cc:1898:  At least two spaces is best between code and comments  [whitespace/comments] [2]
cvmfs/telemetry_aggregator.cc:28:  Missing space before ( in switch(  [whitespace/parens] [5]
cvmfs/telemetry_aggregator.h:69:  Line ends in whitespace.  Consider deleting these extra spaces.  [whitespace/end_of_line] [4]
cvmfs/telemetry_aggregator_influx.cc:123:  Should have a space between // and comment  [whitespace/comments] [4]
cvmfs/telemetry_aggregator_influx.cc:160:  Line ends in whitespace.  Consider deleting these extra spaces.  [whitespace/end_of_line] [4]

cernvm-bot · 2023-01-09T15:22:05Z

linter finished with errors:

cvmfs/cvmfs.cc:2164:  At least two spaces is best between code and comments  [whitespace/comments] [2]
cvmfs/telemetry_aggregator.cc:88:  Line ends in whitespace.  Consider deleting these extra spaces.  [whitespace/end_of_line] [4]

HereThereBeDragons · 2023-01-10T07:05:49Z

@cernvm-bot cloudtest

cernvm-bot · 2023-01-10T07:24:13Z

building for cloudtests finished: SUCCESS
https://lcgapp-services.cern.ch/cvmfs-jenkins/job/CvmfsFullBuildDocker/3619/

HereThereBeDragons · 2023-01-10T08:33:59Z

@cernvm-bot cloudtest

cernvm-bot · 2023-01-10T09:09:49Z

building for cloudtests finished: SUCCESS
https://lcgapp-services.cern.ch/cvmfs-jenkins/job/CvmfsFullBuildDocker/3622/

cernvm-bot · 2023-01-10T09:56:28Z

cloudtests finished: FAILURE
https://lcgapp-services.cern.ch/cvmfs-jenkins/job/CvmfsCloudTesting/1517/

cernvm-bot · 2023-01-10T10:36:57Z

cloudtests finished: FAILURE
https://lcgapp-services.cern.ch/cvmfs-jenkins/job/CvmfsCloudTesting/1520/

jblomer

Cool! Some comments on the implementation, regarding tests I'd suggest

Unit test for MakePayload, MakeDeltaPayload
Integration test with netcat as an influx listener
The tests need to ignore the timestamp value.

cvmfs/mountpoint.cc

cvmfs/statistics.h

cvmfs/telemetry_aggregator.h

cvmfs/telemetry_aggregator_influx.cc

cernvm-bot · 2023-01-18T16:55:05Z

linter finished with errors:

cvmfs/telemetry_aggregator.h:46:  "public:" should be preceded by a blank line  [whitespace/blank_line] [3]
cvmfs/telemetry_aggregator_influx.h:30:  "public:" should be preceded by a blank line  [whitespace/blank_line] [3]
test/unittests/t_telemetry_aggregator.cc:25:  Line ends in whitespace.  Consider deleting these extra spaces.  [whitespace/end_of_line] [4]
test/unittests/t_telemetry_aggregator.cc:63:  Line ends in whitespace.  Consider deleting these extra spaces.  [whitespace/end_of_line] [4]
test/unittests/t_telemetry_aggregator.cc:81:  Missing space before ( in for(  [whitespace/parens] [5]
test/unittests/t_telemetry_aggregator.cc:99:  Line ends in whitespace.  Consider deleting these extra spaces.  [whitespace/end_of_line] [4]
test/unittests/t_telemetry_aggregator.cc:103:  Line ends in whitespace.  Consider deleting these extra spaces.  [whitespace/end_of_line] [4]
test/unittests/t_telemetry_aggregator.cc:112:  Line ends in whitespace.  Consider deleting these extra spaces.  [whitespace/end_of_line] [4]
test/unittests/t_telemetry_aggregator.cc:131:  Missing space before ( in for(  [whitespace/parens] [5]
test/unittests/t_telemetry_aggregator.cc:152:  Line ends in whitespace.  Consider deleting these extra spaces.  [whitespace/end_of_line] [4]
test/unittests/t_telemetry_aggregator.cc:171:  Missing space before ( in for(  [whitespace/parens] [5]
test/unittests/t_telemetry_aggregator.cc:201:  Missing space before ( in for(  [whitespace/parens] [5]
test/unittests/t_telemetry_aggregator.cc:215:  Lines should be <= 80 characters long  [whitespace/line_length] [2]
test/unittests/t_telemetry_aggregator.cc:219:  Lines should be <= 80 characters long  [whitespace/line_length] [2]
test/unittests/t_telemetry_aggregator.cc:240:  Missing space before ( in for(  [whitespace/parens] [5]
test/unittests/t_telemetry_aggregator.cc:266:  At least two spaces is best between code and comments  [whitespace/comments] [2]

HereThereBeDragons · 2023-01-19T13:38:00Z

@jblomer have a look

HereThereBeDragons · 2023-01-19T13:38:24Z

@cernvm-bot cloudtest

cernvm-bot · 2023-01-19T13:54:22Z

building for cloudtests finished: FAILURE
https://lcgapp-services.cern.ch/cvmfs-jenkins/job/CvmfsFullBuildDocker/3645/

HereThereBeDragons · 2023-01-20T16:59:18Z

Documentation PR here: cvmfs/doc-cvmfs#195

cernvm-bot · 2023-01-20T17:17:26Z

linter finished with errors:

test/unittests/t_telemetry_aggregator.cc:30:  Using C-style cast.  Use reinterpret_cast<void*>(...) instead  [readability/casting] [4]

HereThereBeDragons · 2023-01-24T08:59:02Z

@jblomer have a look

test/unittests/t_telemetry_aggregator.cc

jblomer

One small issue in the unit tests, then this is good to go.

test/unittests/t_telemetry_aggregator.cc

cernvm-bot · 2023-01-26T16:39:05Z

linter finished with errors:

test/unittests/t_telemetry_aggregator.cc:45:  Is this a non-const reference? If so, make const or use a pointer: std::vector<std::string> &vec  [runtime/references] [2]

HereThereBeDragons · 2023-01-27T08:28:53Z

@jblomer have a look

jblomer · 2023-01-27T10:15:38Z

@cernvm-bot cloudtest

cernvm-bot · 2023-01-27T10:34:03Z

building for cloudtests finished: SUCCESS
https://lcgapp-services.cern.ch/cvmfs-jenkins/job/CvmfsFullBuildDocker/3664/

cernvm-bot · 2023-01-27T12:01:23Z

cloudtests finished: FAILURE
https://lcgapp-services.cern.ch/cvmfs-jenkins/job/CvmfsCloudTesting/1543/

jblomer · 2023-01-27T12:11:57Z

@HereThereBeDragons Some issues in the integration tests: netcat is missing on some platforms. You can install it in the test/cloud_testing/platforms/*setup.sh scripts. Also, I'd suggest to rebase the PR to the latest devel branch, which should give you a green baseline for the integration tests.

HereThereBeDragons · 2023-01-30T12:43:36Z

@cernvm-bot cloudtest

cernvm-bot · 2023-01-30T13:02:52Z

building for cloudtests finished: SUCCESS
https://lcgapp-services.cern.ch/cvmfs-jenkins/job/CvmfsFullBuildDocker/3667/

cernvm-bot · 2023-01-30T14:32:09Z

cloudtests finished: FAILURE
https://lcgapp-services.cern.ch/cvmfs-jenkins/job/CvmfsCloudTesting/1545/

HereThereBeDragons · 2023-01-31T14:27:40Z

@cernvm-bot cloudtest

cernvm-bot · 2023-01-31T14:46:22Z

building for cloudtests finished: SUCCESS
https://lcgapp-services.cern.ch/cvmfs-jenkins/job/CvmfsFullBuildDocker/3669/

cernvm-bot · 2023-01-31T15:18:24Z

cloudtests finished: ABORTED
https://lcgapp-services.cern.ch/cvmfs-jenkins/job/CvmfsCloudTesting/1547/

HereThereBeDragons · 2023-01-31T15:18:40Z

@cernvm-bot cloudtest

cernvm-bot · 2023-01-31T15:37:31Z

building for cloudtests finished: SUCCESS
https://lcgapp-services.cern.ch/cvmfs-jenkins/job/CvmfsFullBuildDocker/3670/

HereThereBeDragons · 2023-01-31T16:28:27Z

@cernvm-bot cloudtest

cernvm-bot · 2023-01-31T16:47:02Z

building for cloudtests finished: SUCCESS
https://lcgapp-services.cern.ch/cvmfs-jenkins/job/CvmfsFullBuildDocker/3673/

cernvm-bot · 2023-01-31T17:02:56Z

cloudtests finished: FAILURE
https://lcgapp-services.cern.ch/cvmfs-jenkins/job/CvmfsCloudTesting/1548/

cernvm-bot · 2023-01-31T18:10:44Z

cloudtests finished: FAILURE
https://lcgapp-services.cern.ch/cvmfs-jenkins/job/CvmfsCloudTesting/1550/

test/src/098-telemetry/udp_listener.py

HereThereBeDragons · 2023-02-01T08:35:21Z

@cernvm-bot cloudtest

cernvm-bot · 2023-02-01T08:53:56Z

building for cloudtests finished: SUCCESS
https://lcgapp-services.cern.ch/cvmfs-jenkins/job/CvmfsFullBuildDocker/3677/

cernvm-bot · 2023-02-01T10:20:27Z

cloudtests finished: FAILURE
https://lcgapp-services.cern.ch/cvmfs-jenkins/job/CvmfsCloudTesting/1554/

jblomer changed the title ~~internal affairs telemetry framework + aggregator for influx~~ Telemetry framework + aggregator for influx Dec 1, 2022

jblomer requested changes Dec 2, 2022

View reviewed changes

jblomer requested changes Jan 12, 2023

View reviewed changes

jblomer reviewed Jan 26, 2023

View reviewed changes

test/unittests/t_telemetry_aggregator.cc Outdated Show resolved Hide resolved

jblomer requested changes Jan 26, 2023

View reviewed changes

test/unittests/t_telemetry_aggregator.cc Outdated Show resolved Hide resolved

test/unittests/t_telemetry_aggregator.cc Outdated Show resolved Hide resolved

test/unittests/t_telemetry_aggregator.cc Outdated Show resolved Hide resolved

jblomer approved these changes Jan 27, 2023

View reviewed changes

HereThereBeDragons added 2 commits January 30, 2023 13:40

working version (but no tests)

601fac3

small influx format fix

20e96fc

HereThereBeDragons added 5 commits January 30, 2023 13:40

tidy.. maybe..

b67b5ef

maybe nolint is the solution..

c8e7c05

use posix wrapper to appease tidy?

e53c651

change requests

8e0ed1d

appease tidy

03b7349

HereThereBeDragons force-pushed the feature_intaffairs_telemetry#3096 branch from 6037496 to 03b7349 Compare January 30, 2023 12:41

replace netcat with custom python udp listener

cc08e24

udp listener does not care if python2 or python3

7b1caea

harmonize output between python2 and python3

aa22960

jblomer reviewed Jan 31, 2023

View reviewed changes

test/src/098-telemetry/udp_listener.py Outdated Show resolved Hide resolved

fix to choose systemavailable python version

d7728df

add new line for udp_listener.py

f0f85ee

jblomer merged commit 33f416e into cvmfs:devel Feb 1, 2023

HereThereBeDragons deleted the feature_intaffairs_telemetry#3096 branch February 1, 2023 10:35

HereThereBeDragons added this to the 2.11 milestone Jul 7, 2023

Telemetry framework + aggregator for influx #3098

Telemetry framework + aggregator for influx #3098

Uh oh!

Conversation

HereThereBeDragons commented Nov 9, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HereThereBeDragons commented Nov 9, 2022

Uh oh!

jblomer left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cernvm-bot commented Dec 14, 2022

Uh oh!

cernvm-bot commented Jan 9, 2023

Uh oh!

HereThereBeDragons commented Jan 10, 2023

Uh oh!

cernvm-bot commented Jan 10, 2023

Uh oh!

HereThereBeDragons commented Jan 10, 2023

Uh oh!

cernvm-bot commented Jan 10, 2023

Uh oh!

cernvm-bot commented Jan 10, 2023

Uh oh!

cernvm-bot commented Jan 10, 2023

Uh oh!

jblomer left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cernvm-bot commented Jan 18, 2023

Uh oh!

HereThereBeDragons commented Jan 19, 2023

Uh oh!

HereThereBeDragons commented Jan 19, 2023

Uh oh!

cernvm-bot commented Jan 19, 2023

Uh oh!

HereThereBeDragons commented Jan 20, 2023

Uh oh!

cernvm-bot commented Jan 20, 2023

Uh oh!

HereThereBeDragons commented Jan 24, 2023

Uh oh!

Uh oh!

jblomer left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cernvm-bot commented Jan 26, 2023

Uh oh!

HereThereBeDragons commented Jan 27, 2023

Uh oh!

jblomer commented Jan 27, 2023

Uh oh!

cernvm-bot commented Jan 27, 2023

HereThereBeDragons commented Nov 9, 2022 •

edited

Loading

jblomer left a comment •

edited

Loading