Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Decoder for Base64 Encoded Binary data input. #14950

Closed
rc-networks opened this issue Mar 7, 2024 · 10 comments · Fixed by #14961
Closed

Decoder for Base64 Encoded Binary data input. #14950

rc-networks opened this issue Mar 7, 2024 · 10 comments · Fixed by #14961
Labels
feature request Requests for new plugin and for new features to existing plugins

Comments

@rc-networks
Copy link

Use Case

Needed for decoding Base64 Encoded Binary data input.
For example, we have settings like this:

  [[inputs.gnmi.subscription]]
    name = "ps"
    origin = "openconfig"
    path = "/components/component/power-supply/state"
    subscription_mode = "sample"
    sample_interval = "60s"

and the data coming in looks like this:

app.company.network.device.ps,
device=DEV-SW-01,
host=dev-server,
name=PowerSupply1/B,
telemetry_source=network 
openconfig_platform_psu:input_voltage="Q1HAAA==",
openconfig_platform_psu:output_current="P3YEGQ==",
openconfig_platform_psu:output_voltage="QmF3zw==",
openconfig_platform_psu:output_power="QlMAAA==",
openconfig_platform_psu:enabled=true,
openconfig_platform_psu:capacity="RDLAAA==",
openconfig_platform_psu:input_current="Psk3TA==" 1709529211201225000

Model opeconfig-platform-cpu specifies that we should expect a binary data as stated here:
image
(image taken from YANGSUITE app)

The values for most of the fields are Base64 encoded strings,
which needs to be convert to binary data,
which we then need to convert to float (ieeefloat32) as specified in the specification above.

Expected behavior

Base64 Encoded String Data comes in.
Process field values to be converted to Binary Data.
Binary Data converted to type float before sending to the output plugin.

Actual behavior

Currently not possible as far as I am informed.
Issue also discussed on Slack

Additional info

No response

@rc-networks rc-networks added the feature request Requests for new plugin and for new features to existing plugins label Mar 7, 2024
@srebhan
Copy link
Contributor

srebhan commented Mar 7, 2024

@rc-networks could you please add dump_responses = true to your [[inputs.gnmi]] section and run telegraf ... --test --debug once? Please post the lines starting with D! [inputs.gnmi] Got update_!?

@rc-networks
Copy link
Author

rc-networks commented Mar 8, 2024

Here is the output with --test

dev@dev-server:~/streaming_telemetry$ ./run.sh 
2024-03-08T02:32:44Z I! Loading config: /home/dev/streaming_telemetry/telegraf.conf
2024-03-08T02:32:44Z I! Loading config: /home/dev/streaming_telemetry/telegraf.d/ios_xe.conf
2024-03-08T02:32:44Z I! Loading config: /home/dev/streaming_telemetry/telegraf.d/output.conf
2024-03-08T02:32:44Z I! Loading config: /home/dev/streaming_telemetry/telegraf.d/processor.conf
2024-03-08T02:32:44Z I! Starting Telegraf 1.29.5 brought to you by InfluxData the makers of InfluxDB
2024-03-08T02:32:44Z I! Available plugins: 241 inputs, 9 aggregators, 30 processors, 24 parsers, 60 outputs, 6 secret-stores
2024-03-08T02:32:44Z I! Loaded inputs: gnmi
2024-03-08T02:32:44Z I! Loaded aggregators: 
2024-03-08T02:32:44Z I! Loaded processors: converter (3x) enum rename (6x)
2024-03-08T02:32:44Z I! Loaded secretstores: 
2024-03-08T02:32:44Z W! Outputs are not used in testing mode!
2024-03-08T02:32:44Z I! Tags enabled: host=dev-server telemetry_source=network
2024-03-08T02:32:44Z D! [agent] Initializing plugins
2024-03-08T02:32:44Z D! [inputs.gnmi] Internal alias mapping: map[openconfig:/components/component/power-supply/state:ps]
2024-03-08T02:32:44Z D! [agent] Starting service inputs
2024-03-08T02:32:44Z D! [agent] Stopping service inputs
2024-03-08T02:32:44Z D! [agent] Input channel closed
2024-03-08T02:32:44Z D! [agent] Processor channel closed
2024-03-08T02:32:44Z D! [agent] Processor channel closed
2024-03-08T02:32:44Z D! [agent] Processor channel closed
2024-03-08T02:32:44Z D! [agent] Processor channel closed
2024-03-08T02:32:44Z D! [agent] Processor channel closed
2024-03-08T02:32:44Z D! [agent] Processor channel closed
2024-03-08T02:32:44Z D! [agent] Processor channel closed
2024-03-08T02:32:44Z D! [agent] Processor channel closed
2024-03-08T02:32:44Z D! [agent] Processor channel closed
2024-03-08T02:32:44Z D! [agent] Processor channel closed
2024-03-08T02:32:44Z D! [agent] Stopped Successfully

I don't see the D! [inputs.gnmi] Got update_ though.
What are we looking for?
Tried removing the --test, Got update_ still not visible:

dev@dev-server:~/streaming_telemetry$ ./run.sh 
2024-03-08T02:41:45Z I! Loading config: /home/dev/streaming_telemetry/telegraf.conf
2024-03-08T02:41:45Z I! Loading config: /home/dev/streaming_telemetry/telegraf.d/ios_xe.conf
2024-03-08T02:41:45Z I! Loading config: /home/dev/streaming_telemetry/telegraf.d/output.conf
2024-03-08T02:41:45Z I! Loading config: /home/dev/streaming_telemetry/telegraf.d/processor.conf
2024-03-08T02:41:45Z I! Starting Telegraf 1.29.5 brought to you by InfluxData the makers of InfluxDB
2024-03-08T02:41:45Z I! Available plugins: 241 inputs, 9 aggregators, 30 processors, 24 parsers, 60 outputs, 6 secret-stores
2024-03-08T02:41:45Z I! Loaded inputs: gnmi
2024-03-08T02:41:45Z I! Loaded aggregators: 
2024-03-08T02:41:45Z I! Loaded processors: converter (3x) enum rename (6x)
2024-03-08T02:41:45Z I! Loaded secretstores: 
2024-03-08T02:41:45Z I! Loaded outputs: file
2024-03-08T02:41:45Z I! Tags enabled: host=dev-server telemetry_source=network
2024-03-08T02:41:45Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"dev-server", Flush Interval:10s
2024-03-08T02:41:45Z D! [agent] Initializing plugins
2024-03-08T02:41:45Z D! [inputs.gnmi] Internal alias mapping: map[openconfig:/components/component/power-supply/state:ps]
2024-03-08T02:41:45Z D! [agent] Connecting outputs
2024-03-08T02:41:45Z D! [agent] Attempting connection to [outputs.file]
2024-03-08T02:41:45Z D! [agent] Successfully connected to outputs.file
2024-03-08T02:41:45Z D! [agent] Starting service inputs
2024-03-08T02:41:45Z D! [inputs.gnmi] Connection to gNMI device DEV-SW-01:50052 established
2024-03-08T02:41:55Z D! [outputs.file] Wrote batch of 2 metrics in 339.24µs
2024-03-08T02:41:55Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2024-03-08T02:42:05Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2024-03-08T02:42:15Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2024-03-08T02:42:25Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2024-03-08T02:42:35Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2024-03-08T02:42:45Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2024-03-08T02:42:55Z D! [outputs.file] Wrote batch of 2 metrics in 230.006µs
2024-03-08T02:42:55Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
^C2024-03-08T02:43:04Z D! [agent] Stopping service inputs
2024-03-08T02:43:04Z D! [inputs.gnmi] Connection to gNMI device DEV-SW-01:50052 closed
2024-03-08T02:43:04Z D! [agent] Input channel closed
2024-03-08T02:43:04Z D! [agent] Processor channel closed
2024-03-08T02:43:04Z D! [agent] Processor channel closed
2024-03-08T02:43:04Z D! [agent] Processor channel closed
2024-03-08T02:43:04Z D! [agent] Processor channel closed
2024-03-08T02:43:04Z D! [agent] Processor channel closed
2024-03-08T02:43:04Z D! [agent] Processor channel closed
2024-03-08T02:43:04Z D! [agent] Processor channel closed
2024-03-08T02:43:04Z D! [agent] Processor channel closed
2024-03-08T02:43:04Z D! [agent] Processor channel closed
2024-03-08T02:43:04Z D! [agent] Processor channel closed
2024-03-08T02:43:04Z I! [agent] Hang on, flushing any cached metrics before shutdown
2024-03-08T02:43:04Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2024-03-08T02:43:04Z I! [agent] Stopping running outputs
2024-03-08T02:43:04Z D! [agent] Stopped Successfully

outputs without the --test:

app.woven.network.device.ps,device=DEV-SW-01,host=dev-server,name=PowerSupply1/A,telemetry_source=network openconfig_platform_psu:capacity="RDLAAA==",openconfig_platform_psu:input_current="PvEm6Q==",openconfig_platform_psu:input_voltage="Q1CAAA==",openconfig_platform_psu:output_current="P5n75w==",openconfig_platform_psu:output_voltage="QmF3zw==",openconfig_platform_psu:output_power="QoeAAA==",openconfig_platform_psu:enabled=true 1709866326661974000
app.woven.network.device.ps,device=DEV-SW-01,host=dev-server,name=PowerSupply1/B,telemetry_source=network openconfig_platform_psu:output_current="P2wIMQ==",openconfig_platform_psu:output_voltage="QmF3zw==",openconfig_platform_psu:output_power="Qk+AAA==",openconfig_platform_psu:enabled=true,openconfig_platform_psu:capacity="RDLAAA==",openconfig_platform_psu:input_current="Ps3S8g==",openconfig_platform_psu:input_voltage="Q1HAAA==" 1709866326661974000
app.woven.network.device.ps,device=DEV-SW-01,host=dev-server,name=PowerSupply1/A,telemetry_source=network openconfig_platform_psu:output_power="QoeAAA==",openconfig_platform_psu:enabled=true,openconfig_platform_psu:capacity="RDLAAA==",openconfig_platform_psu:input_current="PvEm6Q==",openconfig_platform_psu:input_voltage="Q1CAAA==",openconfig_platform_psu:output_current="P5n75w==",openconfig_platform_psu:output_voltage="QmF3zw==" 1709866386738724000
app.woven.network.device.ps,device=DEV-SW-01,host=dev-server,name=PowerSupply1/B,telemetry_source=network openconfig_platform_psu:output_voltage="QmF3zw==",openconfig_platform_psu:output_power="QlMAAA==",openconfig_platform_psu:enabled=true,openconfig_platform_psu:capacity="RDLAAA==",openconfig_platform_psu:input_current="Ps3S8g==",openconfig_platform_psu:input_voltage="Q1GAAA==",openconfig_platform_psu:output_current="P3AgxQ==" 1709866386738724000

@srebhan
Copy link
Contributor

srebhan commented Mar 11, 2024

Oh sorry @rc-networks I think you need to do with --debug but without --test as we need to wait until data arrives from the device...

@srebhan
Copy link
Contributor

srebhan commented Mar 11, 2024

@rc-networks can you please test the binary in #14961, available once CI finished the tests!? You should be able to convert the data above using

[[inputs.file]]
  files = ["./test_configs/gnmi_issue_14950.influx"]
  data_format = "influx"

[[processors.parser]]
  parse_fields = ["openconfig_platform_psu:capacity"]
  merge = "override"

  data_format = "binary"
  endianness = "be"
  binary_encoding = "base64"
  [[processors.parser.binary]]
    entries = [{ name = "openconfig_platform_psu:capacity", type = "float32" }]

[[processors.parser]]
  parse_fields = ["openconfig_platform_psu:input_current"]
  merge = "override"

  data_format = "binary"
  endianness = "be"
  binary_encoding = "base64"
  [[processors.parser.binary]]
    entries = [{ name = "openconfig_platform_psu:input_current", type = "float32" }]

[[processors.parser]]
  parse_fields = ["openconfig_platform_psu:input_voltage"]
  merge = "override"

  data_format = "binary"
  endianness = "be"
  binary_encoding = "base64"
  [[processors.parser.binary]]
    entries = [{ name = "openconfig_platform_psu:input_voltage", type = "float32" }]

[[processors.parser]]
  parse_fields = ["openconfig_platform_psu:output_current"]
  merge = "override"

  data_format = "binary"
  endianness = "be"
  binary_encoding = "base64"
  [[processors.parser.binary]]
    entries = [{ name = "openconfig_platform_psu:output_current", type = "float32" }]

[[processors.parser]]
  parse_fields = ["openconfig_platform_psu:output_power"]
  merge = "override"

  data_format = "binary"
  endianness = "be"
  binary_encoding = "base64"
  [[processors.parser.binary]]
    entries = [{ name = "openconfig_platform_psu:output_power", type = "float32" }]

[[processors.parser]]
  parse_fields = ["openconfig_platform_psu:output_voltage"]
  merge = "override"

  data_format = "binary"
  endianness = "be"
  binary_encoding = "base64"
  [[processors.parser.binary]]
    entries = [{ name = "openconfig_platform_psu:output_voltage", type = "float32" }]

@rc-networks
Copy link
Author

rc-networks commented Mar 12, 2024

Thank you!
Looks like the given binary is working.
There's just too much redundant code blocks for the same endpoint.

I tried putting multiple entries on the parse_fields and entries and it showed the same values for everything.
i.e.:

[[processors.parser]]
  parse_fields = ["openconfig_platform_psu:output_voltage", "openconfig_platform_psu:output_power"]
  merge = "override"

  data_format = "binary"
  endianness = "be"
  binary_encoding = "base64"
  [[processors.parser.binary]]
    entries = [{ name = "openconfig_platform_psu:output_voltage", type = "float32" }, {openconfig_platform_psu:output_power}]

@srebhan
Copy link
Contributor

srebhan commented Mar 12, 2024

Yes, because the parser gets the data field-by-field so your config will not work. Do you know if transporting floats as base64 encoded strings is a standard method in GNMI? If so, could you please point me to some official documentation?

@rc-networks
Copy link
Author

@srebhan
Thanks for working on this.
Got sick and did not managed to respond.

As for the question above, it's not about the GNMI, it's about the model that is being used.
Here is the standard link which states:

[9.8.2](https://datatracker.ietf.org/doc/html/rfc6020#section-9.8.2).  Lexical Representation

   Binary values are encoded with the base64 encoding scheme (see
   [[RFC4648], Section 4](https://datatracker.ietf.org/doc/html/rfc4648#section-4)).

@srebhan
Copy link
Contributor

srebhan commented Mar 18, 2024

@rc-networks so if this encoding is used, is it used for all data reported by the device or is this a per-subscription thing?

@rc-networks
Copy link
Author

@srebhan It would be used on all data that is using models that return binary data.

@srebhan
Copy link
Contributor

srebhan commented Mar 19, 2024

@rc-networks thanks! Could you please open another feature request for the GNMI plugin, providing the link above, an (redacted) example configuration and some data with dump_responses = true set running Telegraf in debug mode (i.e. telegraf ... --debug)!? I would like to support this mode of operation in the GNMI input if it's according to the standard...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request Requests for new plugin and for new features to existing plugins
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants