Streaming telemetry protobuf archive

In this repository we track proto files for model driven telemetry, and provide scripts to generate the language binding, coupled with an autogenerated thin layer to serve the objects to unmarshall against for a given yang encoding path.

The directory proto_archive contains a set of telemetry messages defined in .proto files. The .proto files are organised in a directory structure which matches the yang model.

Note: Model driven telemetry project is work in progress, and both the on and off box components of are likely to evolve.

Content in proto_archive as follows:

• cisco_ios_xr_.*oper, root of native Cisco IOS XR models
• .*_yang2proto_map.json, a mapping of YANG encoding path to message set definition models; files of this type do not need to be used directly, but would be used by decoding support code (see Support for unmarshalling telemetry messages below).
• telemetry.proto, top level messages used with streaming telemetry and including the unified header used with GPB and GPB K/V encoding
• mdt_grpc_dialin/out, gRPC service specification for streaming telemetry server and client side streaming
• old, messages retained for backwards compatibility

├── proto_archive
│   ├── mdt_grpc_dialin
│   │   └── mdt_grpc_dialin.proto
│   ├── mdt_grpc_dialout
│   │   └── mdt_grpc_dialout.proto
│   └── telemetry.proto

Support for unmarshaling telemetry messages

When GPB encoded messages are received, in order to decode the content in data_gpb, receiver must retrieve the correct automatically generated native binding structure to unmarshal or decode the byte stream against.

This repository includes support for the automatic generation of bindings and serving code to make the operation as simple as possible for the receiver.

Telemetry messages include an encoding path in the header - this encoding path is the key to retrieving the objects to unmarshal against:

{
        "node_id_str": "uut",
        "subscription_id_str": "test",
        "encoding_path": "Cisco-IOS-XR-infra-statsd-oper:infra-statistics/interfaces/interface/latest/generic-counters",
        "collection_id": "111808",
        "collection_start_time": "1471785717681",
...

Once the collector code extracts this encoding path, a simple invocation of a service provided by this repository will provide the caller with the message set to unmarshal against.

In future, a model version will also be sent along in the same header above, and when it is, this can be used to qualify the correct version of message set to unmarshal against for cases where collector must support multiple versions concurrently.

Note that the serving code for all languages is built with help from metadata in proto/*_yang2proto_map.json for native platform models. Should a new proto file be added, a corresponding entry would need to be added to metadata; e.g.

{
 "message_keys": "inv_xml_fru_info_KEYS",
 "message": "inv_xml_fru_info",
 "package": "cisco_ios_xr_plat_chas_invmgr_oper.platform_inventory.racks.rack.slots.slot.cards.card.port_slots.port_slot.attributes.fru_info",
 "file": "cisco_ios_xr_plat_chas_invmgr_oper/platform_inventory/racks/rack/slots/slot/cards/card/port_slots/port_slot/attributes/fru_info/inv_xml_fru_info.proto",
 "encoding_path": "Cisco-IOS-XR-plat-chas-invmgr-oper:platform-inventory/racks/rack/slots/slot/cards/card/port-slots/port-slot/attributes/fru-info"
}

The message and message_keys field reflect the top level messages in the proto. These are the messages serialised and carried by the keys and content fields in TelemetryRowGPB in telemetry.proto:

message TelemetryRowGPB {
   uint64 timestamp = 1;
   bytes keys = 10;
   bytes content = 11;
}

The field package reflects the .proto package name, file points at where in the hierarchy the .proto file lives, while encoding path reflects the value of field encoding_path in the message Telemetry in telemetry.proto which would carry the messages referenced in messages and message_keys as payload.

Do note that meta data will be collected from any file of the form *_yang2proto_map.json in the staging directory.

Customising models supported by generated code

In order to contain bloat of library package supporting unmarshalling, it is possible to pick and choose which models are included. In order to support a model, all that is required is to copy the model from the proto_archive directory to the staging directly. Conversely, removing a model or part of a model from the staging directory will result in the model not being supported.

Decoding support in all language bindings will support the ability to report which (YANG gather paths, model versions) are supported by the generated package. For example in golang, EncodingPathsSupported() is provided to allow collector code to report what is supported in a given build/configuration.

Bindings

Use OS with large max argument length given large number of protos in staging directory. Current Binding generation has been tested on CentOS 7 and Ubuntu 16.04.

Binding: Golang

Go

Binding: Java

Coming soon.