Switch branches/tags
Find file History
noahgoldman and sjung-stripe Automate code generation for gRPC forwarding (#527)
* Automate code generation for gRPC forwarding

In an earlier PR (#439) I added in a few different protobuf and gRPC
definitions, as well as the generated packages for each of them.  I had
generated them locally, but I didn't add the proper "protoc" invocations
to "generate.go" to make it easy to regenerate these in the future.

This change adds the proper commands to the Travis build and
"generate.go" to allow you to run "go generate" both locally and in the
CI environment.  Specifically it adds the "protoc-gen-gogofaster"
binary, which generates code from protobuf definitions that is supposed
to be a bit leaner than "protoc-gen-gofast".  There isn't any particular
reason that we need to use "gogofaster", it was just what I used to
originally generate all the protobuf-related code, so changing it would
generate a large diff.

I'm happy to change it back to the normal "protoc-gen-gofast" if that is
preferable.

* Use protoc-gen-gogofaster for all protobufs

This change switches "go generate" to generate code using
protoc-gen-gogofaster.  I also updated ".travis.yml" to remove sections
that are no longer needed, now that we are just using a single binary.
Latest commit 3847602 Aug 7, 2018
Permalink
..
Failed to load latest commit information.
README.md fix SSF Go link (#514) Jul 30, 2018
doc.go Add a less-gross package doc for package ssf Jan 29, 2018
example_randomly_sample_test.go Make a capsule structure for SSFSample arrays Feb 1, 2018
sample.pb.go Automate code generation for gRPC forwarding (#527) Aug 7, 2018
sample.proto Rejigger order, add Sample docs Feb 18, 2018
samples.go
samples_test.go Make ssf.RandomlySample a million times faster (#466) Apr 25, 2018

README.md

Sensor Sensibility Format

The Sensor Sensibility Format — or SSF for short — is a language agnostic format for transmitting observability data such as trace spans, metrics, events and more.

Why?

SSF is based on prior art of metrics formats and protocols mentioned in the inspiration section. Unlike each of these wonderful formats, which we have used for years and benefited greatly from, SSF is a binary format utilizing Protocol Buffers. It is emission-, collection- and storage-agnostic. It is merely a protocol for transmitting information. It is developed as part of a suite of technologies around Veneur.

Why A New Format?

Because we want to:

  • leverage protobuf so we no longer have to write buggy, string-based marshaling and unmarshaling code in clients libraries and server implementations
  • benefit from the efficiency of protobuf
  • collect and combine the great ideas from our inspiration
  • add some of our own ideas

Using SSF

  • You can send SSF as UDP packets to Veneur's ssf_listen_addresses.
  • Veneur can also read SSF as a wire protocol over connection protocols such as TCP or UNIX domain sockets
  • You can use the CLI tool veneur-emit.
  • You can use an SSF trace client in Go or Ruby.

Philosophy

We've got some novel ideas that we've put in to SSF. It might help to be familiar with the concepts in our inspiration. Here is our founding philosophy of SSF:

  • a timer is a span
  • a log line is a span, especially if it's structured
  • events are also spans
  • therefore, the core unit of all observability data is a span, or a unit of a trace.
    • spans might be single units, or units of a larger whole
  • other point metrics (e.g. counters and gauges) can be constituents of a span
    • it's more valuable to know the depth of a queue in the context of a span than alone
    • improve the context of counters and gauges, as they are part of a span
  • provide a format containing the superset of many backend's features

Components

You can examine the protobuf definition, but in a nutshell SSF provides the following:

A metric field describing its type as one of COUNTER, GAUGE, HISTOGRAM (supplanting a timer), SET and STATUS. Rounding out the traditional fields are name, value, sample_rate and timestamp. There is a map of tags string key-value pairs.

Others

Beyond these StatsD-stye fields are also message for including an arbitrary string such as a log message and unit as a string describing the unit of the message such as seconds. Note that SSF does not have defined units at present. Only strings!

STATUS Samples

A Metric of STATUS is most like a Nagios check result.

Log Samples?

Since all fields are optional, one could leave out many fields and represent a log line in SSF by setting timestamp, name with a canonical name and tags for parameters. This is intended to be used in the future for Veneur to unify observability primitives.

Inspiration

We build on the shoulders of giants, and are proud to have used and been inspired by these marvelous tools: