Skip to content
c2966cf
Compare
Choose a tag to compare

Release date: 2022-05-09

Adds plumbing for modular encoding, but more importantly, an enrichment
updater sender.

The sender "enrichmentupdater" and the corresponding changes to the
enrichment transformer will allow on-line updates to the enrichment
database using regular Skogul pipelines. You can load a JSON file at
start-up with an initial database of enrichment, then update that database
on the fly over an HTTP interface for example, if you set up a
receiver-pipeline that ends in an enrichment updater.

Both the enrichment updater and the enrichment transformer is STILL
experimental, but I'm feeling good about them, and they will be a major
addition to medium to large-scale Skogul deployments.

PS: I botched the NEWS in v0.15.1 so let's go lucky v0.15.2.

db87cde
Compare
Choose a tag to compare

Release date: 2022-04-10

Add an experimental enrichment transformer, and a slightly less
experimental switch sender.

The enrichment transformer WILL change, but needs testing, so it is
included. It allows an operator to provide a JSON file to enrich metrics
based on metadata fields, e.g.: Add customer name to interface stats, or
add if description. To use it, see docs/examples/tester_to_stdout_enrich.json ,
you will need a JSON file that has a list of metadata fields to match, and
the data field will be the fields added to metadata (yes, this is
confusing, this is one of the reasons it's considered experimental).

The switch sender is simpler, but also probably a bit immature, but testing
is required. It takes a list of conditions and checks if a metric(!)
matches this, and if it matches a complete set, data will be sent to the
associated sender. Use case: Dupe sender sends data to your own influxdb,
and to a switch sender, the switch sender then selectively streams
customer-specific data to a customer.

bdb58ae
Compare
Choose a tag to compare

Release date: 2022-04-01

Introduce float flattening to the cast transformer, which allows flattening
floating point numbers into "non-exponential" string representations. E.g.:
For serial numbers, instead of having them encoded as 3.141519e+X when sent
to influx as tags, they'll be encoded as 3141519.

Introduces much better request logging as well for the HTTP receiver.

86a4b86
Compare
Choose a tag to compare

Release date: 2022-03-22

Introduce IgnorePartialErrors in handlers to let a handler modify a
container by removing invalid metrics instead of dropping the entire
container.

Also introduces several small but important log-quality fixes.

Compare
Choose a tag to compare

Release date: 2022-03-22

Add Cast transformer, which allows specifying a shallow schema for input.
It can cast data and metadatafields to strings, integers and 64-bit floats.

It currently ignores missing fields, and for fields already of the correct
value, it "just works" with no added costs. For fields needing conversion
it will somewhat cumbersomely go through fmt.Sscanf(fmt.Sprintf(...),...),
which is somewhat costly, but also ensures that it can convert anything
that fmt knows how to handle reasonably well.

It is meant to use as a "we can't fix the source data", not as a "we don't
want to fix the source provider".

Compare
Choose a tag to compare

Release date: 2022-03-22

Updates juniper telemetry interface, including a change of protobuf
compiler. Tested on various routers.

Greatly improves error messages, specially for protobuf, this is the
beginning of a major revamp to use wrapped errors, which will hopefully
both reduce log spam, and make the log messages more useful for real life
trouble shooting. For protobuf, this means actually useful errors when
marshaling fails, including what router and the underlying errors. Next up:
Workaround for -Infinity.

Also adds http stats, courtesy of eiriktaa eirikamdal@yahoo.com.

Also switches rpm builds from CentOS 8 which no longer works at all, to
Rocky Linux 8. This should have no impact on end users, regardless of
platform.

PS: This is a verbatim copy of the entire v0.13.x debacle, since it failed
to build the release. There are 0 code changes between 0.13.0 and 0.14.0,
and this is just to get the actual NEWS section fixed and a clean start.

Compare
Choose a tag to compare

Release date: 2022-03-22

Only build fixes since 0.13.0 failed to check on our CI pipeline. See
v0.13.0 for actual news.

b3ade5e
Compare
Choose a tag to compare

Release date: 2021-12-10

Fixes a bug in the stats receiver where it would not run transformers before
sending data.

f1f77fa
Compare
Choose a tag to compare

Release date: 2021-12-10

Adds a Stats receiver for shipping internal skogul stats. This is a bit more
detailed than the Stats sender, and will be further extended in the future.
For now there are stats for the UDP receiver, protobuf parser and HTTP sender.
Furthermore, in the future we will try to get rid of more log messages about
errors and rather use stats for that.

Fixes the Structured Data parser to support more formats, and tackle all
formats with the same efficiency and priority.

Adds a concept of skogul.Identity to identify a skogul module by name. The
name is fetched from the configuration, and is typically the key used to
define a module.

This release also adds CopyFromData, a transformer to copy a field from data
to metadata without removing it. The ExtractFromData transformer is marked as
obsolete and will be removed in the future. To achieve the same effect, use
CopyFromData togheter with a Remove transformer.

Compare
Choose a tag to compare

Release date: 2021-12-02

The Batch sender now has a concept of a burning metrics if the sender is
too slow. If a "Burner" sender is configured, the batch sender will
redirect metrics to the burner if the regular sender would otherwise block,
e.g.: If the backlog is too big. This is recommended for setups where data
is duplicated to multiple backends, such as to a staging and test
environment, where you don't want the staging and test environment to slow
down data replication to the production environment.

Additionally, the UDP Read buffer size can now be configured for the UDP
receiver, which can be very useful for bursty telemetry data.

On a side note, shout out to Lasse Haugen for inspiring the DictSplitter
included in v0.10.9.