Configuration unit tests

[LucioFranco]

As a user I want tools that I can use to test vector locally without much setup, this may mean creating a cli tool to submit and accept logs from vector. This should allow the user to play around with different sources, transforms and sinks locally with minimal setup.

[Jeffail]

Dumping some thoughts before writing specs out.

Goals

Ability to complement configs with unit tests

I think we should aim for a solid specification for unit testing Vector configs, something that we can recommend as best practices for maintaining large deployment projects.

Simplified execution

It should be simple to run a group of unit tests. Something like vector --test ./configs, maybe expanding to vector --test ./configs/**/*.toml and/or vector --test ./.... Our spec therefore needs to make the relationship between test definitions and their target config files explicit.

Tests should not need to cover an entire config

It will overwhelmingly be the case that users want to test the interactions between transforms with their tests, we therefore need to accommodate that. However, Vector makes it very easy to express rather complex graphs of components and the more complex a system being unit tested is the more brittle the tests become.

Our spec should allow tests to be defined for an explicit subset of a config, allowing users to break them down into logical groups and test them in isolation.

Tests should be easy to find

The relationship between a config and its test definitions should be clear to a user editing that config. We want to make it easy for large teams to maintain their Vector configs.

For example, we could enforce that test definition files are named according to the config file they are testing (foo.test.toml to your foo.toml), or we could enforce that tests are defined at the bottom of the config file they are designed to test.

[Jeffail]

Proposal 1

DISCLAIMER: This is only a partial proposal and likely to get fleshed out over time.

Unit tests for a config file are defined within the config file itself at the bottom. Each test allows you to specify one input event, to be injected in the place of a component (as a pseudo source), and >1 output events, each to be expected from a component in order for the test to pass.

[sources.foo]
  type = "kafka"
  bootstrap_servers = "10.14.22.123:9092,10.14.23.332:9092"
  group_id = "consumer-group-name"
  topics = ["topic-1"]

[sources.buz]
  type = "some_other_thing"

[transforms.bar]
  type = "something"
  inputs = [ "foo" ]

[transforms.baz]
  type = "something_else"
  inputs = [ "bar", "buz" ]

[sinks.qux]
  type = "kafka"
  inputs = ["baz"]
  bootstrap_servers = ["10.14.22.123:9092", "10.14.23.332:9092"]
  encoding = "json"
  key_field = "TODO"
  topic = "topic-1234"

[[test]]
  name = "foo test"

  [test.input_event]
    insert_at = "foo"
    type = "log"
    [test.input_event.log_fields]
      foo = "bar"
      baz = "buz"

  [[test.output_event]]
    expect_from = "bar"
    type = "log"
    [test.output_event.log_fields]
      foo = "bar"
      baz = "buz"

  [[test.output_event]]
    expect_from = "baz"
    type = "log"
    [test.output_event.log_fields]
      foo = "bar"
      baz = "buz"

[[test]]
  name = "bar test"

  [test.input_event]
    insert_at = "buz"
    type = "metric"
    [test.input_event.metric]
      type = "counter"
      name = "foocounter"
      val = 1
      timestamp = "TODO"
      [test.input_event.metric.tags]
        tag_one = "foo"
        tag_two = "bar"

  [[test.output_event]]
    expect_from = "baz"
    type = "log"
    [test.output_event.log_fields]
      foo = "bar"
      baz = "buz"

Problem: Test input data

Users need to be able to define their input data when writing tests, what should that look like? This ties in with #704. For now I've mocked up a spec that allows explicit event typing, but I think most users would want to simply specify lines of raw data like this:

[[test]]
  name = "baz test"
  insert_at = "bar"
  input = '{"field":"this is way simpler"}'

  [[test.output_event]]
    expect_from = "baz"
    value = '{"field":"but Im not sure of the feasibility of this"}'

[binarylogic]

Thanks, @Jeffail! These are good proposals and agree with all of your goals 👍 . I also agree that the original example is a little verbose, the JSON example is much nicer.

One comment:

Users need to be able to define their input data when writing tests, what should that look like? This ties in with #704.

For all intents and purposes the user should think the event is nested. The fact that we flatten it internally should not be exposed to the user.

Onto the testing design. I find proposal 1 to be rather verbose. Most of that is probably due to defining the test input data (which you pointed out). What do you think about this?

Proposal 2

# 
# Pipeline
#

[sources.in]
  type = "stdin"

[transforms.parse]
  type = "tokenizer"
  inputs = [ "info" ]
  field_names = ["timestamp", "level", "message"]

  [transforms.parse.types]
    timestamp = "timestamp|%Y-%m-%dT%H:%m:%sZ"

[sinks.out]
  type = "console"

#
# Tests
#

[tests.parse]
  # Sytax is {input|output}.<component-id>.<type>.<field-name>
  input.parse.log.message = '2019-10-19T12:22:22Z debug "I have the best words"'
  output.parse.log.timestamp = 2019-10-19T12:22:22Z
  output.parse.log.level = "debug"
  output.parse.log.message = "I have the best words"

# Another example demonstrating different assertions
[tests.conditions]
  input.parse.log.message = '2019-10-19T12:22:22Z debug "I have the best words"'
  output.parse.log.message.ne = "Does not equal this"
  output.parse.log.message.starts_with = "I have"
  output.parse.log.timestamp.gt = 0

# Another example demonstrating asserting the entire event
[tests.full_document]
  input.parse.log.message = '2019-10-19T12:22:22Z debug "I have the best words"'
  output.parse.log = '{"timestamp": "2019-10-19T12:22:22Z", "level": "debug", "message": "I have the best words"}'

This is just a rough example. I realize there are many more edge cases we need to cover, but this is a starting point for a more succinct syntax.

CLI

In terms of the CLI. I think this will be fairly straight-forward:

vector test

Targeting specific test(s):

vector test --test=parse --test=conditions

Targeting specific components:

vector test --component=parse

[lukesteensen]

I like Proposal 1 and am not too worried about being verbose at this point.

I do think a lot of these tests will involve some kind of parsing, so it will likely be common to supply only a raw log message as input. We can probably special case that to avoid the extra nesting under input_event.log_fields.

In general, I think there's plenty of work to be done just getting this working. We can iterate on the UI and come up with a more succinct representation once we have some experience writing these tests.