Validate topology.yaml

[conorbros]

• every transform chain ends in a terminating transform
• a terminating transform is not followed by a non-terminating transform
  transforms are compatible with each other e.g. RedisTimestampTagger -> CassandraDestination will make a follow up PR for this as this one is large enough

[conorbros]

Example output is below:

Topology errors:
redis_chain:
  Terminating transform "Null" is not last in chain
  Terminating transform "Null" is not last in chain
  Non-terminating transform "Printer" is last in chain

other_chain:
    chain_3:
        sub_chain_2:
          Terminating transform "Null" is not last in chain
    chain_1:
      Terminating transform "Null" is not last in chain
      Non-terminating transform "Printer" is last in chain
    chain_2:
      Terminating transform "Null" is not last in chain


thread 'Shotover-Proxy-Thread' panicked at 'Topology is not valid', shotover-proxy/src/config/topology.rs:138:13
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Added to the Transform trait:

• is_terminating: default implementation is set to false because it's far more common. Terminating transforms override this.
• validate: default implementation checks whether the transform's position in the chain is valid using the is_terminating method. Transforms that require more logic (such as running their subchain's validation) override this.

Added to TransformChain:

• validate which will call each Transform's validate method.

Errors are passed back using a Vec<String>, Runner will check to see if this is not empty, dumps to stderr and panics.

[rukai]

Heres some feedback, but I think its best to keep focusing on #275 before we come back to this.

[github-actions]

1 benchmarks reported regressed performance. Please check the benchmark workflow logs for details: https://github.com/shotover/shotover-proxy/actions/runs/1352605458

[github-actions]

1 benchmarks reported regressed performance. Please check the benchmark workflow logs for details: https://github.com/shotover/shotover-proxy/actions/runs/1357988606

[github-actions]

2 benchmarks reported regressed performance. Please check the benchmark workflow logs for details: https://github.com/shotover/shotover-proxy/actions/runs/1362058174

[github-actions]

2 benchmarks reported regressed performance. Please check the benchmark workflow logs for details: https://github.com/shotover/shotover-proxy/actions/runs/1362662111

[github-actions]

1 benchmarks reported regressed performance. Please check the benchmark workflow logs for details: https://github.com/shotover/shotover-proxy/actions/runs/1366164776

[github-actions]

1 benchmarks reported regressed performance. Please check the benchmark workflow logs for details: https://github.com/shotover/shotover-proxy/actions/runs/1366591294

[github-actions]

1 benchmarks reported regressed performance. Please check the benchmark workflow logs for details: https://github.com/shotover/shotover-proxy/actions/runs/1366611104

[github-actions]

3 benchmarks reported regressed performance. Please check the benchmark workflow logs for details: https://github.com/shotover/shotover-proxy/actions/runs/1370307857

[rukai]

So currently we have 3 layers of tests:

• highest level integration test - does shotover shutdown when validation fails.
• mid level test - testing that we get specific nicely formatted errors in response to validation failures.
• lowest level unit tests - test specific transform/chain validation implementation to ensure we get full coverage of its validation implementation.

Im happy for all of these layers to exist.
However, duplicate tests add a lot of maintenance burden so its good to avoid when we can.
I'm noticing that the lowest level unit tests do a lot of testing of other transforms/chain validation logic, such tests should be moved into the unittests for the other transform/chain or moved into a mid level test.

Would like to hear your thoughts before you go rewriting lots of stuff.
Maybe you had your own reasoning for how you wrote them?

[github-actions]

1 benchmarks reported regressed performance. Please check the benchmark workflow logs for details: https://github.com/shotover/shotover-proxy/actions/runs/1378879865

[github-actions]

4 benchmarks reported regressed performance. Please check the benchmark workflow logs for details: https://github.com/shotover/shotover-proxy/actions/runs/1379219498

[github-actions]

1 benchmarks reported regressed performance. Please check the benchmark workflow logs for details: https://github.com/shotover/shotover-proxy/actions/runs/1379259994

[github-actions]

4 benchmarks reported regressed performance. Please check the benchmark workflow logs for details: https://github.com/shotover/shotover-proxy/actions/runs/1379406911

[conorbros]

Probably not, I'd imagine some of them are quite out of date. I can make another PR for updating the examples and testing any that are not already run as part of tests.

[benbromhead]

sounds good

[rukai]

good catch with the rebase :)

[github-actions]

1 benchmarks reported regressed performance. Please check the benchmark workflow logs for details: https://github.com/shotover/shotover-proxy/actions/runs/1383728363