enhancement(lading_payload): add new service topology-based OTLP traces generator

[tobz]

What does this PR do?

This PR introduces a new service topology-based OpenTelemetry Traces generator, aimed at generating more realistic workload outputs.

Motivation

The current OpenTelemetry Traces generator resembles many of the existing payload generators in lading: specification-driven and fuzzer-esque. This is perfectly acceptable for some payload formats, but OpenTelemetry traces, and traces in general, aren't well served by this model. Principally, while we can (and do) generate valid traces/spans with the current generator, the traces it generates are not particularly useful for benchmarking. Given their fuzzer-esque qualities, they aren't suitable for exercising certain behaviors or codepaths in trace receivers, such as evaluating optimizations for handling repetitive attributes, or exercising rare code paths by controlling the frequency of emission of those rare traces/spans.

This PR introduces a new OpenTelemetry Traces generator that's geared towards generating traces/spans based on a user-defined "service topology." Traces are ultimately designed for representing spans within a distributed systems, so why shouldn't we also define our trace/span generation in terms of what the hypothetical distributed system that we're simulating looks like?

Our new generator requires users to specify the catalog of services that traces/spans will be generated for, the operations those services expose, and the suboperations (calls to other service operations) that each operation will make. Services can be one of three fixed types: HTTP, gRPC, or database. This allows us to provide reasonable, out-of-the-box shortcuts for quickly scaffolding a service definition, and also lets us provide out-of-the-box resource attributes that map to the correct OpenTelemetry Semantic Conventions for the given service. Services, and operations defined for those services, can be extended to add additional resource attributes. Attributes defined at any level are able to succinctly describe their value using either fixed values, randomly-generated values, or randomly-selected values from a set pre-defined dictionaries: cloud regions, environments, HTTP methods/status codes, and gRPC status codes. Again, the goal is to make scaffolding these service definitions quick and easy, with a structure that is self-describing.

Below is an example configuration with a three-tiered system, demonstrating how services are defined, linked together, as well as things like randomly-generated and pre-defined dictionary values, and also randomized suboperation selection:

grpc:
  seed: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 6]
  target_uri: http://127.0.0.1:4317/opentelemetry.proto.collector.trace.v1.TraceService/Export
  variant:
    opentelemetry_traces:
      error_rate: 0.01
      services:
        - name: api-gateway
          service_type: http
          scope_name: com.example.gateway
          resource_attributes:
            - key: deployment.environment
              value: production
            - key: cloud.region
              value:
                dictionary: cloud_regions
          operations:
            - id: get-product
              method: GET
              route: '/api/v1/products/{id}'
              suboperations:
                - to: product-service/get-product
            - id: list-products
              method: GET
              route: /api/v1/products
              suboperations:
                - to: product-service/list-products
            - id: create-order
              method: POST
              route: /api/v1/order
              suboperations:
                - to: order-service/create-order
        - name: product-service
          service_type: grpc
          grpc:
            service: ProductService
          scope_name: com.example.products
          operations:
            - id: get-product
              method: GetProduct
              suboperations:
                - to: product-cache/get-product-by-id
                - to: product-db/select-product-by-id
                  rate: 0.1
            - id: list-products
              method: ListProducts
              suboperations:
                - to: product-cache/get-products
                - to: product-db/select-products
                  rate: 0.1
        - name: order-service
          service_type: grpc
          grpc:
            service: OrderService
          scope_name: com.example.orders
          operations:
            - id: create-order
              method: CreateOrder
              suboperations:
                - to: product-db/select-product-by-id
                  max_repeat: 5
                - to: order-db/create-order
              attributes:
                - key: order.total_price_cents
                  value:
                    random_int:
                      min: 350
                      max: 99999
        - name: product-cache
          service_type: database
          database:
            system: redis
          operations:
            - id: get-product-by-id
              query: 'GET products:by_id:$1'
            - id: get-products
              query: 'GET products:full'
        - name: product-db
          service_type: database
          database:
            system: postgresql
            name: products
          operations:
            - id: select-product-by-id
              table: products
              query: SELECT * FROM products WHERE id = $1
            - id: select-products
              table: products
              query: SELECT * FROM products LIMIT 50
        - name: order-db
          service_type: database
          database:
            system: postgresql
            name: orders
          operations:
            - id: create-order
              table: orders
              query: 'INSERT INTO orders VALUES ($1, $2, $3, $4, $5)'

Related issues

Additional Notes

[blt]

I'm really liking where this is going. I'd even be interested in swapping this for the other implementation.

[tobz]

🙇🏻

Yeah, it's working well so far. It's still a little verbose for my taste, and I want to explore how we might shrink down how we define the operations/links.

[goxberry]

Very cool! Looking forward to seeing this generator in action.

[blt]

This is cool. Very different from what we had previously, like you point out, but I find this approach compelling. I have some netflow work coming up soon and I'll build on this notion.

[blt]

We don't have a convention for this yet but I could buy a opentelemetry::trace::validation module, just to keep things tidy.