span.http.status_code is a string(not converted to int) in TraceQL

[andreivaloshyn]

Helm chart version: 1.2.3
Grafana: 9.4.3

I migrated from tempo 1.5.x(helm chart 0.28.0) to the latest chart version
Find out that { span.http.status_code >= 200 && span.http.status_code < 300 } show nothing, because of span.http.status_code not being converted to int, which is weird.

NOTE: All traces below are generated after migration.

Config

target: ingester
http_api_prefix: ""
server:
    http_listen_network: tcp
    http_listen_address: ""
    http_listen_port: 3100
    http_listen_conn_limit: 0
    grpc_listen_network: tcp
    grpc_listen_address: ""
    grpc_listen_port: 9095
    grpc_listen_conn_limit: 0
    tls_cipher_suites: ""
    tls_min_version: ""
    http_tls_config:
        cert_file: ""
        key_file: ""
        client_auth_type: ""
        client_ca_file: ""
    grpc_tls_config:
        cert_file: ""
        key_file: ""
        client_auth_type: ""
        client_ca_file: ""
    register_instrumentation: true
    graceful_shutdown_timeout: 30s
    http_server_read_timeout: 30s
    http_server_write_timeout: 30s
    http_server_idle_timeout: 2m0s
    grpc_server_max_recv_msg_size: 16777216
    grpc_server_max_send_msg_size: 16777216
    grpc_server_max_concurrent_streams: 100
    grpc_server_max_connection_idle: 2562047h47m16.854775807s
    grpc_server_max_connection_age: 2562047h47m16.854775807s
    grpc_server_max_connection_age_grace: 2562047h47m16.854775807s
    grpc_server_keepalive_time: 2h0m0s
    grpc_server_keepalive_timeout: 20s
    grpc_server_min_time_between_pings: 10s
    grpc_server_ping_without_stream_allowed: true
    log_format: logfmt
    log_level: debug
    log_source_ips_enabled: false
    log_source_ips_header: ""
    log_source_ips_regex: ""
    log_request_at_info_level_enabled: false
    http_path_prefix: ""
distributor:
    ring:
        kvstore:
            store: memberlist
            prefix: collectors/
            consul:
                host: localhost:8500
                acl_token: ""
                http_client_timeout: 20s
                consistent_reads: false
                watch_rate_limit: 1
                watch_burst_size: 1
                cas_retry_delay: 1s
            etcd:
                endpoints: []
                dial_timeout: 10s
                max_retries: 10
                tls_enabled: false
                tls_cert_path: ""
                tls_key_path: ""
                tls_ca_path: ""
                tls_server_name: ""
                tls_insecure_skip_verify: false
                tls_cipher_suites: ""
                tls_min_version: ""
                username: ""
                password: ""
            multi:
                primary: ""
                secondary: ""
                mirror_enabled: false
                mirror_timeout: 2s
        heartbeat_period: 5s
        heartbeat_timeout: 5m0s
        instance_id: tempo-ingester-0
        instance_interface_names:
            - eth0
            - en0
        instance_port: 0
        instance_addr: ""
    receivers:
        jaeger:
            protocols:
                grpc:
                    endpoint: 0.0.0.0:14250
                thrift_binary:
                    endpoint: 0.0.0.0:6832
                thrift_compact:
                    endpoint: 0.0.0.0:6831
                thrift_http:
                    endpoint: 0.0.0.0:14268
        otlp:
            protocols:
                grpc:
                    endpoint: 0.0.0.0:4317
                http:
                    endpoint: 0.0.0.0:55681
        zipkin: null
    override_ring_key: distributor
    log_received_traces: false
    log_received_spans:
        enabled: false
        include_all_attributes: true
        filter_by_status_error: true
    forwarders: []
    extend_writes: true
    search_tags_deny_list: []
ingester_client:
    pool_config:
        checkinterval: 15s
        healthcheckenabled: true
        healthchecktimeout: 1s
        maxconcurrenthealthchecks: 0
    remote_timeout: 5s
    grpc_client_config:
        max_recv_msg_size: 104857600
        max_send_msg_size: 104857600
        grpc_compression: snappy
        rate_limit: 0
        rate_limit_burst: 0
        backoff_on_ratelimits: false
        backoff_config:
            min_period: 100ms
            max_period: 10s
            max_retries: 10
        tls_enabled: false
        tls_cert_path: ""
        tls_key_path: ""
        tls_ca_path: ""
        tls_server_name: ""
        tls_insecure_skip_verify: false
        tls_cipher_suites: ""
        tls_min_version: ""
metrics_generator_client:
    pool_config:
        checkinterval: 15s
        healthcheckenabled: true
        healthchecktimeout: 1s
        maxconcurrenthealthchecks: 0
    remote_timeout: 5s
    grpc_client_config:
        max_recv_msg_size: 104857600
        max_send_msg_size: 104857600
        grpc_compression: snappy
        rate_limit: 0
        rate_limit_burst: 0
        backoff_on_ratelimits: false
        backoff_config:
            min_period: 100ms
            max_period: 10s
            max_retries: 10
        tls_enabled: false
        tls_cert_path: ""
        tls_key_path: ""
        tls_ca_path: ""
        tls_server_name: ""
        tls_insecure_skip_verify: false
        tls_cipher_suites: ""
        tls_min_version: ""
querier:
    search:
        query_timeout: 30s
        prefer_self: 10
        external_endpoints: []
        external_hedge_requests_at: 8s
        external_hedge_requests_up_to: 2
    trace_by_id:
        query_timeout: 10s
    max_concurrent_queries: 20
    frontend_worker:
        frontend_address: tempo-query-frontend-discovery:9095
        dns_lookup_duration: 10s
        parallelism: 2
        match_max_concurrent: true
        id: ""
        grpc_client_config:
            max_recv_msg_size: 104857600
            max_send_msg_size: 16777216
            grpc_compression: gzip
            rate_limit: 0
            rate_limit_burst: 0
            backoff_on_ratelimits: false
            backoff_config:
                min_period: 100ms
                max_period: 1s
                max_retries: 5
            tls_enabled: false
            tls_cert_path: ""
            tls_key_path: ""
            tls_ca_path: ""
            tls_server_name: ""
            tls_insecure_skip_verify: false
            tls_cipher_suites: ""
            tls_min_version: ""
    query_relevant_ingesters: false
query_frontend:
    max_outstanding_per_tenant: 2000
    querier_forget_delay: 0s
    max_retries: 2
    search:
        concurrent_jobs: 1000
        target_bytes_per_job: 104857600
        default_result_limit: 100
        max_result_limit: 5000
        max_duration: 0s
        query_backend_after: 15m0s
        query_ingesters_until: 30m0s
    trace_by_id:
        query_shards: 50
        hedge_requests_at: 2s
        hedge_requests_up_to: 2
compactor:
    ring:
        kvstore:
            store: ""
            prefix: collectors/
            consul:
                host: localhost:8500
                acl_token: ""
                http_client_timeout: 20s
                consistent_reads: false
                watch_rate_limit: 1
                watch_burst_size: 1
                cas_retry_delay: 1s
            etcd:
                endpoints: []
                dial_timeout: 10s
                max_retries: 10
                tls_enabled: false
                tls_cert_path: ""
                tls_key_path: ""
                tls_ca_path: ""
                tls_server_name: ""
                tls_insecure_skip_verify: false
                tls_cipher_suites: ""
                tls_min_version: ""
                username: ""
                password: ""
            multi:
                primary: ""
                secondary: ""
                mirror_enabled: false
                mirror_timeout: 2s
        heartbeat_period: 5s
        heartbeat_timeout: 1m0s
        wait_stability_min_duration: 1m0s
        wait_stability_max_duration: 5m0s
        instance_id: tempo-ingester-0
        instance_interface_names:
            - eth0
            - en0
        instance_port: 0
        instance_addr: ""
        wait_active_instance_timeout: 10m0s
    compaction:
        v2_in_buffer_bytes: 5242880
        v2_out_buffer_bytes: 20971520
        v2_prefetch_traces_count: 1000
        compaction_window: 1h0m0s
        max_compaction_objects: 6000000
        max_block_bytes: 107374182400
        block_retention: 2160h0m0s
        compacted_block_retention: 1h0m0s
        retention_concurrency: 10
        max_time_per_tenant: 5m0s
        compaction_cycle: 30s
    override_ring_key: compactor
ingester:
    lifecycler:
        ring:
            kvstore:
                store: memberlist
                prefix: collectors/
                consul:
                    host: localhost:8500
                    acl_token: ""
                    http_client_timeout: 20s
                    consistent_reads: false
                    watch_rate_limit: 1
                    watch_burst_size: 1
                    cas_retry_delay: 1s
                etcd:
                    endpoints: []
                    dial_timeout: 10s
                    max_retries: 10
                    tls_enabled: false
                    tls_cert_path: ""
                    tls_key_path: ""
                    tls_ca_path: ""
                    tls_server_name: ""
                    tls_insecure_skip_verify: false
                    tls_cipher_suites: ""
                    tls_min_version: ""
                    username: ""
                    password: ""
                multi:
                    primary: ""
                    secondary: ""
                    mirror_enabled: false
                    mirror_timeout: 2s
            heartbeat_timeout: 5m0s
            replication_factor: 3
            zone_awareness_enabled: false
            excluded_zones: ""
        num_tokens: 128
        heartbeat_period: 5s
        heartbeat_timeout: 1m0s
        observe_period: 0s
        join_after: 0s
        min_ready_duration: 15s
        interface_names:
            - eth0
        final_sleep: 0s
        tokens_file_path: /var/tempo/tokens.json
        availability_zone: ""
        unregister_on_shutdown: true
        readiness_check_ring_health: true
        address: ""
        port: 0
        id: tempo-ingester-0
    concurrent_flushes: 4
    flush_check_period: 10s
    flush_op_timeout: 5m0s
    trace_idle_period: 10s
    max_block_duration: 30m0s
    max_block_bytes: 268435456
    complete_block_timeout: 10m0s
    override_ring_key: ring
metrics_generator:
    ring:
        kvstore:
            store: memberlist
            prefix: collectors/
            consul:
                host: localhost:8500
                acl_token: ""
                http_client_timeout: 20s
                consistent_reads: false
                watch_rate_limit: 1
                watch_burst_size: 1
                cas_retry_delay: 1s
            etcd:
                endpoints: []
                dial_timeout: 10s
                max_retries: 10
                tls_enabled: false
                tls_cert_path: ""
                tls_key_path: ""
                tls_ca_path: ""
                tls_server_name: ""
                tls_insecure_skip_verify: false
                tls_cipher_suites: ""
                tls_min_version: ""
                username: ""
                password: ""
            multi:
                primary: ""
                secondary: ""
                mirror_enabled: false
                mirror_timeout: 2s
        heartbeat_period: 5s
        heartbeat_timeout: 1m0s
        instance_id: tempo-ingester-0
        instance_interface_names:
            - eth0
            - en0
        instance_addr: ""
    processor:
        service_graphs:
            wait: 10s
            max_items: 10000
            workers: 10
            histogram_buckets:
                - 0.1
                - 0.2
                - 0.4
                - 0.8
                - 1.6
                - 3.2
                - 6.4
                - 12.8
            dimensions: []
        span_metrics:
            histogram_buckets:
                - 0.002
                - 0.004
                - 0.008
                - 0.016
                - 0.032
                - 0.064
                - 0.128
                - 0.256
                - 0.512
                - 1.02
                - 2.05
                - 4.1
            intrinsic_dimensions:
                service: true
                span_name: true
                span_kind: true
                status_code: true
            dimensions: []
    registry:
        collection_interval: 15s
        stale_duration: 15m0s
        max_label_name_length: 1024
        max_label_value_length: 2048
    storage:
        path: /var/tempo/wal
        wal:
            wal_segment_size: 0
            wal_compression: false
            stripe_size: 0
            truncate_frequency: 0s
            min_wal_time: 0
            max_wal_time: 0
            no_lockfile: false
        remote_write_flush_deadline: 1m0s
        remote_write:
            - url: http://victoria-metrics-cluster-vminsert.victoria-metrics-cluster:8480/insert/749340814140/prometheus/api/v1/write
              remote_timeout: 30s
              name: victoria-metrics
              follow_redirects: true
              enable_http2: true
              queue_config:
                capacity: 10000
                max_shards: 150
                min_shards: 50
                max_samples_per_send: 2000
                batch_send_deadline: 5s
                min_backoff: 30ms
                max_backoff: 5s
              metadata_config:
                send: true
                send_interval: 1m
                max_samples_per_send: 500
    metrics_ingestion_time_range_slack: 30s
storage:
    trace:
        pool:
            max_workers: 400
            queue_depth: 20000
        wal:
            path: /var/tempo/wal
            completedfilepath: /var/tempo/wal/completed
            blocksfilepath: /var/tempo/wal/blocks
            v2_encoding: snappy
            search_encoding: none
            ingestion_time_range_slack: 2m0s
            version: vParquet
        block:
            bloom_filter_false_positive: 0.01
            bloom_filter_shard_size_bytes: 102400
            version: vParquet
            search_encoding: snappy
            search_page_size_bytes: 1048576
            v2_index_downsample_bytes: 1048576
            v2_index_page_size_bytes: 256000
            v2_encoding: zstd
            parquet_row_group_size_bytes: 100000000
        search:
            chunk_size_bytes: 1000000
            prefetch_trace_count: 1000
            read_buffer_count: 32
            read_buffer_size_bytes: 1048576
            cache_control:
                footer: false
                column_index: false
                offset_index: false
        blocklist_poll: 5m0s
        blocklist_poll_concurrency: 50
        blocklist_poll_fallback: true
        blocklist_poll_tenant_index_builders: 2
        blocklist_poll_stale_tenant_index: 0s
        blocklist_poll_jitter_ms: 0
        backend: s3
        local:
            path: ""
        gcs:
            bucket_name: ""
            chunk_buffer_size: 10485760
            endpoint: ""
            hedge_requests_at: 0s
            hedge_requests_up_to: 2
            insecure: false
            object_cache_control: ""
            object_metadata: {}
        s3:
            bucket: digitaltest01-grafana-tempo
            endpoint: s3.amazonaws.com
            region: ""
            access_key: ""
            secret_key: ""
            insecure: false
            insecure_skip_verify: false
            part_size: 0
            hedge_requests_at: 0s
            hedge_requests_up_to: 2
            signature_v2: false
            forcepathstyle: false
            tags: {}
            storage_class: ""
            metadata: {}
        azure:
            storage_account_name: ""
            storage_account_key: ""
            use_managed_identity: false
            user_assigned_id: ""
            container_name: ""
            endpoint_suffix: blob.core.windows.net
            max_buffers: 4
            buffer_size: 3145728
            hedge_requests_at: 0s
            hedge_requests_up_to: 2
        cache: memcached
        cache_min_compaction_level: 0
        cache_max_block_age: 0s
        background_cache:
            writeback_goroutines: 10
            writeback_buffer: 10000
        memcached:
            host: tempo-memcached
            service: memcached-client
            addresses: ""
            timeout: 500ms
            max_idle_conns: 16
            max_item_size: 0
            update_interval: 1m0s
            consistent_hash: true
            circuit_breaker_consecutive_failures: 0
            circuit_breaker_timeout: 0s
            circuit_breaker_interval: 0s
            ttl: 0s
        redis: null
overrides:
    ingestion_rate_strategy: local
    ingestion_rate_limit_bytes: 15000000
    ingestion_burst_size_bytes: 20000000
    search_tags_allow_list: null
    max_traces_per_user: 0
    max_global_traces_per_user: 0
    max_search_bytes_per_trace: 0
    forwarders: []
    metrics_generator_ring_size: 0
    metrics_generator_processors:
        - service-graphs
        - span-metrics
    metrics_generator_max_active_series: 0
    metrics_generator_collection_interval: 0s
    metrics_generator_disable_collection: false
    metrics_generator_forwarder_queue_size: 0
    metrics_generator_forwarder_workers: 0
    metrics_generator_processor_service_graphs_histogram_buckets: []
    metrics_generator_processor_service_graphs_dimensions: []
    metrics_generator_processor_span_metrics_histogram_buckets: []
    metrics_generator_processor_span_metrics_dimensions: []
    metrics_generator_processor_span_metrics_intrinsic_dimensions: {}
    block_retention: 0s
    max_bytes_per_tag_values_query: 5000000
    max_search_duration: 0s
    max_bytes_per_trace: 5000000
    per_tenant_override_config: /conf/overrides.yaml
    per_tenant_override_period: 10s
memberlist:
    node_name: ""
    randomize_node_name: true
    stream_timeout: 10s
    retransmit_factor: 2
    pull_push_interval: 30s
    gossip_interval: 1s
    gossip_nodes: 2
    gossip_to_dead_nodes_time: 30s
    dead_node_reclaim_time: 0s
    compression_enabled: false
    advertise_addr: ""
    advertise_port: 7946
    cluster_label: ""
    cluster_label_verification_disabled: false
    join_members:
        - tempo-gossip-ring
    min_join_backoff: 1s
    max_join_backoff: 1m0s
    max_join_retries: 10
    abort_if_cluster_join_fails: false
    rejoin_interval: 0s
    left_ingesters_timeout: 5m0s
    leave_timeout: 20s
    message_history_buffer_bytes: 0
    bind_addr: []
    bind_port: 7946
    packet_dial_timeout: 2s
    packet_write_timeout: 5s
    tls_enabled: false
    tls_cert_path: ""
    tls_key_path: ""
    tls_ca_path: ""
    tls_server_name: ""
    tls_insecure_skip_verify: false
    tls_cipher_suites: ""
    tls_min_version: ""
usage_report:
    reporting_enabled: true
    backoff:
        min_period: 100ms
        max_period: 10s
        max_retries: 0

See screenshots above which shows that regular old style search is able to search both string and int values, but TraceQL works in that particular case only with string.

Example of Grafana Inspector trace Json:

[
  {
    "schema": {
      "refId": "A",
      "meta": {
        "preferredVisualisationType": "trace"
      },
      "name": "Trace",
      "fields": [
        {
          "name": "traceID",
          "type": "string",
          "typeInfo": {
            "frame": "string"
          },
          "config": {}
        },
        {
          "name": "spanID",
          "type": "string",
          "typeInfo": {
            "frame": "string"
          },
          "config": {}
        },
        {
          "name": "parentSpanID",
          "type": "string",
          "typeInfo": {
            "frame": "string"
          },
          "config": {}
        },
        {
          "name": "operationName",
          "type": "string",
          "typeInfo": {
            "frame": "string"
          },
          "config": {}
        },
        {
          "name": "serviceName",
          "type": "string",
          "typeInfo": {
            "frame": "string"
          },
          "config": {}
        },
        {
          "name": "serviceTags",
          "type": "other",
          "typeInfo": {
            "frame": "json.RawMessage"
          },
          "config": {}
        },
        {
          "name": "startTime",
          "type": "number",
          "typeInfo": {
            "frame": "float64"
          },
          "config": {}
        },
        {
          "name": "duration",
          "type": "number",
          "typeInfo": {
            "frame": "float64"
          },
          "config": {}
        },
        {
          "name": "logs",
          "type": "other",
          "typeInfo": {
            "frame": "json.RawMessage"
          },
          "config": {}
        },
        {
          "name": "references",
          "type": "other",
          "typeInfo": {
            "frame": "json.RawMessage"
          },
          "config": {}
        },
        {
          "name": "tags",
          "type": "other",
          "typeInfo": {
            "frame": "json.RawMessage"
          },
          "config": {}
        }
      ]
    },
    "data": {
      "values": [
        [
          "e380bf6bf5412cbb7b6ec1e049529f26"
        ],
        [
          "7b6ec1e049529f26"
        ],
        [
          ""
        ],
        [
          "productpage.default.svc.cluster.local:9080/productpage"
        ],
        [
          "ingress-gateway-private.istio-system"
        ],
        [
          [
            {
              "value": "ingress-gateway-private.istio-system",
              "key": "service.name"
            }
          ]
        ],
        [
          1678269472593.3281
        ],
        [
          21.431
        ],
        [
          [
            {
              "timestamp": 1678269472614.833,
              "fields": [
                {
                  "value": "cr",
                  "key": "message"
                }
              ]
            }
          ]
        ],
        [
          null
        ],
        [
          [
            {
              "value": "-",
              "key": "downstream_cluster"
            },
            {
              "value": "outbound|9080||productpage.default.svc.cluster.local",
              "key": "upstream_cluster.name"
            },
            {
              "value": "latest",
              "key": "istio.canonical_revision"
            },
            {
              "value": "proxy",
              "key": "component"
            },
            {
              "value": "fd57a6e0-4f72-9740-adff-c83020e823b0",
              "key": "guid:x-request-id"
            },
            {
              "value": "10.35.91.61",
              "key": "peer.address"
            },
            {
              "value": "200",
              "key": "http.status_code"
            },
            {
              "value": "HTTP/1.1",
              "key": "http.protocol"
            },
            {
              "value": "0",
              "key": "request_size"
            },
            {
              "value": "router~10.35.90.203~ingress-gateway-private-876fb4dc7-zvs7k.istio-system~istio-system.svc.cluster.local",
              "key": "node_id"
            },
            {
              "value": "curl/7.86.0",
              "key": "user_agent"
            },
            {
              "value": "cluster.local",
              "key": "istio.mesh_id"
            },
            {
              "value": "-",
              "key": "response_flags"
            },
            {
              "value": "0",
              "key": "response_size"
            },
            {
              "value": "ingress-gateway-private",
              "key": "istio.canonical_service"
            },
            {
              "value": "outbound|9080||productpage.default.svc.cluster.local",
              "key": "upstream_cluster"
            },
            {
              "value": "istio-system",
              "key": "istio.namespace"
            },
            {
              "value": "10.35.90.203",
              "key": "net.host.ip"
            },
            {
              "value": "HEAD",
              "key": "http.method"
            },
            {
              "value": "http://bookinfo.digitaldev03.travisperkins.digital/productpage",
              "key": "http.url"
            },
            {
              "value": "client",
              "key": "span.kind"
            },
            {
              "value": 0,
              "key": "status.code"
            }
          ]
        ]
      ]
    }
  },
  {
    "schema": {
      "meta": {
        "preferredVisualisationType": "nodeGraph"
      },
      "fields": [
        {
          "name": "id",
          "type": "string",
          "config": {}
        },
        {
          "name": "title",
          "type": "string",
          "config": {}
        },
        {
          "name": "subtitle",
          "type": "string",
          "config": {}
        },
        {
          "name": "mainstat",
          "type": "string",
          "config": {
            "displayName": "Total time (% of trace)"
          }
        },
        {
          "name": "secondarystat",
          "type": "string",
          "config": {
            "displayName": "Self time (% of total)"
          }
        },
        {
          "name": "color",
          "type": "number",
          "config": {
            "color": {
              "mode": "continuous-GrYlRd"
            },
            "displayName": "Self time / Trace duration"
          }
        }
      ]
    },
    "data": {
      "values": [
        [
          "7b6ec1e049529f26"
        ],
        [
          "ingress-gateway-private.istio-system"
        ],
        [
          "productpage.default.svc.cluster.local:9080/productpage"
        ],
        [
          "21.43ms (100%)"
        ],
        [
          "21.43ms (100%)"
        ],
        [
          1.0000042833870655
        ]
      ]
    }
  },
  {
    "schema": {
      "meta": {
        "preferredVisualisationType": "nodeGraph"
      },
      "fields": [
        {
          "name": "id",
          "type": "string",
          "config": {}
        },
        {
          "name": "target",
          "type": "string",
          "config": {}
        },
        {
          "name": "source",
          "type": "string",
          "config": {}
        }
      ]
    },
    "data": {
      "values": [
        [],
        [],
        []
      ]
    }
  }
]

[joe-elliott]

for performance reasons there are currently no plans to attempt data type conversions in traceql. this will return nothing b/c the condition is false:

{ "1" = 1 }

we are considering supporting range operators for strings though. in this case we'd use string semantics to compare the following:

{ span.http.status_code < "300" }

[andreivaloshyn]

@joe-elliott thank you for your answer.
Am I right thinking that in this example https://grafana.com/docs/tempo/latest/traceql/ span.http.status_code have int value type? And seems in my case app returns string instead of int.

[joe-elliott]

that is correct. for whatever reason your app is storing http status codes with type string. if this is auto instrumentation i would encourage you to file an issue with the client. if it's your own instrumentation you should be able to easily change it to an int.

[andreivaloshyn]

Got it. Thanks again for your time.