feat: return real data in detected fields endpoint #12421

trevorwhitney · 2024-04-01T16:22:46Z

What this PR does / why we need it:

This PR calculates real responses for the new experimental detected fields endpoint. It does this in an unsustainable, brute force way by parsing each line (up to a limit) in the querier. Ideally, long term, we'll figure out a way to store this data at ingest, but this is a short term solution to vet the API and data we're surfacing to the UI.

Which issue(s) this PR ~~fixes~~ references:
RE: #12339

trevorwhitney · 2024-04-01T16:46:55Z

pkg/loghttp/labels.go

@@ -99,17 +99,3 @@ func ParseDetectedLabelsQuery(r *http.Request) (*logproto.DetectedLabelsRequest,
 		Query: query(r),
 	}, nil
 }
-
-func ParseDetectedFieldsQuery(r *http.Request) (*logproto.DetectedFieldsRequest, error) {


moved to query.go in the same package

trevorwhitney · 2024-04-01T16:47:17Z

pkg/loghttp/query.go

@@ -617,6 +617,93 @@ func ParseVolumeRangeQuery(r *http.Request) (*VolumeRangeQuery, error) {
 	}, nil
 }

+func ParseDetectedFieldsQuery(r *http.Request) (*logproto.DetectedFieldsRequest, error) {


moved from labels.go and additional fields populated

trevorwhitney · 2024-04-01T16:48:43Z

pkg/querier/querier.go

@@ -900,18 +906,6 @@ func (q *SingleTenantQuerier) Volume(ctx context.Context, req *logproto.VolumeRe
 	return seriesvolume.Merge(responses, req.Limit), nil
 }

-func (q *SingleTenantQuerier) DetectedFields(_ context.Context, _ *logproto.DetectedFieldsRequest) (*logproto.DetectedFieldsResponse, error) {


moved below DetectedLables so so it was next to the private functions it calls

trevorwhitney · 2024-04-01T21:27:35Z

currently not returning any data in loki-dev-005, will publish for review once it's working

cyriltovena · 2024-04-02T10:30:20Z

pkg/querier/querier.go

+		return logproto.DetectedFieldDuration
+	}
+
+	if _, err := humanize.ParseBytes(value); err == nil {


this could be a string.Contains like we do for the level label detection. I suspect a small benchmark with some special log line will help. Might also be true for duration.

cyriltovena

Love the direction !

trevorwhitney · 2024-04-02T17:46:51Z

Alright, got it working locally pointing at loki-dev-005 data

curl http://localhost:3100/loki/api/v1/detected_fields\?end\=1711929720000000000\&query\=%7Bjob%3D%22loki-dev-005%2Fquery-frontend%22%7D+%7C%3D+%22metrics.go%22\&start\=1711929600000000000 -H "X-Scope-OrgID: 29"

{
  "fields": [
    {
      "label": "query_hash",
      "type": "int",
      "cardinality": 14
    },
    {
      "label": "query",
      "type": "string",
      "cardinality": 14
    },
    {
      "label": "cache_stats_results_download_time",
      "type": "duration",
      "cardinality": 1
    },
    {
      "label": "cache_volume_results_req",
      "type": "int",
      "cardinality": 1
    },
    {
      "label": "throughput",
      "type": "bytes",
      "cardinality": 13
    },
    {
      "label": "range_type",
      "type": "string",
      "cardinality": 1
    },
    {
      "label": "cache_stats_results_req",
      "type": "int",
      "cardinality": 1
    },
    {
      "label": "total_bytes",
      "type": "bytes",
      "cardinality": 15
    },
    {
      "label": "cache_index_hit",
      "type": "int",
      "cardinality": 1
    },
    {
      "label": "lines_per_second",
      "type": "int",
      "cardinality": 15
    },
    {
      "label": "cache_index_download_time",
      "type": "duration",
      "cardinality": 1
    },
    {
      "label": "cache_result_hit",
      "type": "int",
      "cardinality": 1
    },
    {
      "label": "cache_chunk_bytes_stored",
      "type": "int",
      "cardinality": 1
    },
    {
      "label": "cache_chunk_bytes_fetched",
      "type": "int",
      "cardinality": 11
    },
    {
      "label": "index_post_bloom_filter_chunks",
      "type": "int",
      "cardinality": 1
    },
    {
      "label": "queue_time",
      "type": "duration",
      "cardinality": 15
    },
    {
      "label": "cache_chunk_req",
      "type": "int",
      "cardinality": 2
    },
    {
      "label": "limit",
      "type": "int",
      "cardinality": 1
    },
    {
      "label": "index_total_chunks",
      "type": "int",
      "cardinality": 1
    },
    {
      "label": "length",
      "type": "duration",
      "cardinality": 1
    },
    {
      "label": "ingester_post_filter_lines",
      "type": "int",
      "cardinality": 5
    },
    {
      "label": "congestion_control_latency",
      "type": "duration",
      "cardinality": 1
    },
    {
      "label": "shards",
      "type": "int",
      "cardinality": 1
    },
    {
      "label": "cache_result_req",
      "type": "int",
      "cardinality": 1
    },
    {
      "label": "cache_chunk_download_time",
      "type": "duration",
      "cardinality": 12
    },
    {
      "label": "duration",
      "type": "duration",
      "cardinality": 16
    },
    {
      "label": "cache_chunk_hit",
      "type": "int",
      "cardinality": 2
    },
    {
      "label": "query_type",
      "type": "string",
      "cardinality": 1
    },
    {
      "label": "returned_lines",
      "type": "int",
      "cardinality": 1
    },
    {
      "label": "cache_volume_results_download_time",
      "type": "duration",
      "cardinality": 1
    },
    {
      "label": "ingester_chunk_matches",
      "type": "int",
      "cardinality": 2
    },
    {
      "label": "disable_pipeline_wrappers",
      "type": "boolean",
      "cardinality": 1
    },
    {
      "label": "start_delta",
      "type": "duration",
      "cardinality": 16
    },
    {
      "label": "total_lines",
      "type": "int",
      "cardinality": 14
    },
    {
      "label": "ts",
      "type": "string",
      "cardinality": 16
    },
    {
      "label": "traceID",
      "type": "string",
      "cardinality": 15
    },
    {
      "label": "ingester_chunk_compressed_bytes",
      "type": "bytes",
      "cardinality": 2
    },
    {
      "label": "total_bytes_structured_metadata",
      "type": "bytes",
      "cardinality": 14
    },
    {
      "label": "level",
      "type": "string",
      "cardinality": 1
    },
    {
      "label": "ingester_chunk_downloaded",
      "type": "int",
      "cardinality": 1
    },
    {
      "label": "status",
      "type": "int",
      "cardinality": 1
    },
    {
      "label": "sampled",
      "type": "boolean",
      "cardinality": 1
    },
    {
      "label": "ingester_chunk_decompressed_bytes",
      "type": "bytes",
      "cardinality": 5
    },
    {
      "label": "store_chunks_download_time",
      "type": "duration",
      "cardinality": 12
    },
    {
      "label": "index_bloom_filter_ratio",
      "type": "float",
      "cardinality": 1
    },
    {
      "label": "ingester_chunk_head_bytes",
      "type": "bytes",
      "cardinality": 1
    },
    {
      "label": "cache_result_query_length_served",
      "type": "duration",
      "cardinality": 1
    },
    {
      "label": "post_filter_lines",
      "type": "int",
      "cardinality": 14
    },
    {
      "label": "query_referenced_structured_metadata",
      "type": "boolean",
      "cardinality": 1
    },
    {
      "label": "token_id",
      "type": "string",
      "cardinality": 2
    },
    {
      "label": "ingester_requests",
      "type": "int",
      "cardinality": 2
    },
    {
      "label": "end_delta",
      "type": "duration",
      "cardinality": 15
    },
    {
      "label": "chunk_refs_fetch_time",
      "type": "duration",
      "cardinality": 15
    },
    {
      "label": "splits",
      "type": "int",
      "cardinality": 1
    },
    {
      "label": "cache_stats_results_hit",
      "type": "int",
      "cardinality": 1
    },
    {
      "label": "caller",
      "type": "string",
      "cardinality": 1
    },
    {
      "label": "ingester_chunk_refs",
      "type": "int",
      "cardinality": 1
    },
    {
      "label": "cache_index_req",
      "type": "int",
      "cardinality": 1
    },
    {
      "label": "component",
      "type": "string",
      "cardinality": 1
    },
    {
      "label": "step",
      "type": "duration",
      "cardinality": 1
    },
    {
      "label": "latency",
      "type": "string",
      "cardinality": 1
    },
    {
      "label": "org_id",
      "type": "int",
      "cardinality": 2
    },
    {
      "label": "cache_volume_results_hit",
      "type": "int",
      "cardinality": 1
    },
    {
      "label": "cache_result_download_time",
      "type": "duration",
      "cardinality": 1
    },
    {
      "label": "total_entries",
      "type": "int",
      "cardinality": 3
    }
  ]
}

cyriltovena

LGTM

parse log lines in the querier to return data about detected fields. this is not sustainable long term, but a short term solution to validate the data we want for the frontend.

trevorwhitney requested a review from a team as a code owner April 1, 2024 16:22

pull-request-size bot added size/XL size/XXL and removed size/XL labels Apr 1, 2024

github-actions bot added the type/docs Issues related to technical documentation; the Docs Squad uses this label across many repositories label Apr 1, 2024

pull-request-size bot added size/L and removed size/XXL labels Apr 1, 2024

github-actions bot removed the type/docs Issues related to technical documentation; the Docs Squad uses this label across many repositories label Apr 1, 2024

trevorwhitney commented Apr 1, 2024

View reviewed changes

trevorwhitney changed the title ~~Detected fields data~~ feat: return real data in detected fields endpoint Apr 1, 2024

trevorwhitney force-pushed the detected-fields-data branch from f0c88e9 to 0f84f6b Compare April 1, 2024 20:16

trevorwhitney marked this pull request as draft April 1, 2024 21:27

cyriltovena reviewed Apr 2, 2024

View reviewed changes

trevorwhitney marked this pull request as ready for review April 2, 2024 17:46

trevorwhitney added 2 commits April 2, 2024 12:53

fix: always include type in json response

5aa30ec

feat: cast to constrained type

5ab6f97

cyriltovena approved these changes Apr 2, 2024

View reviewed changes

trevorwhitney added 8 commits April 2, 2024 13:06

feat: parse detected fields in querier

04849d4

chore: clean up

557fba2

chore: remove comment

926d433

fix: linting

673ff4e

fix: detected fields unwrapping

a080471

fix: one more wrapping

ecbcf9c

why so many places?

72f9e41

wip: add debug logging

314165d

trevorwhitney added 4 commits April 2, 2024 13:14

chore: more debug logging

f7ce3ff

fix: missing new line

5cedd73

chore: clean up after final testing

5415fef

chore: clean-up after main rebase

7079f3b

trevorwhitney force-pushed the detected-fields-data branch from 9535a13 to 7079f3b Compare April 2, 2024 19:16

chore: fix linting

439099c

trevorwhitney enabled auto-merge (squash) April 2, 2024 19:54

trevorwhitney merged commit a53a0cc into main Apr 2, 2024
10 checks passed

trevorwhitney deleted the detected-fields-data branch April 2, 2024 19:57

cyriltovena mentioned this pull request Apr 2, 2024

feat: remove logfmt in log volume queries grafana/explore-logs#112

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: return real data in detected fields endpoint #12421

feat: return real data in detected fields endpoint #12421

trevorwhitney commented Apr 1, 2024

trevorwhitney Apr 1, 2024

trevorwhitney Apr 1, 2024

trevorwhitney Apr 1, 2024

trevorwhitney commented Apr 1, 2024

cyriltovena Apr 2, 2024

cyriltovena left a comment

trevorwhitney commented Apr 2, 2024

cyriltovena left a comment

feat: return real data in detected fields endpoint #12421

feat: return real data in detected fields endpoint #12421

Conversation

trevorwhitney commented Apr 1, 2024

trevorwhitney Apr 1, 2024

Choose a reason for hiding this comment

trevorwhitney Apr 1, 2024

Choose a reason for hiding this comment

trevorwhitney Apr 1, 2024

Choose a reason for hiding this comment

trevorwhitney commented Apr 1, 2024

cyriltovena Apr 2, 2024

Choose a reason for hiding this comment

cyriltovena left a comment

Choose a reason for hiding this comment

trevorwhitney commented Apr 2, 2024

cyriltovena left a comment

Choose a reason for hiding this comment