feat(caching): Support caching `/series` and `/labels` query results #11539

kavirajk · 2023-12-20T23:25:32Z

What this PR does / why we need it:
Add support for caching metadata queries (both series and labels).
caching happens after splitting similar to other types of queries.

This pr adds the following configs to enable them.

cache_series_results: true|false (default false)
cache_label_results: true|false (default false)

And the cache backend for them can be configured using series_results_cache and label_results_cache blocks under the query_range section.

Currently the split interval for metadata queries is fixed and defaults to 24h, this pr makes it configurable by introducing split_metadata_queries_by_interval

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:

Checklist

Reviewed the CONTRIBUTING.md guide (required)
Documentation added
Tests updated
CHANGELOG.md updated
- If the change is worth mentioning in the release notes, add add-to-release-notes label
Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
For Helm chart changes bump the Helm chart version in production/helm/loki/Chart.yaml and update production/helm/loki/CHANGELOG.md and production/helm/loki/README.md. Example PR
If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

github-actions · 2023-12-20T23:28:47Z

Trivy scan found the following vulnerabilities:

HIGH, Target: docker.io/grafana/loki:main-d2f4378 (alpine 3.18.4), Type: alpine openssl: Incorrect cipher key and IV length processing in libcrypto3 v3.1.3-r0. Fixed in v3.1.4-r0
HIGH, Target: docker.io/grafana/loki:main-d2f4378 (alpine 3.18.4), Type: alpine openssl: Incorrect cipher key and IV length processing in libssl3 v3.1.3-r0. Fixed in v3.1.4-r0
\nTo see more details on these vulnerabilities, and how/where to fix them, please run docker build -t grafana/loki:main-d2f4378 -f cmd/loki/Dockerfile .
trivy i grafana/loki:main-d2f4378 on your branch. If these were not introduced by your PR, please considering fixing them in via a subsequent PR. Thanks!

Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>

docs/sources/configure/_index.md

Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>

dannykopping

Overall looks great! Nice work fellas

dannykopping · 2024-01-03T09:17:09Z

docs/sources/configure/_index.md

+
+# If series_results_cache is not configured and cache_series_results is true,
+# the config for the results cache is used.
+series_results_cache:


I don't see much value in having separate cache configs for this, label results, and normal result cache. IMHO we should err on the side of simplicity until we find a very compelling case for needing separate caches for all 3.

Our config is already monstrously complex; we should do what we can to not make it more so.

agree to this. one downside is that we would lose granularity with stats and metric collection since they are tried to the cache instance. labels, series and results caching all would be clubbed together.

How valuable are the metrics? Could we use a different label based on which "subcomponent" is observing metrics?

Good point. Thought about this when introducing it. Now we have so much repeated configs for normal results cache (metric range queries), index stats, volume, labels, series and for metric instant queries in future. All have exactly same configs.

One downside is it's hard to clean this up without breaking changes I think. Other option is to introduce shared config with "subcomponent" label but only for labels and series without breaking existing configs, then we loose the consistency with how we configure different results cache.

I would be fine with leaving this as is for now but combining them all in v3.0 with breaking changes, thoughts?

Good idea. Adding it to the epic 👍

another thing that I am worried about is blowing up the size of stats proto. does encoding ignore fields that are not set? if that's the case we don't have to worry about this

only one of these would be set for a given request, I guess we could just collapse this into a single field with a new cache type field

loki/pkg/logqlmodel/stats/stats.proto

Lines 40 to 60 in 40d64ef

Cache result = 3 [

(gogoproto.nullable) = false,

(gogoproto.jsontag) = "result"

];

Cache statsResult = 4 [

(gogoproto.nullable) = false,

(gogoproto.jsontag) = "statsResult"

];

Cache volumeResult = 5 [

(gogoproto.nullable) = false,

(gogoproto.jsontag) = "volumeResult"

];

Cache seriesResult = 6 [

(gogoproto.nullable) = false,

(gogoproto.jsontag) = "seriesResult"

];

Cache labelResult = 7 [

(gogoproto.nullable) = false,

(gogoproto.jsontag) = "labelResult"

];

}

pkg/logql/metrics.go

pkg/loki/config_wrapper.go

pkg/querier/queryrange/codec.go

pkg/querier/queryrange/labels_cache.go

pkg/querier/queryrange/roundtrip.go

pkg/querier/queryrange/series_cache.go

cmd/loki/loki-local-with-memcached.yaml

Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>

dannykopping

LGTM

…11539) **What this PR does / why we need it**: Add support for caching metadata queries (both series and labels). caching happens after splitting similar to other types of queries. This pr adds the following configs to enable them. ``` cache_series_results: true|false (default false) cache_label_results: true|false (default false) ``` And the cache backend for them can be configured using `series_results_cache` and `label_results_cache` blocks under the `query_range` section. Currently the split interval for metadata queries is fixed and defaults to 24h, this pr makes it configurable by introducing `split_metadata_queries_by_interval` **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](0d4416a) --------- Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> Co-authored-by: Ashwanth Goli <iamashwanth@gmail.com>

…rafana#11539) **What this PR does / why we need it**: Add support for caching metadata queries (both series and labels). caching happens after splitting similar to other types of queries. This pr adds the following configs to enable them. ``` cache_series_results: true|false (default false) cache_label_results: true|false (default false) ``` And the cache backend for them can be configured using `series_results_cache` and `label_results_cache` blocks under the `query_range` section. Currently the split interval for metadata queries is fixed and defaults to 24h, this pr makes it configurable by introducing `split_metadata_queries_by_interval` **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](grafana@d10549e) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](grafana@0d4416a) --------- Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> Co-authored-by: Ashwanth Goli <iamashwanth@gmail.com>

pull-request-size bot added the size/XL label Dec 20, 2023

kavirajk and others added 10 commits December 21, 2023 18:37

feature(cache): Support caching for metadata query results

54b42a2

Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>

idk

6139807

Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>

fix TestSeriesCache

fe12193

Complete series cache tests

ec99784

have separate loki config to run with memcached

7158994

Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>

Add cache hit metrics and configs to test

e4d1302

Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>

fix memcached config

4cb7145

Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>

stats.proto change to support series cache

55a7c8a

Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>

tidy up the tests

64175cf

update failing tests to include the new stats

d716461

ashwanthgoli force-pushed the kavirajk/metadata-caching branch from 3d2bf8d to d716461 Compare December 21, 2023 13:13

ashwanthgoli added 3 commits December 26, 2023 17:51

apply default configs to series cache

01f4792

add test for GenerateCacheKey

13ff271

make doc

634b43b

github-actions bot added the type/docs Issues related to technical documentation; the Docs Squad uses this label across many repositories label Dec 26, 2023

ashwanthgoli added 6 commits December 26, 2023 19:16

Merge branch 'main' into kavirajk/metadata-caching

c654c82

preserve cache prefix

ed3db1a

fixup! preserve cache prefix

836b9ee

s/querier.cache-series-results/frontend.cache-series-results

dfe174f

retain headers when merging series response

ee9667f

add label results cache

2cc04cf

pull-request-size bot added size/XXL and removed size/XL labels Dec 29, 2023

ashwanthgoli added 2 commits December 29, 2023 12:34

make format && make doc

bb2280d

add CHANGELOG

40f69e3

ashwanthgoli marked this pull request as ready for review December 29, 2023 10:01

ashwanthgoli requested a review from a team as a code owner December 29, 2023 10:01

Merge branch 'main' into kavirajk/metadata-caching

db966bd

ashwanthgoli added 2 commits January 2, 2024 13:32

introduce split_metadata_queries_by_interval

ccd4a55

make doc

32f4d20

ashwanthgoli reviewed Jan 2, 2024

View reviewed changes

docs/sources/configure/_index.md Outdated Show resolved Hide resolved

kavirajk and others added 2 commits January 2, 2024 15:03

Make flags prefix consistent with rest of result cache flags

92a7e43

Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>

nit

8706162

dannykopping reviewed Jan 3, 2024

View reviewed changes

slim-bean reviewed Jan 3, 2024

View reviewed changes

cmd/loki/loki-local-with-memcached.yaml Outdated Show resolved Hide resolved

PR remarks

40d64ef

Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>

kavirajk requested review from dannykopping, ashwanthgoli and slim-bean January 4, 2024 08:14

dannykopping approved these changes Jan 4, 2024

View reviewed changes

kavirajk merged commit ce57448 into main Jan 4, 2024
8 checks passed

kavirajk deleted the kavirajk/metadata-caching branch January 4, 2024 12:12

kavirajk mentioned this pull request Jan 29, 2024

feat: Support split align and caching for instant metric query results #11814

Merged

8 tasks

This was referenced Feb 13, 2024

Series API should be cacheable #2168

Closed

Labels API should partition by time & be cacheable #2169

Closed

loki-gh-app bot mentioned this pull request Mar 27, 2024

chore(add-major-release-workflow): release 3.0.0-rc.1 #12380

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(caching): Support caching `/series` and `/labels` query results #11539

feat(caching): Support caching `/series` and `/labels` query results #11539

kavirajk commented Dec 20, 2023 •

edited by ashwanthgoli

github-actions bot commented Dec 20, 2023 •

edited

dannykopping left a comment

dannykopping Jan 3, 2024

ashwanthgoli Jan 3, 2024

dannykopping Jan 3, 2024

kavirajk Jan 4, 2024

dannykopping Jan 4, 2024

kavirajk Jan 4, 2024

ashwanthgoli Jan 4, 2024

dannykopping left a comment

	Cache result = 3 [
	(gogoproto.nullable) = false,
	(gogoproto.jsontag) = "result"
	];
	Cache statsResult = 4 [
	(gogoproto.nullable) = false,
	(gogoproto.jsontag) = "statsResult"
	];
	Cache volumeResult = 5 [
	(gogoproto.nullable) = false,
	(gogoproto.jsontag) = "volumeResult"
	];
	Cache seriesResult = 6 [
	(gogoproto.nullable) = false,
	(gogoproto.jsontag) = "seriesResult"
	];
	Cache labelResult = 7 [
	(gogoproto.nullable) = false,
	(gogoproto.jsontag) = "labelResult"
	];
	}

feat(caching): Support caching /series and /labels query results #11539

feat(caching): Support caching /series and /labels query results #11539

Conversation

kavirajk commented Dec 20, 2023 • edited by ashwanthgoli

github-actions bot commented Dec 20, 2023 • edited

dannykopping left a comment

Choose a reason for hiding this comment

dannykopping Jan 3, 2024

Choose a reason for hiding this comment

ashwanthgoli Jan 3, 2024

Choose a reason for hiding this comment

dannykopping Jan 3, 2024

Choose a reason for hiding this comment

kavirajk Jan 4, 2024

Choose a reason for hiding this comment

dannykopping Jan 4, 2024

Choose a reason for hiding this comment

kavirajk Jan 4, 2024

Choose a reason for hiding this comment

ashwanthgoli Jan 4, 2024

Choose a reason for hiding this comment

dannykopping left a comment

Choose a reason for hiding this comment

feat(caching): Support caching `/series` and `/labels` query results #11539

feat(caching): Support caching `/series` and `/labels` query results #11539

kavirajk commented Dec 20, 2023 •

edited by ashwanthgoli

github-actions bot commented Dec 20, 2023 •

edited