store: Add Groupcache as a cache backend #4818

akanshat · 2021-10-30T18:18:34Z

I added CHANGELOG entry for this change.
Change is not relevant to the end user.

Changes

Added Groupcache.
Implemented Cache interface to groupcache.
Added a new package cachekey.

Verification

Existing tests passed.

GiedriusS

Seems like you have some unrelated changes here? Could you please rebase on main and don't forget the DCO ;P

akanshat · 2021-10-30T19:01:00Z

Okay, I was hoping you won't see this. 😆

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

Signed-off-by: akanshat <akanshat1999@gmail.com>

pkg/cache/groupcache.go

GiedriusS

Looks good, just need to convert to https://github.com/vimeo/galaxycache now + add some e2e tests together with metrics? 😄

akanshat · 2021-11-01T15:59:06Z

I am planning to make groupcache work properly first, and write a test for it too. Then I want to convert it to galaxycache.

Signed-off-by: akanshat <akanshat1999@gmail.com>

pkg/cache/groupcache.go

GiedriusS

After converting those two methods to have a pointer as the receiver, the metrics appear and I now see the following error when running the e2e tests:

             unexpected error: unable to find metrics [thanos_cache_groupcache_hits_total] with expected values after 50 retries. Last error: <nil>. Last values: [31]

This is probably because Store "refreshed" the view of blocks asynchronously hence increasing that counter (:

akanshat · 2021-11-17T12:11:04Z

@GiedriusS What did you mean by .Get(), I couldn't find any such method here.

GiedriusS · 2021-11-17T12:13:55Z

@GiedriusS What did you mean by .Get(), I couldn't find any such method here.

Get() is a method on AtomicInt which is the type of stats.Gets and so on. So, it should be:

	ch <- prometheus.MustNewConstMetric(s.gets, prometheus.CounterValue, float64(stats.Gets.Get()))

And so on, I think.

Signed-off-by: akanshat <akanshat1999@gmail.com>

hanjm · 2021-11-17T12:45:29Z

Awesome feature! I am headache in lots of memcache timeout log, groupcache will save me 😄

pkg/cache/groupcache.go

test/e2e/store_gateway_test.go

onprem · 2021-11-17T13:30:47Z

test/e2e/store_gateway_test.go

+			},
+		)
+
+		testutil.Ok(t, store1.WaitSumMetricsWithOptions(e2e.Greater(0), []string{`thanos_cache_groupcache_loads_total`}))


Lets check for the generic metrics we check in memcached test here as well.

pkg/cache/groupcache.go

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

pkg/cache/groupcache.go

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

Signed-off-by: akanshat <akanshat1999@gmail.com>

Adds TTL support to galaxycache as per this pull request: thanos-community/galaxycache#1 Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

Signed-off-by: akanshat <akanshat1999@gmail.com>

move cache TTLs to a struct

cache: add TTL support

GiedriusS

It became a big PR in the end but we've done it in small iterations so it's all good 🎉

GiedriusS · 2022-01-06T14:33:10Z

Dockerfile.multi-stage

@@ -1,6 +1,6 @@
 # By default we pin to amd64 sha. Use make docker to automatically adjust for arm64 versions.
 ARG BASE_DOCKER_SHA="14d68ca3d69fceaa6224250c83d81d935c053fb13594c811038c461194599973"
-FROM golang:1.16-alpine3.12 as builder
+FROM golang:1.17-alpine3.15 as builder


This change also seems unrelated. Probably just needs a rebase since this exact version is used in main. Or you can do git merge origin/main to avoid rewriting the history. I'd prefer that ;P

CHANGELOG.md

…cket Signed-off-by: akanshat <akanshat1999@gmail.com>

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

GiedriusS

🎉 🎉 🎉
#5037 made this to track the next steps

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

GiedriusS

Just made some random changes to make the linters happy, I hope you don't mind 👍

akanshat · 2022-01-06T20:22:33Z

Thanks!

GiedriusS · 2022-01-06T20:36:23Z

Only docs are failing so I'll merge with admin rights 🎉

akanshat force-pushed the groupcache_caching_bucket branch from eb6b62d to 14247ce Compare October 30, 2021 18:48

GiedriusS reviewed Oct 30, 2021

View reviewed changes

cache: add groupcache

0ae3021

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

akanshat force-pushed the groupcache_caching_bucket branch from 14247ce to b7bc057 Compare October 30, 2021 19:03

akanshat added 3 commits October 31, 2021 03:02

implement Cache interface on groupcache

a99bd9e

Signed-off-by: akanshat <akanshat1999@gmail.com>

move BucketCacheKey to a new package

2bf5358

Signed-off-by: akanshat <akanshat1999@gmail.com>

add copyright to new pkg cachekey

87e8f64

Signed-off-by: akanshat <akanshat1999@gmail.com>

akanshat force-pushed the groupcache_caching_bucket branch from b7bc057 to 87e8f64 Compare October 30, 2021 21:33

GiedriusS reviewed Nov 1, 2021

View reviewed changes

pkg/cache/groupcache.go Outdated Show resolved Hide resolved

GiedriusS reviewed Nov 1, 2021

View reviewed changes

akanshat added 4 commits November 6, 2021 20:59

add a fix to return partial results if Get fails

d139fef

Signed-off-by: akanshat <akanshat1999@gmail.com>

migrate from groupcache to galaxycache

0b7ca17

Signed-off-by: akanshat <akanshat1999@gmail.com>

instrument metrics for galaxyCache

6bca1ff

Signed-off-by: akanshat <akanshat1999@gmail.com>

add e2e test for store with groupcache

8f8c261

Signed-off-by: akanshat <akanshat1999@gmail.com>

GiedriusS reviewed Nov 15, 2021

View reviewed changes

pkg/cache/groupcache.go Outdated Show resolved Hide resolved

GiedriusS reviewed Nov 15, 2021

View reviewed changes

pkg/cache/groupcache.go Outdated Show resolved Hide resolved

GiedriusS reviewed Nov 15, 2021

View reviewed changes

pkg/cache/groupcache.go Outdated Show resolved Hide resolved

GiedriusS reviewed Nov 15, 2021

View reviewed changes

fix collector interface on CacheStatsCollector

8621c68

Signed-off-by: akanshat <akanshat1999@gmail.com>

akanshat marked this pull request as ready for review November 17, 2021 12:26

onprem reviewed Nov 17, 2021

View reviewed changes

pkg/cache/groupcache.go Show resolved Hide resolved

cache: fix / clean up tests

a6f1945

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

onprem reviewed Nov 17, 2021

View reviewed changes

pkg/cache/groupcache.go Outdated Show resolved Hide resolved

groupcache: support IterVerb

63b212b

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

GiedriusS and others added 12 commits November 17, 2021 18:00

groupcache: changes according to comments

992e2e1

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

Merge branch 'main' into groupcache_caching_bucket

2bf5594

Signed-off-by: akanshat <akanshat1999@gmail.com>

store: fix groupcache test

3c12c2e

Signed-off-by: akanshat <akanshat1999@gmail.com>

cache: add TTL support

088b9d3

Adds TTL support to galaxycache as per this pull request: thanos-community/galaxycache#1 Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

move caching_bucket_config to pkg cache

b15440e

Signed-off-by: akanshat <akanshat1999@gmail.com>

modify groupcacheCfg

c7c1460

Signed-off-by: akanshat <akanshat1999@gmail.com>

remove duplicate CachingBucketConfig

57b1ab5

Signed-off-by: akanshat <akanshat1999@gmail.com>

remove duplicate calls to NewCachingBucketConfig

13eaa61

Signed-off-by: akanshat <akanshat1999@gmail.com>

refactor cache configuration

1e59562

Signed-off-by: akanshat <akanshat1999@gmail.com>

implement suggestions from comments

5f2f7cf

Signed-off-by: akanshat <akanshat1999@gmail.com>

Merge pull request #2 from akanshat/add_ttl

d0ef6f2

move cache TTLs to a struct

Merge pull request #1 from GiedriusS/add_ttl

a8ba37f

cache: add TTL support

pull-request-size bot added the size/XXL label Jan 6, 2022

GiedriusS reviewed Jan 6, 2022

View reviewed changes

akanshat and others added 2 commits January 6, 2022 22:56

Merge remote-tracking branch 'origin/main' into groupcache_caching_bu…

e6c3d73

…cket Signed-off-by: akanshat <akanshat1999@gmail.com>

*: formatting changes

d61e97c

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

GiedriusS mentioned this pull request Jan 6, 2022

Next steps with groupcache (galaxycache) #5037

Open

7 tasks

GiedriusS previously approved these changes Jan 6, 2022

View reviewed changes

*: linter fixes

16ac6fd

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

GiedriusS dismissed their stale review via 16ac6fd January 6, 2022 20:16

GiedriusS approved these changes Jan 6, 2022

View reviewed changes

GiedriusS enabled auto-merge (squash) January 6, 2022 20:21

GiedriusS disabled auto-merge January 6, 2022 20:36

GiedriusS merged commit 4d24bf8 into thanos-io:main Jan 6, 2022

akanshat mentioned this pull request Feb 12, 2022

Added DefaultTransport function to package exthttp #4655

Closed

2 tasks

akanshat mentioned this pull request Feb 21, 2022

update caching_bucket according to thanos changes cortexproject/cortex#4651

Closed

3 tasks

matej-g mentioned this pull request Feb 25, 2022

*: Cut 0.25.0-rc.0 #5184

Merged

saswatamcode mentioned this pull request Jun 10, 2022

Expose tsdb status in receiver #5402

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

store: Add Groupcache as a cache backend #4818

store: Add Groupcache as a cache backend #4818

akanshat commented Oct 30, 2021 •

edited by onprem

GiedriusS left a comment

akanshat commented Oct 30, 2021

GiedriusS left a comment

akanshat commented Nov 1, 2021

GiedriusS left a comment

akanshat commented Nov 17, 2021

GiedriusS commented Nov 17, 2021

hanjm commented Nov 17, 2021

onprem Nov 17, 2021

GiedriusS left a comment

GiedriusS Jan 6, 2022 •

edited

GiedriusS left a comment

GiedriusS left a comment

akanshat commented Jan 6, 2022

GiedriusS commented Jan 6, 2022

store: Add Groupcache as a cache backend #4818

store: Add Groupcache as a cache backend #4818

Conversation

akanshat commented Oct 30, 2021 • edited by onprem

Changes

Verification

GiedriusS left a comment

Choose a reason for hiding this comment

akanshat commented Oct 30, 2021

GiedriusS left a comment

Choose a reason for hiding this comment

akanshat commented Nov 1, 2021

GiedriusS left a comment

Choose a reason for hiding this comment

akanshat commented Nov 17, 2021

GiedriusS commented Nov 17, 2021

hanjm commented Nov 17, 2021

onprem Nov 17, 2021

Choose a reason for hiding this comment

GiedriusS left a comment

Choose a reason for hiding this comment

GiedriusS Jan 6, 2022 • edited

Choose a reason for hiding this comment

GiedriusS left a comment

Choose a reason for hiding this comment

GiedriusS left a comment

Choose a reason for hiding this comment

akanshat commented Jan 6, 2022

GiedriusS commented Jan 6, 2022

akanshat commented Oct 30, 2021 •

edited by onprem

GiedriusS Jan 6, 2022 •

edited