Cache secrets in interceptors with Reflector #594

tragiclifestories · 2020-06-03T09:54:55Z

Changes

This is an alternative implementation of #585 using client-go's cache package. We create a reflector for secrets across all namespaces and use it as a caching layer for secrets in both the Github/lab webhook parsers and CEL compareSecret calls.

I'm putting this up as a draft to get immediate feedback. Still need to test. Also, now that it's also backing CEL functions, WebhookSecretStore is probably the wrong name ...

Submitter Checklist

These are the criteria that every PR should meet, please check them off as you
review them:

Includes tests (if functionality changed/added)
Includes docs (if user facing)
Commit messages follow commit message best practices

See the contribution guide for more details.

Release Notes

Describe any user facing changes here, or delete this block.

Examples of user facing changes:
- API changes
- Bug fixes
- Any changes in behavior

linux-foundation-easycla · 2020-06-03T09:54:58Z

The committers are authorized under a signed CLA.

✅ Lawrence Jones (61ddc25)
✅ James Turley (8dfdd85, e20bb92, 88e7977)

tekton-robot · 2020-06-03T09:55:09Z

Hi @tragiclifestories. Thanks for your PR.

I'm waiting for a tektoncd member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

pkg/interceptors/interceptors.go

jace-ys · 2020-06-03T10:06:54Z

pkg/interceptors/interceptors.go

+	Get(sr triggersv1.SecretRef) ([]byte, error)
+}
+
+type webhookSecretStore struct {


To address your question, maybe just secretStore would work?

Yes, which of course was what it was when I first wrote this struct def out 😅

jace-ys · 2020-06-03T10:13:35Z

pkg/interceptors/interceptors.go

-		ns = eventListenerNamespace
+// Get returns the secret value for a given SecretRef.
+func (ws *WebhookSecretStore) Get(sr triggersv1.SecretRef) ([]byte, error) {
+	cachedObj, ok, _ := ws.store.GetByKey(getKey(sr))


Not very familiar with the cache store, so curious why we're ignoring the error here?

It turns out that if you spelunk into the code, error is always nil from this particular store implementation. However, I will probably handle it anyway, since it may turn out that a different store makes more sense in future.

pkg/interceptors/interceptors.go

bigkevmcd · 2020-06-24T08:49:35Z

pkg/sink/sink.go

 		case i.GitLab != nil:
-			interceptor = gitlab.NewInterceptor(i.GitLab, r.KubeClientSet, r.EventListenerNamespace, log)
+			interceptor = gitlab.NewInterceptor(i.GitHub, r.WebhookSecretStore, log)


This looks like a copy/paste mistake?

it does indeed ...

bigkevmcd · 2020-06-24T08:55:45Z

Definitely would help with scaling both volume of requests, and triggers that use secrets.

This caching layer is something we do need to consider if we go for a process-based plugin mechanism.

tragiclifestories · 2020-06-25T11:36:39Z

@bigkevmcd I'll be picking this up again. There are actually a whole host of uncached requests going on in processTrigger and it seems like a lot of caching will be necessary to keep this performant with a lot of triggers.

tragiclifestories · 2020-06-30T14:15:26Z

So, on further investigation, the fix for this problem implemented in #595 did not work. The reason for this seems to be that the the triggers are executed asynchronously, and thus all immediately stampede to get the secret at the very top of the ExecuteTrigger methods, so in practice you get 100% cache misses, 100% of the time. Which was not what was intended, to put it mildly.

This version speeds things up very significantly compared to master in my tests - more thorough than last time around ;-). It takes about 3s to process the hideous 500-github-trigger YAML I put in the examples in this PR, as opposed to something like 3 minutes on master - and most of that is now due to another, far less severe performance problem with compiling CEL expressions (will raise an issue when I get a second).

tragiclifestories · 2020-06-30T15:38:29Z

pkg/interceptors/interceptors.go

 	}

-	return make(map[string]interface{})
+	lw := cache.NewListWatchFromClient(cs.CoreV1().RESTClient(), "secrets", metav1.NamespaceAll, fields.Everything())


Using NamespaceAll here is probably not a long term solution, since it requires cluster-wide list secrets in RBAC. I guess we could wrap a map of stores here - one per namespace referenced in calls to Get ...

tragiclifestories · 2020-06-30T15:40:13Z

cmd/eventlistenersink/main.go

@@ -80,12 +82,15 @@ func main() {
 		logger.Fatal(err)
 	}

+	webhookSecretStore := interceptors.NewWebhookSecretStore(kubeClient, sinkArgs.ElNamespace, 5*time.Second, stopCh)


The interval should probably be configurable here ...

tekton-robot · 2020-07-07T10:22:53Z

@tragiclifestories: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

r.URL is nil sometimes, in which case this code will panic. This fix just handles the nil case.

Co-authored-by: Jace Tan <jaceys.tan@gmail.com>

tekton-robot · 2020-08-15T06:49:18Z

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

Send feedback to tektoncd/plumbing.

tekton-robot · 2020-08-15T06:49:19Z

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.

/lifecycle rotten

Send feedback to tektoncd/plumbing.

tekton-robot · 2020-08-15T06:49:20Z

@tekton-robot: Closed this PR.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

Send feedback to tektoncd/plumbing.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

tragiclifestories · 2020-08-15T09:20:51Z

/reopen

tekton-robot · 2020-08-15T09:20:59Z

@tragiclifestories: Reopened this PR.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

tekton-robot · 2020-08-15T09:21:03Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please assign wlynch
You can assign the PR to them by writing /assign @wlynch in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

tragiclifestories · 2020-08-15T09:22:19Z

I hope I'll get some time to update this in the next week or so. Any ideas on how to write tests for this - and why they don't currently pass - very much welcome. FWIW, we've a version of this PR running in production for over a month with no incident.

tekton-robot · 2021-01-11T16:46:09Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale with a justification.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle stale

Send feedback to tektoncd/plumbing.

tragiclifestories · 2021-01-11T17:29:36Z

/remove-lifecycle stale Promise I'll get to this soon ...

tragiclifestories · 2021-01-11T17:29:49Z

/remove-lifecycle stale

tekton-robot · 2021-03-10T15:03:42Z

@tragiclifestories: Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our release note process to remove it.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

bobcatfish · 2021-05-06T19:41:08Z

hey @tragiclifestories ! do you have any strong feelings about us closing this PR for now? You can absolutely re-open it (or open a new one) when you're ready to get back to it

tragiclifestories · 2021-05-08T09:28:28Z

At this point the rebase would probably take longer than redoing it from scratch 😅 . I think this PR can be closed.

I will try to find a moment to see if the bug still exists - it may well not for all I know ...

tekton-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 3, 2020

tekton-robot requested review from bobcatfish and vtereso June 3, 2020 09:54

tekton-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jun 3, 2020

tekton-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jun 3, 2020

tragiclifestories mentioned this pull request Jun 3, 2020

Cache trigger secrets for the duration of request #585

Closed

3 tasks

jace-ys reviewed Jun 3, 2020

View reviewed changes

tragiclifestories force-pushed the cache-trigger-secrets-reflector branch 2 times, most recently from 9ae0f53 to 6eacf24 Compare June 3, 2020 16:38

bigkevmcd reviewed Jun 24, 2020

View reviewed changes

tragiclifestories force-pushed the cache-trigger-secrets-reflector branch from 2ce7e18 to e5f068d Compare June 25, 2020 14:16

tekton-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jun 30, 2020

tragiclifestories force-pushed the cache-trigger-secrets-reflector branch from 0155770 to 95c412b Compare June 30, 2020 15:32

tragiclifestories commented Jun 30, 2020

View reviewed changes

tragiclifestories marked this pull request as ready for review June 30, 2020 15:38

tekton-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 30, 2020

tragiclifestories commented Jun 30, 2020

View reviewed changes

tragiclifestories force-pushed the cache-trigger-secrets-reflector branch from 4406021 to 1a33db2 Compare June 30, 2020 18:03

tekton-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 7, 2020

James Turley and others added 4 commits July 9, 2020 09:42

Check that URL exists before stringifying it in CEL context

9fc9d5c

r.URL is nil sometimes, in which case this code will panic. This fix just handles the nil case.

wip

1eaeed3

use webhook secret store

a203f5b

Change error message capitalisation

799568e

Co-authored-by: Jace Tan <jaceys.tan@gmail.com>

wip

2a7dd42

tragiclifestories force-pushed the cache-trigger-secrets-reflector branch 2 times, most recently from cb3018a to 3b4cc64 Compare July 9, 2020 11:16

James Turley added 2 commits July 9, 2020 15:24

non-passing unit test

02fdba0

Still doesn't work

e35c790

tragiclifestories force-pushed the cache-trigger-secrets-reflector branch from 3b4cc64 to e35c790 Compare July 9, 2020 14:46

tekton-robot closed this Aug 15, 2020

tekton-robot added the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label Aug 15, 2020

tekton-robot reopened this Aug 15, 2020

dibyom removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label Oct 13, 2020

dibyom mentioned this pull request Oct 16, 2020

TEP-0026- Add TEP for interceptor plugins tektoncd/community#229

Merged

tekton-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jan 11, 2021

tekton-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jan 11, 2021

Base automatically changed from master to main March 10, 2021 15:03

tekton-robot added the do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. label Mar 10, 2021

tragiclifestories closed this May 8, 2021

dibyom mentioned this pull request Jan 20, 2022

Avoid watching all secrets in the cluster #1274

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cache secrets in interceptors with Reflector #594

Cache secrets in interceptors with Reflector #594

tragiclifestories commented Jun 3, 2020

linux-foundation-easycla bot commented Jun 3, 2020 •

edited

Loading

tekton-robot commented Jun 3, 2020

jace-ys Jun 3, 2020 •

edited

Loading

tragiclifestories Jun 3, 2020

jace-ys Jun 3, 2020

tragiclifestories Jun 3, 2020

bigkevmcd Jun 24, 2020

tragiclifestories Jun 25, 2020

bigkevmcd commented Jun 24, 2020

tragiclifestories commented Jun 25, 2020

tragiclifestories commented Jun 30, 2020 •

edited

Loading

tragiclifestories Jun 30, 2020

tragiclifestories Jun 30, 2020

tekton-robot commented Jul 7, 2020

tekton-robot commented Aug 15, 2020

tekton-robot commented Aug 15, 2020

tekton-robot commented Aug 15, 2020

tragiclifestories commented Aug 15, 2020

tekton-robot commented Aug 15, 2020

tekton-robot commented Aug 15, 2020

tragiclifestories commented Aug 15, 2020

tekton-robot commented Jan 11, 2021

tragiclifestories commented Jan 11, 2021

tragiclifestories commented Jan 11, 2021

tekton-robot commented Mar 10, 2021

bobcatfish commented May 6, 2021

tragiclifestories commented May 8, 2021

Cache secrets in interceptors with Reflector #594

Cache secrets in interceptors with Reflector #594

Conversation

tragiclifestories commented Jun 3, 2020

Changes

Submitter Checklist

Release Notes

linux-foundation-easycla bot commented Jun 3, 2020 • edited Loading

tekton-robot commented Jun 3, 2020

jace-ys Jun 3, 2020 • edited Loading

Choose a reason for hiding this comment

tragiclifestories Jun 3, 2020

Choose a reason for hiding this comment

jace-ys Jun 3, 2020

Choose a reason for hiding this comment

tragiclifestories Jun 3, 2020

Choose a reason for hiding this comment

bigkevmcd Jun 24, 2020

Choose a reason for hiding this comment

tragiclifestories Jun 25, 2020

Choose a reason for hiding this comment

bigkevmcd commented Jun 24, 2020

tragiclifestories commented Jun 25, 2020

tragiclifestories commented Jun 30, 2020 • edited Loading

tragiclifestories Jun 30, 2020

Choose a reason for hiding this comment

tragiclifestories Jun 30, 2020

Choose a reason for hiding this comment

tekton-robot commented Jul 7, 2020

tekton-robot commented Aug 15, 2020

tekton-robot commented Aug 15, 2020

tekton-robot commented Aug 15, 2020

tragiclifestories commented Aug 15, 2020

tekton-robot commented Aug 15, 2020

tekton-robot commented Aug 15, 2020

tragiclifestories commented Aug 15, 2020

tekton-robot commented Jan 11, 2021

tragiclifestories commented Jan 11, 2021

tragiclifestories commented Jan 11, 2021

tekton-robot commented Mar 10, 2021

bobcatfish commented May 6, 2021

tragiclifestories commented May 8, 2021

linux-foundation-easycla bot commented Jun 3, 2020 •

edited

Loading

jace-ys Jun 3, 2020 •

edited

Loading

tragiclifestories commented Jun 30, 2020 •

edited

Loading