fix: do reconcile secrets with konghq.com/credential label

[pmalek]

What this PR does / why we need it:

Do reconcile secrets with konghq.com/credential label.

Which issue this PR fixes:

Closes: #5398

Special notes for your reviewer:

PR Readiness Checklist:

Complete these before marking the PR as ready to review:

• the CHANGELOG.md release notes have been updated to reflect any significant (and particularly user-facing) changes introduced by this PR

[codecov]

Codecov Report

Attention: Patch coverage is 88.88889% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 73.9%. Comparing base (2570997) to head (a9dcc45).
Report is 1 commits behind head on main.

Files	Patch %	Lines
...ion/validation/consumers/credentials/validation.go	50.0%	1 Missing ⚠️

Additional details and impacted files

@@          Coverage Diff          @@
##            main   #5816   +/-   ##
=====================================
  Coverage   73.9%   73.9%           
=====================================
  Files        176     176           
  Lines      18207   18210    +3     
=====================================
+ Hits       13457   13469   +12     
+ Misses      3748    3738   -10     
- Partials    1002    1003    +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[rainest]

Okay, yes, technically for unit test coverage but,

"konghq.com/ca-cert": "false"

WHOMST?

On a less joking note, don't we technically want the reference behavior? I think we want to pull in only Secrets we'll use, just actually trigger the reconciles properly and pull them in immediately when a KongConsumer asks for them. This would pull in others instances' Secrets in a multi-install environment.

That said, I'm not sure exactly what's going on with the existing reconcile behavior that apparently does demonstrably cause issues. My best guess is that there's a race condition if you create a KongConsumer and its credential Secret simultaneously, and that we're not requeuing a KongConsumer reconcile if we can't find a requested Secret, so it happily stores the KongConsumer without its Secret if the latter isn't available in the API server first.

The main reduction in ingested Secrets we get from having the label at all still outweighs the old "ingest every Secret" behavior, so excluding other instances' Secrets is maybe more a nice to have. Maybe create an issue for it, but I'm thinking err on the side of handling it in the simpler way until there's demand for it/someone hits a situation where they have enough tenants that ingest across all is indeed a performance concern.

[pmalek]

@rainest Created #5819 to track the issue you mentioned. Feel free to update it or comment on it.