Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

daemon: move circular initialization of policy.Repository to hive #24073

Merged
merged 3 commits into from Mar 9, 2023

Conversation

lmb
Copy link
Contributor

@lmb lmb commented Feb 28, 2023

IPCache, CachingIdentityAllocator and policy.Repository (really SelectorCache) have a gnarly circular dependency during intialization. Do the simplest possible thing and lift them into hive by providing a constructor that encapsulates the neccessary logic.

This way we can add dependencies on these without having to modularize them first.

Pre-requisite for #23046.

@lmb lmb added release-note/misc This PR makes changes that have no direct user impact. area/modularization labels Feb 28, 2023
Copy link
Contributor

@joamaki joamaki left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah let's do this. Small incremental steps like this to unblock other work seems fine to me (rather than rabbit hole'ing into refactoring ipcache and policy repo :D)

daemon/cmd/policy.go Outdated Show resolved Hide resolved
daemon/cmd/cells.go Outdated Show resolved Hide resolved
daemon/cmd/cells.go Show resolved Hide resolved
@lmb lmb force-pushed the egw-mod-dependencies branch 5 times, most recently from 902e3e9 to 021babd Compare March 1, 2023 12:32
@lmb
Copy link
Contributor Author

lmb commented Mar 1, 2023

/test

@lmb lmb marked this pull request as ready for review March 1, 2023 16:05
@lmb lmb requested review from a team as code owners March 1, 2023 16:05
@lmb lmb requested review from nebril, gandro and aspsk March 1, 2023 16:05
daemon/cmd/policy.go Outdated Show resolved Hide resolved
daemon/cmd/policy.go Show resolved Hide resolved
@lmb lmb force-pushed the egw-mod-dependencies branch 2 times, most recently from cd160bb to f8f1b83 Compare March 2, 2023 16:21
@borkmann
Copy link
Member

borkmann commented Mar 6, 2023

@jibi given egress GW would be great if you could take a look

@lmb
Copy link
Contributor Author

lmb commented Mar 6, 2023

/test

Job 'Cilium-PR-K8s-1.25-kernel-4.19' failed:

Click to show.

Test Name

K8sDatapathServicesTest Checks E/W loadbalancing (ClusterIP, NodePort from inside cluster, etc) with L4 policy Tests NodePort with L4 Policy

Failure Output

FAIL: Request from testclient-xwr4g pod to service tftp://[fd04::11]:30519/hello failed

If it is a flake and a GitHub issue doesn't already exist to track it, comment /mlh new-flake Cilium-PR-K8s-1.25-kernel-4.19 so I can create one.

Job 'Cilium-PR-K8s-1.26-kernel-net-next' failed:

Click to show.

Test Name

K8sDatapathConfig IPv4Only Check connectivity with IPv6 disabled

Failure Output

FAIL: Failure while waiting for all cilium endpoints to reach ready state: Error: context deadline exceeded

If it is a flake and a GitHub issue doesn't already exist to track it, comment /mlh new-flake Cilium-PR-K8s-1.26-kernel-net-next so I can create one.

//
// Qeues must be allocated via [Repository.Start]. The function serves to
// satisfy hive invariants.
func NewStoppedPolicyRepository(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does "Stopped" mean? I found out what it means by reading further along in the code, but it would be good to document it in this function's godoc.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

$ git grep NewPolicyReposit | grep -v test.go
daemon/cmd/daemon.go:           // Must be done before calling policy.NewPolicyRepository() below.
pkg/policy/repository.go:// NewPolicyRepository creates a new policy repository.
pkg/policy/repository.go:func NewPolicyRepository(
$ git grep NewPolicyReposit | grep test.go | wc -l
68

It looks like the main NewPolicyRepository() is now just a helper for test code. It's a bit of a code smell to me to call this NewStoppedPolicyRepository() rather than just having a NewPolicyRepository() + Start(), but I appreciate that all those other test code references are a bit of a pain to fix up if they're all relying on this code as-is to initialize+start the policy repo. I'm OK to defer this for now.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've updated the function doc. I think Jussi's preference was to pass a hive.Lifecycle into the constructor, but that causes a big ripple effect, NewPolicyRepository is used a lot. So I did this botch to keep the changes small(er).

@christarazi christarazi added area/daemon Impacts operation of the Cilium daemon. sig/policy Impacts whether traffic is allowed or denied based on user-defined policies. labels Mar 6, 2023
Copy link
Member

@joestringer joestringer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this looks like a nice step forward. Just one minor nit in something that I think functionally changed during the refactoring.

daemon/cmd/policy.go Outdated Show resolved Hide resolved
lmb added 2 commits March 7, 2023 09:17
Introduce a small wrapper around chan struct{} to be able to
provide it via hive to egressgateway and ipcache.

Signed-off-by: Lorenz Bauer <lmb@isovalent.com>
Add a constructor for Repository that doesn't allocate any queues.
This is necessary to satisfy hive invariants.

Signed-off-by: Lorenz Bauer <lmb@isovalent.com>
IPCache, CachingIdentityAllocator and policy.Repository (really
SelectorCache) have a gnarly circular dependency during initialization.
Do the simplest possible thing and lift them into hive by providing
a constructor that encapsulates the necessary logic.

This way we can add dependencies on these without having to modularize
them first.

During shutdown of the IPCache we first cancel the context
and then call IPCache.Shutdown, see cilium#21676.

Signed-off-by: Lorenz Bauer <lmb@isovalent.com>
@lmb
Copy link
Contributor Author

lmb commented Mar 7, 2023

/test

@lmb lmb added the ready-to-merge This PR has passed all tests and received consensus from code owners to merge. label Mar 8, 2023
@jibi
Copy link
Member

jibi commented Mar 9, 2023

Required reviews are in, CI is green, merging

@jibi jibi merged commit d6503fa into cilium:master Mar 9, 2023
@lmb lmb deleted the egw-mod-dependencies branch March 9, 2023 10:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/daemon Impacts operation of the Cilium daemon. area/modularization ready-to-merge This PR has passed all tests and received consensus from code owners to merge. release-note/misc This PR makes changes that have no direct user impact. sig/policy Impacts whether traffic is allowed or denied based on user-defined policies.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

8 participants