cilium policy delete root takes a long time

[tgraf]

The policy regeneration takes a long time which slows down all policy API interactions.

[aanm]

When the root node gets deleted, all endpoints need to recalculate their policy (as expected), unfortunately this happens sequentially, any reason not to do it in parallel? I'm doing some tests to see out it goes.

[tgraf]

When the root node gets deleted, all endpoints need to recalculate their policy (as expected), unfortunately this happens sequentially, any reason not to do it in parallel? I'm doing some tests to see out it goes.

I think it is the actual rebuilding of programs that is expensive. The policy calculation is probably quick but the rebuilding should be moved to go subroutines. We should probably have a go subroutine listening on a channel for build requests and then spawn n number of build workers.

I think we should limit it to one build per endpoint though and to n parallel builds in total.

A first good step is to measure how much of the time is spent retrieving the identity from kvstore, policy calc and then actual building.

All of this is done within endpointsMU lock which blocks out any read operation. This is also why endpoint list eventually takes a long time.

[aanm]

Fixed by #424