-
Notifications
You must be signed in to change notification settings - Fork 550
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BUG] flytescheduler
not starting with istio authorization policy
#2562
Comments
Thank you for opening your first issue here! 🛠 |
Hi @bstadlbauer , doesn't the k8 backoff policy kick in even in this case and eventually bringup the scheduler container. |
@pmahindrakar-oss / @bstadlbauer what if we just remove the init container? I think scheduler will |
@pmahindrakar-oss Not a K8s expert, but from what I see ATM, the @kumare3 That would be nice! What is the purpose of |
@bstadlbauer @kumare3 we are using precheck as kind of readiness check where it doesn't initialize the main pod until the init-container precheck completes. It validates that flyteadmin is in SERVING status. This indirectly also validates that DB is up which is used by the scheduler to read the schedules .
We can probably add a flag to skip the precheck and you can pass this from your values file. Will that work. I want to avoid removing this check as it might start sending alerts on schedule failures using FailedExecutionCounter metric. For quick verification you can try removing the init container and check if the scheduler pod eventually comes up |
Init containers are ran in the order they appear in the pod spec Hashicorp Vault gets around this by modifying the yaml so that the Vault init container runs before all other init containers: There could be an annotation within Flyte to determine ordering in the same way |
Within Istio's mesh config, there is a @bstadlbauer We can test this by modifying the labs service mesh with this value |
Describe the bug
When using istio's Authorization Policiy in conjunction with the
flyte-core
helm chart, theflytescheduler
pod won't start, as theflytescheduler-check
init container tries to accessflyteadmin
before the istio proxy gets injected, thus getting aRBAC: access denied
error.Looking at options, I've found this conversation, as well as istio/istio#11130, from which I gathered that there might be no workaround ATM.
A possible solution they are suggesting would be to move all init-logic into a
real
(non-init) container, as theistio
proxy should be ready by then. Do you have any thoughts on this or another possible workaround? We are currently not blocked by this as we're not using the scheduler right now.Expected behavior
The
flytescheduler
pod coming up.Additional context to reproduce
No response
Screenshots
No response
Are you sure this issue hasn't been raised already?
Have you read the Code of Conduct?
The text was updated successfully, but these errors were encountered: