Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix (profile controller): Profile controller fails to update profile owner, fixes #6576 and #5449 #7276

Closed
wants to merge 7 commits into from

Conversation

tzstoyanov
Copy link
Contributor

@tzstoyanov tzstoyanov commented Sep 12, 2023

Fixes issue #6576
Fixes issue #5449

This PR fixes the problem, described in issue #6576. Currently, the profile-controller fails to update the owner as it cannot verify the namespace owner. The PR implements solutions, suggested in the issue.

Additional logic is added, to address issue #5449. If the namespace already exist, and it is not owned by any other profile - check if the namespace has annotation transferToKubeflow: true. If there is such annotation, take the ownership of that namespace and use it for the profile.

@tzstoyanov
Copy link
Contributor Author

/hold still work in progress

@google-oss-prow google-oss-prow bot added size/L and removed size/M labels Sep 13, 2023
@tzstoyanov tzstoyanov changed the title [WIP] Fix (profile controller): Profile controller fails to update profile owner, fixes #6576 Fix (profile controller): Profile controller fails to update profile owner, fixes #6576 Sep 13, 2023
@tzstoyanov
Copy link
Contributor Author

/remove-hold

@juliusvonkohout
Copy link
Member

@tzstoyanov amazing effort!

@thesuperzapper
Copy link
Member

Thanks @tzstoyanov!

@juliusvonkohout @kimwnasptd we need to do some thorough testing of this, but it looks promising.

My main questions before we dive in with testing are:

  • In your testing, I assume that the other resources listed in profile controller fails to update profile owner #6576 are now also updated correctly:
    • RoleBinding/namespaceAdmin
    • AuthorizationPolicy/ns-owner-access-istio
    • ResourceQuota/kf-resource-quota
  • While we are here, we may as well provide a way to explicitly allow the profile controller to take over a namespace that does not already have it listed as an ownerReference:
    • Does anyone have any thoughts about how best to achieve this?

Separately, perhaps we should make a Ready status condition which reflects if the Namespace is healthy and managed by the profile controller (which will allow systems like Helm/ArgoCD to wait until the profile is ready before trying to deploy resources in the namespace).

Effectively it's about the controller creating an element of status.conditions[] with a:

  • reason (think event type, e.g. "Ready")
  • status flag (the current value of this event type, e.g. "True", "False", "Unknown")
  • message (a short human-readable explanation of why it's in this state)

We do something very similar in the Notebooks controller.

It's a bit confusing, but is described here in Kubernetes Style Guide, and cert-manager does something similar.

@tzstoyanov
Copy link
Contributor Author

tzstoyanov commented Sep 17, 2023

Thanks @tzstoyanov!

@juliusvonkohout @kimwnasptd we need to do some thorough testing of this, but it looks promising.

My main questions before we dive in with testing are:

* In your testing, I assume that the other resources listed in [profile controller fails to update profile owner #6576](https://github.com/kubeflow/kubeflow/issues/6576) are now also updated correctly:
  
  * `RoleBinding/namespaceAdmin`
  * `AuthorizationPolicy/ns-owner-access-istio`
  * `ResourceQuota/kf-resource-quota`

The owner of RoleBinding/namespaceAdmin and AuthorizationPolicy/ns-owner-access-istio is updated correctly, but I cannot see any reference to the old user in ResourceQuota/kf-resource-quota.

* While we are here, we may as well provide a way to explicitly allow the profile controller to take over a namespace that does not already have it listed as an `ownerReference`:
  
  * Does anyone have any thoughts about how best to achieve this?

I think that the easiest approach is the users to annotate manually the namespaces that have to be hijacked. The controller can take control of a namespace, if it is not already owned by a profile controller and there is an annotation
transferToKubeflow: true
or something like that.

@juliusvonkohout
Copy link
Member

@tzstoyanov can you present in the next meeting on September 28 and @thesuperzapper can you join as well? I have also found someone else who wants to work on the workbench stuff as well.

@tzstoyanov
Copy link
Contributor Author

@tzstoyanov can you present in the next meeting on September 28 and @thesuperzapper can you join as well? I have also found someone else who wants to work on the workbench stuff as well.

yes, I'll join the meeting to discuss this PR, thanks Julius

@tzstoyanov tzstoyanov changed the title Fix (profile controller): Profile controller fails to update profile owner, fixes #6576 Fix (profile controller): Profile controller fails to update profile owner, fixes #6576 and #5449 Oct 9, 2023
…owner

When the user name in the profile is changed, the controller fails to
apply the changes, as cannot verify the namespace owner. Instead of
using the owner annotation of the namespace, check the if the profile
owns the namespace using the ownerReferences section.

Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
When applying profile changes, update the owner in the namespace
annotations, if the profile user name was changed.

Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
Move the code for Namespace update into a separate function, to make
the Reconcile() implementation more straightforward.

Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
If there is a change in the name of the profile user, update the user in
the ns-owner-access-istio AuthorizationPolicy.

Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
If there is a change in the name of the profile user, update the user in
the namespaceAdmin RoleBinding.

Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
When a new profile is created, usually it creates a new namespace for
that user. There are cases where an existing namespace should be used,
instead of creating new one. Added new logic for that:
 - If a new profile is created and the namespace with that name already
   exist, check if it has a transferToKubeflow:true annotation.
- If there is such annotation, take the ownership of that namespace and
  use it for the profile.

Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
The latest versions of used components requires recent golang version.

Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>
@google-oss-prow
Copy link

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign yanniszark for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@rimolive
Copy link
Member

@tzstoyanov Can you rebase it so we can lgtm/approve?

@tzstoyanov tzstoyanov closed this Apr 11, 2024
@tzstoyanov tzstoyanov deleted the 6576 branch April 11, 2024 03:43
@tzstoyanov
Copy link
Contributor Author

@rimolive - rebased in a new PR #7547

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants