Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bug 1878776: ingressnodesavailable: add controller that checks if router can schedule pods #344

Merged
merged 1 commit into from Sep 14, 2020
Merged

Bug 1878776: ingressnodesavailable: add controller that checks if router can schedule pods #344

merged 1 commit into from Sep 14, 2020

Conversation

mfojtik
Copy link
Member

@mfojtik mfojtik commented Sep 11, 2020 •
edited

The purpose of this controller is to check whether nodes are available to schedule ingress pods (which require workload schedulable nodes).
Not having ingress available is very common failure case for authentication but it lacks a good signal (from ingress operator) and the authentication operator is not available resulting in longer bug triage and red herring.

This controller handle also case when masters are schedulable, however the workers schedulable is best-effort as there could be taints and toleration that can take effect and cause router pods to not be able to schedule.

sttts
sttts reviewed Sep 11, 2020
View reviewed changes
pkg/operator/starter.go Outdated Show resolved Hide resolved
@marun
Copy link
Contributor

marun commented Sep 11, 2020

Seems more than a little cray that the auth operator needs to take responsibility for this.

@mfojtik mfojtik changed the title workers: add controller that checks if router can schedule pods ingressnodesavailable: add controller that checks if router can schedule pods Sep 14, 2020
@mfojtik mfojtik force-pushed the worker-controller branch 3 times, most recently from d97cb0b to 7f98165 Compare September 14, 2020 07:42
sttts
sttts reviewed Sep 14, 2020
View reviewed changes
pkg/controllers/ingressnodesavailable/ingress_nodes_available_controller.go Outdated
Type: "ReadyIngressNodesAvailable",
Status: operatorv1.ConditionFalse,
Reason: "NoReadyIngressNodes",
Message: fmt.Sprintf("Authentication require functional ingress which require at least one schedulable and ready node. Got %d worker nodes and %d master nodes (none are schedulable or ready for ingress pods).",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

requires

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

@sttts
Copy link
Contributor

sttts commented Sep 14, 2020

Make edge team owner of this.

@sttts
Copy link
Contributor

sttts commented Sep 14, 2020

/lgtm
/approve

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Sep 14, 2020
@openshift-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mfojtik, sttts

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci-robot openshift-ci-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. do-not-merge/invalid-owners-file Indicates that a PR should not merge because it has an invalid OWNERS file in it. labels Sep 14, 2020
@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

3 similar comments
@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@mfojtik
Copy link
Member Author

mfojtik commented Sep 14, 2020

/retest

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

1 similar comment
@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-ci-robot openshift-ci-robot removed the lgtm Indicates that a PR is ready to be merged. label Sep 14, 2020
@openshift-ci-robot
Copy link
Contributor

New changes are detected. LGTM label has been removed.

@openshift-ci-robot
Copy link
Contributor

The following users are mentioned in OWNERS file(s) but are untrusted for the following reasons. One way to make the user trusted is to add them as members of the openshift org. You can then trigger verification by writing /verify-owners in a comment.

  • pravisankar
    • User is not a member of the org. User is not a collaborator. Satisfy at least one of these conditions to make the user trusted.
    • pkg/controllers/ingressnodesavailable/OWNERS
  • ramr
    • User is not a member of the org. User is not a collaborator. Satisfy at least one of these conditions to make the user trusted.
    • pkg/controllers/ingressnodesavailable/OWNERS

@mfojtik
Copy link
Member Author

mfojtik commented Sep 14, 2020

/retest

@mfojtik mfojtik added the lgtm Indicates that a PR is ready to be merged. label Sep 14, 2020
@mfojtik
Copy link
Member Author

mfojtik commented Sep 14, 2020

adding lgtm back, fixed stucked informer

@mfojtik mfojtik changed the title ingressnodesavailable: add controller that checks if router can schedule pods Bug 1878776:ingressnodesavailable: add controller that checks if router can schedule pods Sep 14, 2020
@openshift-ci-robot openshift-ci-robot added the bugzilla/severity-high Referenced Bugzilla bug's severity is high for the branch this PR is targeting. label Sep 14, 2020
@openshift-ci-robot
Copy link
Contributor

@mfojtik: This pull request references Bugzilla bug 1878776, which is invalid:

  • expected the bug to target the "4.6.0" release, but it targets "---" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Bug 1878776:ingressnodesavailable: add controller that checks if router can schedule pods

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci-robot openshift-ci-robot added the bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. label Sep 14, 2020
@mfojtik mfojtik changed the title Bug 1878776:ingressnodesavailable: add controller that checks if router can schedule pods Bug 1878776: ingressnodesavailable: add controller that checks if router can schedule pods Sep 14, 2020
@mfojtik
Copy link
Member Author

mfojtik commented Sep 14, 2020

/bugzilla refresh

@openshift-ci-robot openshift-ci-robot added bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. and removed bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. labels Sep 14, 2020
@openshift-ci-robot
Copy link
Contributor

@mfojtik: This pull request references Bugzilla bug 1878776, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (4.6.0) matches configured target release for branch (4.6.0)
  • bug is in the state NEW, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@mfojtik
Copy link
Member Author

mfojtik commented Sep 14, 2020

OWNERS file copied from ingress-operator

@mfojtik mfojtik removed the do-not-merge/invalid-owners-file Indicates that a PR should not merge because it has an invalid OWNERS file in it. label Sep 14, 2020
@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-merge-robot openshift-merge-robot merged commit ee0dce6 into openshift:master Sep 14, 2020
@openshift-ci-robot
Copy link
Contributor

@mfojtik: All pull requests linked via external trackers have merged:

Bugzilla bug 1878776 has been moved to the MODIFIED state.

In response to this:

Bug 1878776: ingressnodesavailable: add controller that checks if router can schedule pods

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

wking added a commit to wking/cluster-authentication-operator that referenced this pull request Oct 30, 2020
@wking
Revert "workers: add controller that checks if router can schedule pods"
ad6f899 
This reverts commit 88e6eba, openshift#344.

The logic assumes the router will be scheduled on "worker"-labeled
nodes.  That leads to false-positives when there are no vanilla
'worker' compute nodes, but are schedulable compute nodes that have
custom names ('infra', 'compute', etc.) [1,2].  Instead of trying to
second-guess the scheduler and the ingress-operator, let the ingress
operator handle reporting this issue [3].

[1]: https://github.com/openshift/machine-config-operator/blob/0170e082a8b8228373bd841d17555fff2cfb51b7/docs/custom-pools.md
[2]: https://bugzilla.redhat.com/show_bug.cgi?id=1893386
[3]: https://bugzilla.redhat.com/show_bug.cgi?id=1881155
@wking wking mentioned this pull request Oct 30, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sttts sttts sttts left review comments

@deads2k deads2k Awaiting requested review from deads2k

@stlaz stlaz Awaiting requested review from stlaz

Assignees

@sttts sttts

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. bugzilla/severity-high Referenced Bugzilla bug's severity is high for the branch this PR is targeting. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. lgtm Indicates that a PR is ready to be merged.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
6 participants
@mfojtik @marun @sttts @openshift-ci-robot @openshift-bot @openshift-merge-robot