Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

search filter out not ready cluster #3010

Merged

Conversation

yanfeng1992
Copy link
Member

@yanfeng1992 yanfeng1992 commented Dec 29, 2022

Signed-off-by: huangyanfeng huangyanfeng1992@gmail.com

What type of PR is this?
/kind bug

What this PR does / why we need it:

I have three cluster member1, member2, member3,member1 status is not ready

0880c4c80710e3e9b808

I did a few steps

step1 create a resourceregistries for all cluster

lx_clip1672312193776

step 2 kubectl get --raw /apis/search.karmada.io/v1alpha1/search/cache/api/v1/nodes --kubeconfig /etc/karmada/karmada-apiserver.config, but get nothing

lx_clip1672312276036

error log in karmada-search

lx_clip1672312065007

I think the not ready cluster should be filtered out

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

karmada-search: filter out not ready clusters

@karmada-bot karmada-bot added the kind/bug Categorizes issue or PR as related to a bug. label Dec 29, 2022
@karmada-bot karmada-bot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Dec 29, 2022
@codecov-commenter
Copy link

codecov-commenter commented Dec 29, 2022

Codecov Report

Merging #3010 (dd5232c) into master (434b7b6) will decrease coverage by 0.03%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##           master    #3010      +/-   ##
==========================================
- Coverage   38.66%   38.62%   -0.04%     
==========================================
  Files         206      206              
  Lines       18815    18818       +3     
==========================================
- Hits         7274     7269       -5     
- Misses      11114    11120       +6     
- Partials      427      429       +2     
Flag Coverage Δ
unittests 38.62% <0.00%> (-0.04%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
pkg/search/proxy/controller.go 83.97% <0.00%> (-1.65%) ⬇️
...kg/scheduler/framework/runtime/metrics_recorder.go 82.85% <0.00%> (-5.72%) ⬇️
pkg/util/worker.go 66.66% <0.00%> (-4.77%) ⬇️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

pkg/search/controller.go Show resolved Hide resolved
pkg/search/proxy/controller.go Outdated Show resolved Hide resolved
Copy link
Member

@XiShanYongYe-Chang XiShanYongYe-Chang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/cc @ikaven1024 @liys87x

pkg/search/controller.go Show resolved Hide resolved
@ikaven1024
Copy link
Member

Thanks @yanfeng1992 , it's LGTM.

While I found a serious problems: Because cluster ready status is updated periodically, before control panel detects a member cluster down, search component will still request to the down cluster and get an error.

@karmada-bot karmada-bot added the do-not-merge/contains-merge-commits Indicates a PR which contains merge commits. label Dec 30, 2022
Signed-off-by: huangyanfeng <huangyanfeng1992@gmail.com>

search filter out not ready cluster fix ut

Signed-off-by: huangyanfeng <huangyanfeng1992@gmail.com>

search filter out not ready cluster fix lint

Signed-off-by: huangyanfeng <huangyanfeng1992@gmail.com>

search filter out not ready cluster fix log and add comment

Signed-off-by: huangyanfeng <huangyanfeng1992@gmail.com>

search filter out not ready cluster fix  comment

Signed-off-by: huangyanfeng <huangyanfeng1992@gmail.com>
@karmada-bot karmada-bot removed the do-not-merge/contains-merge-commits Indicates a PR which contains merge commits. label Dec 30, 2022
@yanfeng1992
Copy link
Member Author

Thanks @yanfeng1992 , it's LGTM.

While I found a serious problems: Because cluster ready status is updated periodically, before control panel detects a member cluster down, search component will still request to the down cluster and get an error.

ok, i squash the commits into one @ikaven1024

@XiShanYongYe-Chang
Copy link
Member

While I found a serious problems: Because cluster ready status is updated periodically, before control panel detects a member cluster down, search component will still request to the down cluster and get an error.

One thing I don't understand is why getting the cache of other clusters fails when one cluster fails.

@ikaven1024
Copy link
Member

While I found a serious problems: Because cluster ready status is updated periodically, before control panel detects a member cluster down, search component will still request to the down cluster and get an error.

One thing I don't understand is why getting the cache of other clusters fails when one cluster fails.

When client list pods with resourceVersion=, proxy will traverse all the member clusters, fail to connect the down cluster and return error.

for clusterIdx, cluster := range clusters {
n, cont, err := listFunc(cluster)
if err != nil {
return nil, err
}

@yanfeng1992
Copy link
Member Author

Is there anything else I need to modify in this PR? @XiShanYongYe-Chang

@XiShanYongYe-Chang
Copy link
Member

Is there anything else I need to modify in this PR? @XiShanYongYe-Chang

I thinks it's okay,
Ask @ikaven1024 and @liys87x to confirm again.

@ikaven1024
Copy link
Member

/lgtm

@karmada-bot karmada-bot added the lgtm Indicates that a PR is ready to be merged. label Jan 3, 2023
@XiShanYongYe-Chang
Copy link
Member

XiShanYongYe-Chang commented Jan 3, 2023

Hi @yanfeng1992, can you help update the release note to:

karmada-search: filter out not ready clusters

We may need to cherry-pick this patch to the previous release v1.2-v1.4. Can you help do it?

@yanfeng1992
Copy link
Member Author

yanfeng1992 commented Jan 3, 2023

Hi @yanfeng1992, can you help update the release note to:

karmada-search: filter out not ready clusters

We may need to cherry-pick this patch to the previous release v1.2-v1.4. Can you help do it?

i have updated @XiShanYongYe-Chang

@XiShanYongYe-Chang
Copy link
Member

i have updated @XiShanYongYe-Chang

Thanks~
/approve

@karmada-bot
Copy link
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: XiShanYongYe-Chang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@karmada-bot karmada-bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 3, 2023
@karmada-bot karmada-bot merged commit d34a33e into karmada-io:master Jan 3, 2023
@XiShanYongYe-Chang
Copy link
Member

Hi @yanfeng1992, can you help cherry-pick this patch to the previous release? You can refer to the https://karmada.io/docs/contributor/cherry-picks/

@yanfeng1992
Copy link
Member Author

Hi @yanfeng1992, can you help cherry-pick this patch to the previous release? You can refer to the https://karmada.io/docs/contributor/cherry-picks/

release1.2 #3018
release1.3 #3019
release1.4 #3016

@XiShanYongYe-Chang

@XiShanYongYe-Chang
Copy link
Member

release1.2 #3018 release1.3 #3019 release1.4 #3016

@XiShanYongYe-Chang

Thanks a lot~

karmada-bot added a commit that referenced this pull request Jan 4, 2023
…010-upstream-release-1.3

Automated cherry pick of #3010: search filter out not ready cluster
helen-frank pushed a commit to helen-frank/karmada that referenced this pull request Jan 4, 2023
…Ready-cluster

search filter out not ready cluster
karmada-bot added a commit that referenced this pull request Jan 4, 2023
…010-upstream-release-1.2

Automated cherry pick of #3010: search filter out not ready cluster
karmada-bot added a commit that referenced this pull request Jan 5, 2023
…010-upstream-release-1.4

Automated cherry pick of #3010: search filter out not ready cluster
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. kind/bug Categorizes issue or PR as related to a bug. lgtm Indicates that a PR is ready to be merged. size/S Denotes a PR that changes 10-29 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

7 participants