Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Multi mesh setup results in an error when fetching workloads #6772

Closed
jshaughn opened this issue Oct 24, 2023 Discussed in #6761 · 2 comments · Fixed by #6782
Closed

Multi mesh setup results in an error when fetching workloads #6772

jshaughn opened this issue Oct 24, 2023 Discussed in #6761 · 2 comments · Fixed by #6782
Assignees
@jshaughn
Labels
backlog Triaged Issue added to backlog bug Something isn't working

Comments

@jshaughn
Copy link
Collaborator

Discussed in #6761

Originally posted by InverseIntegral October 23, 2023
I'm running a multi-mesh / multi-cluster setup with two instances of Kiali. Whenever I access Kiali on cluster A, I get the following error:

2023-10-23T16:32:21Z INF Kiali: Version: v1.75.0, Commit: 45d72534b0556c0425f52a1488811eb372c92e43, Go: 1.20.5
2023-10-23T16:32:21Z INF Using authentication strategy [anonymous]
2023-10-23T16:32:21Z WRN Kiali auth strategy is configured for anonymous access - users will not be authenticated.
2023-10-23T16:32:21Z INF Some validation errors will be ignored [KIA1301]. If these errors do occur, they will still be logged. If you think the validation errors you see are incorrect, please report them to the Kiali team if you have not done so already and provide the details of your scenario. This will keep Kiali validations strong for the whole community.
2023-10-23T16:32:21Z INF Initializing Kiali Cache
2023-10-23T16:32:21Z INF Adding a RegistryRefreshHandler
2023-10-23T16:32:21Z INF [Kiali Cache] Waiting for cluster-scoped cache to sync
2023-10-23T16:32:21Z INF [Kiali Cache] Started
2023-10-23T16:32:21Z INF [Kiali Cache] Kube cache is active for cluster: [A]
2023-10-23T16:32:21Z INF Server endpoint will start at [:20001/kiali]
2023-10-23T16:32:21Z INF Server endpoint will serve static content from [/opt/kiali/console]
2023-10-23T16:32:21Z INF Starting Metrics Server on [:9090]
2023-10-23T16:32:26Z INF [Prom Cache] Enabled
2023-10-23T16:32:26Z ERR grafana URL is not set in Kiali configuration
2023-10-23T16:32:26Z INF Create jaeger GRPC client tracing.istio-system:16685
2023-10-23T16:32:26Z ERR This controller DeploymentRevision is assigned in a Pod and it's not properly managed
2023-10-23T16:32:26Z ERR This controller DeploymentRevision is assigned in a Pod and it's not properly managed
2023-10-23T16:32:28Z ERR This controller DeploymentRevision is assigned in a Pod and it's not properly managed
2023-10-23T16:32:28Z ERR This controller DeploymentRevision is assigned in a Pod and it's not properly managed
2023-10-23T16:32:28Z ERR This controller DeploymentRevision is assigned in a Pod and it's not properly managed
2023-10-23T16:32:28Z ERR Error fetching Workloads per namespace <namespace>: Cluster [B] is not found or is not accessible for Kiali
2023-10-23T16:32:28Z ERR Cluster [B] is not found or is not accessible for Kiali: goroutine 806 [running]:
runtime/debug.Stack()
    /opt/hostedtoolcache/go/1.20.5/x64/src/runtime/debug/stack.go:24 +0x65
github.com/kiali/kiali/handlers.handlePanic({0x23967e0, 0xc00216a810})
    /home/runner/work/kiali/kiali/handlers/graph.go:86 +0x219
panic({0x1c78aa0, 0xc002014de0})
    /opt/hostedtoolcache/go/1.20.5/x64/src/runtime/panic.go:884 +0x213
github.com/kiali/kiali/graph.CheckError(...)
    /home/runner/work/kiali/kiali/graph/util.go:38
github.com/kiali/kiali/graph/telemetry/istio/appender.getWorkloadList({0xc000f1c780, 0x10}, {0xc000f1c7b0, 0xe}, 0xc00217fb90)
    /home/runner/work/kiali/kiali/graph/telemetry/istio/appender/appender.go:345 +0x40a
github.com/kiali/kiali/graph/telemetry/istio/appender.getAppWorkloads({0xc000f1c780, 0x10}, {0xc000f1c7b0, 0xe}, {0xc000529158, 0x16}, {0xc000f1c770, 0x6}, 0xc0020da8e8?)
    /home/runner/work/kiali/kiali/graph/telemetry/istio/appender/appender.go:373 +0x14c
github.com/kiali/kiali/graph/telemetry/istio/appender.(*MeshCheckAppender).applyMeshChecks(0xc0020daaf0, 0x33b39e0?, 0x7fe7166c7010?, 0xc001631050)
    /home/runner/work/kiali/kiali/graph/telemetry/istio/appender/mesh_check.go:73 +0x254
github.com/kiali/kiali/graph/telemetry/istio/appender.MeshCheckAppender.AppendGraph({0xc00217f8f0}, 0x9?, 0xc000d3e330?, 0x12?)
    /home/runner/work/kiali/kiali/graph/telemetry/istio/appender/mesh_check.go:34 +0x3e
github.com/kiali/kiali/graph/telemetry/istio.BuildNamespacesTrafficMap({_, _}, {0xc00217f8f0, {0x0, {0xc000a726c0, 0x6, 0x6}}, 0x0, 0x1, 0xc00217f830, ...}, ...)
    /home/runner/work/kiali/kiali/graph/telemetry/istio/istio.go:80 +0x6a3
github.com/kiali/kiali/graph/api.graphNamespacesIstio({_, _}, _, _, {{0x20932cd, 0x9}, {0x208d8a0, 0x5}, {{0xc0006946b9, 0x15}, ...}, ...})
    /home/runner/work/kiali/kiali/graph/api/api.go:52 +0x126
github.com/kiali/kiali/graph/api.GraphNamespaces({_, _}, _, {{0x20932cd, 0x9}, {0x208d8a0, 0x5}, {{0xc0006946b9, 0x15}, {0xdf8475800, ...}}, ...})
    /home/runner/work/kiali/kiali/graph/api/api.go:33 +0x2fb
github.com/kiali/kiali/handlers.GraphNamespaces({0x23967e0, 0xc00216a810}, 0xc001f73000)
    /home/runner/work/kiali/kiali/handlers/graph.go:51 +0x165
net/http.HandlerFunc.ServeHTTP(0x209b862?, {0x23967e0?, 0xc00216a810?}, 0x7fe716873b00?)
    /opt/hostedtoolcache/go/1.20.5/x64/src/net/http/server.go:2122 +0x2f
github.com/kiali/kiali/routing.metricHandler.func1({0x23a55a0?, 0xc0007267e0}, 0xc001f10a01?)
    /home/runner/work/kiali/kiali/routing/router.go:145 +0x13b
net/http.HandlerFunc.ServeHTTP(0x23a6a38?, {0x23a55a0?, 0xc0007267e0?}, 0xc001f10a80?)
    /opt/hostedtoolcache/go/1.20.5/x64/src/net/http/server.go:2122 +0x2f
github.com/kiali/kiali/handlers.AuthenticationHandler.Handle.func1({0x23a55a0, 0xc0007267e0}, 0xc001f72e00)
    /home/runner/work/kiali/kiali/handlers/authentication.go:79 +0x3fd
net/http.HandlerFunc.ServeHTTP(0x23a6a38?, {0x23a55a0?, 0xc0007267e0?}, 0x2379c18?)
    /opt/hostedtoolcache/go/1.20.5/x64/src/net/http/server.go:2122 +0x2f
github.com/kiali/kiali/server.plainHttpMiddleware.func1({0x23a55a0?, 0xc0007267e0?}, 0xc00217f650?)
    /home/runner/work/kiali/kiali/server/server.go:157 +0x65
net/http.HandlerFunc.ServeHTTP(0xc001f72d00?, {0x23a55a0?, 0xc0007267e0?}, 0x2?)
    /opt/hostedtoolcache/go/1.20.5/x64/src/net/http/server.go:2122 +0x2f
github.com/gorilla/mux.(*Router).ServeHTTP(0xc00052c300, {0x23a55a0, 0xc0007267e0}, 0xc001f72c00)
    /home/runner/go/pkg/mod/github.com/gorilla/mux@v1.8.0/mux.go:210 +0x1cf
github.com/NYTimes/gziphandler.GzipHandlerWithOpts.func1.1({0x23a5450, 0xc0010be0e0}, 0x0?)
    /home/runner/go/pkg/mod/github.com/!n!y!times/gziphandler@v1.1.1/gzip.go:336 +0x24e
net/http.HandlerFunc.ServeHTTP(0xc001394480?, {0x23a5450?, 0xc0010be0e0?}, 0x40d92a?)
    /opt/hostedtoolcache/go/1.20.5/x64/src/net/http/server.go:2122 +0x2f
net/http.(*ServeMux).ServeHTTP(0xc000694660?, {0x23a5450, 0xc0010be0e0}, 0xc001f72c00)
    /opt/hostedtoolcache/go/1.20.5/x64/src/net/http/server.go:2500 +0x149
net/http.serverHandler.ServeHTTP({0xc001dab8f0?}, {0x23a5450, 0xc0010be0e0}, 0xc001f72c00)
    /opt/hostedtoolcache/go/1.20.5/x64/src/net/http/server.go:2936 +0x316
net/http.(*conn).serve(0xc00088b3b0, {0x23a6a38, 0xc000a0d8c0})
    /opt/hostedtoolcache/go/1.20.5/x64/src/net/http/server.go:1995 +0x612
created by net/http.(*Server).Serve
    /opt/hostedtoolcache/go/1.20.5/x64/src/net/http/server.go:3089 +0x5ed

I install Kiali as described here by applying the addon resources. I've tried to disable the clustering feature since I'm currently not interested in having a single Kiali. Do you have an idea how I could resolve this issue? Thank you in advance.
@jshaughn jshaughn added bug Something isn't working backlog Triaged Issue added to backlog labels Oct 24, 2023
@jshaughn jshaughn self-assigned this Oct 25, 2023
jshaughn added a commit to jshaughn/kiali that referenced this issue Oct 25, 2023
@jshaughn
Make sure mesh_check appender only tries to check nodes in accessible…
a9002b9 
… namespaces.

Otherwise the fetch of workload information will fail.

kiali#6772
@jshaughn jshaughn mentioned this issue Oct 25, 2023
Merged
jshaughn added a commit to jshaughn/kiali that referenced this issue Oct 25, 2023
@jshaughn
Make sure mesh_check appender only tries to check nodes in accessible…
7518079 
… namespaces.

Otherwise the fetch of workload information will fail.

- Fix tests
  - update some incorrectly defined AccessibleNamespace structures
  - ensure config properly defines cluster

kiali#6772
jshaughn added a commit that referenced this issue Oct 26, 2023
@jshaughn
Make sure mesh_check appender only tries to check nodes in accessible…
fcd670c 
… namespaces (#6782)

- make accessible namespace check cluster-sensitive, otherwise the fetch of workload information will fail.
- add some type-safety and avoid de-decentralized key-string construction
- Fix tests
  - update some incorrectly defined AccessibleNamespace structures
  - ensure config properly defines cluster

#6772
@InverseIntegral
Copy link

InverseIntegral commented Oct 31, 2023 •
edited

@jshaughn thank you for fixing this so quickly! I've tested it on our cluster(s) and it seems to work now 🎉

@jshaughn
Copy link
Collaborator Author

@InverseIntegral Thanks for verifying!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jshaughn jshaughn

Labels
backlog Triaged Issue added to backlog bug Something isn't working
Projects
Kiali Sprint 24-5 | Kiali 1.84
Archived in project
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Make sure mesh_check appender only tries to check nodes in accessible namespaces jshaughn/kiali
2 participants
@jshaughn @InverseIntegral