Dashboard can't connect rgw service when deploying rgw with 'hostNetwork: false'

[mc956759]

I'm trying to deploy multi rgw instances on one node. So I config 'hostNetwork: false' in CephObjectStore yaml, and use node port service to expose rgw for clients outside k8s cluster. It works, the s5cmd commands could access the rgw, but I can't see the rgw service info via dashboard. I checked the mgr logs, it seems that mgr try to connect fixed url 'host:8080', but the 8080 port just valid in the pod, not in the node. logs:
RGW REST API failed GET, connection error (url=http://node1:8080/admin/metadata/user?myself): [errno: 111] Connection refused
RGW REST API cannot be reached: Connection refused [errno 111]. Please check your configuration and that the API endpoint is accessible

I want to keep 'hostNetwork: false'+nodeport service deployment method. How could I watch the rgw service via dashbord? Thanks for any help!
rook version: 1.10.7
ceph version: 17.2.6
k8s version: 1.24.9

gateway.yaml:
apiVersion: ceph.rook.io/v1
kind: CephObjectStore
metadata:
name: my-store-a
namespace: rook-ceph
spec:
preservePoolsOnDelete: true
gateway:
port: 8081
hostNetwork: false
instances: 8
zone:
name: my-zone

service.yaml
apiVersion: v1
kind: Service
metadata:
name: rgw-external
namespace: rook-ceph # namespace:cluster
labels:
app: rook-ceph-rgw
rook_cluster: rook-ceph # namespace:cluster
rook_object_store: my-store-a
spec:
ports:
- name: rgw
port: 8081 # service port mentioned in object store crd
protocol: TCP
nodePort: 30080
selector:
app: rook-ceph-rgw
rook_cluster: rook-ceph # namespace:cluster
rook_object_store: my-store-a
sessionAffinity: None
type: NodePort

[travisn]

@mc956759 There have been multiple improvements to the ceph dashboard in v18.2.1. Could you test with the latest Rook and Ceph and see if the issue is fixed? If not, please open a new issue so we can investigate for the dashboard.

[LittleFox94]

@travisn
I had a curious case where (apparently) the rook orchestrator added my 4 RGW pods as RGW service instances, but somehow configured the node IP for one of them instead of the pod IP - at least that seems to be the only way to explain what I actually saw: dashboard showing the same as in the screenshot above, mgr logs showing a failed request to http://$nodeIP:8080/admin/metadata/user?myself (connection refused) and after deleting all RGW pods (and the new ones running) everything was fine

running 18.2.1, am not aware of any networking problems on my cluster, hence guessing at the orchestrator here.
first try to fix this was upgrading rook to 1.13.2 (from 1.13.1), but that didn't change anything. Unloading and reloading the rook manager module also didn't fix this.

[travisn]

@rkachach ptal