Dashboard port formatting CI failure

[jacobtomlinson]

Recently one test on the classic KubeCluster has started failing in CI.

 =================================== FAILURES ===================================
______________________ test_diagnostics_link_env_variable ______________________

k8s_cluster = <pytest_kind.cluster.KindCluster object at 0x7f3d38ab58e0>
pod_spec = {'api_version': None,
 'kind': None,
 'metadata': {'annotations': {},
              'cluster_name': None,
            ...       'value': 'worker'}],
          'topology_spread_constraints': None,
          'volumes': None},
 'status': None}
user_env = None

    @pytest.mark.asyncio
    async def test_diagnostics_link_env_variable(k8s_cluster, pod_spec, user_env):
        pytest.importorskip("bokeh")
        with dask.config.set({"distributed.dashboard.link": "foo-{USER}-{port}"}):
            async with KubeCluster(pod_spec, asynchronous=True) as cluster:
                port = cluster.scheduler_info["services"]["dashboard"]
    
>               assert str(port) in cluster.dashboard_link
E               AssertionError: assert '8787' in 'foo-runner-52351'
E                +  where '8787' = str(8787)
E                +  and   'foo-runner-52351' = KubeCluster(dask-runner-04578db5-9, 'tcp://localhost:63965', workers=0, threads=0, memory=0 B).dashboard_link

dask_kubernetes/classic/tests/test_async.py:147: AssertionError

it seems like the port listed under the dashboard service doesn't match up with the port being formatted into the daskboard_link.

https://github.com/dask/dask-kubernetes/runs/7413643070?check_suite_focus=true

Explored a little in #535 but haven't found a solution yet.

[jacobtomlinson]

Minimal reproducer

In [3]: import dask.config
   ...: from dask_kubernetes.classic import KubeCluster
   ...: from dask_kubernetes import make_pod_spec
   ...:
   ...: pod_spec = make_pod_spec(image='ghcr.io/dask/dask:latest',
   ...:                          memory_limit='4G', memory_request='4G',
   ...:                          cpu_limit=1, cpu_request=1)
   ...:
   ...: dask.config.set({"distributed.dashboard.link": "foo-{USER}-{port}"})
   ...:
   ...: cluster = KubeCluster(pod_spec)
Creating scheduler pod on cluster. This may take some time.

In [4]: cluster.scheduler_info["services"]["dashboard"]
Out[4]: 8787

In [5]: cluster.dashboard_link
Out[5]: 'foo-jtomlinson-54144'

[jacobtomlinson]

This seems to be related to the changes in #489. The port showing in the dashboard_link is the forwarded_dashboard_port but the port showing in cluster.scheduler_info["services"]["dashboard"] is the true port.

That makes complete sense, the forwarded port won't match the actual port. What I'm confused about is why this test only started failing recently.

cc @mmccarty