[Dashboard] Select port in dashboard

[fyrestone]

Why are these changes needed?

• Problem
  • The dashboard test cases may fail due to port conflict. It's not stable to check port is free in services.py, then bind port in dashboard.
• Solution
  • Dashboard select an available port by itself, then services.py get the bound port from redis.
• Other
  • Fix the dashboard may hangs at exit.
  • Fix test_stats_collector.py::test_get_all_node_details.
  • Fix test_multi_node_3.py::test_calling_start_ray_head may fail.
  • Move some cases from test_advanced_3.py to test_advanced_2.py due to timeout.

Related issue number

Closes #13351

Checks

• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[rkooo567]

We have the same issue for the dashboard agent, and it is really hard for it to choose the same strategy for that (because other cpp components need to know the port number of the agent). Do you have any good idea to solve it?

[fyrestone]

We have the same issue for the dashboard agent, and it is really hard for it to choose the same strategy for that (because other cpp components need to know the port number of the agent). Do you have any good idea to solve it?

raylet knows the port of dashboard agent because agent calls RegisterAgent after started, the request message is https://github.com/ray-project/ray/blob/master/src/ray/protobuf/agent_manager.proto#L28. Which cpp component needs to know the port of the agent?

[rkooo567]

Lmk when the PR is ready

[fyrestone]

Lmk when the PR is ready

I am not sure if CI failure is related to this PR.

[fyrestone]

Lmk when the PR is ready

All cases passed. The PR is ready.

[fyrestone]

@rkooo567 Is there any problem with this PR?

[rkooo567]

Window tests seem to fail

[fyrestone]

Window tests seem to fail

The dashboard failure will be reported, so Windows CI should build the dashboard frontend before running tests. I found the failures in Windows CI logs:

2021-02-04T08:34:07.1279655Z E                   Exception: Failed to start the dashboard, return code 0. The last 10 lines of C:\Users\RUNNER~1\AppData\Local\Temp\ray\session_2021-02-04_08-33-29_280416_4784\logs\dashboard.log:
2021-02-04T08:34:07.1280906Z E                   2021-02-04 08:33:30,592	WARNING dashboard.py:220 -- The dashboard on node fv-az177-581 failed with the following error:
2021-02-04T08:34:07.1281711Z E                   Traceback (most recent call last):
2021-02-04T08:34:07.1282415Z E                     File "d:\a\ray\ray\python\ray\new_dashboard\dashboard.py", line 201, in <module>
2021-02-04T08:34:07.1283074Z E                       log_dir=args.log_dir)
2021-02-04T08:34:07.1283711Z E                     File "d:\a\ray\ray\python\ray\new_dashboard\dashboard.py", line 85, in __init__
2021-02-04T08:34:07.1284376Z E                       build_dir = setup_static_dir()
2021-02-04T08:34:07.1285143Z E                     File "d:\a\ray\ray\python\ray\new_dashboard\dashboard.py", line 46, in setup_static_dir
2021-02-04T08:34:07.1285842Z E                       "&& npm run build)", build_dir)
2021-02-04T08:34:07.1287263Z E                   FileNotFoundError: [Errno 2] Dashboard build directory not found. If installing from source, please follow the additional steps required to build the dashboard(cd python/ray/new_dashboard/client && npm install && npm ci && npm run build): 'd:\\a\\ray\\ray\\python\\ray\\new_dashboard\\client\\build'
2021-02-04T08:34:07.1288564Z 

But, the npm build is skipped on Windows:

build_dashboard_front_end() {
  if [ "${OSTYPE}" = msys ]; then
    { echo "WARNING: Skipping dashboard due to NPM incompatibilities with Windows"; } 2> /dev/null
  else
    (
      cd ray/new_dashboard/client

      if [ -z "${BUILDKITE-}" ]; then
        set +x  # suppress set -x since it'll get very noisy here
        . "${HOME}/.nvm/nvm.sh"
        nvm use --silent node
      fi
      install_npm_project
      npm run -s build
    )
  fi
}

If the npm is incompatible with Windows, I have to make the dashboard starts silently without frontend.

[fyrestone]

@rkooo567 It's hard to make all test cases green.

[rkooo567]

@fyrestone no reason to make it all green. Just need to make sure failed tests are unrelated to this PR or also failing in the master

[rkooo567]

LGTM. Btw this is only for the dashboard, not agents right? I've seen similar issues from agents too I think

[fyrestone]

LGTM. Btw this is only for the dashboard, not agents right? I've seen similar issues from agents too I think

This PR is only for the dashboard. I will look into the problem of dashboard agent port conflicts.

[rkooo567]

Sounds good. I’ve seen issues from agents a lot recently. The issue is it is a bit trickier than others because raylet need the port of it (to report metrics). Are you planning to work on it soon?

[fyrestone]

Sounds good. I’ve seen issues from agents a lot recently. The issue is it is a bit trickier than others because raylet need the port of it (to report metrics). Are you planning to work on it soon?

I will fix the dashboard agent port conflicts after this PR is merged.