Custom dev server no longer reconnects

[mkilpatrick]

Describe the bug

When using the latest beta version of Vite (3.0.0-beta.10) and the latest plugin-react (2.0.0-beta.1), after you npm run dev then stop the server and restart it again, the polling in the browser never reconnects with the following errors:

client.ts:297          GET http://localhost:24678/ net::ERR_FAILED 426
waitForSuccessfulPing @ client.ts:297
await in waitForSuccessfulPing (async)
(anonymous) @ client.ts:104
localhost/:1 Access to fetch at 'http://localhost:24678/' from origin 'http://localhost:5173' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.

I assume it might be related to this change.

You can easily reproduce it via the playground/ssr-react example.

Reproduction

N/A

System Info

System:
    OS: macOS 12.4
    CPU: (10) arm64 Apple M1 Max
    Memory: 722.75 MB / 64.00 GB
    Shell: 5.1.16 - /usr/local/bin/bash
  Binaries:
    Node: 17.9.0 - ~/.nvm/versions/node/v17.9.0/bin/node
    npm: 8.5.5 - ~/.nvm/versions/node/v17.9.0/bin/npm
  Browsers:
    Chrome: 103.0.5060.114
    Safari: 15.5
  npmPackages:
    @vitejs/plugin-react: 2.0.0-beta.1 => 2.0.0-beta.1 

### Used Package Manager

npm

### Logs

_No response_

### Validations

- [X] Follow our [Code of Conduct](https://github.com/vitejs/vite/blob/main/CODE_OF_CONDUCT.md)
- [X] Read the [Contributing Guidelines](https://github.com/vitejs/vite/blob/main/CONTRIBUTING.md).
- [X] Read the [docs](https://vitejs.dev/guide).
- [X] Check that there isn't [already an issue](https://github.com/vitejs/vite/issues) that reports the same bug to avoid creating a duplicate.
- [X] Make sure this is a Vite issue and not a framework-specific issue. For example, if it's a Vue SFC related bug, it should likely be reported to [vuejs/core](https://github.com/vuejs/core) instead.
- [X] Check that this is a concrete bug. For Q&A open a [GitHub Discussion](https://github.com/vitejs/vite/discussions) or join our [Discord Chat Server](https://chat.vitejs.dev/).
- [X] The provided reproduction is a [minimal reproducible example](https://stackoverflow.com/help/minimal-reproducible-example) of the bug.

[mkilpatrick]

Adding an hmr.clientPort of 5173 seems to fix it, but then results in a separate issue where the page itself randomly reloads while the server is running.