fix(server): keep port when modifying the config file

[ltaoo]

Description

console.log('start vite server');
const server = await vite.createServer({
  root: process.cwd(),
});
server.listen(8000);

I use createServer directly like above, when I modify vite.config.js file, the port is no longer 8000 but the default 3000.

Or make listen not support the port param.

Additional context

---

What is the purpose of this pull request?

• Bug fix
• New Feature
• Documentation update
• Other

Before submitting the PR, please make sure you do the following

• Read the Contributing Guidelines.
• Read the Pull Request Guidelines and follow the Commit Convention.
• Check that there isn't already a PR that solves the problem the same way to avoid creating a duplicate.
• Provide a description in this PR that addresses what the PR is solving, or reference the issue that it solves (e.g. fixes #123).
• Ideally, include relevant tests that fail without this PR but pass with it.

[Shinigami92]

Could you link a bug issue together with a reproduction?

[ltaoo]

bug description #4464 @Shinigami92