Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(browser): fix browser testing url for https #4855

Merged
merged 24 commits into from
Jan 16, 2024
Merged
Show file tree
Hide file tree
Changes from 20 commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -143,6 +143,7 @@ jobs:
strategy:
matrix:
browser: [[chrome, chromium], [firefox, firefox], [edge, webkit]]
fail-fast: false
Copy link
Contributor Author

@hi-ogawa hi-ogawa Jan 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar to the above test job, I thought it's better to run all matrix without fail-fast so that it's easier to identify potential browser-specific quirks.


timeout-minutes: 30

Expand Down Expand Up @@ -186,6 +187,7 @@ jobs:
strategy:
matrix:
browser: [[chrome, chromium], [edge, webkit]]
fail-fast: false

timeout-minutes: 30

Expand Down
3 changes: 2 additions & 1 deletion packages/vitest/src/node/logger.ts
Original file line number Diff line number Diff line change
Expand Up @@ -149,7 +149,8 @@ export class Logger {
const name = project.getName()
const output = project.isCore() ? '' : ` [${name}]`

this.log(c.dim(c.green(` ${output} Browser runner started at http://${project.config.browser.api?.host || 'localhost'}:${c.bold(`${project.browser.config.server.port}`)}`)))
const url = new URL('/', project.browser.resolvedUrls?.local[0])
this.log(c.dim(c.green(` ${output} Browser runner started at ${url}`)))
})

if (this.ctx.config.ui)
Expand Down
2 changes: 1 addition & 1 deletion packages/vitest/src/node/pools/browser.ts
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ export function createBrowserPool(ctx: Vitest): ProcessPool {
const provider = project.browserProvider!
providers.add(provider)

const origin = `http://${ctx.config.browser.api?.host || 'localhost'}:${project.browser!.config.server.port}`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a possibility that this might be null if server is restarted (due to a config change): vitejs/vite#15450

Does it affect us?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the reference. There could be something sketchy scenario. I'll look into it.

Also another thing, actually it might not be preferred to use resolveUrls since it looks like Vite has some inconsistency when users set vite config like server.host: "local.example.com".

Let me think about it.

Copy link
Contributor Author

@hi-ogawa hi-ogawa Jan 10, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just remembered this issue about Vite's --open default behavior:

Essentially Vite does a following when ViteDevServer.openBrowser:

new URL(options.open, resolvedUrls.local[0] ?? resolvedUrls.network[0])

https://github.com/vitejs/vite/blob/2687dbbd4e19c86f9888ee784c9b51598e8b79ca/packages/vite/src/node/server/index.ts#L484-L492

This also means that provider: "none" is already forcing url origin to be resolvedUrls.local[0] ?? resolvedUrls.network[0], so aligning this behavior for all providers should not be that bad.
(Ah, actually that's not the case. new URL(x, y) doesn't change origin to y if x already includes origin.)

Copy link
Contributor Author

@hi-ogawa hi-ogawa Jan 16, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regarding Vite server's restart behavior, I don't think that's relevant for Vitest browser mode since browser mode uses a dedicated Vite dev server, which doesn't watch files on its own and is completely re-spawned when Vitest's main server restarts:

// watch is handled by Vitest
server: {
hmr: false,
watch: {
ignored: ['**/**'],
},
},

async initBrowserServer(configFile: string | undefined) {
if (!this.isBrowserEnabled())
return
await this.browser?.close()
this.browser = await createBrowserServer(this, configFile)
}

So, relying on resolvedUrls should be safe before landing vitejs/vite#15450 in Vite v5.1.

Currently the log looks like this for playwright provider:

Restarting due to config changes...

 DEV  v1.2.0 /home/hiroshi/code/others/vitest/test/browser/fixtures/server-url
      Browser runner started at http://localhost:5174/

But webdriverio actually doesn't support restart at all since it kills a whole process when Vitest main server tries to restart and close all existing pools:

this.pool?.close?.()

// TODO: right now process can only exit with timeout, if we use browser
// needs investigating
process.exit()

const origin = project.browser?.resolvedUrls?.local[0]
const paths = files.map(file => relative(project.config.root, file))

if (project.config.browser.isolate) {
Expand Down
12 changes: 12 additions & 0 deletions pnpm-lock.yaml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

5 changes: 5 additions & 0 deletions test/browser/fixtures/server-url/basic.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
import { expect, test } from "vitest";

test("basic", () => {
expect(1).toBe(1);
})
26 changes: 26 additions & 0 deletions test/browser/fixtures/server-url/vitest.config.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
import path from 'node:path'
import { fileURLToPath } from 'node:url'
import { defineConfig } from 'vitest/config'
import basicSsl from '@vitejs/plugin-basic-ssl'

// test https by
// TEST_HTTPS=1 pnpm test-fixtures --root fixtures/server-url

const provider = process.env.PROVIDER || 'webdriverio';
const browser = process.env.BROWSER || (provider === 'playwright' ? 'chromium' : 'chrome');

export default defineConfig({
plugins: [
!!process.env.TEST_HTTPS && basicSsl(),
],
test: {
browser: {
enabled: true,
provider,
name: browser,
},
},
// separate cacheDir from test/browser/vite.config.ts
// to prevent pre-bundling related flakiness on Webkit
cacheDir: path.join(path.dirname(fileURLToPath(import.meta.url)), "node_modules/.vite")
Comment on lines +23 to +25
Copy link
Contributor Author

@hi-ogawa hi-ogawa Jan 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As commented in #4879 (comment), I found that webkit flakiness is related to Vite pre-bundling somehow. For now, separating vite cacheDir can workaround the issue, so I added this here.

To verify this, I repeated pnpm run test:browser:playwright 20 times consecutively on CI and it didn't fail https://github.com/vitest-dev/vitest/actions/runs/7454247103/job/20281233631#step:11:1

I'm still not sure the same issue could be reproduced outside of Vitest since this issue might be related to Vitest being a linked dep for this in-repository tests.

})
2 changes: 2 additions & 0 deletions test/browser/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,11 @@
"test:webdriverio": "PROVIDER=webdriverio node --test --test-concurrency=1 specs/",
"test:playwright": "PROVIDER=playwright node --test --test-concurrency=1 specs/",
"test:safaridriver": "PROVIDER=webdriverio BROWSER=safari node --test --test-concurrency=1 specs/",
"test-fixtures": "vitest",
"coverage": "vitest --coverage.enabled --coverage.provider=istanbul --browser.headless=yes"
},
"devDependencies": {
"@vitejs/plugin-basic-ssl": "^1.0.2",
"@vitest/browser": "workspace:*",
"@vitest/cjs-lib": "link:./cjs-lib",
"execa": "^7.1.1",
Expand Down
27 changes: 27 additions & 0 deletions test/browser/specs/server-url.test.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
import assert from 'node:assert'
import test from 'node:test'
import { execa } from 'execa'

test('server-url http', async () => {
const result = await execa('npx', ['vitest', 'run', '--root=./fixtures/server-url', '--browser.headless'], {
env: {
CI: '1',
NO_COLOR: '1',
},
})
assert.match(result.stdout, /Browser runner started at http:\/\/localhost:5173\//)
assert.match(result.stdout, /Test Files {2}1 passed/)
})

// this test is skipped since browser warns self-signed https and it requires manual interaction.
// you can toggle "skip" to verify it locally.
test('server-url https', { skip: true }, async () => {
const result = await execa('npx', ['vitest', 'run', '--root=./fixtures/server-url'], {
env: {
NO_COLOR: '1',
TEST_HTTPS: '1',
},
})
assert.match(result.stdout, /Browser runner started at https:\/\/localhost:5173\//)
assert.match(result.stdout, /Test Files {2}1 passed/)
})
3 changes: 2 additions & 1 deletion test/browser/tsconfig.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
{
"compilerOptions": {
"rootDir": ".",
"module": "node16",
"module": "ESNext",
"moduleResolution": "Bundler",
"paths": {
"#src/*": ["./src/*"]
},
Expand Down