[Bug]: Mismatched test id between onTestEnd custom reporter hook and generated HTML test report

[picakia]

Version

1.43.1

Steps to reproduce

1. Clone https://github.com/picakia/playwright-testid-repro
2. Run npx playwright test
3. Compare test.id logged in the console with these shown in the html report
4. (optional) Checkout to initial commit and re run npx playwright test on 1.42.1 to verify that the problem does not happen

Expected behavior

I expect to have the same test id between HTML report and custom reporter.

Actual behavior

The test id returned by custom reporter are not the same as shown in HTML reporter. This was not the case in 1.42.1.

Additional context

Screenshot showing the same IDs from 1.42.1:

Screenshot showing different IDs from 1.43.1:

Environment

This happens on playwright docker image as well as on my system:
  System:
    OS: Linux 6.7 Arch Linux
    CPU: (16) x64 11th Gen Intel(R) Core(TM) i7-11700K @ 3.60GHz
    Memory: 3.90 GB / 7.56 GB
    Container: Yes
  Binaries:
    Node: 18.18.2 - ~/.config/nvm/versions/node/v18.18.2/bin/node
    Yarn: 1.22.19 - ~/.bun/bin/yarn
    npm: 9.8.1 - ~/.config/nvm/versions/node/v18.18.2/bin/npm
    bun: 1.1.3 - ~/.bun/bin/bun
  IDEs:
    VSCode: 1.87.2 - /usr/bin/code
  Languages:
    Bash: 5.2.26 - /usr/bin/bash
  npmPackages:
    @playwright/test: ^1.43.1 => 1.43.1

[yury-s]

The format of the html report url is an implementation details and it doesn't have to match test ids from the reporter API. What is the use case where you need this two ids to be the same?

[picakia]

I'm running the tests in ci/cd every 2 hours. After tests finish the resulting logs from onTestEnd are uploaded to ElasticSearch using my custom reporter that also generates a URL pointing to exact test case in the HTML report. The generated HTML report is uploaded to our http server to the correct directory that matches the URL sent to Elastic. Next we have a grafana dashboard that shows the test results using elastic as data source. Finally we have a team of people monitoring grafana for failures in tests and using that link with ID they can jump immediately from filtered grafana results to failed test report (this is especially useful when we sometimes have over 500 failed tests). Below is the simplified fragment of code responsible for building log for ElasticSearch:

onTestEnd(test: any, result: any) {
	const prefixUrl = "https://example-site.com";
	const reportAddress = "https://test-results.companydomain.com" ;
	const projectShortName = "mon";
        const reportDate = new Date();
	const testState = result.status;
	// Split subjectString to get Platform, subject, type and path (Desktop/Nawigacja/Funkcjonalne/nazwa-testu.test.ts)
	const subjectString = this.getSubjectString(test.parent).split('/'); //helper method
	let element = {
		'@timestamp': reportDate,
                state: testState,  
                browser: test._projectId, 
                execTime: result.duration,
                description: `${test.parent.title} > ${test.title}`,
                prefixUrl: prefixUrl,
                reason: 'Unknown',
                platform: subjectString[0],
                subject: subjectString[1],
                type: subjectString[2],
                path: subjectString[3],
                report: reportAddress
                        ? `${reportAddress}/report/pw-${projectShortName}-${reportDate}/#?testId=${test.id}`
                        : 'No report',
                        // this will create a link similar to this:
                        // https://test-results.companydomain.com/report/pw-mon-2024-02-20T02-01/#?testId=a69f37cf692d9914279a-bc34705686b9e05ba911
                        // then later someone who views the test logs can click the link directly from for example grafana dashboard
               };
    
    // some modifications to the element goes here
    
    // logic for sending element to elastic goes here
    
    // after tests the resulting HTML report is sent to our server from where people can see the report
    // this is done in ci/cd so the link is very helpful in navigating
    } 

If needed I can create a visual demo (short video) of this usecase to better show how this works in action

[picakia]

I believe that this commit caused the IDs to be mismatched: d0cc587
in packages/playwright/src/reporters/html.ts

[yury-s]

Instead of the test id attribute you can construct a filter in the search query of the URL for the test/spec file that you want to highlight, see this comment for examples. Will that work for you?

[picakia]

Thanks for suggestion! I will need to check if it will be possible to adapt my work flow for this, but it will still be one more click to do instead of presenting the report directly after clicking.

But i still don't quite understand this change. Why the testIds needs to be different between custom reporters and built in reporters? Can you explain a little bit more why this chance was introduced? And if it's theoretically possible to revert back to previous behavior in the future updates?

Thank you very much for your help and time, I really appreciate it ☺

[picakia]

Thank you very much for quick fix 😃. I will update from 1.42.1 to 1.44 as soon as it releases and test if it works in my workflow.