Screenshots aren't picked up when they don't have a body

[turchiad]

Problem

In v0.0.18 of playwright-ctrf-json-reporter with v1.49.1 of playwright, screenshots are (sometimes?) omitted from the resulting CTRF .json file when the attachments property contains elements with no body property.

Background

Environment: Windows 10
NodeJs version: v20.15.1
Playwright version: v1.49.1
Playwright CTRF JSON Reporter version: v0.0.18

I ran into this problem when trying to use playwright-crtf-json-reporter for the first time with playwright. I can't show my real code, but I can give some portions:

playwright.config.js

// ...
    reporter: [
        ['list', { printSteps: true }],
        // ['html'],
        ['json', {
            outputFile: 'test-results/standard.json',
        }],
        ['playwright-ctrf-json-reporter', {
            outputDir: 'test-results',
            outputFile: 'results.json',
            minimal: false,
            screenshot: true,
        }],
    ],
// ...

standard.json (Playwright's Standard JSON reporter)

                  "attachments": [
                    {
                      "name": "screenshot",
                      "contentType": "image/png",
                      "path": "C:\\path\\to\\screenshot.png"
                    }
                  ],

results.json (CRTF report)

      {
        "name": "Some Test Name",
        "status": "failed",
        "duration": 123,
        "start": 123,
        "stop": 123,
        "message": "...",
        "trace": "...",
        "rawStatus": "failed",
        "tags": [],
        "type": "e2e",
        "filePath": "C:\\Some\\Path\\To\\The\\File.spec.js",
        "retries": 0,
        "flaky": false,
        "steps": [
          {
            "name": "login",
            "status": "passed"
          },
          ...
        ],
        "suite": "Tests > Path\\To\\The\\File.spec.js",
        "extra": {
          "annotations": []
        }
      }

Hypothesis

I've drilled down into the code a little bit and I believe the issue lives in generate-report.js at extractScreenshotBase64, which accepts testResult as an argument.

I've adjusted the code with some debug logs.

generate-report.js

// Edited for debugging
    extractScreenshotBase64(testResult) {
        var _a;
        console.log("Extracting screenshot...");
        console.log("Test result has the following keys: ", Object.keys(testResult));
        console.log("And inside the attachments key: ", testResult.attachments);
        const screenshotAttachment = testResult.attachments.find((attachment) => attachment.name === 'screenshot' &&
            (attachment.contentType === 'image/jpeg' ||
                attachment.contentType === 'image/png'));
        console.log("And the value we find is: ", screenshotAttachment);]
        // nevermind the `undefined`s here; that was just for my own readability purposes
        const returnValue = (_a = screenshotAttachment === null || screenshotAttachment === undefined ? undefined : screenshotAttachment.body) === null || _a === undefined ? undefined : _a.toString('base64');
        console.log("But the value we're returning is: ", returnValue);
        return returnValue;
    }

From the test runtime:

Extracting screenshot...
Test result has the following keys:  [
  'retry',       'parallelIndex',
  'workerIndex', 'duration',
  'startTime',   'stdout',
  'stderr',      'attachments',
  'status',      'steps',
  'errors',      'error'
]
And inside the attachments key:  [
  {
    name: 'screenshot',
    path: 'C:\\Path\\To\\The\Image.png',
    contentType: 'image/png',
    body: undefined
  }
]
And the value we find is:  {
  name: 'screenshot',
  path: 'C:\\Path\\To\\The\Image.png',
  contentType: 'image/png',
  body: undefined
}
But the value we're returning is:  undefined

I really couldn't tell you why Playwright isn't returning a body with its attachments array, but their documentation indicates body is an '(optional)' field of this object: https://playwright.dev/docs/api/class-testinfo#test-info-attachments

With that in mind, this should probably be changed. I'm trying to run some automation using a CRTF report and the lack of screenshots is a dealbreaker 😓

Etc.

I managed to get the screenshots field to blip up once somehow, but haven't been able to reproduce it. I imagine it had to do with whether playwright passed a body or a path. Maybe an I/O race condition?

[Ma11hewThomas]

Hi @turchiad,

I can't seem to recreate this issue.

Did you add the returned screenshot buffer as the body like in the example below?

import { test, expect } from '@playwright/test'

test('basic test', async ({ page }, testInfo) => {
  await page.goto('https://playwright.dev')
  const screenshot = await page.screenshot({ quality: 50, type: 'jpeg' })

  await testInfo.attach('screenshot', {
    body: screenshot,
    contentType: 'image/jpeg',
  })
})

If you attach the buffer, the screenshot should be included in the report as base64. Please let me know if this helps resolve your issue or if you're still having problems.

Thanks

[turchiad]

Hey @Ma11hewThomas, thanks for having a look.

In the case of my test setup, screenshots are not created in the format above, rather they are automatically produced by Playwright according to the use: { screenshot: ... } property in playwright.config.js

Here's a sample from my playwright.config.js

    /* Shared settings for all the projects below. See https://playwright.dev/docs/api/class-testoptions. */
    use: {
        /* Screenshot behavior. See https://playwright.dev/docs/test-use-options#recording-options */
        screenshot: 'on-first-failure',
        /* Collect trace when retrying the failed test. See https://playwright.dev/docs/trace-viewer */
        trace: 'on-first-retry',
    },

I've tested this with screenshot being 'only-on-failure' as well and observed the same behavior. It would seem to me in my tests that screenshots produced by this method do not use the body field in most cases; but I'm certain this could come down to a variety of reasons.

Let me know if that changes the situation, in any matter I don't think screenshots as you've suggested above are applicable with my use case.

[Ma11hewThomas]

Thanks for sharing @turchiad.

While the current implementation doesn’t seem to align with your use case, the details you provided are really helpful in deciding the future direction regarding attachments. We’ve received multiple requests for CTRF to include an attachments property for screenshots, logs, traces etc. We've introduced an approach similar to the standard you referenced from the Playwright JSON format:

"attachments": [
	{
		"name": "screenshot",
                 "contentType": "image/png",
                 "path": "C:\\path\\to\\screenshot.png"
         }
]

I will add this to the playwright reporter shortly, which might better align with your use case.

Thanks

[Ma11hewThomas]

#16