Skip to content

Plugin: screenshotOnFail: Data driven scenarios ignore the setting "uniqueScreenshotNames: true" #5298

New issue
New issue
Open
#5299
Open
Plugin: screenshotOnFail: Data driven scenarios ignore the setting "uniqueScreenshotNames: true"#5298
#5299
@mirao

Description

@mirao
mirao
opened on Nov 11, 2025

Config codecept.conf.ts (note the setting uniqueScreenshotNames: true):

import { setHeadlessWhen, setCommonPlugins } from '@codeceptjs/configure';
// turn on headless mode when running with HEADLESS=true environment variable
// export HEADLESS=true && npx codeceptjs run
setHeadlessWhen(process.env.HEADLESS);

// enable all common plugins https://github.com/codeceptjs/configure#setcommonplugins
setCommonPlugins();

export const config: CodeceptJS.MainConfig = {
  tests: './*_test.ts',
  output: './output',
  helpers: {
    Playwright: {
      browser: 'chromium',
      url: 'http://localhost',
      show: true
    }
  },
  include: {
    I: './steps_file'
  },
  plugins: {
    htmlReporter: {
      enabled: true
    },
    screenshotOnFail: {
      enabled: true,
      uniqueScreenshotNames: true,
    },
  },
  name: 'my'
}

test:

Feature('Screenshots');

const testCases = new DataTable(['nr', 'url']);

testCases.add(['1', 'http://codecept.io']);
testCases.add(['2', 'http://playwright.dev']);

Data(testCases).Scenario('test something',  ({ I, current }) => {
    I.amOnPage(current.url);
    I.see("Blabla");
});

Run the test:

$ codeceptjs run --verbose
***************************************
nodeInfo:  20.19.5
osInfo:  Linux 6.14 Ubuntu 24.04.3 LTS 24.04.3 LTS (Noble Numbat)
cpuInfo:  (16) x64 AMD Ryzen 7 9700X 8-Core Processor
chromeInfo:  142.0.7444.134
edgeInfo:  "N/A"
firefoxInfo:  undefined
safariInfo:  N/A
playwrightBrowsers:  "chromium: 141.0.7390.37, firefox: 142.0.1, webkit: 26.0"
If you need more detailed info, just run this: npx codeceptjs info
***************************************
CodeceptJS v3.7.5 #StandWithUkraine
Using test root "/home/mirao/workspace/my"
Helpers: Playwright
Plugins: screenshotOnFail, htmlReporter, retryFailedStep, eachElement

Screenshots --
/home/mirao/workspace/my/Screenshots_test.ts
    [1]  Starting recording promises
    Timeouts: 
 › [Session] Starting singleton browser session
  test something | {"nr":"1","url":"http://codecept.io"}
 › [New Session] {"ignoreHTTPSErrors":false,"acceptDownloads":true}
  Scenario()
    I am on page "http://codecept.io"
    I see "Blabla"
    [1]  Error (Non-Terminated) | Error | err => { step.status = 'failed' step.endTime = +Da...
    [1] Error | Error undefined...
HTML Reporter: Test finished - test something | {"nr":"1","url":"http://codecept.io"}, State: failed, Retries: 0
HTML Reporter: Test test something | {"nr":"1","url":"http://codecept.io"} artifacts at test.finished: []
HTML Reporter: Added new test - test something | {"nr":"1","url":"http://codecept.io"}, State: failed
    [1] <teardown>  Stopping recording promises
 › <screenshotOnFail> Test failed, try to save a screenshot
 › [Screenshot] output/test_something.failed.png
  ✖ FAILED in 1384ms

    [2]  Starting recording promises
  test something | {"nr":"2","url":"http://playwright.dev"}
 › [New Session] {"ignoreHTTPSErrors":false,"acceptDownloads":true}
  Scenario()
    I am on page "http://playwright.dev"
    I see "Blabla"
    [2]  Error (Non-Terminated) | Error | err => { step.status = 'failed' step.endTime = +Da...
    [2] Error | Error undefined...
HTML Reporter: Test finished - test something | {"nr":"2","url":"http://playwright.dev"}, State: failed, Retries: 0
HTML Reporter: Test test something | {"nr":"2","url":"http://playwright.dev"} artifacts at test.finished: []
HTML Reporter: Added new test - test something | {"nr":"2","url":"http://playwright.dev"}, State: failed
    [2] <teardown>  Stopping recording promises
 › <screenshotOnFail> Test failed, try to save a screenshot
 › [Screenshot] output/test_something.failed.png
  ✖ FAILED in 565ms

    [3]  Starting recording promises

-- FAILURES:

  1) Screenshots
       test something | {"nr":"1","url":"http://codecept.io"}:

      expected web application to include "Blabla"
      + expected - actual

      - CodeceptJS 💖 Support Us Quickstart Guides Helpers Plugins API Releases Community 🇺🇦 CodeceptJS was created in Ukraine. #StandWithUkraine CodeceptJS ‐ supercharged End 2 End Testing First AI-Powered Testing Framework 🪄 Quickstart CodeceptJS is opensource MIT licensed testing framework. Works with your favorite frontend frameworks → Scenario Driven Write acceptance tests from user's perspective. Make tests readable and easy to follow. Driver Agnostic Run your tests via Playwright, WebDriver, Puppeteer, TestCafe, Protractor, Appium. The code is the same. Learn More Interactive Debug Control tests as they run. Pause tests at any point and execute commands to try locators. Learn More Rich Locators Use semantic locators, CSS, XPath to find elements on page Learn More PageObjects PageObjects are essential to write stable and reusable code! Learn More Web & Mobile Testing Test native mobile apps using Appium or Detox. Learn More Cucumber-like BDD Automate business scenarios as you do in CucumberJS Learn More API Testing Write tests for REST and GraphQL APIs Learn More Data Management Create fake data and clean it up via REST API Learn More Parallel Testing Tests are split into chunks and executed in multiple processes. Learn More Reduced Flackiness Automatically retry failed steps Learn More AI-powered Use OpenAI GPT to heal failing tests Learn More # Write a Test with Interactive Pause Open a browser on an empty page and pause execution. Type in commands to complete the test scenario. Successful commands will be saved into a file. Scenario('Checkout test', ({ I }) => { I.amOnPage('/checkout'); pause(); }) Copy commands from a file into a test. A test is ready! # Print a Test Each executed step will be printed on screen when running with --steps Scenario('Checkout test', ({ I }) => { I.amOnPage('/checkout'); I.fillField('First name', 'davert'); I.fillField('#lastName', 'mik'); I.fillField('Promo code', '123345') //... }) # Powered with IntelliSense Use auto-completion writing a test fast. We use TypeScript type definitions that are automatically updated for custom steps and page objects. Writing a test in Visual Studio Code is as easy as picking a correct action and putting a parameter. It's really that nice! # Watch & Run Tests from UI We have a flexible interactive web runner which allows you to watch, debug, and write your tests in a web mode. Features: Toggle headless/window mode with one click See HTML snapshot of each step Works with WebDriver, Puppeteer, TestCafe Shows step-by-step execution Integrated with your local IDE Try CodeceptUI → # Write Tests from UI With CodeceptUI you can write your tests without closing a browser at all. Write initial commands and execute a test. An interactive pause will be started when test finishes. Share one browser accross test runs to save time on opening a browser. Try CodeceptUI → # Realworld Example Can we use it for long scenarios? Sure! const { faker } = require('@faker-js/faker'); // Use 3rd-party JS code Feature('Store'); Scenario('Create a new store', async ({ I, login, SettingsPage }) => { const storeName = faker.lorem.slug(); login('customer'); // Login customer from saved cookies SettingsPage.open(); // Use Page objects I.dontSee(storeName, '.settings'); // Assert text not present inside an element (located by CSS) I.click('Add', '.settings'); // Click link by text inside element (located by CSS) I.fillField('Store Name', storeName); // Fill fields by labels or placeholders I.fillField('Email', faker.internet.email()); I.fillField('Telephone', faker.phone.phoneNumberFormat()); I.selectInDropdown('Status', 'Active'); // Use custom methods I.retry(2).click('Create'); // Retry flaky step I.waitInUrl('/settings/setup/stores'); // Explicit waiter I.see(storeName, '.settings'); // Assert text present inside an element (located by CSS) const storeId = await I.grabTextFrom('#store-id'); // Use await to get information from browser I.say(`Created a store with ${storeId}`); // Print custom comments }).tag('stores');`; Trusted By Enterprises Loved By Teams We have been using CodeceptJS as our UI testing framework, and it has made writing tests so simp le for us, the amount of options and features available in CodeceptJS just out of the box are perfect for us to test an application like Percona Monitoring and Management (PMM), with so many dashboards & Metric plots. We would recommend CodeceptJS to anyone who is looking for a Javascript based testing framework. Puneet Kala Frontend QA Automation Engineer at Percona We were searching for a solution to write tests which are good to read and easy to write. It must be able to run on several browsers and understandable across different teams with different knowledge and different frameworks in usage. CodeceptJS helps us with all this and much more at Porsche and we are happy that we made that decision. Mitko Tschimev Frontend Tech Lead at My Porsche Core 🚀 Get more of CodeceptJS ✅ Try Testomat.io, next-gen Test Management system for automated & manual tests. Brought to you by creators of CodeceptJS. Docs Quickstart Getting Started CodeceptJS & Playwright CodeceptJS & WebDriver Community GitHub GitHub discussions Slack Chat Forum Twitter Commercial Support Consulting Trainings Hire Engineers Try Testomat.io → Powerful Test Case Management for CodeceptJS from its authors CODECEPTJS - SUPERCHARGED END 2 END TESTING FRAMEWORK FOR NODEJS © 2025
      +Blabla
      
  
  
  
  ◯ File: file:///home/mirao/workspace/my/Screenshots_test.ts
  
  ◯ Scenario Steps:
  ✖ I.see("Blabla") at Test.<anonymous> (./Screenshots_test.ts:10:7)
  ✔ I.amOnPage("http://codecept.io") at Test.<anonymous> (./Screenshots_test.ts:9:7)
  
  ◯ Artifacts:
  - screenshot: /home/mirao/workspace/my/output/test_something.failed.png
  
  ◯ Metadata:
  - browser: chromium
  - browserVersion: 141.0.7390.37
  - windowSize: 1280x720

  2) Screenshots
       test something | {"nr":"2","url":"http://playwright.dev"}:

      expected web application to include "Blabla"
      + expected - actual

      -Skip to main content Playwright Docs API Node.js Community Search K Playwright enables reliable end-to-end testing for modern web apps. GET STARTED Star 79k+ Any browser • Any platform • One API Cross-browser. Playwright supports all modern rendering engines including Chromium, WebKit, and Firefox. Cross-platform. Test on Windows, Linux, and macOS, locally or on CI, headless or headed. Cross-language. Use the Playwright API in TypeScript, JavaScript, Python, .NET, Java. Test Mobile Web. Native mobile emulation of Google Chrome for Android and Mobile Safari. The same rendering engine works on your Desktop and in the Cloud. Resilient • No flaky tests Auto-wait. Playwright waits for elements to be actionable prior to performing actions. It also has a rich set of introspection events. The combination of the two eliminates the need for artificial timeouts - the primary cause of flaky tests. Web-first assertions. Playwright assertions are created specifically for the dynamic web. Checks are automatically retried until the necessary conditions are met. Tracing. Configure test retry strategy, capture execution trace, videos, screenshots to eliminate flakes. No trade-offs • No limits Browsers run web content belonging to different origins in different processes. Playwright is aligned with the modern browsers architecture and runs tests out-of-process. This makes Playwright free of the typical in-process test runner limitations. Multiple everything. Test scenarios that span multiple tabs, multiple origins and multiple users. Create scenarios with different contexts for different users and run them against your server, all in one test. Trusted events. Hover elements, interact with dynamic controls, produce trusted events. Playwright uses real browser input pipeline indistinguishable from the real user. Test frames, pierce Shadow DOM. Playwright selectors pierce shadow DOM and allow entering frames seamlessly. Full isolation • Fast execution Browser contexts. Playwright creates a browser context for each test. Browser context is equivalent to a brand new browser profile. This delivers full test isolation with zero overhead. Creating a new browser context only takes a handful of milliseconds. Log in once. Save the authentication state of the context and reuse it in all the tests. This bypasses repetitive log-in operations in each test, yet delivers full isolation of independent tests. Powerful Tooling Codegen. Generate tests by recording your actions. Save them into any language. Playwright inspector. Inspect page, generate selectors, step through the test execution, see click points, explore execution logs. Trace Viewer. Capture all the information to investigate the test failure. Playwright trace contains test execution screencast, live DOM snapshots, action explorer, test source, and many more. Chosen by companies and open source projects Learn Getting started Playwright Training Learn Videos Feature Videos Community Stack Overflow Discord Twitter LinkedIn More GitHub YouTube Blog Ambassadors Copyright © 2025 Microsoft
      +Blabla
      
  
  
  
  ◯ File: file:///home/mirao/workspace/my/Screenshots_test.ts
  
  ◯ Scenario Steps:
  ✖ I.see("Blabla") at Test.<anonymous> (./Screenshots_test.ts:10:7)
  ✔ I.amOnPage("http://playwright.dev") at Test.<anonymous> (./Screenshots_test.ts:9:7)
  
  ◯ Artifacts:
  - screenshot: /home/mirao/workspace/my/output/test_something.failed.png
  
  ◯ Metadata:
  - browser: chromium
  - browserVersion: 141.0.7390.37
  - windowSize: 1280x720


  FAIL  | 0 passed, 2 failed   // 2s
HTML Reporter: Processing artifacts for 2 tests after all async tasks complete
HTML Reporter: Processing test "test something | {"nr":"1","url":"http://codecept.io"}" (ID: 0fd8d181)
HTML Reporter: Test test something | {"nr":"1","url":"http://codecept.io"} final artifacts: []
HTML Reporter: Using array artifacts: 0 items
HTML Reporter: No artifacts from plugin, trying filesystem for test "test something | {"nr":"1","url":"http://codecept.io"}"
HTML Reporter: Original test title: "test something | {"nr":"1","url":"http://codecept.io"}"
HTML Reporter: CodeceptJS filename: "test_something"
HTML Reporter: Checking 18 possible screenshot names for "test_something"
HTML Reporter: Checking directory: /home/mirao/workspace/my/output
HTML Reporter: Found 3 files in /home/mirao/workspace/my/output
HTML Reporter: Found screenshot: /home/mirao/workspace/my/output/test_something.failed.png
HTML Reporter: Checking directory: /home/mirao/workspace/my/output
HTML Reporter: Found 3 files in /home/mirao/workspace/my/output
HTML Reporter: Checking directory: /home/mirao/workspace/my/output
HTML Reporter: Found 3 files in /home/mirao/workspace/my/output
HTML Reporter: Checking directory: /home/mirao/workspace/my/output
HTML Reporter: Found 3 files in /home/mirao/workspace/my/output
HTML Reporter: Checking directory: /home/mirao/workspace/my/_output
HTML Reporter: Directory does not exist: /home/mirao/workspace/my/_output
HTML Reporter: Checking directory: /home/mirao/workspace/my/output
HTML Reporter: Found 3 files in /home/mirao/workspace/my/output
HTML Reporter: Checking directory: /home/mirao/workspace/my/qa/output
HTML Reporter: Directory does not exist: /home/mirao/workspace/my/qa/output
HTML Reporter: Checking directory: /home/mirao/workspace/qa/output
HTML Reporter: Directory does not exist: /home/mirao/workspace/qa/output
HTML Reporter: Collected 1 screenshots from filesystem for failed test "test something | {"nr":"1","url":"http://codecept.io"}"
HTML Reporter: Filesystem screenshots for "test something | {"nr":"1","url":"http://codecept.io"}": ["/home/mirao/workspace/my/output/test_something.failed.png"]
HTML Reporter: Final artifacts for "test something | {"nr":"1","url":"http://codecept.io"}": ["/home/mirao/workspace/my/output/test_something.failed.png"]
HTML Reporter: Processing test "test something | {"nr":"2","url":"http://playwright.dev"}" (ID: 301605b5)
HTML Reporter: Test test something | {"nr":"2","url":"http://playwright.dev"} final artifacts: []
HTML Reporter: Using array artifacts: 0 items
HTML Reporter: No artifacts from plugin, trying filesystem for test "test something | {"nr":"2","url":"http://playwright.dev"}"
HTML Reporter: Original test title: "test something | {"nr":"2","url":"http://playwright.dev"}"
HTML Reporter: CodeceptJS filename: "test_something"
HTML Reporter: Checking 18 possible screenshot names for "test_something"
HTML Reporter: Checking directory: /home/mirao/workspace/my/output
HTML Reporter: Found 3 files in /home/mirao/workspace/my/output
HTML Reporter: Found screenshot: /home/mirao/workspace/my/output/test_something.failed.png
HTML Reporter: Checking directory: /home/mirao/workspace/my/output
HTML Reporter: Found 3 files in /home/mirao/workspace/my/output
HTML Reporter: Checking directory: /home/mirao/workspace/my/output
HTML Reporter: Found 3 files in /home/mirao/workspace/my/output
HTML Reporter: Checking directory: /home/mirao/workspace/my/output
HTML Reporter: Found 3 files in /home/mirao/workspace/my/output
HTML Reporter: Checking directory: /home/mirao/workspace/my/_output
HTML Reporter: Directory does not exist: /home/mirao/workspace/my/_output
HTML Reporter: Checking directory: /home/mirao/workspace/my/output
HTML Reporter: Found 3 files in /home/mirao/workspace/my/output
HTML Reporter: Checking directory: /home/mirao/workspace/my/qa/output
HTML Reporter: Directory does not exist: /home/mirao/workspace/my/qa/output
HTML Reporter: Checking directory: /home/mirao/workspace/qa/output
HTML Reporter: Directory does not exist: /home/mirao/workspace/qa/output
HTML Reporter: Collected 1 screenshots from filesystem for failed test "test something | {"nr":"2","url":"http://playwright.dev"}"
HTML Reporter: Filesystem screenshots for "test something | {"nr":"2","url":"http://playwright.dev"}": ["/home/mirao/workspace/my/output/test_something.failed.png"]
HTML Reporter: Final artifacts for "test something | {"nr":"2","url":"http://playwright.dev"}": ["/home/mirao/workspace/my/output/test_something.failed.png"]
HTML Reporter: Calculated stats - Tests: 2, Passes: 0, Failures: 2
HTML Reporter: Collected 2 tests in reportData
HTML Reporter: Failures array has 2 items
HTML Reporter: Retries array has 0 items
HTML Reporter: testRetryAttempts Map size: 0
HTML Reporter: Test in reportData - test something | {"nr":"1","url":"http://codecept.io"}, State: failed, Retries: 0
HTML Reporter: Test in reportData - test something | {"nr":"2","url":"http://playwright.dev"}, State: failed, Retries: 0
***************************************
nodeInfo:  20.19.5
osInfo:  Linux 6.14 Ubuntu 24.04.3 LTS 24.04.3 LTS (Noble Numbat)
cpuInfo:  (16) x64 AMD Ryzen 7 9700X 8-Core Processor
chromeInfo:  142.0.7444.134
edgeInfo:  "N/A"
firefoxInfo:  undefined
safariInfo:  N/A
playwrightBrowsers:  "chromium: 141.0.7390.37, firefox: 142.0.1, webkit: 26.0"
If you need more detailed info, just run this: npx codeceptjs info
***************************************
HTML Reporter: Processing 1 artifacts, isFailedTest: true
HTML Reporter: Artifacts: ["/home/mirao/workspace/my/output/test_something.failed.png"]
HTML Reporter: Processing artifact: /home/mirao/workspace/my/output/test_something.failed.png (type: string)
HTML Reporter: Found screenshot: /home/mirao/workspace/my/output/test_something.failed.png
HTML Reporter: Found 1 screenshots and 0 other artifacts
HTML Reporter: Screenshot test_something.failed.png -> test_something.failed.png
HTML Reporter: Processing 1 artifacts, isFailedTest: true
HTML Reporter: Artifacts: ["/home/mirao/workspace/my/output/test_something.failed.png"]
HTML Reporter: Processing artifact: /home/mirao/workspace/my/output/test_something.failed.png (type: string)
HTML Reporter: Found screenshot: /home/mirao/workspace/my/output/test_something.failed.png
HTML Reporter: Found 1 screenshots and 0 other artifacts
HTML Reporter: Screenshot test_something.failed.png -> test_something.failed.png
HTML Report saved to: /home/mirao/workspace/my/output/report.html

Expected result:

  • two different screenshots in the format test_something_<unique_string>.failed.png (for every Data() run)

Actual result

  • only one screenshot test_something.failed.png - each Data() run overwrites it

It works well for simple (non Data()) scenario. Then the screeenshot name is e.g. test_something_TezX4weoIav9Od+F9o6ddYXBJZF9FiNXYHYl4iUs_c.failed

Used SW:

  • CodeceptJS 3.7.5
  • Playwright 1.56.1

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions