Make path pattern smarter for errors screenshots

[fmancardi]

What is your Scenario?

I would like that screenshots on failure generate a file name with the testcase name + a progressive.
Probably this a non-sense because when test script fails it can not take more than one screenshot.

This is my .testcaferc.json

{
  "selectorTimeout": 2000,
  "skipJsErrors": true,

  "screenshots": {
    "takeOnFails": true,
    "pathPattern": "${DATE}_${TIME}/test-${TEST}/${USERAGENT}/${TEST}-${FILE_INDEX}.png"
  }
}

What is the Current behavior?

for the screenshots taken usin .takeScreenshot() it work as expected generating

inside the TCF-004-errors folder, file name is the standard

What is the Expected behavior?

inside the TCF-004-errors folder, file name respect the mask

${TEST}-${FILE_INDEX}.png

What is your public website URL? (or attach your complete example)

You can access our test site

This is the example

What is your TestCafe test code?

import { Selector } from 'testcafe';

fixture `New Fixture`
    .page `https://platform70-at.tesisquare.com/core/framework/login.cfm`;

test
.meta({'TCID': 'TCF-004',
          'WKFSTATUS': 'ready',
          '6.3': 'yes',
          '6.4': 'yes',
          '7.0': 'yes'})
('TCF-004', async t => {
    await t
        .click('#cookieChoiceDismiss')
        .typeText('#userlogin', 'admin')
        .typeText('#password', 'qqq')
        .click('#login')
        .click('#btn-toggle-menu')
        .click(Selector('#A_ADMIN_SYS-GESANAG_AN span').withText('ANAGRAFICHE'))
        .click('#A_ADMIN_SYS-GESANAG_AN_1')
        .click('#inserisci')
        .switchToIframe('#frame-for-tab')
        .typeText('#IdAnag', 'qazwsxedcrfvtgbyhnuj')
        .takeScreenshot()
        .typeText('#DescIndirizzoPF', 'qq')
        .typeText('#DescCittaPF', 'qq')
        .takeScreenshot()
        .typeText('#DescRagSocPF', 'qq')
        .typeText('#CapPF', 'qq')
        .typeText('#IdProvinciaPF', 'qq')
        .takeScreenshot()
        .click('#select2-Stato-container')
        .typeText('#fieldset_g1 .select2-search__field', 'Valido')
        .pressKey('enter')
        .click('#select2-idlingua-container')
        .typeText('#fieldset_g1 .select2-search__field', 'Italiano')
        .pressKey('enter')
        .click('#select2-IdNazionePF-container')
        .typeText('#fieldset_g1 .select2-search__field', 'ITALY')
        .pressKey('enter')
        .click('#inserisci')
        .expect(Selector('#pagecontainer div')
        	     .withText('Error ID: PLATFORM_STANDARD')
        	     .exists).eql(false)
        .expect(Selector('#pagecontainer a')
        	      .withText('Return to the Homepage')
        	      .exists).eql(false)
        .debug();
});

Your complete configuration file

{
  "selectorTimeout": 2000,
  "skipJsErrors": true,

  "screenshots": {
    "takeOnFails": true,
    "pathPattern": "${DATE}_${TIME}/test-${TEST}/${USERAGENT}/${TEST}-${FILE_INDEX}.png"
  }
}

Your complete test report

No response

Screenshots

No response

Steps to Reproduce

N.A.

TestCafe version

1.18.2

Node.js version

v14.15.5

Command-line arguments

testcafe 'chrome --start-fullscreen' --test-meta WKFSTATUS=ready,6.4=yes --cfg=at-7.0.json ./test/testcafe/TCF-006-multiple-sshots.js

Browser name(s) and version(s)

No response

Platform(s) and version(s)

Windows 10, using Gitbash to run the script

Other

No response

[Aleksey28]

Hi @fmancardi,

This behavior is expected. TestCafe replaces any ${FILE_INDEX} for the error with ${ERRORS_FOLDER}/${FILE_INDEX}. This decision was made because a file can contain a lot of tests with errors. If you want to screenshot only an error and you have only one test in the file, you can use ${TEST_INDEX} instead of ${FILE_INDEX}

[fmancardi]

I'm going to provide a more detailed explanation, because IMHO replace is not working as expected for errors

[Aleksey28]

Fine, please share more details, but before you do that, take a look at this paragraph. In the example with the default path pattern, you can see that the errors/string is put before ${FILE_INDEX}. Testcafe works with custom patterns in the same way.

[fmancardi]

Please give a look to my first explanation, the errors folder does not exist anymore, it's name is changed to TC-004-errors.
This is not the expectation.
folder name must remain errors
file name inside the folder have to have TC-004 as prefix then the progressive.

regards

[Aleksey28]

Thank you for the explanation. It happens because you separate ${TEST} and ${FILE_INDEX} with -. Thus, Testcafe doesn't handle your case and just replaces ${FILE_INDEX} with the errors/${FILE_INDEX}. We will think about ways to improve this behavior and update this thread once we have any news.

[fmancardi]

I've tried with
"pathPattern": "${DATE}_${TIME}/test-${TEST}/${USERAGENT}/${TEST}${FILE_INDEX}.png"

same effect
errors folder is renamed but images inside no

[Aleksey28]

It doesn't change anything. In case of errors, testcafe replaces part of the {FILE_INDEX} pattern with errors/${FILE_INDEX} each time. It doesn't matter what you put before or after.
I have an idea. You can swap {FILE_INDEX} and ${TEST} if it's appropriate to you like this: ${DATE}_${TIME}/test-${TEST}/${USERAGENT}/${FILE_INDEX}-${TEST}.png. The names of the files with errors will be changed.

[fmancardi]

Using your suggestions things work as expected.
We can consider this a workaround.
It will be great if in the future using the mask ${TEST}-${FILE_INDEX}.png things also work.
I understand this is not a critic issue, but having fixed will be great.

May be a more flexible approach can be to have two pathPatters:

• 'pathPattern'
• 'errorsPathPattern' or 'pathPatternOnFails'

"pathPattern": "${DATE}_${TIME}/test-${TEST}/${USERAGENT}/${TEST}-${FILE_INDEX}.png"
"pathPatternOnFails": "${DATE}_${TIME}/test-${TEST}/${USERAGENT}/MyErrorsFolder/myPrefix-${TEST}-${FILE_INDEX}.png"

Best regards

[Aleksey28]

Thank you for the suggestion. We will consider ways to improve this behavior and update this thread once we have news.

[github-actions]

Release v3.5.0-rc.1 addresses this.

[github-actions]

Release v3.5.0-rc.1 addresses this.