Skip to content

[BUG] aria snapshot's regex marked as errors if the test fails #34555

New issue
New issue
Open
Open
[BUG] aria snapshot's regex marked as errors if the test fails#34555
Assignees
agg23
Labels
v1.56
@umer-shahzaddar

Description

@umer-shahzaddar
umer-shahzaddar
opened on Jan 30, 2025

hello, i'm using aria snapshots to validate some sections and when a check fails in the error message all the regex are shown as differences making it difficult to find identify the error.

in the example below the real error is:

- - button "Colore neutro Garantisce solo modelli di colore neutro + € 1 "
+ - button "Colore neutro Garantisce solo modelli di colore neutro + € 0 "

and as you can see also all the regex are marked as errors.

if in this text i change the "1" to "0" the test is marked as passed

My code:

await expect(this.page.locator(this.boxConfigurationContainer)).toMatchAriaSnapshot(`
        - text: Configura La tua Subbyx Box
        - img
        - text: /\\d+ % Cosa potresti ricevere/
        - button "" [disabled]
        - text: /Galaxy S24\\+ Risparmi \\d+,\\d+ € al mese 👍🏼 Molto probabile/
        - button ""
        - button "Standard Contiene usato PreLoved di qualsiasi grado estetico"
        - button /Excellent Contiene solo usato PreLoved di grado estetico Eccellente \\+ \\d+,\\d+ €/:
            - paragraph: /\\+ \\d+,\\d+ €/
        - link " Quali sono i gradi dei PreLoved?"
        - paragraph: Blocca una
        - paragraph: Preferenza
        - paragraph: Max 1 selezione
        - button "Marca Garantisce un modello della marca selezionata + € 0  Seleziona un brand "
        - button "Colore neutro Garantisce solo modelli di colore neutro + € 1 "
        - button "Schermo base Garantisce uno schermo di dimensioni standard o più ridotte + € 2 "
        - button "Schermo grande Garantisce uno schermo più grande rispetto al modello base + € 3 "
        - button /Tanta memoria Garantisce una memoria da \\d+ GB o superiore \\+ € 2 /
        - button "Continua senza preferenze Ricevi una configurazione casuale Gratis "
        - button "Abbonati senza vincoli"
            `);

The error:

Locator: locator('//*[@class="w-full flex flex-col justify-center items-center px-5 md:px-[10%] lg:px-[12%] py-14 bg-sbx-gray-light gap-5"]')
    - Expected  - 6
    + Received  + 7

      - text: Configura La tua Subbyx Box
      - img
    - - text: /\d+ % Cosa potresti ricevere/
    + - text: 34 % Cosa potresti ricevere
      - button "" [disabled]
    - - text: /Galaxy S24\+ Risparmi \d+,\d+ € al mese 👍🏼 Molto probabile/
    + - text: Galaxy S24+ Risparmi 13,92 € al mese 👍🏼 Molto probabile
      - button ""
      - button "Standard Contiene usato PreLoved di qualsiasi grado estetico"
    - - button /Excellent Contiene solo usato PreLoved di grado estetico Eccellente \+ \d+,\d+ €/:
    + - button "Excellent Contiene solo usato PreLoved di grado estetico Eccellente + 3,00 €":
    -     - paragraph: /\+ \d+,\d+ €/
    +   - text: Excellent Contiene solo usato PreLoved di grado estetico Eccellente
    +   - paragraph: + 3,00 €
      - link " Quali sono i gradi dei PreLoved?"
      - paragraph: Blocca una
      - paragraph: Preferenza
      - paragraph: Max 1 selezione
      - button "Marca Garantisce un modello della marca selezionata + € 0  Seleziona un brand "
    - - button "Colore neutro Garantisce solo modelli di colore neutro + € 1 "
    + - button "Colore neutro Garantisce solo modelli di colore neutro + € 0 "
      - button "Schermo base Garantisce uno schermo di dimensioni standard o più ridotte + € 2 "
      - button "Schermo grande Garantisce uno schermo più grande rispetto al modello base + € 3 "
    - - button /Tanta memoria Garantisce una memoria da \d+ GB o superiore \+ € 2 /
    + - button "Tanta memoria Garantisce una memoria da 256 GB o superiore + € 2 "
      - button "Continua senza preferenze Ricevi una configurazione casuale Gratis "
      - button "Abbonati senza vincoli"

Activity

agg23

agg23 commented on Jan 30, 2025

@agg23
agg23
on Jan 30, 2025
Contributor

Please break this down into individual cases that we can run and evaluate each scenario. Without a full, minimal repro we are unable to help.

umer-shahzaddar

umer-shahzaddar commented on Jan 30, 2025

@umer-shahzaddar
umer-shahzaddar
on Jan 30, 2025
Author

This is already an individual case. The issue occurs when performing a validation using an ARIA snapshot. If there is a difference between the actual and expected output, the error log also highlights regex patterns as differences, which is unexpected. You can see this behavior in the log I attached.

agg23

agg23 commented on Jan 30, 2025

@agg23
agg23
on Jan 30, 2025
Contributor

It is an individual case, but it is very large, and you have not provided any of the backing DOM or the actual test contents. I need to be able to run the test and get the same output as you. Please provide a smaller ARIA snapshot so it's more readable.

umer-shahzaddar

umer-shahzaddar commented on Jan 30, 2025

@umer-shahzaddar
umer-shahzaddar
on Jan 30, 2025
Author

You can reproduce the issue by running the following code.

import { test, expect } from '@playwright/test';

test('test', async ({ page }) => {
  await page.goto('https://www.subbyx.com/en/box/premium?id=subbyxboxsmartphoneandroid');
  await page.getByLabel('Close this notice').click();
  await expect(page.locator('(//*[@class="w-full flex flex-col gap-3"])[1]')).toMatchAriaSnapshot(`
    - button "Standard Contains PreLoved devices of any aesthetic gradeE"
    - button /Excellent Contains only PreLoved devices with Excellent aesthetic grade \\+ €\\d+\\.\\d+/:
      - paragraph: /\\+ €\\d+\\.\\d+/
    - link " What are the grades of PreLoveds?"
    `);
});

In this test, the only expected difference between the actual and expected output should be:

Expected: "gradeE"
Actual: "grade"
However, the error log also highlights regex patterns as differences, which should not happen.

To confirm that regex handling is correct, you can modify the expected value from "gradeE" to "grade", and the test will pass. This proves that the regex patterns themselves are valid and functioning correctly.

agg23
added
v1.51
on Jan 31, 2025
agg23
self-assigned this
on Jan 31, 2025
mxschmitt
added
v1.52
and removed
v1.51
on Feb 4, 2025
NFranky97

NFranky97 commented on Mar 11, 2025

@NFranky97
NFranky97
on Mar 11, 2025 · edited by NFranky97

Just wanting to add on to this.
Any removed attributes that you do not want to assert against are also shown in the diff even though they are not causing an error such as the following example.

      - rowgroup:
-          - row:
+         - row "table-cell-initials table-cell-profileData table-cell-date table-cell-testName table-cell-position table-cell-reps":
               - cell "table-cell-initials":
               - paragraph: QP
-     - cell "table-cell-date": /\d+\/\d+\/\d+ \d+:\d+ (AM|PM)/
+    - cell "table-cell-date": 12/03/2025 8:44 AM
-     - img: /66 ° 66 ° 73 ° 62 °/
+    - img: 66 ° 66 ° 62 ° 73 °

In the above example the only line causing an error is the img: as the numbers are in an incorrect order. However the report showing a diff for both the excluded properties from the row; and any regex makes it harder to identify the problem.

agg23
added
v1.53
and removed
v1.52
on Mar 25, 2025
agg23
mentioned this on May 27, 2025
agg23
added
v1.54
and removed
v1.53
on May 29, 2025
agg23
added
v1.55
and removed
v1.54
on Jun 30, 2025
agg23
added
v1.56
and removed
v1.55
on Aug 4, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

v1.56

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

    Development

    Participants

    @agg23@mxschmitt@NFranky97@umer-shahzaddar

    Issue actions
      [BUG] aria snapshot's regex marked as errors if the test fails · Issue #34555 · microsoft/playwright