Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Expose the step API to fixtures #27059

Open
dvoytenko opened this issue Sep 13, 2023 · 5 comments
Open

[Feature] Expose the step API to fixtures #27059

dvoytenko opened this issue Sep 13, 2023 · 5 comments
Labels
feature-test-runner Playwright test specific issues

Comments

@dvoytenko
Copy link

Our fixture performs some advanced mocking and it would be very convenient if it could output information, similar to how "route" information is output as steps. The information that could be displayed for a mocking operation would be:

  • Call info: API name, API parameters
  • Call result
  • Additional log messages and error status

We're using the test.step() API right now, but it doesn't allow setting most of these parameters. We've also tried to use the private _runAsStep(), which allows setting all the data, but not a public API.

@dgozman
Copy link
Contributor

dgozman commented Sep 13, 2023

@dvoytenko Could you share a short snippet illustrating your usage of _runAsStep()? Perhaps you can put all the information into a step title and call test.step() instead?

@dvoytenko
Copy link
Author

@dgozman

@dvoytenko Could you share a short snippet illustrating your usage of _runAsStep()? Perhaps you can put all the information into a step title and call test.step() instead?

Here's an example from Playwright UI of what we're doing in our fixture:

image

And here's an example of a failed mock:

image

The only way we found how to do this is by using the _runAsStep internal API:

await testInfo._runAsStep(
  {
    category: 'mock',
    apiName: `mock.fulfilled`,
      params: {
        service: serviceName,
        ...serviceArgs, 
  },
  async (step) => {
    if (serviceFailure) {
       step.complete({ error: { message: serviceFailure } });
    }
  },
);

We'd definitely use test.step if it allowed us to add all this metadata and the error status.

An aside on the error status. Ideally, a step can be marked as failing without interrupting the complete test. Currently, if the test.step callback fails - the test is fully interrupted. Hence the use of step.complete({ error }) above.

@dgozman
Copy link
Contributor

dgozman commented Sep 14, 2023

@dvoytenko We can probably add more options to test.step() like key-value parameters. Sounds like a nice feature request. I'll leave this issue open for prioritization.

Ideally, a step can be marked as failing without interrupting the complete test

You should be able to catch the step:

await test.step('title', async () => {
  throw new Error();
}).catch(() => {});

@dgozman dgozman added feature-test-runner Playwright test specific issues and removed triaging labels Sep 14, 2023
@dvoytenko
Copy link
Author

Thanks! I think this would be a great feature.

@dvoytenko
Copy link
Author

@dgozman I also noticed that there's a "LOGS" section inside the "Call" tab. Not critical, but if that could also be exposed - that'd help some use cases.

kodiakhq bot pushed a commit to vercel/next.js that referenced this issue Sep 18, 2023
This feature eases debugging significantly.

This is what an intercepted fetch looks like in Playwright debugger:

<img width="342" alt="image" src="https://github.com/vercel/next.js/assets/726049/e9fe4304-36b9-4d6d-b4f3-66d649464a35">
<img width="368" alt="image" src="https://github.com/vercel/next.js/assets/726049/db49e18e-3fc5-4a77-abf8-465d925082bc">

And here's what a failing fetch looks like:

<img width="314" alt="image" src="https://github.com/vercel/next.js/assets/726049/41be212a-e414-4b28-a0cb-d9b618b3f2ea">

You can inspect request and response's headers and bodies.

The main drawback: it uses an internal Playwright API. However, they are open to opening part of it publicly in  microsoft/playwright#27059.
Ethan-Arrowood added a commit to vercel/next.js that referenced this issue Apr 17, 2024
This pr removes a console.log that is making testmode tests very noisy.

This relates to an upstream feature request:
microsoft/playwright#27059

Closes NEXT-3139
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature-test-runner Playwright test specific issues
Projects
None yet
Development

No branches or pull requests

2 participants