Playwright Integration improvement and integration with playwright test

[Romarionijim]

Hello, First of all, kudos for making an integration with playwright, I've been diving into automation performance/load testing, I currently use Playwright with TS and I'm researching a tool that I can perform load testing while integrating Playwright and leverage existing framework, I have gone through the documentation and articles and I came across issues that I would love for them to be added in order to consider implementing artillery at my workplace.

1. add integration with playwright/test and leverage existing POM framework:

• at the current moment from what I understand is the integration is only done with Playwright library and not playwright test, I have an existing POM framework written with Playwright TS (btw each class in the page object is already exported with the export keyword before each class ) - the problem is in this case I cannot reuse functions that I already have in my page objects because in order for me to use them, I need to create a new JS file and export the async function and duplicate the code that I already have inside it and only then call it in the yml file, I would appreciate to leverage existing function without creating files and duplicating functions just to call them in the yml since I already exported the class that has these functions that I wanna use and leverage for load testing.

2. support of TS:

• at the moment per documentation - TS is experimental and not fully supported, today TS is the standard and the default language used with playwright, so at the current moment if you have a TS playwright framework in order to use the function you need to create a JS file like said in point 1 above instead of leveraging TS because TS may or may not work since it is experimental.

3. support for creating tests in playwright style instead of yml:

• I would also appreciate to write the tests in a playwright style format instead of building yml files.

4. provide a clear way and documentation on how to assert a test:

• in the docs - it is not that clear how do I assert if a test have passed or failed, maybe I'm missing something but I don't see an assertion on certain metrics such as response time, user count and etc, I suggest to improve the documentation a bit more to let testers that are just getting into performance and load testing know how to use it without complicating things too much.

While the first two suggestions are more important and critical for me (leverage existing POM framework and 100% support for TS and playwright test) - I would very much appreciate if you could address these issues and make these improvements, I think these improvements will very much help to decide to choose artillery over any other performance tool that has integration with playwright.

Thank you!

[hassy]

thanks for the feedback @Romarionijim! we're going to address most of the items over time.

You're right in that some glue code needs to be written to hook up existing Playwright code to Artillery for load testing. Most of the time the amount of work that needs to be done is fairly small/straightforward. It will reduce over time.

TypeScript support is marked as experimental, but it works for most existing test suites. Have you tried it with your code?

I would also appreciate to write the tests in a playwright style format instead of building yml files.

The YAML part is Artillery-specific and that won't go away because ultimately running a load test is different and requires a different configuration from running an end-to-end test. Even when we provide a JavaScript/TypeScript interface instead of the YAML-based one, you'll still have to provide the same configuration.

provide a clear way and documentation on how to assert a test:

We'll improve the docs around this. The recommended way is to set success conditions with the ensure plugin: https://www.artillery.io/docs/reference/extensions/ensure

This is one area where load testing is very different from end-to-end testing. The definition of a successful load test is very application/test specific. It's not as binary as pass/fail for other types of tests. The ensure plugin allows you to define custom conditions for that.

[hassy]

(moving to Discussions)

[afsal-backer]

@hassy You mentioned Ensure plugin but I see it only supports http.

One strange thing I noticed is that if I set my URL to http version I don’t get the full metrics on the console. If I set it to https everything is printed but Ensure assertions doesn’t work. How can we get around that?

[bernardobridge]

Hi @afsal-backer 👋 .

Can you please open a new discussion or bug report (if you believe it's a bug), including your YAML config+script? It's the best way for us to be able to understand what's wrong and help you and others that might run into this.

P.S. The ensure plugin should work with any metric type, not just HTTP engine ones.

[Romarionijim]

@hassy Thank you for the response.

regarding point 1: the thing is that it is still an overhead and a code duplication, for example here is a login class that performs a login flow:

import { LumaMainPage } from "../LumaMainPage";
import { ClientSideValiationErrorOptionalParamsInterface } from "../../helpers/optionalParamsInterfaces/OptionalParams";

export class LoginPage extends LumaMainPage {
  private emailFieldLocator = '[name="login[username]"]';
  private passwordFieldLocator = '[name="login[password]"]';

  public async login(email: string = process.env.EMAIL as string, password: string = process.env.PASSWORD as string,
    options?: ClientSideValiationErrorOptionalParamsInterface & { negativeTest?: boolean, expectedErrorCount?: number }) {
    const emailField = this.page.locator(this.emailFieldLocator);
    const passwordField = this.page.locator(this.passwordFieldLocator);
    const loggedInState = await this.getLoggedInState();
    if (loggedInState === 'not-logged-in') {
      await this.clickSignIn();
      await this.fillText(emailField, email);
      await this.fillText(passwordField, password);
      const signInButton = this.page.getByRole('button', { name: 'Sign In' });
      await this.clickElement(signInButton);
      await this.page.waitForTimeout(2500);
    }
    if (options?.negativeTest && options.expectedErrorCount !== undefined) {
      await this.handleClientSideValidationErrors(options.expectedErrorCount, [emailField, passwordField], options)
    }
  }
}  

in order for me to use it in Artillery yml, I need to create login-flow.js file:

export async function login() {
 const emailField = this.page.locator(this.emailFieldLocator);
    const passwordField = this.page.locator(this.passwordFieldLocator);
    const loggedInState = await this.getLoggedInState();
    if (loggedInState === 'not-logged-in') {
      await this.clickSignIn();
      await this.fillText(emailField, email);
      await this.fillText(passwordField, password);
      const signInButton = this.page.getByRole('button', { name: 'Sign In' });
      await this.clickElement(signInButton);
      await this.page.waitForTimeout(2500);
} 

and then call this login function from this js file

I could also just create an object of the login page inside the async function in the js file but it is still a sort of duplication since I have the same exact functionality in 2 different files.
What I would expect in this case is to reference the function in the yml file from the already exported login class and choose the function by name instead of doing all of this which could make a HUGE difference IMO.
Will there be an improvement on this matter?

As for TS - I still have not tried it but I'm a little doubt in it since it is marked as experimental because if my whole codebase is in TS and some may not work, then I need to convert to JS and make these adjustments.

[hassy]

As @bernardobridge said just try out TS support. If something breaks we can take a look at the actual issue then. We use esbuild to convert TypeScript to JavaScript under the hood, so most existing codebases should just work.

[Romarionijim]

@hassy Got it but what about having to create the function again like shown in the above code? can you address if there will be improvement there?

[bernardobridge]

Hi @Romarionijim,

You don't need to convert it to a Javascript file, you should just try out the Typescript support.

There's an example here of reusing Typescript code: https://github.com/artilleryio/artillery/tree/main/examples/browser-playwright-reuse-typescript . The general idea is that you abstract anything you need into reusable code that can be used by both your Functional and Load Tests, which sounds very similar to what you want anyway.

I understand it's marked as Experimental, but there are currently no open issues we are aware of that users have hit with Typescript support.

[bernardobridge]

Hi @Rio517 - glad it was useful ! The docs repository is not currently public. We are already planning on linking all examples in there, so it'll be done soon! Thank you for the feedback 🙇‍♂️.

[enxoco]

This example unfortunately is not working for me. When I run artillery run search-for-ts-doc.yml I get an error about not being able to find @playwright/test. What am I missing here? This seems to be related to import { Page, expect } from '@playwright/test';. If I remove expect and comment out the related lines in the test then everything works. I am seeing this same behavior in a project where I am trying to refactor it to follow these practices.

$ playwright --version
Version 1.45.3

$ artillery --version
        ___         __  _ ____
  _____/   |  _____/ /_(_) / /__  _______  __ ___
 /____/ /| | / ___/ __/ / / / _ \/ ___/ / / /____/
/____/ ___ |/ /  / /_/ / / /  __/ /  / /_/ /____/
    /_/  |_/_/   \__/_/_/_/\___/_/   \__  /
                                    /____/


VERSION INFO:

Artillery: 2.0.18
Node.js:   v20.16.0
OS:        linux

[bernardobridge]

Hi @enxoco 👋. I'll be looking at this issue this week, as there is someone else running into this. A couple of questions:

• Are you trying to run inside a docker container by any chance? Or is it just in your local environment?
• In your project, how are you installing Artillery? Is it globally installed, or as a dependency?
• If you try to install @playwright/test yourself, does it work?

Thanks!

[MSauchanka]

@bernardobridge I have the same issue running example browser-playwright-reuse-authentication inside docker container in the same way as it described in the doc - https://www.artillery.io/docs/docker

browser-playwright-reuse-authentication docker run --rm -it -v ${PWD}:/scripts \
artilleryio/artillery:latest \
run /scripts/scenario.yml
Test run id: trbjd_3dk4g6tr7fz5w6n9p6y9bqct6r4dk_w8bb
⠋ node:internal/modules/cjs/loader:1140
  const err = new Error(message);
              ^

Error: Cannot find module '@playwright/test'
Require stack:
- /scripts/flow.js
- /home/node/artillery/node_modules/@artilleryio/int-core/lib/runner.js
- /home/node/artillery/node_modules/@artilleryio/int-core/index.js
- /home/node/artillery/lib/artillery-global.js
- /home/node/artillery/bin/run
    at Module._resolveFilename (node:internal/modules/cjs/loader:1140:15)
    at Module._load (node:internal/modules/cjs/loader:981:27)
    at Module.require (node:internal/modules/cjs/loader:1231:19)
    at require (node:internal/modules/helpers:177:18)
    at Object.<anonymous> (/scripts/flow.js:1:20)
    at Module._compile (node:internal/modules/cjs/loader:1364:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1422:10)
    at Module.load (node:internal/modules/cjs/loader:1203:32)
    at Module._load (node:internal/modules/cjs/loader:1019:12)
    at Module.require (node:internal/modules/cjs/loader:1231:19) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/scripts/flow.js',
    '/home/node/artillery/node_modules/@artilleryio/int-core/lib/runner.js',
    '/home/node/artillery/node_modules/@artilleryio/int-core/index.js',
    '/home/node/artillery/lib/artillery-global.js',
    '/home/node/artillery/bin/run'
  ]
}

[enxoco]

@bernardobridge Yeah that was on me. I guess I was taking the example a little too literally. Installing the packages worked for me. I do have a separate issue with another project I am working on that uses yarn workspaces but I also recognize that is probably not a recommended setup.

[Romarionijim]

@bernardobridge I understand, I will try it with TS for sure but My issue is not with the programming language, My issue is the code duplication and having to create another abstraction which seems a bit redundant IMO because I'm already abstracting the code and logic in my page objects, in case of artillery your aiming to achieve the same exact flow in the "abstracted function" while you already have it abstracted, so why not use the function directly from the page object inside the yml file? similar to end to end.
in end to end you create an object and call the function that you defined once, why not reference the same thing in the yml file instead of just creating another abstraction of the same thing?

[bernardobridge]

@Romarionijim If you have page objects like you described, there's nothing preventing you from importing those page objects directly (as long as you're using the Typescript support). For example, using the code you put above:

Your Page Object:

import { LumaMainPage } from "../LumaMainPage";
import { ClientSideValiationErrorOptionalParamsInterface } from "../../helpers/optionalParamsInterfaces/OptionalParams";

export class LoginPage extends LumaMainPage {
  private emailFieldLocator = '[name="login[username]"]';
  private passwordFieldLocator = '[name="login[password]"]';

  public async login(email: string = process.env.EMAIL as string, password: string = process.env.PASSWORD as string,
    options?: ClientSideValiationErrorOptionalParamsInterface & { negativeTest?: boolean, expectedErrorCount?: number }) {
    const emailField = this.page.locator(this.emailFieldLocator);
    const passwordField = this.page.locator(this.passwordFieldLocator);
    const loggedInState = await this.getLoggedInState();
    if (loggedInState === 'not-logged-in') {
      await this.clickSignIn();
      await this.fillText(emailField, email);
      await this.fillText(passwordField, password);
      const signInButton = this.page.getByRole('button', { name: 'Sign In' });
      await this.clickElement(signInButton);
      await this.page.waitForTimeout(2500);
    }
    if (options?.negativeTest && options.expectedErrorCount !== undefined) {
      await this.handleClientSideValidationErrors(options.expectedErrorCount, [emailField, passwordField], options)
    }
  }
}  

Your Artillery Processor (processor.ts):

import { LoginPage } from './PATH_TO_TS_FILE.ts'

export async function loginFlow() {
  const loginPage = new LoginPage();
   loginPage.login();
} 

---

What I meant about abstracting any code you want to reuse is for example if you wanted to reuse a Playwright test flow directly in Artillery. That's when you'd need to abstract that into helpers instead. So for example let's say your Playwright test performs a login and then a search, you could abstract that into a helper/POM/whatever, and then use that in Artillery in a similar way to what I described above.

[Romarionijim]

@bernardobridge Yes I understand that, but that is exactly my point, why import it into another file and create a function that does the same thing which is login in my case? it is the exact same flow, why not reference the LoginPage class as the processor in the yml file and call/reference the login function as the targetFunction ?
My point is, I already have functions with their logic in my POM, my suggestion is, instead of creating another processor file and export a function that does the exact same thing as an existing function in my page objects, to create a yml file, reference the page object class file and call the target function, easy and straightforward.

[bernardobridge]

@Romarionijim because that's how Artillery is currently designed - there's a single entrypoint (processor) which exports the functions needed. The function used by your Artillery test needs to follow Artillery's function signature, so there is currently a need to create a specific function for Artillery because you may need Artillery-specific functionality in it.

In that sense it's not much different from any abstraction you create in your code that needs to be used by two consumers - functionA and functionB may both use the same abstraction but each has their own function signatures and properties. In this case your abstraction is going to be used by both Artillery and Playwright, which both have slight differences to them.

---

We appreciate the feedback though and will take that into consideration for future improvements. I'm just trying to show that there is a way to do what you described, and with not much additional abstraction needed, as you can import your POM directly if you're using Typescript anyway. If that breaks or is not the case, we'll take a look at it!

[Romarionijim]

@bernardobridge @hassy and If I have a backend framework using playwright as well (it also supports api testing) and my framework is written in OOP style, similar to POM but for the backend, in this case, can I leverage playwright as well?

for example, I have this ApiClient class the serves as a BaseClass for Api, here is some of the code, can I create a processer from these functions and use the playwright engine or do I need to recreate the backend from the other suggested engines such as HTTP engine?

code example:

`
export class ApiClient {

constructor(public request: APIRequestContext) {
this.request = request;
}

public async paginateRequest(method: RequestMethod, url: string, pagintionType: PaginationType, options: ApiOptionalParams) {
try {
let response = await this.paginateBy(method, url, pagintionType, options);
return response;
} catch (error) {
throw new Error(an error occured in the paginate request function: ${error})
}
}

public async get<T>(url: string, options?: ApiOptionalParams<T>) {
    let response = await this.makeRequest(RequestMethod.GET, url, options)
    return response;
}

public async post<T>(url: string, options?: ApiOptionalParams<T>) {
    let response = await this.makeRequest(RequestMethod.POST, url, options)
    return response;
}

public async put<T>(url: string, options?: ApiOptionalParams<T>) {
    let response = await this.makeRequest(RequestMethod.PUT, url, options)
    return response;
}

public async patch<T>(url: string, options?: ApiOptionalParams<T>) {
    let response = await this.makeRequest(RequestMethod.PATCH, url, options)
    return response;
}

public async delete<T>(url: string, options?: ApiOptionalParams<T>) {
    let response = await this.makeRequest(RequestMethod.DELETE, url, options)
    return response;
}

}
`

[Beliefg]

I did this with just the Playwright Engine in Javascript, with just the library implementation of fetch.