[Feature] support route aka request interception for service workers #1090

Francois-Esquire · 2020-02-24T20:12:29Z

Hi,

I'd like to be able to include markup directly instead of using page.goto to a remote address. I currently have a small example here of what I've tried out but it appears that 'serviceWorker' in navigator is returning false and registering a service worker fails.

I've tried page.setContent as well and I had the same results.

Can anyone tell me what I'm missing? The example I am running:

const { chromium } = require('playwright');

const msg = 'hi';

const serviceWorkerPath = '/sw.js';
const serviceWorkerScope = '/';
const serviceWorker = `console.log("${msg}");`;

const html = `<!doctype html>
<html>
  <head>
    <title>One Service Worker</title>
  </head>

  <body>
    <script type="text/javascript">
      console.log('serviceWorker' in navigator);

      try {
        navigator.serviceWorker.register('${serviceWorkerPath}', {
          scope: '${serviceWorkerScope}'
        });
      } catch (e) {
        console.error(e);
      }
    </script>
  </body>
</html>
`.trim();

(async () => {
  const browser = await chromium.launch();
  const page = await browser.newPage();

  await page.route('**/*.html', async request =>
    request.fulfill({
      status: 200,
      contentType: 'text/html',
      body: html,
    }),
  );

  await page.route(serviceWorkerPath, request =>
    request.fulfill({
      status: 200,
      contentType: 'text/javascript',
      body: serviceWorker,
    }),
  );

  await page.goto('http://example.com/index.html', {
    waitUntil: 'load',
  });

  const sw = await page.evaluate(`window.navigator.serviceWorker`);
  
  console.log(sw);

  if (sw) {
    const swTarget = await browser.waitForTarget(target => target.type() === 'service_worker');
    const worker = await browser.serviceWorker(swTarget);
    // use worker target/handle
  }

  await browser.close();
})();

The text was updated successfully, but these errors were encountered:

Francois-Esquire · 2020-02-25T18:35:15Z

Any takers?

arjunattam · 2020-02-25T19:45:44Z

Hi @Francois-Esquire, service workers are only available for secure contexts, which is not the case for your example. Running console.log(await page.evaluate("window.isSecureContext")) returns false.

You could potentially serve the html over localhost with something like the http-server NPM package. Would love to learn why you want to use inline HTML instead.

Francois-Esquire · 2020-02-25T23:34:47Z

Hey @arjun27 , appreciate you taking time to respond!

Hmm, I originally had set up a demo server on localhost and that was working as expected. I didn’t consider the secure context, but makes complete sense now in how I was using it... thanks for pointing that out. Maybe I could add:

--unsafely-treat-insecure-origin-as-secure=http://example.com

To the arguments, or change the destination url to localhost?

The goal essentially is to create a function I can reuse to execute code inside the service worker; on demand and with a portable solution not requiring an http server in place. I just need to register a service worker, thus a simple html page.

Is there a better way of achieving this?

pavelfeldman · 2020-02-27T23:22:44Z

page.route has no effect on the service workers since those live outside of page, inside the browser context. We will eventually get there. Two things are missing:

move page.route to be browserContext.route
attach to service workers and support interception there

Francois-Esquire · 2020-03-02T17:38:52Z

@pavelfeldman Thanks for addressing that, and really excited to hear the plan with that! Love the API design with that, very intuitive.

What I'm trying to do is register a service worker without a supporting web server to run playwright with - thanks to @arjun27 bringing up the point on secure context, I was able to have the navigator.serviceWorker in the browser context using inline html document, however the service worker registration hangs until it times out with this setup. I should add, I am intercepting the path to the service worker and fetching it returns the correct script content.

What could be the cause to this? Am I missing anything else? Again, this works with a conventional web server backing the playwright usage.

Aaron-Pool · 2021-01-15T17:56:25Z

Coming from #5014 . I added my thumbs up to the issue, but I also think it's worth explicitly pointing out that the inability to use .route with service worker requests makes playwright unable to listen to requests that have been intercepted and handled by mswjs. MSW is a pretty big newcomer to the network mocking scene, thanks in large part to the articles and advocacy of Kent C. Dodds' (the guy who wrote testing library and is pretty much the premier voice in the frontend testing world). I think compatibility between playwright and msw should be a pretty important consideration in playwright development.

mszmida · 2021-03-26T11:59:01Z

Hello 😃

First of all I would like to thank the Playwright team for their effort. Great project awaited by me for a long time! Good job 👍

I agree with @Aaron-Pool. The MSW library is a game changer in terms of the mocking of the back-end services and I think that not listening to the service workers requests is a significant drawback. I found this issue when I was looking for myself to integrate the MSW library with the Playwright with no luck. At least now I know the reason behind it.

Another reason why this functionality should be prioritized and implemented is the support for GraphQL. It is used in the company that I work for and currently in the Playwright there is no easy possibility to differentiate the GraphQL requests. The MSW library solves this problem but currently Playwright do not like the service workers. I can easily imagine that a lot of people will have the same problem so this is not some kind of an edge case.

I would very much like this functionality to be implemented in a reasonable time.

Once again thank you for the Playwright 👍

RFC2109 · 2021-04-14T12:23:35Z

Vote up for this feature.
It is very usefull because many websites has service worker.
Disable sw is not a good idea,because it chang the tested system。

Meir017 · 2021-08-08T11:12:07Z

Is it possible to implement this behavior currently using the CDPSession class directly?

Meir017 · 2021-10-22T13:09:03Z

I think a current workaround could me to launch the browser with the "--host-rules='MAP * 127.0.0.1'" argument, and run a local HTTP server side-by-side, and implement your interception logic inside of this local HTTP server

This is the beginning of fixing microsoft#1090. There's still quite a bit to be done, but this at least shows an end-to-end path to accomplish the feature.

Resolves #14666. Relates #1090. Supercedes #14321.

Adds cross-browser support for easily allowing/blocking Service Workers via a Context option. Includes plumbing for Playwright Test's `use`. Resolves #14522. Relates #1090. Supercedes #14321.

kettanaito · 2022-06-16T11:13:13Z

Hey. MSW author here. I'd very much love to see Playwright and MSW work together, which includes supporting more use cases. We've been testing MSW with Playwright for years and didn't have any issues. I think one of the things that confuses people the most is the execution context difference between your tests and the browser runtime in Playwright.

That being said, I'd love to migrate to @playwright/test in our repo and maybe have a designated smoke test for Playwright support. We have a few such tests now but they are largely outdated as there's not enough hands to keep everything up-to-date.

Let me know if I can be of any help when it comes to Service Worker nuances.

Aaron-Pool · 2022-06-16T13:54:41Z

Hey. MSW author here. I'd very much love to see Playwright and MSW work together, which includes supporting more use cases. We've been testing MSW with Playwright for years and didn't have any issues. I think one of the things that confuses people the most is the execution context difference between your tests and the browser runtime in Playwright.

That being said, I'd love to migrate to @playwright/test in our repo and maybe have a designated smoke test for Playwright support. We have a few such tests now but they are largely outdated as there's not enough hands to keep everything up-to-date.

Let me know if I can be of any help when it comes to Service Worker nuances.

Playwright + MSW ❤️ 🔥 ❤️ 🔥

These are two of my favorite testing tools and I'd love to see them work together seamlessly.

rwoll · 2022-06-17T18:07:34Z

Thanks @kettanaito! Happy to hop on a chat and learn more about MSW from your perspective. You can find me on in the Playwright Slack.

At this point, this issue is tracking work in Playwright to ensure Network Requests occurring in SW's show up in Playwright's mocking/routing via {context,page}.route.

kettanaito · 2022-06-17T20:49:31Z

Would love to, @rwoll! Alas, that Slack link is no longer active. Could you per chance generate a new one?

I've just had an idea yesterday that may allow Playwright to know when requests are happening in the worker. Would love to tell you more about it and discuss whether it's something Playwright would want to support out of the box or as an extension.

rwoll · 2022-06-17T21:50:30Z

Would love to, @rwoll! Alas, that Slack link is no longer active. Could you per chance generate a new one?

I've just had an idea yesterday that may allow Playwright to know when requests are happening in the worker. Would love to tell you more about it and discuss whether it's something Playwright would want to support out of the box or as an extension.

Oops—sorry about that! It should be updated now! 😄

kettanaito · 2022-06-20T10:50:14Z

Okay, I'm sharing my proposal below.

Context

A brief context to bring everybody onboard.

Service Workers execute in their own thread, and have their own scope. Because of this, Playwright doesn’t know when requests are issued by the worker because there’s no standard means to know that. Although the worker’s requests appear in the Network, they still execute in their own (worker) scope.

Proposal

I propose for Playwright to register a proxy worker that would look somewhat like this:

// sw-proxy.js
self.addEventListener('install', () => {
  sendToClient({ type: 'event', name: 'install' })
})

self.addEventListener('fetch', (event) => {
  sendToClient({
    type: 'request',
    request: serializeRequest(request)
  })
})

The goal is to signal about the worker events back to the client (Playwright). You can use both MessageChannel and BroadcastChannel to do that, we can discuss which one is more suited for this particular case later.

There’s one catch though: there can only be 1 worker controlling the page. So, obviously, introducing sw-proxy.js means that any worker registered as a part of user’s tests will be ignored. To account for this, the proxy should wrap any existing worker in itself.

One of the ways to achieve this wrapping is by using the importScripts:

// sw-proxy.
// ...the rest of the proxy worker.

// Import the actual tested worker as the last thing.
// The order of imports affects the order of event
// propagation, so the proxy must always add its
// listeners first.
importScripts('/actual-user-worker.js')

When the workers are combined, the same events will bubble through all of them based on the listener declaration order. If the proxy worker adds its listeners first, it will receive events first. On the premise that the proxy must treat events as readonly (never affect their outcome, for example, in the fetch event), we can find out when events happen, notify Playwright, and allow them to propagate to the underlying user worker to actually be resolved.

This order of execution may also allow us to influence event resolution in some cases. For example, if a request event matching a defined .route() pattern is received, Playwright may send back the mocked response, which the worker will use to respond to the request event:

// sw-proxy.js
self.addEventListener('fetch', (event) => {
  signalToClient(
    { type: 'request', ... },
    (data) => {
      const mockedResponse = JSON.parse(data)
      event.respondWith(mockedResponse)
    }
  )
})

Keep in mind this is a pseudo-code, since delegating event response to a callback creates a race condition between the proxy request listener and the user worker request listener. You really want to use promises and await things properly but see the difficulties this brings below.

In a nutshell, this is how MSW works.

There are 2 technical difficulties with this approach:

The “fetch” event listener must be synchronous. Any async logic must be deleted to the event.respondWith() that accepts a Promise as an argument.
Coming from the first point, if you call event.respondWith(), then you are handling the request no matter the response outcome. This will halt the event propagation to the user worker even when there’s no “route” defined for an intercepted request.

Request spying is the most tricky part of this entire proposal.

Alternative route

I’m not aware how much Playwright is in control over the underlying Chromium but if it could patch the Service Worker’s global scope, then we could spy on the listeners without having to manage worker nesting and event propagation. Affecting the worker’s scope is rather dangerous so this must be approached with caution.

Closing thoughts

In the end, the challenge is to know when certain worker events happen. And while tools like MSW already implement this client-worker channel that allows to intercept requests, they do not care for other worker events (like life-cycle events), which may be of interest to people who would want to test their workers extensively. At the same time, MSW operates with a single worker, and whenever people have their own worker we just recommend to use importScript manually in the mockServiceWorker.js. This isn’t a good strategy to adopt in Playwright, neither it’s a working strategy for it.

kettanaito · 2022-06-20T10:58:51Z

As a general route, given the scope difference between the client and the worker, it may be a good idea to consider a different API to control worker events/requests other than .route().

page.serviceWorker.route('/user')

There may be cases when the same resource is fetched on the client and on the worker, and you may want to control those cases differently.

rwoll · 2022-06-20T18:40:01Z

@kettanaito Thanks for the proposal! I think it's important to separate the discussion into at least two components:

Network Mocking
Testing/Interacting with Service Workers themselves

Each can be solved different ways, and patching the browsers themselves is in scope (as needed). As part of #14716 —which allows you to intercept all requests (including the main SW script as well as requests being made inside the SW) via Playwright's native {context,page}.route(…) interception/mocking, there's ~6 patches to CR upstream so far that builds on existing upstream work.¹

Proxying the SW is a neat idea worth exploring—but hitting limitations and interfering with user code as you've alluded to worries me. In general, we prefer native instrumentation, because it does not interfere with user code and brings us more capabilities.

If folks are looking for more ways to test Service Workers themselves, that has yet to be explored as I've mainly focused on their networking components, but PW APIs that give you visibility and hooks into the SW lifecycle/versions etc. will likely show up at some point, but let's have that discussion in a distinct issue to keep this focused on solving the issue of mocking network requests.

As a work around today—until the patch lands—you can block SWs so all the requests show up in the native PW {context,page}.route. ↩

kettanaito · 2022-06-21T08:09:47Z

In general, we prefer native instrumentation, because it does not interfere with user code and brings us more capabilities.

I totally understand and share this direction.

but PW APIs that give you visibility and hooks into the SW lifecycle/versions etc. will likely show up at some point

Really looking forward to this! I know it'd help me test some of my worker logic.

…#14716) Adds Chromium support for Service Worker Networking (interception/routing, Request/Response events, and HAR). Resolves #1090. Depends on #14714 and #14714. Supercedes #14321. Follow up #14711. Landed upstream patches: - https://chromium-review.googlesource.com/c/chromium/src/+/3510917 - https://chromium-review.googlesource.com/c/chromium/src/+/3526571 - https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/3566669 - https://chromium-review.googlesource.com/c/chromium/src/+/3544685 - https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/3610924 - https://chromium-review.googlesource.com/c/chromium/src/+/3689949

rwoll · 2022-07-02T02:48:47Z

#14716 landed which means—on Chromium-only for now—network requests made within Service Workers to external resources (and the initial Service Worker main script request), are now instrumented and visible via context.route(…) and context.on('request', …).

More exhaustive docs are still in the works, but for folks who want to try it out, you can try ToT npm i @playwright/test@next (or npm i playwright@next if you are using PW as a lib) and skim through the test(s) for examples of usage:

playwright/tests/library/chromium/chromium.spec.ts

Line 214 in 71dcad3

    
           test('should intercept only serviceworker request, not page', async ({ context, page, server }) => {

playwright/tests/library/chromium/chromium.spec.ts

Line 42 in 71dcad3

    
           test('serviceWorker(), and fromServiceWorker() work', async ({ context, page, server }) => {

playwright/tests/library/chromium/chromium.spec.ts

Line 83 in 71dcad3

    
           test('should intercept service worker requests (main and within)', async ({ context, page, server }) => {

…and more in that file :)

If you have questions—or bugs 🐛 —feel free to file a new issue under the question type and refer to this issue #1090. Please include your use case when filing so we can understand it in context.

rwoll · 2022-07-22T19:59:15Z

Playwright v1.24.0 was released this week. As part of the new release, we've updated the Playwright Network guide: https://playwright.dev/docs/network#missing-network-events-and-service-workers pertaining to this issue.

Please check it out and let us know if you have any questions or if your use case is not addressed by commenting on #15684, or filing a new issue with a description of your use case.

Thanks!

For the initial Service Worker script, `Network.requestWillBeBeSent` is observed, but the corresponding `Fetch.requestPaused` event when performing interception was missing a `networkId` param. This prevented a CDP consumer from definitively associating the two events. Requests from within the Service Worker appear to have proper `networkId` attribution, so it appears it was just for the script itself. Discovered while working on Playwright: microsoft/playwright#1090 R=caseq Bug: 1304381 Change-Id: I55446db8b305ae81767c100efa27aced22152da0 Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/3510917 Reviewed-by: Andrey Kosyakov <caseq@chromium.org> Reviewed-by: Dmitry Gozman <dgozman@chromium.org> Commit-Queue: Dmitry Gozman <dgozman@chromium.org> Cr-Commit-Position: refs/heads/main@{#979924} NOKEYCHECK=True GitOrigin-RevId: 5e489de274b6f1676d81669aac86c5145c561826

Service Worker Main Request CDP Network events had some IDs generated in the Browser process, and others later generated in the Renderer, so they did not align. Discovered while working on microsoft/playwright#1090. This relates to previous patches: * https://chromium-review.googlesource.com/c/chromium/src/+/3526571 * https://chromium-review.googlesource.com/c/chromium/src/+/3510917 Change-Id: I544e7b7c6d780fa0b11a6a16a39b360e675fe7d8 Bug: 1304795 Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/3544685 Reviewed-by: Dmitry Gozman <dgozman@chromium.org> Reviewed-by: Andrey Kosyakov <caseq@chromium.org> Reviewed-by: Asami Doi <asamidoi@chromium.org> Commit-Queue: Andrey Kosyakov <caseq@chromium.org> Cr-Commit-Position: refs/heads/main@{#996961} NOKEYCHECK=True GitOrigin-RevId: ab1268c40421a390b722774f40f103cf4b2b04cc

Francois-Esquire changed the title ~~[Question] inline html document instead of navigating to a server address, while using service worker~~ [Question] inline html document rather than navigate to a remote server address, while using service worker Feb 24, 2020

pavelfeldman changed the title ~~[Question] inline html document rather than navigate to a remote server address, while using service worker~~ [Feature] support route aka request interception for service workers Feb 27, 2020

evandigby mentioned this issue Aug 6, 2020

[Feature] Apply context routes prior to context creation #3326

Closed

dgozman added the P3-collecting-feedback label Nov 24, 2020

yury-s mentioned this issue Jan 15, 2021

[Question] How can I use page.route on a network request initiated by a service worker? #5014

Closed

mxschmitt mentioned this issue Apr 15, 2021

[BUG] webkit route.fulfill dont work after a while at some website (not every website )，while chromium is ok #6045

Closed

aslushnikov mentioned this issue Jul 14, 2021

[Question]: Is route interception on a page supposed to work in every case (Chromium)? #7588

Closed

pavelfeldman added v1.18 and removed P3-collecting-feedback labels Dec 15, 2021

pavelfeldman self-assigned this Dec 15, 2021

pavelfeldman added v1.19 and removed v1.18 labels Jan 3, 2022

pavelfeldman added v1.20 and removed v1.19 labels Feb 1, 2022

Meir017 mentioned this issue Feb 3, 2022

Requests are not intercepted when cached by Service Worker puppeteer/puppeteer#4041

Open

pavelfeldman removed their assignment Feb 18, 2022

aslushnikov added v1.21 and removed v1.20 labels Mar 1, 2022

aslushnikov assigned rwoll Mar 1, 2022

rwoll added a commit to rwoll/playwright that referenced this issue Mar 3, 2022

wip: feat: service worker interception

7ef215f

This is the beginning of fixing microsoft#1090. There's still quite a bit to be done, but this at least shows an end-to-end path to accomplish the feature.

rwoll removed the v1.24 label Jun 6, 2022

This was referenced Jun 8, 2022

feat: add allow/block serviceWorkers option #14714

Merged

feat: add Response.fromServiceWorker flag #14715

Merged

feat: Service Worker Network Instrumentation and Inspection #14716

Merged

pavelfeldman mentioned this issue Jun 8, 2022

[Question] How can I trigger a service worker update by modifying the HTTP response? #14711

Closed

rwoll added a commit that referenced this issue Jun 8, 2022

feat: add Response.fromServiceWorker flag (#14715)

fccee89

Resolves #14666. Relates #1090. Supercedes #14321.

dgozman added v1.24 and removed v1.23 labels Jun 13, 2022

rwoll closed this as completed in #14716 Jul 1, 2022

rwoll reopened this Jul 14, 2022

aslushnikov added v1.25 and removed v1.24 labels Jul 21, 2022

rwoll closed this as completed Jul 22, 2022

brianjenkins94 mentioned this issue Mar 20, 2023

[BUG] Traffic from iframes, nested iframes and service workers not intercepted via CDPSession / Fetch.getResponseBody #21816

Closed

1 task

weblancaster mentioned this issue Jun 21, 2023

[Feature] Mocking API requests for MSW #23277

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature] support route aka request interception for service workers #1090

[Feature] support route aka request interception for service workers #1090

Francois-Esquire commented Feb 24, 2020 •

edited

Loading

Francois-Esquire commented Feb 25, 2020

arjunattam commented Feb 25, 2020

Francois-Esquire commented Feb 25, 2020 •

edited

Loading

pavelfeldman commented Feb 27, 2020

Francois-Esquire commented Mar 2, 2020 •

edited

Loading

Aaron-Pool commented Jan 15, 2021 •

edited

Loading

mszmida commented Mar 26, 2021 •

edited

Loading

RFC2109 commented Apr 14, 2021 •

edited

Loading

Meir017 commented Aug 8, 2021

Meir017 commented Oct 22, 2021

kettanaito commented Jun 16, 2022 •

edited

Loading

Aaron-Pool commented Jun 16, 2022

rwoll commented Jun 17, 2022

kettanaito commented Jun 17, 2022

rwoll commented Jun 17, 2022

kettanaito commented Jun 20, 2022 •

edited

Loading

kettanaito commented Jun 20, 2022

rwoll commented Jun 20, 2022

kettanaito commented Jun 21, 2022

rwoll commented Jul 2, 2022

rwoll commented Jul 22, 2022

[Feature] support route aka request interception for service workers #1090

[Feature] support route aka request interception for service workers #1090

Comments

Francois-Esquire commented Feb 24, 2020 • edited Loading

Francois-Esquire commented Feb 25, 2020

arjunattam commented Feb 25, 2020

Francois-Esquire commented Feb 25, 2020 • edited Loading

pavelfeldman commented Feb 27, 2020

Francois-Esquire commented Mar 2, 2020 • edited Loading

Aaron-Pool commented Jan 15, 2021 • edited Loading

mszmida commented Mar 26, 2021 • edited Loading

RFC2109 commented Apr 14, 2021 • edited Loading

Meir017 commented Aug 8, 2021

Meir017 commented Oct 22, 2021

kettanaito commented Jun 16, 2022 • edited Loading

Aaron-Pool commented Jun 16, 2022

rwoll commented Jun 17, 2022

kettanaito commented Jun 17, 2022

rwoll commented Jun 17, 2022

kettanaito commented Jun 20, 2022 • edited Loading

Context

Proposal

Alternative route

Closing thoughts

kettanaito commented Jun 20, 2022

rwoll commented Jun 20, 2022

Footnotes

kettanaito commented Jun 21, 2022

rwoll commented Jul 2, 2022

rwoll commented Jul 22, 2022

Francois-Esquire commented Feb 24, 2020 •

edited

Loading

Francois-Esquire commented Feb 25, 2020 •

edited

Loading

Francois-Esquire commented Mar 2, 2020 •

edited

Loading

Aaron-Pool commented Jan 15, 2021 •

edited

Loading

mszmida commented Mar 26, 2021 •

edited

Loading

RFC2109 commented Apr 14, 2021 •

edited

Loading

kettanaito commented Jun 16, 2022 •

edited

Loading

kettanaito commented Jun 20, 2022 •

edited

Loading