Expose raw protocol in the API

[paulirish]

While using puppeteer there are still occasions where I want to use the raw protocol.
I'd certainly want to do this against the Page, and potentially the browser as well.

I want to send methods and get responses. Additionally, I want to listen for specific events.

Not sure of the right API and if it needs to be transactional as to not mixup state.

[aslushnikov]

So having just:

var connection = page.connection()

should satisfy the usecase.

[paulirish]

and expose all of what's in https://github.com/GoogleChrome/puppeteer/blob/master/lib/Connection.js ?

i was thinking maybe..

const connection = page.connection();

// send()
const ltree = await connection.send('LayerTree.getLayerTree');

// listen to events
await connection.send('LayerTree.enable');
connection.on('LayerTree.layerTreeChanged', data => { ... });

[paulirish]

Here's a good usecase of why we need to punch through to raw protocol: gr2m/headless-chrome-test#1 Dev wants to toggle offline mode but is currently using simple-headless-chrome

[aslushnikov]

@paulirish It seems to be a good idea to have offline mode in puppeteer to ease service worker testing, filed #63

[paambaati]

Another use case is using the 'Network.setBlockedURLs` API. I run a crawler with an adblock list that blocks all ads; it would be much simpler to use that instead of using the intercept API and building a regex check for each of my adblock patterns.

[ebidel]

Anyone working on this? Getting some requests that folks want to go off road. I'd love to recommend a paved path rather than a dirt road :)

[stilliard]

Currently accessing the raw protocol via page._client like so:

await page._client.send('Log.enable');
page._client.on('Log.entryAdded', console.log);

But the proposed page.connection() sounds like a better solution :)

[aslushnikov]

@ebidel we're working on it!

[ebidel]

Ping again :)

[karanjthakkar]

Can I help here?

[JulianKlug]

Would be good to have for speed testing as well.
Accessing the time every element was loaded would be very useful.

[trentmwillis]

Since I could also use this feature, I've opened a PR to implement this: #1445.

[Everettss]

How will it be solved a problem with enabling some services? Currently, there are some services enabled in source code: https://github.com/GoogleChrome/puppeteer/blob/16320b7ac2943aa04420c67e9df367e7801409ea/lib/Page.js#L47-L50

So currently using page._client.send() technique it's hard to know when to use page._client.send('XYZ.enable'). For example:

 // this will work without sending enable
await page._client.send('Network.clearBrowserCache');

// this one require sending enable
await page._client.send('ServiceWorker.enable'); 
await page._client.send('ServiceWorker.unregister', {
    scopeURL: 'http://localhost:8080/',
});

Is page.connection() will cover this problem? Or it will be in documentation when to send enable?

[GregRos]

@Everettss Does enabling something twice do anything bad? It could be just best practice to enable something in your application if you want to use it.

---

I kind of see the problem here. Basically, puppeteer is high-level enough that it needs to keep an internal state, and sending messages via the raw protocol can break that state.

On the other hand, in JavaScript things breaking is just part of life. Raw protocol access should be something that is outlined as dangerous so people will be wary of it and that it can break internal state, but people should still be able to use it because puppeteer will never be able to implement all the functionality of the raw protocol. And if they break something, well, that's what debugging is for.

(From my perspective, using underscored members is not so much dangerous as not a feature of the library so don't use it).

Otherwise, programmers will have to use another library to send raw protocol messages, which will make the likelihood of breaking even higher.