Runtime.CompileScript throws an error on a long input script

[yarikbratashchuk]

Hello again)

I'm trying to run a script with Runtime.RunScript.
Script itself is about 13000loc.

First i do Runtime.CompileScript and then i do Runtime.RunScript
But it always fails on Runtime.CompileScript, error message differs from time to time:

• cdp.Runtime: CompileScript: write tcp 127.0.0.1:65367->127.0.0.1:9222: use of closed network connection
• cdp.Runtime: CompileScript: rpcc: the connection is closing
• cdp.Runtime: CompileScript: websocket: close 1006 (abnormal closure): unexpected EOF

i tried to run Runtime.CompileScript and Runtime.RunScript with short script that returns promise, and it works fine.
Is there any restriction for the script size? Or maybe there is another way to run such long script?

this is how it looks in code

exp, err := ioutil.ReadFile("./content.js")
if err != nil {
	panic(err)
}
if err := c.Runtime.Enable(ctx); err != nil {
	panic(err)
}
compileReply, err := c.Runtime.CompileScript(context.Background(), &cdpcmd.RuntimeCompileScriptArgs{Expression: string(exp), PersistScript: true})
if err != nil {
        // it panics all the time for a 13000loc script
	panic(err)
}
awaitPromise := true
runReply, err := c.Runtime.RunScript(ctx, &cdpcmd.RuntimeRunScriptArgs{ScriptID: *compileReply.ScriptID, AwaitPromise: &awaitPromise})
if err != nil {
	panic(err)
}

Sincerely, yarik

[mafredri]

Hi Yarik,

this error originates from gorilla/websocket, and seems to be due to the message size being larger than the default WriteBufferSize (4096).

For a quick fix, you can try increasing the buffer size by modifying the websocket.Dialer in rpcc/conn.go#L86.

For example:

WriteBufferSize: 1024 * 1024,

I will look into fixing this properly, either by allowing you to change the write buffer size (e.g. rpcc.Dial(..., rpcc.WithWriteBufferSize(1024 * 1024)) or working around the gorilla/websocket limitation.

[yarikbratashchuk]

@mafredri thanks a lot)
that did work out for me, i increased read and write buffer size.

[mafredri]

Glad to hear 👍. I did a quick test with the golang.org/x/net/websocket package, it does not suffer from the same issue as gorilla/websocket. However, it's possible that this issue is also due to the way chrome handles websocket connections.

I guess there are two options:

1. Switch from gorilla/websocket to x/net/websocket
2. Allow user to configure {Read,Write}BufferSize for gorilla/websocket.Dialer.

Pros / cons:

• + With option 1 the user does not need to change any options
• + We do not need any features from gorilla/websocket that are not available in x/net/websocket
• - x/net/websocket is not maintained

[yarikbratashchuk]

@mafredri, i vote for option 2. Its good to have more configuration. But also good to have default values. My opinion, U decide.

[mafredri]

It looks like some others have run into this issue as well: gorilla/websocket#245.

Further testing also shows that Chrome will disconnect the websocket client if a request reaches 1MB in size (1048576 bytes). Counting JSON encoding, etc, the max size for a script would in this case be ~950kb, possibly less. Problem seems to be in Chrome since this is an issue for both x/net/websocket and gorilla/websocket.

Turning on compression (via EnableCompression) seems to help, allowing larger messages to be sent. This indicates some packet limit imposed by Chrome.

i vote for option 2. Its good to have more configuration. But also good to have default values. My opinion, U decide.

Noted. In my opinion less options is usually better, but not at the cost of usability. Especially since I don't think it's beneficial for the user to be able to change the buffer size in this project (it should just work). I will have to think about this some more, but I will try to get an intermediate fix out ASAP.

We could safely set WriteBufferSize: 1048562 since this is currently the absolute maximum that will work (in my tests). rpcc should also do some form of validation to avoid writing messages that are too large and return a useful error. This would impose a slight memory bump to each connection (1MB) but maybe that wouldn't be such a big deal.

[mafredri]

@yarikbratashchuk I've just pushed an update, 5f9a01f that allows you to change the write buffer size and enable compression.

Usage:

wsURL := "ws://localhost:9222/devtools/page/69e4f3c3-9d68-496f-8486-9583614fd32d"
rpcc.DialContext(ctx, wsURL, rpcc.WithWriteBufferSize(1048562), rpcc.WithCompression())

(Compression is optional, of course)

You could set the size to greater than 1048562 but in my tests this does not help, Chrome will disconnect anyway (if the actual message size bigger).

Note: in e7d1f0e I also improved the error reporting a bit, now when this error is encountered you could see something like:

panic: cdp.Runtime: CompileScript: write tcp 127.0.0.1:62300->127.0.0.1:9222: write: broken pipe
panic: cdp.Runtime: CompileScript: write tcp 127.0.0.1:62497->127.0.0.1:9222: use of closed network connection
panic: cdp.Runtime: CompileScript: write tcp 127.0.0.1:62532->127.0.0.1:9222: write: protocol wrong type for socket

These are still not very descriptive of the problem, but better than "connection is closing".

[yarikbratashchuk]

@mafredri thanks a lot) Using this)

[mafredri]

Ok, so I think I've traced the problem, documenting it here.

1. Chrome does not support websocket fragmentation (continuation frames), they remain unimplemented at the time of writing (net/server/web_socket_encoder.cc#86)

2. The underlying HTTP connection does not support bigger payloads than 1MB, I believe the reason is the kDefaultMaxBufferSize, defined here: net/server/http_connection.h#33

   When we send messages larger than 1MB, Chrome logs the following:

   [89253:55299:0616/194024.005586:ERROR:http_connection.cc(37)] Too large read data is pending: capacity=1048576, max_buffer_size=1048576, read=1048576
   

I will have to think about a proper solution/workaround for cdp, I don't want users to run into this problem, and if they do, I want them to be able to understand why.