[Major Blocker] need way to reliably run code on a page

[max-mapper]

Currently the real world usefulness of this is blocked by some Electron and Chromium limitations

• Browser-Window node integration breaks after POST electron/electron#1959 - pages loaded from a POST break node-integration which means you don't get e.g. require in the web page, meaning you can't use node modules on pages that load after a form submission. I was using require('ipc') to communicate between window and browser, but this breaks that. The upstream chromium issue is https://code.google.com/p/chromium/issues/detail?id=475027
• Disabling Content Security Policy when web-security is disabled electron/electron#2223 - no way to disable content security policy for wss. this means my backup strategy of using a local websocket fails on pages that restrict connections to non-whitelisted domains (see linked issue text for details)
• if a page has no script tag on it, electrons prebuilt script does not run (webview and preload do not work when there is no script tag in HTML electron/electron#1117)
• CSP cannot be disabled for pages that restrict eval (no way to bypass/modify CSP (Content Security Policy) to allow unsafe-eval electron/electron#3430)

This module is a web scraper, so we have the following requirements:

• have nodeIntegration disabled to minimize XSS security problems
• use some technique to reliably establish two way communication between the renderer process and the main process
  • preload scripts and IPC would be one way, but preload is not reliable (see above issues)
  • injecting a websocket and doing remote code execution using eval is another way, but eval is not reliable (see above issues)

So currently there is no way to achieve the above requirements (that I can see)

[max-mapper]

I may be able to figure out a hack such as storing data in localStorage and then parsing the on-disk local storage files that Electron writes out :) Or maybe using cookies https://github.com/atom/electron/blob/master/docs/api/browser-window.md#sessioncookiesgetdetails-callback

[max-mapper]

Update: The CSP and POST issues have been addressed. However, preload is broken for page without a script tag, so we currently cannot disable nodeIntegration, meaning we would have to keep nodeIntegration enabled which means the scraper would be more insecure

[max-mapper]

I guess a workaround would be to use webContents.executeJavaScript from the main process side, and use the websocket to send data back from the renderer side. If you combine these you could fake a duplex stream interface.

[etiktin]

Most preload issues should be solved once session wide preload is ready.

[max-mapper]

@etiktin it sounds like that will solve the preload issue with webview but I think it will have to also be solved separately with the no <script> tag issue as well

[etiktin]

It will make preload work in iframes and windows created by window.open, but yeah, it probably won't fix the no script tag issue.

[etiktin]

As a workaround for both remaining issues we could run a proxy server, intercept the responses and manipulate them (e.g. add a script tag).
This is the approach used by the Tamper project: https://github.com/dutzi/tamper

[xywz]

To make preload work for pages without <script>: electron/electron#1117 (comment).

[emersion]

"CSP cannot be disabled for pages that restrict eval" is now fixed

[max-mapper]

rewrote this to use web-view, v2.0.0 is out now. should fix this issue