Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for testing Electron.js applications #4964

Open
bahmutov opened this issue Aug 9, 2019 · 62 comments

Comments

@bahmutov
Copy link
Collaborator

@bahmutov bahmutov commented Aug 9, 2019

Currently Cypress can open the browser and load websites for testing. Electron.js applications ARE in essence the browser, so it would be cool to let Cypress test them.

Original issue: #2072

  • create basic Electron.js example to be used as a proof of concept
  • figure out how to change the browser launch from Cypress in order to "attach" to an Electron application
    • external process, similar to Chrome, passing remote interface url to be able to control the browser process
  • see if we can implement testing an Electron.js application with minimal core changes, and most of the code as external plugin (faster iteration)
  • figure out the changes / API that a typical Electron.js application would need to be testable from Cypress
@bahmutov bahmutov self-assigned this Aug 9, 2019
@bahmutov

This comment has been minimized.

Copy link
Collaborator Author

@bahmutov bahmutov commented Aug 12, 2019

Hmm, just trying to load our extensions into Electrons browser window

if (args['--load-extension']) {
    const extensions = args['--load-extension'].split(',')
    extensions.forEach((ext) => {
      console.log('loading extension', ext)
      const name = BrowserWindow.addExtension(ext)
      console.log('extension has returned name: %s', name)
    })
    console.log('loaded extensions\n%s', extensions.join('\n'))
  }
  // Create the browser window.
  mainWindow = new BrowserWindow({
    width: 800,
    height: 600,
    webPreferences: {
      // ? should we just preload Cypress scripts if passed
      preload: path.join(__dirname, 'preload.js'),
      nodeIntegration: false,
      nativeWindowOpen: true,
      webSecurity: false,
      devTools: true,
    }
  })

At first, seems ok

loading extension /Users/gleb/Library/Application Support/Cypress/cy/development/browsers/cypress-example-electron/interactive/CypressExtension
extension has returned name: Cypress
loading extension /Users/gleb/git/cypress/packages/extension/theme
extension has returned name: Cypress Theme
loaded extensions
/Users/gleb/Library/Application Support/Cypress/cy/development/browsers/cypress-example-electron/interactive/CypressExtension
/Users/gleb/git/cypress/packages/extension/theme

but then seems to give errors:

[222:0812/112909.516074:ERROR:CONSOLE(7946)] "Skipping extension with invalid URL: chrome-extension://cypress", source: devtools://devtools/bundled/shell.js (7946)
[222:0812/112909.516118:ERROR:CONSOLE(7946)] "Skipping extension with invalid URL: chrome-extension://cypress-theme", source: devtools://devtools/bundled/shell.js (7946)
GET /__cypress/runner/cypress_runner.js 200 2.142 ms - -
@brian-mann

This comment has been minimized.

Copy link
Member

@brian-mann brian-mann commented Aug 12, 2019

You can use the --remote-debugging-port flag documented here: https://electronjs.org/docs/all#supported-chrome-command-line-switches

Put it in environment.coffee and you're good to go. You'll connect to the browser window instance the same way / same logic as we do for chrome RDP.

@bahmutov

This comment has been minimized.

Copy link
Collaborator Author

@bahmutov bahmutov commented Aug 12, 2019

Ok, did a test with Electron - it seems the --remote-debugging-port option is passed directly to the main window, which allows one to connect directly via debugging protocol. The debugging commands are shown below

{
  "scripts": {
    "start": "electron .",
    "debug-main-process": "electron --inspect=5858 .",
    "debug-window-process": "electron --remote-debugging-port=9222 ."
  }
}

The output from both (one for Node debugging, another for the main window browser debugging)

$ npm run debug-main-process

> electron-quick-start@1.0.0 debug-main-process /Users/gleb/git/cypress-example-electron
> electron --inspect=5858 .

Debugger listening on ws://127.0.0.1:5858/8ab0188e-c271-4932-b2f0-f4bb4484dded
For help, see: https://nodejs.org/en/docs/inspector
in /Users/gleb/git/cypress-example-electron/main.js
~/git/cypress-example-electron on master*
$ npm run debug-window-process

> electron-quick-start@1.0.0 debug-window-process /Users/gleb/git/cypress-example-electron
> electron --remote-debugging-port=9222 .

in /Users/gleb/git/cypress-example-electron/main.js

DevTools listening on ws://127.0.0.1:9222/devtools/browser/b2d8ee31-3ad2-4daf-ba29-b624a3b3033a

So Cypress maybe will be able to control external Electron application's main browser window via the remote interface. Luckily we are going this way in this PR #4628

@bahmutov

This comment has been minimized.

Copy link
Collaborator Author

@bahmutov bahmutov commented Aug 12, 2019

Nice, using code from add-cri-4608 I am able to grab the remote interface and control the main window

Screen Shot 2019-08-12 at 1 00 00 PM

The switcher for the browser / app

Screen Shot 2019-08-12 at 1 02 11 PM

@bahmutov

This comment has been minimized.

Copy link
Collaborator Author

@bahmutov bahmutov commented Aug 12, 2019

Added controlling the external Electronjs main window through chrome remote interface, allowing the main window to show and run tests

Screen Shot 2019-08-12 at 3 51 01 PM

Next question - what would the test actually do? For the example application, here is its "normal" behavior

Screen Shot 2019-08-12 at 3 58 34 PM

I would like to have the test be:

cy.visit() // without anything, should load main.js
cy.get('#node-version').should('equal', '12.4.0')
@bahmutov

This comment has been minimized.

Copy link
Collaborator Author

@bahmutov bahmutov commented Aug 12, 2019

Hacking around to get two windows going - one with specs, another the real Elecron app window.

Screen Shot 2019-08-12 at 4 52 14 PM

And now need to figure out how to use that second application window as the test window (instead of the app iframe)

@bahmutov

This comment has been minimized.

Copy link
Collaborator Author

@bahmutov bahmutov commented Aug 13, 2019

In my example electron app, the window under test is loading the index.html directly, which I think might be a common cast. It has document.domain = '' and so the test window has to get the same domain. I have managed to load the plain file in the test window and from the console am able to access the main window to get the element (even if the element is incorrectly shown!)

Screen Shot 2019-08-13 at 9 56 29 AM

If only I could get the script running in the test window, then all would be perfect

@bahmutov

This comment has been minimized.

Copy link
Collaborator Author

@bahmutov bahmutov commented Aug 13, 2019

Instead of loading a static file from the main window, loading localhost website with document.domain=localhost set either in the site itself or in the preload.js script. The first test works

Screen Shot 2019-08-13 at 2 19 18 PM

In the above image, the main window opens test window. The test window has opener pointing at the main window. To get all our functions to work, the spec does the following trick

beforeEach(() => {
  if (typeof top !== 'undefined' && top.opener) {
    cy.state('document', top.opener.document)
  }
})

Everything is good - but how do we "clean up" the main window? For example if we reload the main window, the remaining test window is NO LONGER ITS CHILD WINDOW. For example, if I reload the main window, then rerun the tests - they no longer can access opener.document

Screen Shot 2019-08-13 at 2 22 14 PM

So maybe the relationship should go the other way: open test window, and let it open the main window to load the Electron application.

@bahmutov

This comment has been minimized.

Copy link
Collaborator Author

@bahmutov bahmutov commented Aug 13, 2019

First we are creating test window that we control - via RDP
Before each test it opens child window that loads (via simple http server to have localhost) the main application 'index.html' url - which has node integration enabled, showing process properties in this case.

Screen Shot 2019-08-13 at 3 06 34 PM

@bahmutov

This comment has been minimized.

Copy link
Collaborator Author

@bahmutov bahmutov commented Aug 13, 2019

To better understand what potentially might need testing in Electron applications I have randomly picked first apps in https://electronjs.org/apps list The main questions I wanted to see where:

  • how was the main window initialized?
  • where there multiple windows? (additional windows like config, preferences, about)
  • was the main window loaded using loadUrl(...) or via loadFile(<local filename>)?
  1. https://github.com/yafp/ttth Chat client
    // Create the browser window.
    mainWindow = new BrowserWindow({
        title: "${productName}",
        frame: true, // false results in a borderless window
        show: false, // hide until: ready-to-show
        width: windowWidth,
        height: windowHeight,
        minWidth: 800,
        minHeight: 600,
        backgroundColor: "#ffffff",
        icon: path.join(__dirname, "app/img/icon/icon.png"),
        webPreferences: {
            nodeIntegration: true,
            webviewTag: true, // # see #37
        }
    });

There was config window, initialized similarly.
Both windows were loaded from file mainWindow.loadFile("app/mainWindow.html")

  1. https://github.com/sagargurtu/lector PDF viewer
    win = new BrowserWindow({
        width: 800,
        height: 600,
        minWidth: 300,
        minHeight: 300,
        icon: './src/assets/images/logo.png',
        webPreferences: {
            plugins: true,
            nodeIntegration: true
        },
        frame: false
    });

    // and load the index.html of the app.
    win.loadFile('./src/index.html');
  1. https://github.com/CodeF0x/violin music player
function createWindow() {
  window = new BrowserWindow({
    width: 800,
    minWidth: 800,
    height: 600,
    minHeight: 600,
    titleBarStyle: 'hiddenInset',
    useContentSize: false,
    webPreferences: {
      nodeIntegration: true
    }
  });

  window.setResizable(true);
  window.loadFile('src/index.html');
}
  1. https://github.com/dot-browser/desktop browser
    Subclass of BrowserWindow
export class AppWindow extends BrowserWindow {
  public viewManager: ViewManager = new ViewManager();
  public permissionWindow: PermissionDialog = new PermissionDialog(this);
  public menu: MenuList = new MenuList(this);

  constructor() {
    super({
      frame: false,
      minWidth: 500,
      minHeight: 450,
      width: 1280,
      height: 720,
      show: false,
      backgroundColor: '#1c1c1c',
      title: 'Dot Browser',
      titleBarStyle: 'hiddenInset',
      maximizable: false,
      webPreferences: {
        plugins: true,
        nodeIntegration: true,
        contextIsolation: false,
        experimentalFeatures: true,
        enableBlinkFeatures: 'OverlayScrollbars',
        webviewTag: true,
      },
      icon: resolve(app.getAppPath(), '/icon.png'),
    });
  }
...

Interestingly loads file:// url in production

if (process.env.ENV === 'dev') {
      this.setIcon(nativeImage.createFromPath(resolve(app.getAppPath() + '\\static\\icon.png')))
      this.webContents.openDevTools({ mode: 'detach' });
      this.loadURL('http://localhost:4444/app.html');
    } else {
      this.loadURL(join('file://', app.getAppPath(), 'build/app.html'));
    }
  1. https://github.com/digimezzo/knowte-electron Note-taking application
    mainWindow = new BrowserWindow({
      'x': mainWindowState.x,
      'y': mainWindowState.y,
      'width': mainWindowState.width,
      'height': mainWindowState.height,
      backgroundColor: '#fff',
      frame: false,
      icon: path.join(__dirname, 'build/icon/icon.png'),
      show: false
    });

Again, uses loadUrl to load file:// url

  mainWindow.loadURL(url.format({
        pathname: path.join(__dirname, 'dist/index.html'),
        protocol: 'file:',
        slashes: true
      }));
@bahmutov

This comment has been minimized.

Copy link
Collaborator Author

@bahmutov bahmutov commented Aug 13, 2019

Trying to load main application window via file:// url does not give our localhost test window access to it

Screen Shot 2019-08-13 at 5 30 26 PM

@brian-mann

This comment has been minimized.

Copy link
Member

@brian-mann brian-mann commented Aug 14, 2019

We need to experiment to see if properly enabling the proxy on the browser window / electron routes requests to the file protocol through the cypress proxy. If it does not then we'll need to go down the plugin route and require users pass us the browser window instance to our plugin like so...

const cypressElectron = require('@cypress/electron-plugin')

const win = new BrowserWindow(...)

cypressElectron(win)

Once we receive the browser window, we could override the loadFile function to properly re-route through the proxy, forcing us to receive it.

Alternatively we could monkey patch the electron API's themselves such as BrowserWindow.prototype.loadURL and re-route that through us.

I'm hoping we can just go down the regular network proxy as long as chromium routes requests to the file:// protocol through us.

@bahmutov

This comment has been minimized.

Copy link
Collaborator Author

@bahmutov bahmutov commented Aug 14, 2019

I am not sure what the re-routing loadFile would do. When we re-route HTML load via our proxy it is to insert document.domain = 'localhost' there. When we re-reroute file load and even if the index.html file has the domain set

<script>
  document.domain = 'localhost'
</script>

it does not work - here is a screenshot

Screen Shot 2019-08-14 at 10 18 24 AM

I think we might be able to run OUR window / iframe from domain '' by loading it from a file too

@bahmutov

This comment has been minimized.

Copy link
Collaborator Author

@bahmutov bahmutov commented Aug 14, 2019

Question: can we proxy external domain and load it correctly (by inserting document.domain = 'localhost' into the returned HTML). For example, if the Electron application does

open('http://todomvc.com/examples/vue/', 'mainwindow', replace)

Trying to load, getting cookie error - because we now have 2 windows - the test window and the child Electron window with the app. Our automation works against the test window (where the command log is), but we also need a second connection to drive the application window and set the cookies, everything correctly there.

Hmm, do not see the domain loaded by the open('https://todomvc.com') going through our proxy for some reason.

@bahmutov

This comment has been minimized.

Copy link
Collaborator Author

@bahmutov bahmutov commented Aug 14, 2019

Progress meeting Wed 08-14-2019

Showed demo testing localhost page: loading, accessing Node integration, clicking and typing. The main window loads our test runner, and opens a web window for the main application to load like this

const mw = (window.mw = open(
  'http://localhost:4600',
  'mainWindow',
  replace
))

The test runs as shown in this animation. Note that direct access to the child window allows us to see DOM snapshots, etc

electron-test

Next steps

We are really interested in writing down little "seams" that Cypress test runner should expose for the Electron testing plugin code to use to change the test runner behavior.

  1. how to display + detect we should add electron as a selectable browser?
    • right now it's hard-coded to concat browsers. We probably want to expose in plugins file (future cypress.js file that replaces plugins.js and cypress.json file)
  2. @flotwig to look at how file:// protocol is routed through the proxy?
    • look into custom protocol handler? Or interception at the file system layer or loading file via local HTTP server, so it becomes 'localhost'
  3. expose a way to tap into how cy.visit() is resolved
    • enables you to change from using $autIframe.contentWindow.location.href to using window.open(…). Current trick I do in the spec file is
const mw = (window.mw = open(
  'http://localhost:4600',
  'mainWindow',
  replace
))
// TODO resolve when mw.document is valid
// for now simply wait
setTimeout(() => {
  cy.state('document', mw.document)
  cy.state('window', mw)
  resolve()
}, 100)
  1. control the app's browser window scaling the same way as the $autIframe
    • possibly not even allow the browserwindow to be detached
    • making it work the same as it currently does
    • possibly replace the <iframe> with the <webview>
      • Electron's BrowserView class is the replacement for <webview>
    • maybe allow the user to control whether its detached or not
    • embedding allows us to capture the video in a single place
      Approximately this will look like this:

Screen Shot 2019-08-14 at 12 14 30 PM

  1. normalize to use the RDP instead of launch args for settings things like the network proxy
    • this is immediate work that can be done now
    • this can work the same for both chrome, for electron vanilla browser, or custom 3rd party native electron apps
  2. look at Spectron
    • what APIs do they allow or add to Electron app testing?
    • right now we are treating the loaded site in the test window / view as a normal website. But what about testing Node integration, accessing Electron APIs etc from the test?

Next goal

  • do not worry too much about a proxy for 3rd party sites, aside from localhost.
  • set the proxy on the Electron browser process using RDP command and point back at Cypress process
  • switch the child window to be a subview of the main window.
    • work on cy.visit to do all the work for switching / loading the site url
  • start work on exposing the list of browsers from detected-only per machine to being per project. Probably needs some logic in plugins.js file or more likely in cypress.js
@bahmutov

This comment has been minimized.

Copy link
Collaborator Author

@bahmutov bahmutov commented Aug 14, 2019

Testing proxy CLI arguments to Electron application

  1. When launching Electron with same CLI args as Chrome --proxy-server=http://localhost:5556,--proxy-bypass-list=<-loopback> we get our proxy to see and proxy external sites. For example, if the app window is loading http://todomvc.com
  cypress:server:proxy handling proxied request { url: '/', proxiedUrl: 'http://todomvc.com/', headers: { host: 'todomvc.com', 'proxy-connection': 'keep-alive', 'upgrade-insecure-requests': '1', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.102 Electron/6.0.1 Safari/537.36', accept: 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3', referer: 'http://localhost:5556/__cypress/iframes/integration/spec.js', 'accept-encoding': 'gzip, deflate', 'accept-language': 'en-US', cookie: '_ga=GA1.2.665055498.1565718779' }, remoteState: { auth: undefined, props: null, origin: 'http://localhost:5556', strategy: 'file', visiting: undefined, domainName: 'localhost', fileServer: 'http://localhost:52703' } } +39ms

Without --proxy* CLI arguments the requests from the Electron application do not appear in our proxy.

See Electron issue electron/electron#584 for --proxy-server and the full list of CLI switches https://github.com/electron/electron/blob/master/docs/api/chrome-command-line-switches.md

Conclusion

We can use same proxy CLI arguments with Chrome and Electron external application

@brian-mann

This comment has been minimized.

Copy link
Member

@brian-mann brian-mann commented Aug 14, 2019

We could override the loadFile and loadURL methods on the browser window and force those requests via the file:// protocol to be routed through the proxy. There are lots of ways to solve this.

Regardless though - we have to initially suppress the app from initially loading / inflating - and must have it load cypress instead. This is really simply handled by the user modifying their code (or for instance passing off the browser window instance to us).

Then they would use cy.visit(...) directly to load in their application. The only other problem here is that their code could be instantiating other browser windows, which then have the exact same problem. I believe the solution here would be to require / import us as a module in their code (conditionally using a flag) and then we'd be able to monkey patch the BrowserWindow directly. Alternatively... since we have direct access to the initial BrowserView instances, we may be able to do this ourselves automatically.

@flotwig

This comment has been minimized.

Copy link
Member

@flotwig flotwig commented Aug 15, 2019

I played around some with trying to intercept loadFile and loadURL without overriding the functions.

  • You can intercept loadFile using protocol.interceptHttpProtocol (or protocol.interceptStreamProtocol, presumably others) and attempt to redirect the request to the Cypress proxy:
    protocol.interceptStreamProtocol('file', (req, cb) => {
      console.log('file:// request intercepted', req)
    
      // imagine this `rp` is for Cypress's proxy
      rp({
        headers: req.headers,
        method: req.method,
        url: 'https://httpdump.io/vj6ef?originalFileUrl=' + req.url,
        body: req.uploadData ? req.uploadData : null,
        resolveWithFullResponse: true
      }).then((res) => {
        cb({
          data: res.body,
          statusCode: res.statusCode,
          headers: res.headers
        })
      })
    })
    However, this won't work - with webSecurity on, the BrowserWindow just shows
    image, without it, the BrowserWindow silently shows nothing on a loadFile.
    This could probably only be used to intercept HTTP URLs. There is a protocol.interceptFileProtocol, but it can only be used to serve files from disk in response to a file:// request.
  • Network.setRequestInterception and Fetch.enable from RDP don't seem to intercept file:// requests, so they can't be used.

In light of this, overriding loadFile and loadURL in JS sounds like the cleanest way to intercept those calls.

@bahmutov

This comment has been minimized.

Copy link
Collaborator Author

@bahmutov bahmutov commented Aug 15, 2019

I have looked at using https://electronjs.org/docs/api/browser-view It is like child frameless window, except has several downsides

Example code

// main-browser-view.js
// $ npx electron ./main-browser-view.js
const { app, BrowserView, BrowserWindow } = require('electron')

function createWindow () {
  let win = new BrowserWindow({ width: 800, height: 600 })
  win.on('closed', () => {
    win = null
  })
  win.loadFile('main-browser-view.html')

  let view = new BrowserView()
  win.setBrowserView(view)
  view.setBounds({ x: 0, y: 150, width: 800, height: 450 })
  view.webContents.loadURL('https://electronjs.org')
}

app.on('ready', createWindow)
<body>
  <h1>Browser view demo</h1>
  <p>this page embeds browser view instance below</p>
</body>

The loaded view literally sits on top of everything in the top window, even the DevTools.

Screen Shot 2019-08-15 at 12 30 07 PM

But the browser view instance is "invisible" to the outside window JavaScript, trying to solve this conundrum.

Hmm, seems I cannot get reference to the window object inside a BrowserView - only by using window.open or its equivalent can I get the reference back that allows accessing child window. Need to investigate webview

@bahmutov

This comment has been minimized.

Copy link
Collaborator Author

@bahmutov bahmutov commented Aug 15, 2019

<webview> has been deprecated / disabled by default https://electronjs.org/docs/api/webview-tag#webview-tag ughh.

Back to square 1 - just use our current iframe method to visit the site. Since the first browser window running in the external Electron the iframe has access to the Node stuff too

Screen Shot 2019-08-15 at 1 44 59 PM

@brian-mann

This comment has been minimized.

Copy link
Member

@brian-mann brian-mann commented Aug 15, 2019

We could still instantiate a separate browser window - it would just make recording more difficult. We'd need to multiplex in two different video streams and align them so it appears in a single video. Not the hardest thing, but not an instant win either.

@bahmutov

This comment has been minimized.

Copy link
Collaborator Author

@bahmutov bahmutov commented Aug 15, 2019

Side note

I was thinking about the code I used to open the second window from Electron app

beforeEach(function openSeparateWindow () {
  return new Promise(resolve => {
    const replace = true
    const mw = (window.mw = open(
      'http://example.cypress.io',
      'mainWindow',
      replace
    ))

    // TODO resolve when mw.document is valid
    // for now simply wait
    setTimeout(() => {
      cy.state('document', mw.document)
      cy.state('window', mw)
      resolve()
    }, 1000)
  })
})

as I suspected - this works even today in v3.4.1 with external Chrome browser as well. This might be good for people who have problems testing their web apps in an iframe

Screen Shot 2019-08-15 at 4 55 11 PM

@brian-mann

This comment has been minimized.

Copy link
Member

@brian-mann brian-mann commented Aug 15, 2019

@bahmutov

This comment has been minimized.

Copy link
Collaborator Author

@bahmutov bahmutov commented Aug 15, 2019

Yup, so I think we can advise this as a workaround to people struggling with their site being iframed

@bahmutov

This comment has been minimized.

Copy link
Collaborator Author

@bahmutov bahmutov commented Aug 15, 2019

One other thing - this blows away the window before each test! no more lifecycle leaking callbacks and requests from one test into another

@bahmutov

This comment has been minimized.

Copy link
Collaborator Author

@bahmutov bahmutov commented Aug 15, 2019

BTW, the DevTools just stay open through the tests

devtools-and-child-window

@brian-mann

This comment has been minimized.

Copy link
Member

@brian-mann brian-mann commented Aug 15, 2019

The devtools stay open because the window isn't being closed. It would close if we closed the window.

Architecturally this is no different than an iframe and it would inherit all of the same problems. This doesn't solve lifecycle at all. The problem with lifecycle isn't our ability to close the window or iframe (we could do that now). The challenges are around the API's and controlling when you want to blow away the window vs recreating it and forcing users to cy.visit() over and over again.

@bahmutov

This comment has been minimized.

Copy link
Collaborator Author

@bahmutov bahmutov commented Aug 15, 2019

Right, but since we haven't implemented lifecycle events, I would advise users with random crashes and lots of XHR requests to do the above window open to take control of the window themselves.

@brian-mann

This comment has been minimized.

Copy link
Member

@brian-mann brian-mann commented Aug 15, 2019

Functionally or architecturally it won't change anything. The upsides are purely visual and superficial.

@brian-mann

This comment has been minimized.

Copy link
Member

@brian-mann brian-mann commented Aug 15, 2019

But don't get me wrong - I actually think those visual / ergonomic upsides may be worth exploring because we have a lot of options like embedding the devtools into the cypress window and mounting it against the window and/or iframe (i have to try that out). We could then programmatically control the devtools to do things like automatically opening / closing it when you interact with the command log.

I also think at the very least we could offer opening in a separate window as an option to users that they could control themselves. It wouldn't do this in cypress run mode, so the video would not be affected. Functionally nothing has to change in cypress to support this.

@bahmutov

This comment has been minimized.

Copy link
Collaborator Author

@bahmutov bahmutov commented Aug 16, 2019

Exploring the application's side of testing

Aside from loading and controlling the user application from Cypress, we need to think what the entire E2E test is capable of testing:

  • is it just the browser page layout?
  • is the only reset before each test is the browser page reload in cy.visit() or do the apps need more actions in the Electron source to reset?
  • what does Spectron allow?

Spectron examples

Typical test (via Ava)

import test from 'ava';
import {Application} from 'spectron';

test.beforeEach(async t => {
  t.context.app = new Application({
    path: '/Applications/MyApp.app/Contents/MacOS/MyApp'
  });

  await t.context.app.start();
});

test.afterEach.always(async t => {
  await t.context.app.stop();
});

test(async t => {
  const app = t.context.app;
  await app.client.waitUntilWindowLoaded();

  const win = app.browserWindow;
  t.is(await app.client.getWindowCount(), 1);
  t.false(await win.isMinimized());
  t.false(await win.isDevToolsOpened());
  t.true(await win.isVisible());
  t.true(await win.isFocused());

  const {width, height} = await win.getBounds();
  t.true(width > 0);
  t.true(height > 0);
});
@bahmutov

This comment has been minimized.

Copy link
Collaborator Author

@bahmutov bahmutov commented Aug 16, 2019

Ok, I got it. For all additional resetting that the app might want to do before loading the page, it should just use actions similar to cy.task via Electron's native ipcMain and ipcRenderer communication.

Example:
Imagine the app needs to reset the global counter that the page wants to show

// main.js
global.counter = 0
<h2>Global counter</h2>
<div id="counter"></div>
<script>
  let remote = require('electron').remote
  document.getElementById('counter').innerHTML = remote.getGlobal('counter')
</script>

Then the main.js would change like this:

// main.js
global.counter = 0

// let's give Cypress tests access to the "global.counter"
const tasks = {
  setCounter (n) {
    console.log('setting global.counter to', n)
    global.counter = n
  }
}
// this is all generic code that WE can hide in `@cypress/electron-utils`
ipcMain.on('task', (e, taskName, ...args) => {
  console.log('ipMain task "%s"', taskName)
  if (tasks[taskName]) {
    // TODO handle sync and promise-returning tasks
    const result = tasks[taskName](...args)
    e.returnValue = result
  } else {
    console.error('Unknown task name "%s"', taskName)
    e.returnValue = null
  }
})

and the spec code

// spec.js
Cypress.Commands.add('electronTask', (name, ...args) => {
  // avoid Cypress bundling using its own "require"
  const ipcRenderer = global['require']('electron').ipcRenderer
  return ipcRenderer.sendSync('task', name, ...args)
})
it('resets app counter', () => {
  cy.electronTask('setCounter', 5)
  cy.visit('http://localhost:4600')
  cy.get('#counter').should('have.text', '5')

  cy.electronTask('setCounter', 21)
  cy.reload()
  cy.get('#counter').should('have.text', '21')
})

Screen Shot 2019-08-16 at 3 43 28 PM

@bahmutov

This comment has been minimized.

Copy link
Collaborator Author

@bahmutov bahmutov commented Aug 16, 2019

Hmm, if we allowed system Node that executes cypress/plugins/index.js file to BE the local Electron app, then the plugins could BE the main.js or our wrapper - becoming background.js file like we want in Cypress v5

Then the app's own main.js would initialize the application, pop main window pointing out our test runner url AND each cy.task would directly have access to all app's variables one might want to control during testing. This could be sweet

@bahmutov

This comment has been minimized.

Copy link
Collaborator Author

@bahmutov bahmutov commented Aug 16, 2019

Future work estimate

We probably can limit the initial work to 3 parts

  • a programmatic way to load local Electron application and local main.js script file - a week of work, depending on how much we do for cypress.json to cypress.js transition. This will require at least some Cypress Test Runner changes + documentation
    • investigate if today we can do this using plugin code to manipulate the list of browsers in the config + validate the returned information
    • handle external electron application similar to Chrome browser - control cookies and the rest via Chrome Remote Interface
  • some docs explaining how to use CLI to the electron application to either do "normal" load or let Cypress load in the main window - a day or two
    • we launch electron main.js
    • the app opens browser window - and Cypress can control it because we pass remote interface CLI argument to their Electron
    • the app should show Cypress GUI - maybe via CLI argument
  • a utility for adding cy.task equivalent for Electron applications to work with the main process from the spec file - a week of work, this is part of external plugin + documentation
  • document how users can "pop" the window we spawn instead of using cy.visit inside an app iframe

Limitations

  • probably loading an url at first, I have not checked if Cypress built-in server can serve the local src/index.html as well as Electron can do window.loadFile('src/index.html')
@FrancescoBorzi

This comment has been minimized.

Copy link

@FrancescoBorzi FrancescoBorzi commented Aug 17, 2019

@bahmutov

This comment has been minimized.

Copy link
Collaborator Author

@bahmutov bahmutov commented Aug 17, 2019

@FrancescoBorzi I have just made this repo public, but it not ready, and is really in a state of experimentation

@bahmutov

This comment has been minimized.

Copy link
Collaborator Author

@bahmutov bahmutov commented Sep 4, 2019

Reworked architecture using a private sandbox project:

  • supports the browser window created by the application, not just reusing Cypress window. You have your application window created on-demand via a window factory function.
    • we believe most Electron applications (like the ones on https://electronjs.org/apps) need to create their browser window in order to function properly, and just overwriting cy.visit to load an url is not going to be enough
  • normal workflow
  • Cypress testing workflow
  • steps to implement
    • load a local Electron per-project
    • allow users to control list of browsers #5068
    • allow users to return an external Electron app in the list of browser and launch it similarly to new Chrome (and control via Chrome remote interface, uses #4628)
    • run tests similar to the sandbox demo, polish the experience, release as a plugin for faster iteration

electron-window

We are still thinking about how to cleanly separate loading an application and creating its window to be tested.

@bahmutov

This comment has been minimized.

Copy link
Collaborator Author

@bahmutov bahmutov commented Sep 5, 2019

Updated work estimate

Roughly 3 parts:

  1. use external Electron as a browser
  2. open and control user app's window
  3. ergonomics and additional features

Part 1: use external Electron as a browser: 1 week

  • a programmatic way to load local Electron application and a user-specified local main.js script file - a week of work.
    • there is already PR to let the user control list of browsers from the plugins file #5068
    • allow returning external Electron path and maybe some way of saying "load Cypress Electron plugin to test it". Maybe even path to the factory function that creates the main browser window?
    • Cypress Test Runner changes + documentation
  • run external Electron application when using cypress run command (right now hard-coded to run the built-in Electron browser)

Part 2: open and control user app's window: 1 week

  • handle the external electron application similar to Chrome browser - control cookies, record video and the rest via Chrome Remote Interface via #4628
  • some docs explaining how to use CLI to the electron application to either do "normal" load or let Cypress load in the main window, similar to electron-sandbox demo
  • publish the Electron custom command as a separate plugin @cypress/electron-plugin

Part 3: ergonomics and examples: 1 to 2 weeks

  • Need to polish the ergonomics of testing Electron apps. Everything from app reload, to event listeners, to IPC equivalent of cy.task, etc.
  • list of apps to try testing at https://electronjs.org/apps
  • take care of unexpected bugs found

Plan

  • build Cypress binary with Electron support from a branch (at least Mac binary) on every commit, keep merging separate branches and develop branch into this branch as needed. This allows anyone to install a new version of Cypress using environment variable CYPRESS_INSTALL_BINARY=<url> npm i <url>
  • document and set access to be public for @cypress/electron-plugin package
  • think if we can simplify installation of beta versions of Cypress #5135
@bahmutov

This comment has been minimized.

Copy link
Collaborator Author

@bahmutov bahmutov commented Sep 11, 2019

Used browser list PR #5068 to insert the custom Electron browser from plugins/index.js file in the project

module.exports = (on, config) => {
  // `on` is used to hook into various events Cypress emits
  // `config` is the resolved Cypress config

  // remove "standard" browsers and use
  // our local Electron as a browser
  config.browsers = [
    {
      name: 'electron-sandbox',
      family: 'electron-app',
      displayName: 'electron-sandbox',
      version: pkg.version,
      // return full path to Electron module
      path: path.join(
        __dirname,
        '..',
        '..',
        'node_modules',
        '.bin',
        'electron'
      ),
      // show full package version in the browser dropdown
      majorVersion: `v${pkg.version}`,
      info:
        pkg.description || 'Electron.js app that supports the Cypress launcher'
    }
  ]

  return config
}

nice, only a single browser

Screen Shot 2019-09-11 at 11 51 14 AM

Probably need to pass additional CLI arguments to Electron, like our main file path

@kettanaito

This comment has been minimized.

Copy link

@kettanaito kettanaito commented Sep 11, 2019

Hello. Thank you for the great work behind this feature!

If I may, would such testing be automating an actual Electron app, or the application it's serving? I'm wondering what would be the testing environment: is it native Electron, or Cypress replacing electron and server+automating the application?

@bahmutov

This comment has been minimized.

Copy link
Collaborator Author

@bahmutov bahmutov commented Sep 11, 2019

The way we are doing it right now is Cypress

  • opens YOUR Electron app like an external browser, similarly to how Cypress opens external Chrome browser during testing
  • from the spec files you tell us how to load YOUR JavaScript file that creates your main browser window. We call it for you on cy.visit and then control it from our side while running tests

so your spec would be

it('clicks', () => {
  // window creation
  cy.electronVisitUrl('./main_browser_window.js', 'http://localhost:4600')
  cy.get('button').click().click()
  cy.get('#clicked').should('have.text', '2')
})

where 'main_browser_window.js' is your code that creates BrowserWindow instance. Our trick is that we create your window in a way that our runner can synchronously control during tests (instead of iframing your site)

Screen Shot 2019-09-11 at 3 01 10 PM

@kettanaito

This comment has been minimized.

Copy link

@kettanaito kettanaito commented Sep 11, 2019

@bahmutov that looks very promising! Would it be possible to automate file system interactions, like opening file dialogue and other native behaviors?

@bahmutov

This comment has been minimized.

Copy link
Collaborator Author

@bahmutov bahmutov commented Sep 11, 2019

not initially, but when we start driving everything in the browser window via debugger protocol, then it would be I think. And for the rest of automation - we could add cy.task equivalent via ipc inside the Electron app so you could execute actions in the main process from the spec tests.

@megaras2325

This comment has been minimized.

Copy link

@megaras2325 megaras2325 commented Sep 25, 2019

Hi, I'm trying to automate an electron application. Can you enlighten me a little about this? What files need to be modify and all that stuff. It's pritty hard to understand 'cause there's no project to look at :(
Or an example at least

@kettanaito

This comment has been minimized.

Copy link

@kettanaito kettanaito commented Sep 25, 2019

Hi, @megaras2325. The example repo was mentioned on top, linking once more. As I understood it's the work-in-progress feature, so there may be no official guidelines just yet.

@megaras2325

This comment has been minimized.

Copy link

@megaras2325 megaras2325 commented Sep 25, 2019

Hi, @megaras2325. The example repo was mentioned on top, linking once more. As I understood it's the work-in-progress feature, so there may be no official guidelines just yet.

Thank you, @kettanaito :)

@megaras2325

This comment has been minimized.

Copy link

@megaras2325 megaras2325 commented Sep 26, 2019

Hi, guys. Do you think its possible to update the README of the project. I'm working right now on a project where I need to automate an electron app. This project it's the only place where I'd found a clue about it. I tried to apply your structure to my current project without success :( When you open Cypress to run the tests, are you openning a browser or an electron window?

@bahmutov

This comment has been minimized.

Copy link
Collaborator Author

@bahmutov bahmutov commented Oct 1, 2019

@ivml

This comment has been minimized.

Copy link

@ivml ivml commented Oct 16, 2019

Hi guys,

I'm trying to run the example from the blog post (and the linked readme).

After following the readme, running npx cypress open still shows multiple browser options.

Selecting electron-sandbox 1.0.0 and running a test file results in the following error:

Can't run because you've entered an invalid browser name.

Browser: 'electron-sandbox' was not found on your system.

Available browsers found are: electron-sandbox, chrome, electron

Running the example cy:run command results in the same error, but without electron-sandbox in the list of available browsers.

The same thing happens if I clone the example project. The only difference is that the example tries to call the browser electron-app instead of sandbox.

@SamuelKnoch

This comment has been minimized.

Copy link

@SamuelKnoch SamuelKnoch commented Oct 21, 2019

Hi guys,

I'm trying to run the example from the blog post (and the linked readme).

After following the readme, running npx cypress open still shows multiple browser options.

Selecting electron-sandbox 1.0.0 and running a test file results in the following error:

Can't run because you've entered an invalid browser name.
Browser: 'electron-sandbox' was not found on your system.
Available browsers found are: electron-sandbox, chrome, electron

Running the example cy:run command results in the same error, but without electron-sandbox in the list of available browsers.

The same thing happens if I clone the example project. The only difference is that the example tries to call the browser electron-app instead of sandbox.

Same issue

@warpdesign

This comment has been minimized.

Copy link
Contributor

@warpdesign warpdesign commented Oct 24, 2019

I tried to follow the instructions on the blog post but I'm not sure I understand everything.

If I understand correctly, it's cypress-electron-plugin that's supposed to open the window of my Electron app using MainBrowserWindow.

Should win.loadURL be called inside MainBrowserWindow or is it called by the plugin at another time?

What's the second parameter of cy.electronVisitUrl?

In my app, I am loading the window's url like this: win.loadURL(file://${__dirname}/index.html): can I still use the file: protocol in my tests?

@warpdesign

This comment has been minimized.

Copy link
Contributor

@warpdesign warpdesign commented Oct 25, 2019

After some research I'm almost there. This is what I'm doing in my tests:

        cy.electronVisitUrl(
            "../src/electron/main_browser_window.js",
            "http://localhost:8080/index2.html"
        );

       cy.get(".bp3-navbar-heading").should("have.text", "React-Explorer");

A new Electron window is opened with index2.html but the Electron runner shows an error:

SecurityError: Blocked a frame with origin "http://localhost:57258" from accessing a cross-origin frame.

The port 57258 seems to be random, each time I restart cypress it changes.

The weird thing is that if I replace the local http://localhost:8080/index2.html url with some random https url, like https://github.com/cypress-io/cypress/issues/4964 the error doesn't show up.

Also, I had to run a local web server: ideally I'd simply use file:/// and point to my local html file.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
10 participants
You can’t perform that action at this time.