New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add minimal support for on-device automated testing #13

Closed
jonnor opened this Issue Nov 19, 2013 · 9 comments

Comments

Projects
None yet
2 participants
@jonnor
Member

jonnor commented Nov 19, 2013

Test should be driven from Mocha/JS, and be able to execute things on-device.

An environment variable or command-line switch should be used to select that
on-device tests should be run, and which device it should use.

A minimal test could be one that determines how many nodes we can instantiate on an Arduino, as we've had problems with hitting this out-of-memory condition several times already. Its also hard to debug and is an important measurement of how MicroFlo scales.

Tests should upload new firmware to the device as a setup-step.

Longer term, it should also be possible to run the same tests against the runtime on x86. More details found in https://github.com/jonnor/microflo/blob/master/doc/braindump.md#testing-strategy

@bergie

This comment has been minimized.

Member

bergie commented Nov 19, 2013

You might want to additionally look at https://github.com/noflo/noflo-test

I'm planning to do some work on it soon, so feedback would be welcome

@jonnor

This comment has been minimized.

Member

jonnor commented Nov 19, 2013

Good point. I've already looked at it, and definitely that kind of story I want for components (and with time, maybe subgraphs).

A lot of the work on getting to a minimal level will be the problems around triggering and communicating with a test that runs on a microcontroller device, with a tiny serial connections between the device and host (test driver). There are ties to #6 and #8 here

@jonnor

This comment has been minimized.

Member

jonnor commented Nov 19, 2013

A possible strategy that we may explore is to add test infrastructure like or being noflo-test, and for now, only run the tests on the host (already have the basic infra for that). May be useful when hashing out the component-set.

@jonnor

This comment has been minimized.

Member

jonnor commented Jan 26, 2014

Automated testing of graphs now possible on host
https://twitter.com/jononor/status/427498832012648448

@jonnor

This comment has been minimized.

Member

jonnor commented Feb 3, 2014

When we get to on-device testing, we could use noflo/noflo-ui#70 to host our own test-platform and hook up public Travis CI to drive the tests.

@jonnor

This comment has been minimized.

Member

jonnor commented Apr 23, 2014

Should have a look at what Sauce labs does to integrate with Travis CI, they tunnel communication to let the tests execute on their own infrastrucure: https://saucelabs.com/opensource/travis https://saucelabs.com/docs/connect

@jonnor jonnor modified the milestones: 0.5.0, 0.4.0 May 12, 2014

@jonnor

This comment has been minimized.

Member

jonnor commented May 13, 2015

We now use https://github.com/flowbased/fbp-spec for testing. It connects over the FBP runtime protocol. Currently wired up in http://github.com/microflo/microflo-core component test running on Emscripten/node.js.

This approach should work fine also for on-device testing, but this is currently untested. This is the last piece missing to close this issue.

@jonnor

This comment has been minimized.

Member

jonnor commented May 13, 2015

After some security work on our protocol implemention, it should also be possible to run tests remotely from something like Travis CI, but this is a secondary issue.

@jonnor

This comment has been minimized.

Member

jonnor commented Jan 16, 2017

fbp-spec against device tested working in 0.3.44 on an Arduino Leonardo.

@jonnor jonnor closed this Jan 16, 2017

@jonnor jonnor modified the milestones: 0.4.0, 0.5.0 Jan 16, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment