Challenge tests should be done in browser

[BerkeleyTrue]

At the moment the only challenges that have solutions are bonfires and javascript challenges. I'm not sure jquery and html challenges would be testable at the moment.

The testing framework needs to be moved into a browser window so tests can be done on iFrames if required.

[ltegman]

Can confirm any challenge with tests that rely on editor will not work in the testing framework. Currently there are 326 references to editor within the challenge tests.

[BerkeleyTrue]

Yup. Will need to put on my thinking cap.

[systimotic]

As far as I'm aware the tests are currently done in the browser. Can this issue be closed?
/cc @FreeCodeCamp/moderators

[dhcodes]

@systimotic thanks for looking at these issue deep-dives.

[raisedadead]

I would have loved to chime in, but considering this is a very old one, @BerkeleyTrue is the best judge of what this was about then.

[BerkeleyTrue]

Tests are still not run in the browser.

[raisedadead]

Ahhhhh, now I get it, @systimotic @dhcodes, Berkeley is talking about, the npm run test-challenges, these are "unit tests" that are validated against default solutions in CI.

This is a build validation step that is currently in console environment only.

JS Challenges are tested for breaks in the CI step, by validating against the solutions that you may see with the seed code in the JSON files.

This is currently not possible for the jQuery and HTML challenges, and hence need migration to the browser window.

Hope this clarifies the understanding.

[systimotic]

Ah, thank you for the clarification @raisedadead!
That's certainly an interesting problem. Should we label this help wanted?

[BerkeleyTrue]

I've assigned myself to this one.

[Greenheart]

I've got some ideas on how to solve this. Please check #13016 (comment) 😊

[Bouncey]

An idea I have had for HTML testing is passing the HMTL string to react-render-html, and then use react-test-renderer to parse it into a javascript object which we can test easily enough.

[BerkeleyTrue]

This is something that needs to be done in the browser with the same framework we use to test challenges when a user is using fcc. Abstracts will only introduce more issues and differences that cannot be accounted for.

[raisedadead]

@BerkeleyTrue I have been researching lately for one of my day job assignments where we already have been doing testing on Karma and Jasmine (testing framework) to run tests in browser for over a year, and recently I have started some investigation on Headless browsers.

So, potentially we could see this implemented here too, with great Travis.

Chrome has:
https://developers.google.com/web/updates/2017/04/headless-chrome (working on all major OS vendors)

Mozilla has:
https://developer.mozilla.org/en-US/Firefox/Headless_mode

I know you are planning something, but should you be okay, I would like to take this opportunity and try out some prototype.

Let me know your thoughts.

[BerkeleyTrue]

I have in my mind to move over to ava.js for testing (I've used it in several other projects and it is fantastic), but ava does not support browser testing. It does seem possible to do iframe testing using jsdom. If that is possible it might not be needed to do challenge testing in the browser. This is an untested theory but it would make challenge testing pretty fast as ava runs tests in parallel.

[QuincyLarson]

@BerkeleyTrue What do you think of @raisedadead's proposed tool? Should we give him the green light to experiment with it while you work on other parts of the platform?

[raisedadead]

@BerkeleyTrue yup, ava.js is totally dope, and while I haven't yet official used it in any of my production projects, but having gone through some of the writeups and a tinkering around, I can clearly see why its on rage.

And, to the best of my knowledge, Karma is a (cross-platform) engine which supports any testing framework you plug it into. So, while I am yet to confirm if it will play well with ava.js, I am pretty sure it would be like adding any other library.

My main point here is that will headless browsers, we can work with a real-time browser in a CI environment, and this gives us a fair chance to create tests that make the platform stable(-er).

Meaning, we will not be faking any DOM, but its the actual DOM without a rendering UI, which is taken care of by headless browsers, and the fact that these are official implements of Chrome and Firefox, (maybe edge/ie and safari will catchup), is a good thing to have.

[BerkeleyTrue]

@raisedadead if you can get ava working with Karma that would be great, but I suspect it will not be easy. We may end up using mocha/karma for challenge testing and ava for everything else.

[raisedadead]

@BerkeleyTrue sure thing, I'll give this a try and let you know. Thanks for the feedback.

[QuincyLarson]

@raisedadead Thanks for giving this a try. Have you had a chance to experiment with it much yet?

[raisedadead]

Yup, @QuincyLarson I am checking with the AVA team for support for official adaptor for Karma.

[QuincyLarson]

@raisedadead OK - great. Thanks for keeping us posted on this.

[QuincyLarson]

I'm closing this issue as stale since it hasn't been active lately. If you think this is still relevant to the newly updated platform, please explain why, then reopen it.