Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

100% test coverage #75

Open
piranna opened this issue Oct 27, 2014 · 23 comments
Open

100% test coverage #75

piranna opened this issue Oct 27, 2014 · 23 comments
Assignees
Milestone

Comments

@piranna
Copy link
Member

@piranna piranna commented Oct 27, 2014

If some software really needs to be reliable, that's the OS. We need to document and tests all the processes, and we must work to achieve a 100% testing coverage to be sure there's no hidden bugs.

@hackintoshrao
Copy link
Contributor

@hackintoshrao hackintoshrao commented Nov 30, 2014

Im a tech blogger , here is the link of my tech blog ( www.hackintoshrao.com ) , im interested in documentation and testing , on providing little guidance on how to go about it , i can help you guys out

@jtmarmon
Copy link

@jtmarmon jtmarmon commented Nov 30, 2014

Would be interested in helping but I haven't the slightest clue how you write OS tests - looping in some people that may know how to help

@piranna
Copy link
Member Author

@piranna piranna commented Nov 30, 2014

The main advantage of NodeOS is that the barebones layer is really small, and after that all is mostly build in Javascript and managed by NPM, so testing NodeOS is mostly write tests to its dependencies packages. The other advantage is that being minimalistic by design its footprint is small and there are very few packages to tests, so it could be feasable to have a 100% fully tested OS... :-)

@piranna
Copy link
Member Author

@piranna piranna commented Dec 1, 2014

By using a package.json file] to define the NodeOS dependencies, it would be easy to recursively check them and see what's their code coverage and help them to improve it :-)

@piranna piranna modified the milestone: 0.2.0 Feb 8, 2015
This was referenced Feb 24, 2016
@piranna
Copy link
Member Author

@piranna piranna commented Apr 13, 2016

I've just developed coverdeeps, a small tool that check a module dependencies and give a global tests coverage percentage instead of just only the own module, because having reliable dependencies are as important (or more!) than the module itself. It still lacks a lot of features, only support modules published on npm registry (not local ones or published only on a repository) and tests coverage info from coveralls.io so it's really pesimistic about the projects tests coverage, but at least it's a beginning :-)

And as usual, pull-requests are welcome :-D

@lite20
Copy link
Contributor

@lite20 lite20 commented Apr 15, 2016

This tool is fun to play with! I've been running 'coverdeeps' on everything for the last 20 minutes

@piranna
Copy link
Member Author

@piranna piranna commented Apr 15, 2016

And what's your impression? :-D
El 15/4/2016 4:54 AM, "Lite McFish" notifications@github.com escribió:

This tool is fun to play with! I've been running 'coverdeeps' on
everything for the last 20 minutes


You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#75 (comment)

@lite20
Copy link
Contributor

@lite20 lite20 commented Apr 18, 2016

I get the impression that I have a lot of tests to write for a lot ;-;

@piranna
Copy link
Member Author

@piranna piranna commented Apr 18, 2016

I get the impression that I have a lot of tests to write for a lot ;-;

I think this is a good thing, isn't it? ;-)

@lite20
Copy link
Contributor

@lite20 lite20 commented Apr 19, 2016

hahaha nope! Because that means I have a lot of work, but it will be nice once that number raises haha!

@piranna
Copy link
Member Author

@piranna piranna commented Apr 19, 2016

First of all, make coverdeeps more flexible... and get it to achieve 100%
;-)
El 19/4/2016 6:18, "Lite McFish" notifications@github.com escribió:

hahaha nope! Because that means I have a lot of work, but it will be nice
once that number raises haha!


You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#75 (comment)

@luii
Copy link
Member

@luii luii commented Apr 19, 2016

Can we make a list of all things we need to test because i wann to help

@piranna
Copy link
Member Author

@piranna piranna commented Apr 19, 2016

Coverdeeps should help on this by showing the tests coverage of each
module, but I think we can start with itself. You know, eat our own dog
food :-)
El 19/4/2016 9:23, "Philipp Czarnetzki" notifications@github.com escribió:

Can we make a list of all things we need to test because i wann to help


You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#75 (comment)

@lite20
Copy link
Contributor

@lite20 lite20 commented Apr 26, 2016

For a list of tasks to complete, there is the roadmap. You can find a link
to it on the Wiki

On Tue, Apr 19, 2016, 1:27 AM Jesús Leganés Combarro <
notifications@github.com> wrote:

Coverdeeps should help on this by showing the tests coverage of each
module, but I think we can start with itself. You know, eat our own dog
food :-)
El 19/4/2016 9:23, "Philipp Czarnetzki" notifications@github.com
escribió:

Can we make a list of all things we need to test because i wann to help


You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#75 (comment)


You are receiving this because you were assigned.
Reply to this email directly or view it on GitHub
#75 (comment)

@lite20
Copy link
Contributor

@lite20 lite20 commented Apr 26, 2016

Just for clarification, does NodeOS support native modules (compiled
modules)

On Tue, Apr 26, 2016, 4:09 PM Lite McFish lite260@gmail.com wrote:

For a list of tasks to complete, there is the roadmap. You can find a link
to it on the Wiki

On Tue, Apr 19, 2016, 1:27 AM Jesús Leganés Combarro <
notifications@github.com> wrote:

Coverdeeps should help on this by showing the tests coverage of each
module, but I think we can start with itself. You know, eat our own dog
food :-)
El 19/4/2016 9:23, "Philipp Czarnetzki" notifications@github.com
escribió:

Can we make a list of all things we need to test because i wann to help


You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#75 (comment)


You are receiving this because you were assigned.
Reply to this email directly or view it on GitHub
#75 (comment)

@piranna
Copy link
Member Author

@piranna piranna commented Apr 26, 2016

Not to install them with npm from inside NodeOS itself because we don't have yet a c compiler (probably nodeos-cross-toolchain would be reused for this), but you can add them on build time.

@TheHans255
Copy link

@TheHans255 TheHans255 commented Apr 27, 2016

Quick question: How are you defining coverage? If you're purely going by lines of code (which is what it appears that coverdeeps is doing), then you aren't necessarily covering all expected behaviors, since you have varying branches and whatnot. Are you accounting for branches, including branches implicit in operations such as division (which returns NaN when you try to divide by zero)?

@piranna
Copy link
Member Author

@piranna piranna commented Apr 27, 2016

Good question. At this moment I only trust on what coveralls.io does, so if
it only count lines (I think it do), then that's what we have. Coverdeeps
is really pesimistic about percentage, that's something good since if we
get some better checkers the value given can only decrease... so we have
more things to fix :-) We could expand it to use bithound.io or other
metrics, too...
El 27/4/2016 4:22 PM, "Hans Jorgensen" notifications@github.com escribió:

Quick question: How are you defining coverage? If you're purely going by
lines of code (which is what it appears that coverdeeps is doing), then you
aren't necessarily covering all expected behaviors, since you have varying
branches and whatnot. Are you accounting for branches, including branches
implicit in operations such as division (which throw errors when you try to
divide by zero)?


You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#75 (comment)

@TheHans255
Copy link

@TheHans255 TheHans255 commented Apr 28, 2016

Good question. At this moment I only trust on what coveralls.io does, so if it only count lines (I think it do), then that's what we have. Coverdeeps is really pesimistic about percentage, that's something good since if we get some better checkers the value given can only decrease... so we have more things to fix :-) We could expand it to use bithound.io or other metrics, too...

Those things could work - having a metric like this is definitely a good first step to have.

If you're trying to go for a perfectly bug free OS, however, then something that you might want to consider is actually going in and formally verifying the OS components (i.e. sitting down with the JS source and mathematically proving that each function actually does exactly what it's supposed to do). I believe there are tools that help automate this process (for instance, there's one called Coq that works for Haskell). It is an expensive process and not even possible in some cases (not to mention that Chrome V8 has bugs, so your proof can only be as correct as the V8 engine is), but if you can do it, then you never have to worry about bugs for that module ever again as long as it keeps the same feature set.

And if code is so convoluted that you can't figure out what it's supposed to do, that might be an indication to use a different module.

@piranna
Copy link
Member Author

@piranna piranna commented Apr 28, 2016

If you're trying to go for a perfectly bug free OS, however, then something that you might want to consider is actually going in and formally verifying the OS components (i.e. sitting down with the JS source and mathematically proving that each function actually does exactly what it's supposed to do). I believe there are tools that help automate this process (for instance, there's one called Coq that works for Haskell). It is an expensive process and not even possible in some cases (not to mention that Chrome V8 has bugs, so your proof can only be as correct as the V8 engine is), but if you can do it, then you never have to worry about bugs for that module ever again as long as it keeps the same feature set.

That's definitely dificult to achieve, since we are using both Linux and Node.js (and as you've said, this one use v8), and the two are really big projects, and as far as I've seen there's no way to get tests coverage statistics for them, imagine to get a mathematic audit of the code... The best we could achieve is to claim that there's a perfect score of 100% on NodeOS core modules and dependencies, this itself would bring a HUGE confidence on the system, isn't it? :-)

And if code is so convoluted that you can't figure out what it's supposed to do, that might be an indication to use a different module.

Or split it and make it simpler :-)

@TheHans255
Copy link

@TheHans255 TheHans255 commented Apr 29, 2016

The best we could achieve is to claim that there's a perfect score of 100% on NodeOS core modules and dependencies, this itself would bring a HUGE confidence on the system, isn't it? :-)

Yes, that we could. I'm just saying that if we can prove that the JS that drives NodeOS core is correct, then we know that any bugs in NodeOS that we do find are V8's fault or the Linux kernel's fault, not NodeOS's fault. And then our JS code can serve as a reproducible example that we can draw up really quickly for the bug report.

@piranna
Copy link
Member Author

@piranna piranna commented Apr 29, 2016

That's exactly what I was telling you.

@TheHans255
Copy link

@TheHans255 TheHans255 commented Apr 29, 2016

That's exactly what I was telling you.

OK, that makes sense. :)

luii added a commit to luii/NodeOS that referenced this issue Dec 22, 2017
Recently i encountered the exit code 71 which i traced back to line NodeOS#75 in scripts/build and saw that it used a - (minus) instead of a _ (underscore) so i changed the concatenation of the strings
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
6 participants
You can’t perform that action at this time.