100% test coverage #75

Open
piranna opened this Issue Oct 27, 2014 · 23 comments

Comments

Projects
None yet
6 participants
@piranna
Member

piranna commented Oct 27, 2014

If some software really needs to be reliable, that's the OS. We need to document and tests all the processes, and we must work to achieve a 100% testing coverage to be sure there's no hidden bugs.

@hackintoshrao

This comment has been minimized.

Show comment
Hide comment
@hackintoshrao

hackintoshrao Nov 30, 2014

Contributor

Im a tech blogger , here is the link of my tech blog ( www.hackintoshrao.com ) , im interested in documentation and testing , on providing little guidance on how to go about it , i can help you guys out

Contributor

hackintoshrao commented Nov 30, 2014

Im a tech blogger , here is the link of my tech blog ( www.hackintoshrao.com ) , im interested in documentation and testing , on providing little guidance on how to go about it , i can help you guys out

@jtmarmon

This comment has been minimized.

Show comment
Hide comment
@jtmarmon

jtmarmon Nov 30, 2014

Would be interested in helping but I haven't the slightest clue how you write OS tests - looping in some people that may know how to help

Would be interested in helping but I haven't the slightest clue how you write OS tests - looping in some people that may know how to help

@piranna

This comment has been minimized.

Show comment
Hide comment
@piranna

piranna Nov 30, 2014

Member

The main advantage of NodeOS is that the barebones layer is really small, and after that all is mostly build in Javascript and managed by NPM, so testing NodeOS is mostly write tests to its dependencies packages. The other advantage is that being minimalistic by design its footprint is small and there are very few packages to tests, so it could be feasable to have a 100% fully tested OS... :-)

Member

piranna commented Nov 30, 2014

The main advantage of NodeOS is that the barebones layer is really small, and after that all is mostly build in Javascript and managed by NPM, so testing NodeOS is mostly write tests to its dependencies packages. The other advantage is that being minimalistic by design its footprint is small and there are very few packages to tests, so it could be feasable to have a 100% fully tested OS... :-)

@piranna

This comment has been minimized.

Show comment
Hide comment
@piranna

piranna Dec 1, 2014

Member

By using a package.json file] to define the NodeOS dependencies, it would be easy to recursively check them and see what's their code coverage and help them to improve it :-)

Member

piranna commented Dec 1, 2014

By using a package.json file] to define the NodeOS dependencies, it would be easy to recursively check them and see what's their code coverage and help them to improve it :-)

@piranna piranna modified the milestone: 0.2.0 Feb 8, 2015

This was referenced Feb 24, 2016

@piranna

This comment has been minimized.

Show comment
Hide comment
@piranna

piranna Apr 13, 2016

Member

I've just developed coverdeeps, a small tool that check a module dependencies and give a global tests coverage percentage instead of just only the own module, because having reliable dependencies are as important (or more!) than the module itself. It still lacks a lot of features, only support modules published on npm registry (not local ones or published only on a repository) and tests coverage info from coveralls.io so it's really pesimistic about the projects tests coverage, but at least it's a beginning :-)

And as usual, pull-requests are welcome :-D

Member

piranna commented Apr 13, 2016

I've just developed coverdeeps, a small tool that check a module dependencies and give a global tests coverage percentage instead of just only the own module, because having reliable dependencies are as important (or more!) than the module itself. It still lacks a lot of features, only support modules published on npm registry (not local ones or published only on a repository) and tests coverage info from coveralls.io so it's really pesimistic about the projects tests coverage, but at least it's a beginning :-)

And as usual, pull-requests are welcome :-D

@lite20

This comment has been minimized.

Show comment
Hide comment
@lite20

lite20 Apr 15, 2016

Member

This tool is fun to play with! I've been running 'coverdeeps' on everything for the last 20 minutes

Member

lite20 commented Apr 15, 2016

This tool is fun to play with! I've been running 'coverdeeps' on everything for the last 20 minutes

@piranna

This comment has been minimized.

Show comment
Hide comment
@piranna

piranna Apr 15, 2016

Member

And what's your impression? :-D
El 15/4/2016 4:54 AM, "Lite McFish" notifications@github.com escribió:

This tool is fun to play with! I've been running 'coverdeeps' on
everything for the last 20 minutes


You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#75 (comment)

Member

piranna commented Apr 15, 2016

And what's your impression? :-D
El 15/4/2016 4:54 AM, "Lite McFish" notifications@github.com escribió:

This tool is fun to play with! I've been running 'coverdeeps' on
everything for the last 20 minutes


You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#75 (comment)

@lite20

This comment has been minimized.

Show comment
Hide comment
@lite20

lite20 Apr 18, 2016

Member

I get the impression that I have a lot of tests to write for a lot ;-;

Member

lite20 commented Apr 18, 2016

I get the impression that I have a lot of tests to write for a lot ;-;

@piranna

This comment has been minimized.

Show comment
Hide comment
@piranna

piranna Apr 18, 2016

Member

I get the impression that I have a lot of tests to write for a lot ;-;

I think this is a good thing, isn't it? ;-)

Member

piranna commented Apr 18, 2016

I get the impression that I have a lot of tests to write for a lot ;-;

I think this is a good thing, isn't it? ;-)

@lite20

This comment has been minimized.

Show comment
Hide comment
@lite20

lite20 Apr 19, 2016

Member

hahaha nope! Because that means I have a lot of work, but it will be nice once that number raises haha!

Member

lite20 commented Apr 19, 2016

hahaha nope! Because that means I have a lot of work, but it will be nice once that number raises haha!

@piranna

This comment has been minimized.

Show comment
Hide comment
@piranna

piranna Apr 19, 2016

Member

First of all, make coverdeeps more flexible... and get it to achieve 100%
;-)
El 19/4/2016 6:18, "Lite McFish" notifications@github.com escribió:

hahaha nope! Because that means I have a lot of work, but it will be nice
once that number raises haha!


You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#75 (comment)

Member

piranna commented Apr 19, 2016

First of all, make coverdeeps more flexible... and get it to achieve 100%
;-)
El 19/4/2016 6:18, "Lite McFish" notifications@github.com escribió:

hahaha nope! Because that means I have a lot of work, but it will be nice
once that number raises haha!


You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#75 (comment)

@luii

This comment has been minimized.

Show comment
Hide comment
@luii

luii Apr 19, 2016

Member

Can we make a list of all things we need to test because i wann to help

Member

luii commented Apr 19, 2016

Can we make a list of all things we need to test because i wann to help

@piranna

This comment has been minimized.

Show comment
Hide comment
@piranna

piranna Apr 19, 2016

Member

Coverdeeps should help on this by showing the tests coverage of each
module, but I think we can start with itself. You know, eat our own dog
food :-)
El 19/4/2016 9:23, "Philipp Czarnetzki" notifications@github.com escribió:

Can we make a list of all things we need to test because i wann to help


You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#75 (comment)

Member

piranna commented Apr 19, 2016

Coverdeeps should help on this by showing the tests coverage of each
module, but I think we can start with itself. You know, eat our own dog
food :-)
El 19/4/2016 9:23, "Philipp Czarnetzki" notifications@github.com escribió:

Can we make a list of all things we need to test because i wann to help


You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#75 (comment)

@lite20

This comment has been minimized.

Show comment
Hide comment
@lite20

lite20 Apr 26, 2016

Member

For a list of tasks to complete, there is the roadmap. You can find a link
to it on the Wiki

On Tue, Apr 19, 2016, 1:27 AM Jesús Leganés Combarro <
notifications@github.com> wrote:

Coverdeeps should help on this by showing the tests coverage of each
module, but I think we can start with itself. You know, eat our own dog
food :-)
El 19/4/2016 9:23, "Philipp Czarnetzki" notifications@github.com
escribió:

Can we make a list of all things we need to test because i wann to help


You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#75 (comment)


You are receiving this because you were assigned.
Reply to this email directly or view it on GitHub
#75 (comment)

Member

lite20 commented Apr 26, 2016

For a list of tasks to complete, there is the roadmap. You can find a link
to it on the Wiki

On Tue, Apr 19, 2016, 1:27 AM Jesús Leganés Combarro <
notifications@github.com> wrote:

Coverdeeps should help on this by showing the tests coverage of each
module, but I think we can start with itself. You know, eat our own dog
food :-)
El 19/4/2016 9:23, "Philipp Czarnetzki" notifications@github.com
escribió:

Can we make a list of all things we need to test because i wann to help


You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#75 (comment)


You are receiving this because you were assigned.
Reply to this email directly or view it on GitHub
#75 (comment)

@lite20

This comment has been minimized.

Show comment
Hide comment
@lite20

lite20 Apr 26, 2016

Member

Just for clarification, does NodeOS support native modules (compiled
modules)

On Tue, Apr 26, 2016, 4:09 PM Lite McFish lite260@gmail.com wrote:

For a list of tasks to complete, there is the roadmap. You can find a link
to it on the Wiki

On Tue, Apr 19, 2016, 1:27 AM Jesús Leganés Combarro <
notifications@github.com> wrote:

Coverdeeps should help on this by showing the tests coverage of each
module, but I think we can start with itself. You know, eat our own dog
food :-)
El 19/4/2016 9:23, "Philipp Czarnetzki" notifications@github.com
escribió:

Can we make a list of all things we need to test because i wann to help


You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#75 (comment)


You are receiving this because you were assigned.
Reply to this email directly or view it on GitHub
#75 (comment)

Member

lite20 commented Apr 26, 2016

Just for clarification, does NodeOS support native modules (compiled
modules)

On Tue, Apr 26, 2016, 4:09 PM Lite McFish lite260@gmail.com wrote:

For a list of tasks to complete, there is the roadmap. You can find a link
to it on the Wiki

On Tue, Apr 19, 2016, 1:27 AM Jesús Leganés Combarro <
notifications@github.com> wrote:

Coverdeeps should help on this by showing the tests coverage of each
module, but I think we can start with itself. You know, eat our own dog
food :-)
El 19/4/2016 9:23, "Philipp Czarnetzki" notifications@github.com
escribió:

Can we make a list of all things we need to test because i wann to help


You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#75 (comment)


You are receiving this because you were assigned.
Reply to this email directly or view it on GitHub
#75 (comment)

@piranna

This comment has been minimized.

Show comment
Hide comment
@piranna

piranna Apr 26, 2016

Member

Not to install them with npm from inside NodeOS itself because we don't have yet a c compiler (probably nodeos-cross-toolchain would be reused for this), but you can add them on build time.

Member

piranna commented Apr 26, 2016

Not to install them with npm from inside NodeOS itself because we don't have yet a c compiler (probably nodeos-cross-toolchain would be reused for this), but you can add them on build time.

@Compynerd255

This comment has been minimized.

Show comment
Hide comment
@Compynerd255

Compynerd255 Apr 27, 2016

Quick question: How are you defining coverage? If you're purely going by lines of code (which is what it appears that coverdeeps is doing), then you aren't necessarily covering all expected behaviors, since you have varying branches and whatnot. Are you accounting for branches, including branches implicit in operations such as division (which returns NaN when you try to divide by zero)?

Compynerd255 commented Apr 27, 2016

Quick question: How are you defining coverage? If you're purely going by lines of code (which is what it appears that coverdeeps is doing), then you aren't necessarily covering all expected behaviors, since you have varying branches and whatnot. Are you accounting for branches, including branches implicit in operations such as division (which returns NaN when you try to divide by zero)?

@piranna

This comment has been minimized.

Show comment
Hide comment
@piranna

piranna Apr 27, 2016

Member

Good question. At this moment I only trust on what coveralls.io does, so if
it only count lines (I think it do), then that's what we have. Coverdeeps
is really pesimistic about percentage, that's something good since if we
get some better checkers the value given can only decrease... so we have
more things to fix :-) We could expand it to use bithound.io or other
metrics, too...
El 27/4/2016 4:22 PM, "Hans Jorgensen" notifications@github.com escribió:

Quick question: How are you defining coverage? If you're purely going by
lines of code (which is what it appears that coverdeeps is doing), then you
aren't necessarily covering all expected behaviors, since you have varying
branches and whatnot. Are you accounting for branches, including branches
implicit in operations such as division (which throw errors when you try to
divide by zero)?


You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#75 (comment)

Member

piranna commented Apr 27, 2016

Good question. At this moment I only trust on what coveralls.io does, so if
it only count lines (I think it do), then that's what we have. Coverdeeps
is really pesimistic about percentage, that's something good since if we
get some better checkers the value given can only decrease... so we have
more things to fix :-) We could expand it to use bithound.io or other
metrics, too...
El 27/4/2016 4:22 PM, "Hans Jorgensen" notifications@github.com escribió:

Quick question: How are you defining coverage? If you're purely going by
lines of code (which is what it appears that coverdeeps is doing), then you
aren't necessarily covering all expected behaviors, since you have varying
branches and whatnot. Are you accounting for branches, including branches
implicit in operations such as division (which throw errors when you try to
divide by zero)?


You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#75 (comment)

@Compynerd255

This comment has been minimized.

Show comment
Hide comment
@Compynerd255

Compynerd255 Apr 28, 2016

Good question. At this moment I only trust on what coveralls.io does, so if it only count lines (I think it do), then that's what we have. Coverdeeps is really pesimistic about percentage, that's something good since if we get some better checkers the value given can only decrease... so we have more things to fix :-) We could expand it to use bithound.io or other metrics, too...

Those things could work - having a metric like this is definitely a good first step to have.

If you're trying to go for a perfectly bug free OS, however, then something that you might want to consider is actually going in and formally verifying the OS components (i.e. sitting down with the JS source and mathematically proving that each function actually does exactly what it's supposed to do). I believe there are tools that help automate this process (for instance, there's one called Coq that works for Haskell). It is an expensive process and not even possible in some cases (not to mention that Chrome V8 has bugs, so your proof can only be as correct as the V8 engine is), but if you can do it, then you never have to worry about bugs for that module ever again as long as it keeps the same feature set.

And if code is so convoluted that you can't figure out what it's supposed to do, that might be an indication to use a different module.

Good question. At this moment I only trust on what coveralls.io does, so if it only count lines (I think it do), then that's what we have. Coverdeeps is really pesimistic about percentage, that's something good since if we get some better checkers the value given can only decrease... so we have more things to fix :-) We could expand it to use bithound.io or other metrics, too...

Those things could work - having a metric like this is definitely a good first step to have.

If you're trying to go for a perfectly bug free OS, however, then something that you might want to consider is actually going in and formally verifying the OS components (i.e. sitting down with the JS source and mathematically proving that each function actually does exactly what it's supposed to do). I believe there are tools that help automate this process (for instance, there's one called Coq that works for Haskell). It is an expensive process and not even possible in some cases (not to mention that Chrome V8 has bugs, so your proof can only be as correct as the V8 engine is), but if you can do it, then you never have to worry about bugs for that module ever again as long as it keeps the same feature set.

And if code is so convoluted that you can't figure out what it's supposed to do, that might be an indication to use a different module.

@piranna

This comment has been minimized.

Show comment
Hide comment
@piranna

piranna Apr 28, 2016

Member

If you're trying to go for a perfectly bug free OS, however, then something that you might want to consider is actually going in and formally verifying the OS components (i.e. sitting down with the JS source and mathematically proving that each function actually does exactly what it's supposed to do). I believe there are tools that help automate this process (for instance, there's one called Coq that works for Haskell). It is an expensive process and not even possible in some cases (not to mention that Chrome V8 has bugs, so your proof can only be as correct as the V8 engine is), but if you can do it, then you never have to worry about bugs for that module ever again as long as it keeps the same feature set.

That's definitely dificult to achieve, since we are using both Linux and Node.js (and as you've said, this one use v8), and the two are really big projects, and as far as I've seen there's no way to get tests coverage statistics for them, imagine to get a mathematic audit of the code... The best we could achieve is to claim that there's a perfect score of 100% on NodeOS core modules and dependencies, this itself would bring a HUGE confidence on the system, isn't it? :-)

And if code is so convoluted that you can't figure out what it's supposed to do, that might be an indication to use a different module.

Or split it and make it simpler :-)

Member

piranna commented Apr 28, 2016

If you're trying to go for a perfectly bug free OS, however, then something that you might want to consider is actually going in and formally verifying the OS components (i.e. sitting down with the JS source and mathematically proving that each function actually does exactly what it's supposed to do). I believe there are tools that help automate this process (for instance, there's one called Coq that works for Haskell). It is an expensive process and not even possible in some cases (not to mention that Chrome V8 has bugs, so your proof can only be as correct as the V8 engine is), but if you can do it, then you never have to worry about bugs for that module ever again as long as it keeps the same feature set.

That's definitely dificult to achieve, since we are using both Linux and Node.js (and as you've said, this one use v8), and the two are really big projects, and as far as I've seen there's no way to get tests coverage statistics for them, imagine to get a mathematic audit of the code... The best we could achieve is to claim that there's a perfect score of 100% on NodeOS core modules and dependencies, this itself would bring a HUGE confidence on the system, isn't it? :-)

And if code is so convoluted that you can't figure out what it's supposed to do, that might be an indication to use a different module.

Or split it and make it simpler :-)

@Compynerd255

This comment has been minimized.

Show comment
Hide comment
@Compynerd255

Compynerd255 Apr 29, 2016

The best we could achieve is to claim that there's a perfect score of 100% on NodeOS core modules and dependencies, this itself would bring a HUGE confidence on the system, isn't it? :-)

Yes, that we could. I'm just saying that if we can prove that the JS that drives NodeOS core is correct, then we know that any bugs in NodeOS that we do find are V8's fault or the Linux kernel's fault, not NodeOS's fault. And then our JS code can serve as a reproducible example that we can draw up really quickly for the bug report.

The best we could achieve is to claim that there's a perfect score of 100% on NodeOS core modules and dependencies, this itself would bring a HUGE confidence on the system, isn't it? :-)

Yes, that we could. I'm just saying that if we can prove that the JS that drives NodeOS core is correct, then we know that any bugs in NodeOS that we do find are V8's fault or the Linux kernel's fault, not NodeOS's fault. And then our JS code can serve as a reproducible example that we can draw up really quickly for the bug report.

@piranna

This comment has been minimized.

Show comment
Hide comment
@piranna

piranna Apr 29, 2016

Member

That's exactly what I was telling you.

Member

piranna commented Apr 29, 2016

That's exactly what I was telling you.

@Compynerd255

This comment has been minimized.

Show comment
Hide comment
@Compynerd255

Compynerd255 Apr 29, 2016

That's exactly what I was telling you.

OK, that makes sense. :)

That's exactly what I was telling you.

OK, that makes sense. :)

luii added a commit to luii/NodeOS that referenced this issue Dec 22, 2017

Fix initramfs copy path
Recently i encountered the exit code 71 which i traced back to line #75 in scripts/build and saw that it used a - (minus) instead of a _ (underscore) so i changed the concatenation of the strings
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment