Progressive Testing

[travisperson]

Does anyone know of a better name / the actual name for this?

---

This relates to #45, interface-ipfs-core, as well as other interface testing projects.

The goal of, what I'm calling here at the moment, progressive testing, is to enable a way to run a test suite that may contain failing tests and still get value out of the results. It also aims to remove implementation specific workarounds inside the interface tests themselves, which current exists quite extensively insight of interface-ipfs-core.

The basic idea is to allow failing tests to not effect the outcome of a test suite as long as they have not been previously proven to have passed.

I think this is important because IPFS and libp2p are build around the idea of interfaces and protocols. These ideas enable a powerful way to build and compose software, but I think it's important to also make it easy to consume and work with the tests.

libp2p

• https://github.com/libp2p/interface-transport
• https://github.com/libp2p/interface-record-store
• https://github.com/libp2p/interface-connection
• https://github.com/libp2p/interface-stream-muxer
• https://github.com/libp2p/interface-peer-routing

ipfs

• https://github.com/ipfs/interface-ipfs-core
• https://github.com/ipfs/interface-datastore
• https://github.com/ipfs/interface-pull-blob-store

Protocol Driven Development https://github.com/ipfs/pdd

The issue at the moment is due to the simplicity of must test runner and reporters. A suite of tests is consider to have failed if any single test fails. Generally communicated through the runners exit code.

When starting to work on a new project this means you are in a continue state of failure till a full implementation has been completed. To get around this issue at the moment we rely on making modifications to the tests themselves for each implementation (see interface-ipfs-core).

One possible solution is to define a new kind of reporter. One that has previous knowledge of which tests have passed prior. Any previously passing test failing results in a failure of the suite. If a test has never passed it is considers skipped.

This enables a project to introduce the entire test suite and incrementally move towards 100% test completion.

This does have a drawback though, as it does require tracking previous state of a tests.

@alanshaw has made some progress with this by allowing more finally controlled execution of tests by breaking things down into smaller sections.

ipfs-inactive/interface-js-ipfs-core#290

This is a great step in the right direction, as it enables projects to incrementally build and introduce subsystems. However, we still run into the issue of a new test being added and breaking implementations that have yet to introduce the new feature.

[chriscool]

One potential problem is to reliably track which test is which.

For example with Sharness tests, if we just track each test by its order in its test script, then this breaks when someone introduces a new test between 2 existing tests.

So we should probably use the test name, but we will have to be careful about test naming and renaming and when making an existing test stricter.

[travisperson]

Ya, any name change would be considered a new test. This means that if a name changes, it would then require to pass again prior to it being tracked to fail the suite. This is the one major downside to doing this, it requires a bit of book keeping.

[chriscool]

For Sharness I can implement something that stores the last 5 test results in test-results/txxxx/<test_name>.res where <test_name> is a filename made from the test name. It cannot be the test name as there could be some special characters in the test name.

[chriscool]

The main problem with the above is that I am not sure that on the CI machines where we will run Sharness tests we will be able to save the files where the tests results are stored and then restore these files before running new tests.

These files should also be specific to the branch that is tested as they might not apply well to a branch that is developed in parallel.

[chriscool]

If we store the test results in let's say test-progress/txxxx/<test_name>.res file, what we could do is have Jenkins copy the test-progress folder into a folder named using the branch name.

The issue with this is that when creating a PR a new branch is usually used and we would need to find from which branch this new branch was started to know which test-progress folder should be copied to be used when testing the branch for the PR.

Then we get into another issue. If the branch for the PR was started from let's say master and if master is improved by other PRs at the same time as the PR is reviewed and improved, when testing the PR a new test-progress folder from master would be used instead of the test-progress folder as it was when the branch was started.

So we would perhaps need to store the test-progress folder for each commit in the master branch (and perhaps a few other branches).

[chriscool]

Another way to do it would be to use git-notes to automatically store a note for each commit in the go-ipfs, js-ipfs, rs-ipfs, etc repos that we test using Sharness when the commit passed the tests.

This note would contain all the test results for this commit as well as the sha1 of the commit in ipfs-sharness-tests that was used for testing.

[chriscool]

Yet another way to store the test results would be to use IPFS or IPLD. This way we could avoid giving write access to the Git repo to the user running the tests.

[chriscool]

For now I implemented a new test_expect_unstable() that should replace test_expect_success() when the test can fail:

felipec/sharness#75

test_might_fail() already exists in Sharness but it is for individual commands that can fail, not for a whole test.

[travisperson]

Yet another way to store the test results would be to use IPFS or IPLD. This way we could avoid giving write access to the Git repo to the user running the tests.

When I was first thinking about this, using IPFS was my go to ways to deal with artifacts.

One thing I want to throw out here is that I think we should attempt to keep this idea separate from test frameworks / test runners in the end. I think the process probably should be done after the tests are run. We can take two test outputs and compare them to determine if things are moving forward.

What I want to try and accomplish with this idea is to make it easy to do test (or protocol) driven development, but not have to deal with managing which tests are running.

Hypothetical

Assumptions

• IPNS Publishing is quickish
• Tests are run on master after each merge

When a branch is merged to master tests are ran. The tests results are compared to the previous results of the master branch HEAD. If forward test progression is determined (all previous tests passing continue to pass), then master is considered passing, and the new test results are stored under the commit HEAD points too.

This should mean that for every passing master branch merge, we have a record of those test results under the HEAD commit.

Anytime a PR is opened against master, we should be able to find the test results for the HEAD of master and use it as a previous test result.

The branch is tested against these previous test results. If forward test progression is determine, the branch is valid and can be merged.

Repeat.

The root hash which contains all the commit => test results key/value pairs can be published using IPNS which would allow for outside of CI testing environments (developer computers) to easily run and check forward test progression. They should be able to resolve the IPNS entry and find the commit where they branched off from master.

Test changing

There are main three ways a branch may effect the tests which are run.

1. Tests are added
   • No previous record of the test will exists. The test is permitted to fail when introduced. If the test passed, it will be required to test for forward test progression on all other merged.
2. Tests are removed
   • When tests are removed they are just ignored. A test missing from new test results is not considered to stop forward test progression. (Same as any other testing, deleted test don't do anything)
3. Tests are renamed (removed / added)
   • Because of the way added and removed tests are handled. If a test is renamed, it just looks like a new test. This means that if it was passing, and during the rename it also started to fail, then it will not stop forward test progression.

[chriscool]

One thing I want to throw out here is that I think we should attempt to keep this idea separate from test frameworks / test runners in the end. I think the process probably should be done after the tests are run. We can take two test outputs and compare them to determine if things are moving forward.

We should decide soon if we separate that from test frameworks.