Development flow/process with sharness in it's own repository

[victorb]

Now, with sharness in it's own repository, we need to figure out a development flow that works for everyone. The goal is to allow developers to make changes in sharness without having to update the implementations until they are ready. The idea is similar to how you decouple DB migrations and backend deployments from each other so they can be run when appropriate.

So the current thinking is some flow like this:

• New Feature

  • Add new feature tests in sharness
  • New go-ipfs PRs will run sharness as usual via triggering a build after go-ipfs builds
  • If a test that never passed before is not passing, don't treat this as a test failure[0]
  • If the test have passed before, but is not passing now, it's a failure

• Bugfix

  • If something was fixed as a bugfix (maybe the asserted hash was wrong), we should be able to
    fix it async from other fixes, as to not stop development. Changed tests needs to be marked somehow that they can now fail, but don't treat it as such

• [0] This is to avoid new changes in sharness to now start failing bunch of PRs that are not related to the change at all.

Goal of this issue is to come forward with a process that works for the teams involved (mainly go-ipfs, js-ipfs and rs-ipfs (when that happens))

cc @chriscool @travisperson @diasdavid @whyrusleeping

[chriscool]

If something was fixed as a bugfix (maybe the asserted hash was wrong), we should be able to
fix it async from other fixes, as to not stop development. Changed tests needs to be marked somehow that they can now fail, but don't treat it as such

One hack would be to slightly change the test name, so that it looks like a different test (assuming we identify tests using their name).

[whyrusleeping]

+1 on separating out the sharness tests like this. Though, I strongly disagree that we should have CLI tests for javascript. Javascript doesnt need to be CLI compatible with go, and really, js-ipfs doesnt need a CLI. js-ipfs only runs as a daemon with an http api outside of the browser for testing purposes...

[daviddias]

Yeah, for sharness sake, we should use the go-ipfs cli with --api and point it a js-ipfs daemon.