Converge/clarify testing frameworks

[dgrisham]

There are various efforts to make testing/simulations easier for IPFS + related tools.

• https://github.com/ipfs/test-lab
  • (Description needed)
• https://github.com/ipfs/kubernetes-ipfs
  • Automate interactions between IPFS nodes in a Kubernetes cluster using a DSL
  • Uses Prometheus/Grafana for metrics gathering/display
• https://github.com/whyrusleeping/iptb
  • Manage a local cluster of sandboxed IPFS nodes
• https://github.com/ipfs/ipfs-performance-profiling
  • Benchmarking suite for js-ipfs that can be extended with custom tests
• https://github.com/ipfs/ipfs-metrics
  • Interface for managing event logs (viewing, filtering, managing) of IPFS nodes

This issue is meant to reference and discuss all of these repos in one place. The goal is to clearly define the purpose of each repo, as well as the interface (input + output) that they present. Doing this will help clarify how they fit together and where they overlap in functionality. This also gives us an idea of where any missing pieces might be, so that we can either add that missing functionality to an existing repo or note that we might need a new repo to implement it.

Also let me know if I missed any repos here or mischaracterized any of them in my descriptions.

Pinging @diasdavid @jbenet @whyrusleeping @ForrestWeston @FrankPetrilli @hsanjuan @Kubuxu @pgte @lgierth

[frrist]

I can elaborate a bit more on ipfs-metrics (and update its README since some of this is new).

Input

ipfs-metrics expects a json stream as input from the log/tail endpoint. This means it works with go-ipfs as is, and will work with js-ipfs after this feature is added.

Inputs can be added via a configuration file:

$ ipfs-metrics add --config config.json

Or via the CLI:

$ ipfs-metrics add -i "<IP>:<Port>"

Inputs may have tags added to them, by default a nodeId tag is added to each event stream. This serves as a way to identify which ipfs daemon events are coming from when more than one has been added. Tags can be added via the configuration file, or at the end of a CLI command:

$ ipfs-metrics add -i "<IP>:<Port>" "animal=cat" "fruit=kiwi"

Output

ipfs-metrics can output to an http endpoint, such as an Influxdb instance, or if no output option is given to Stdout (note the above examples will write to Stdout). This can be defined via a configuration file, or via the CLI:

$ ipfs-metrics add -i "<IP>:<Port>" -o "<IP>:<Port>" "animal=cat" "fruit=kiwi"

Outputs can be written in different formats, json is the default, but Line Protocol is also an option. Again this can be defined via a configuration file, or via the CLI:

$ ipfs-metrics add -i "<IP>:<Port>" -o "<IP>:<Port>" --lineprotocol "animal=cat" "fruit=kiwi"

Line Protocol should be used when writing to Influxdb as json was deprecated after 1.0

Json Event Log

{"message":{"duration":449828,"event":"handleFindPeer","peerID":"QmSoLer265NRgSp2LA3dPaeykiS1J6DifTC88f5uVQKNAd","session":"d29d385e-a1a7-42b7-a8d3-1a7c72902336","system":"dht","time":"2017-11-28T20:18:27.891072958Z"},"tags":[{"Name":"nodeId","Value":"QmcJ9RHiEoa1WYeaFAEHVgjc41aXfD52WDEFLZrEcQvbPR"},{"Name":"animal","Value":"cat"},{"Name":"fruit","Value":"kiwi"}]}

Line Protocol Event Log

dht,session=d29d385e-a1a7-42b7-a8d3-1a7c72902336,event=handleFindPeer,nodeId=QmcJ9RHiEoa1WYeaFAEHVgjc41aXfD52WDEFLZrEcQvbPR,animal=cat,fruit=kiwi duration=449828 1511900315311252030