way to integrate runtime validation with travis.ci

[liangchenye]

Once #439 been merged, we will have a very flexible way to verify/certify a runtime. I wrote a demo to explain how we can integrate it with travis.ci.
The approach is similar with code coverage which does these:

1. using prove validate/*.t > output.txt to get a testing report
2. send the testing report to a server
3. add the icon to 'README.md' to show the certification result

I wrote a demo project to show how to achieve this:

1. a ok demo and a fail demo. In the travis, I generate a 'ok.txt' and 'fail.txt'.
2. In the travis, I upload them to a server -- 'ocihub.org' which I applied.
3. In README.md, I added two icons, one is 'ok', one is 'fail'.

In the future, 'runc', 'rkt', runv' and other runtime projects could apply this approach. So both the maintainer/user could be easily know a runtime is conformed with oci spec.

The server code is very simple, I reuse isula/ihub framework.

[wking]

On Wed, Nov 29, 2017 at 11:56:53AM +0000, 梁辰晔 (Liang Chenye) wrote: 2. In the [travis](https://github.com/liangchenye/oci-cert-demo/blob/master/.travis.yml#L16), I upload them to a server -- 'ocihub.org' which I applied.

How will this be secured? Will the admin will hand out per-project credentials to project admins who put those credentials in Travis repository settings [1] and use those settings for their POST? [1]: https://docs.travis-ci.com/user/environment-variables/#Defining-Variables-in-Repository-Settings

[liangchenye]

Yes, the admin will send the 'token' to the user, the user can put it in the travis environment.
Taking codecov for example, a user could login by his/her github account. codecov will generate the token and send it to the user automatically. I don't know how many man-power it will cost to develop such a system.

The certification progress is more official. In that case, we may not need to develop a 'token-generate' system. The certification team could just generate it by hand and hand out pre-project credentials to project admins.

[alban]

I've run the validation tests on runc with CircleCI: opencontainers/runc#1757