etcd CI for ppc64le platform

[mkumatag]

This issues is created based on the discussion with @xiang90 about enabling ppc64le platform in etcd CI.

Requirements for Supporting a new platform

1. An "official" maintainer for the platform with clear motivation; someone must be responsible for taking care of the platform. - Merged part of ppc64le platform support #7238
2. Set up CI for build; etcd must compile. - Added part of Add ppc64le travis builds #6707
3. Set up CI for running unit tests; etcd must pass simple tests. All tests are running and now ppc64le platform is part of jenkins run, https://jenkins-etcd-public.prod.coreos.systems/computer/etcd-ppc64le/
4. Set up CI (TravisCI, SemaphoreCI or Jenkins) for running integration tests; etcd must pass intensive tests. - Added ppc64le vm as slave to etcd jenkins server
5. (Optional) Set up a functional testing cluster; an etcd cluster should survive stress testing.

created #7238 to support ppc64le platform

[heyitsanthony]

https://github.com/coreos/etcd/blob/master/Documentation/op-guide/supported-platform.md#supporting-a-new-platform

[mkumatag]

@heyitsanthony updated the description where we stand now with respect to requirement list for supporting new platform, can you please help me proceed further on this?

[xiang90]

please let me know if you have any CI infrastructure where we can include ppc64le platform(travis does not support ppc64le platform as of now)

We do not know actually. Huawei plans to setup their jenkins for arm64 support. Are you able to do similar thing?

[mkumatag]

IMO no need to create multiple jenkins server, we can just add ppc64le slave into same jenkins server and make it as one more platform into configuration matrix.

[xiang90]

@mkumatag We do not have any jenkins master setup right now. You need to setup all stuff. But we might setup jenkins in the future. Then we can add ppc64le and arm64 ones as jenkins slaves.

[mkumatag]

@xiang90 I see a jenkins CI entry in recent PR's, I tried clicking it but couldn't get the access. Can you please help me get access for that jenkins CI server.?

[mkumatag]

ping @xiang90 @heyitsanthony

[heyitsanthony]

@mkumatag OK, I fixed the build URL to point to the public address (https://jenkins-etcd-public.prod.coreos.systems/) on PRs.

[mkumatag]

Thanks @heyitsanthony. Can you help me understand the requirements to add ppc64le slave into this jenkins setup. The expectation is to run all the tests on ppc64le slave on every PR.

[heyitsanthony]

@mkumatag sure, the jenkins worker needs ssh, a user account for jenkins, java, and go 1.7.

If you email the IP for the machine to team-etcd@coreos.com, I can send you the key for the authorized_keys file and configure the jenkins master on our side to talk to it (and help debug it if it doesn't go smoothly). Thanks!

[mkumatag]

@heyitsanthony Do you have any constraints about Operating System, Memory, CPU, Diskspace etc..?

[heyitsanthony]

@mkumatag no experience running it on ppc64, so I can't give any hard numbers. As a point of comparison, the virtual machine running the jenkins CI tests is a n1-standard-2 (2 sandy bridge vCPUs, 7.5 GB memory) with 32GB of storage. PPC may not need as much memory since there's no race detection support for that arch.

So long as ./test reliably passes on the machine it should be OK.

[mkumatag]

cool, I should be able to get a VM with mentioned requirements. Will drop you a mail once get a VM ready. Fingers crossed.

[mkumatag]

Ran the test and found following issues. Any idea what could be issue?

ok github.com/coreos/etcd/pkg/monotime 0.007s coverage: 50.0% of statements 2017-01-20 02:21:21.619114 I | pkg/netutil: resolving infra0.example.com:4001 to 10.0.1.10:4001 2017-01-20 02:21:21.619212 I | pkg/netutil: resolving infra0.example.com:2379 to 10.0.1.10:2379 2017-01-20 02:21:21.619227 I | pkg/netutil: resolving infra0.example.com:7001 to 10.0.1.10:7001 2017-01-20 02:21:21.619236 I | pkg/netutil: resolving infra0.example.com:2380 to 10.0.1.10:2380 2017-01-20 02:21:21.619274 W | pkg/netutil: failed resolving host infra0.example.com:4001 (cannot resolve host.); retrying in 1s 2017-01-20 02:21:22.619399 W | pkg/netutil: failed resolving host infra0.example.com:4001 (cannot resolve host.); retrying in 1s 2017-01-20 02:21:22.619442 E | pkg/netutil: could not resolve host infra0.example.com:4001 2017-01-20 02:21:22.619578 I | pkg/netutil: resolving example.com:2379 to 10.0.10.1:2379 2017-01-20 02:21:22.619615 I | pkg/netutil: resolving example.com:2379 to 10.0.10.1:2379 2017-01-20 02:21:22.619629 I | pkg/netutil: resolving example.com:2379 to 10.0.10.1:2379 2017-01-20 02:21:22.619654 I | pkg/netutil: resolving example.com:2379 to 10.0.10.1:2379 2017-01-20 02:21:22.619688 I | pkg/netutil: resolving example.com:2380 to 10.0.10.1:2380 2017-01-20 02:21:22.619710 I | pkg/netutil: resolving example.com:2379 to 10.0.10.1:2379 2017-01-20 02:21:22.619741 I | pkg/netutil: resolving example.com:2379 to 10.0.10.1:2379 2017-01-20 02:21:22.619779 I | pkg/netutil: resolving example.com:2379 to 10.0.10.1:2379 2017-01-20 02:21:22.619803 I | pkg/netutil: resolving first.com:2379 to 10.0.11.1:2379 2017-01-20 02:21:22.619812 I | pkg/netutil: resolving second.com:2380 to 10.0.11.2:2380 2017-01-20 02:21:22.619836 I | pkg/netutil: resolving second.com:2380 to 10.0.11.2:2380 2017-01-20 02:21:22.619845 I | pkg/netutil: resolving first.com:2379 to 10.0.11.1:2379 --- FAIL: TestGetDefaultInterface (0.00s) routes_linux_test.go:24: default host not supported on linux_ppc64le 2017-01-20 02:21:22.620111 I | pkg/netutil: resolving infra0.example.com:4001 to 10.0.1.10:4001 2017-01-20 02:21:22.620122 I | pkg/netutil: resolving infra0.example.com:2379 to 10.0.1.10:2379 2017-01-20 02:21:22.620136 I | pkg/netutil: resolving infra0.example.com:7001 to 10.0.1.10:7001 2017-01-20 02:21:22.620145 I | pkg/netutil: resolving infra0.example.com:2380 to 10.0.1.10:2380 2017-01-20 02:21:22.620178 W | pkg/netutil: failed resolving host infra0.example.com:4001 (cannot resolve host.); retrying in 1s 2017-01-20 02:21:23.620271 E | pkg/netutil: could not resolve host infra0.example.com:4001 2017-01-20 02:21:23.620507 I | pkg/netutil: resolving example.com:2379 to 10.0.10.1:2379 2017-01-20 02:21:23.620544 I | pkg/netutil: resolving example.com:2379 to 10.0.10.1:2379 2017-01-20 02:21:23.620558 I | pkg/netutil: resolving example.com:2379 to 10.0.10.1:2379 2017-01-20 02:21:23.620583 I | pkg/netutil: resolving example.com:2379 to 10.0.10.1:2379 2017-01-20 02:21:23.620609 I | pkg/netutil: resolving example.com:2380 to 10.0.10.1:2380 2017-01-20 02:21:23.620633 I | pkg/netutil: resolving example.com:2379 to 10.0.10.1:2379 2017-01-20 02:21:23.620663 I | pkg/netutil: resolving example.com:2379 to 10.0.10.1:2379 2017-01-20 02:21:23.620699 I | pkg/netutil: resolving example.com:2379 to 10.0.10.1:2379 2017-01-20 02:21:23.620722 I | pkg/netutil: resolving first.com:2379 to 10.0.11.1:2379 2017-01-20 02:21:23.620731 I | pkg/netutil: resolving second.com:2380 to 10.0.11.2:2380 2017-01-20 02:21:23.620757 I | pkg/netutil: resolving second.com:2380 to 10.0.11.2:2380 2017-01-20 02:21:23.620766 I | pkg/netutil: resolving first.com:2379 to 10.0.11.1:2379 --- FAIL: TestGetDefaultInterface (0.00s) routes_linux_test.go:24: default host not supported on linux_ppc64le 2017-01-20 02:21:23.621012 I | pkg/netutil: resolving infra0.example.com:4001 to 10.0.1.10:4001 2017-01-20 02:21:23.621022 I | pkg/netutil: resolving infra0.example.com:2379 to 10.0.1.10:2379 2017-01-20 02:21:23.621036 I | pkg/netutil: resolving infra0.example.com:7001 to 10.0.1.10:7001 2017-01-20 02:21:23.621045 I | pkg/netutil: resolving infra0.example.com:2380 to 10.0.1.10:2380 2017-01-20 02:21:23.621081 W | pkg/netutil: failed resolving host infra0.example.com:4001 (cannot resolve host.); retrying in 1s 2017-01-20 02:21:24.621206 E | pkg/netutil: could not resolve host infra0.example.com:4001 2017-01-20 02:21:24.621418 I | pkg/netutil: resolving example.com:2379 to 10.0.10.1:2379 2017-01-20 02:21:24.621457 I | pkg/netutil: resolving example.com:2379 to 10.0.10.1:2379 2017-01-20 02:21:24.621471 I | pkg/netutil: resolving example.com:2379 to 10.0.10.1:2379 2017-01-20 02:21:24.621500 I | pkg/netutil: resolving example.com:2379 to 10.0.10.1:2379 2017-01-20 02:21:24.621527 I | pkg/netutil: resolving example.com:2380 to 10.0.10.1:2380 2017-01-20 02:21:24.621551 I | pkg/netutil: resolving example.com:2379 to 10.0.10.1:2379 2017-01-20 02:21:24.621581 I | pkg/netutil: resolving example.com:2379 to 10.0.10.1:2379 2017-01-20 02:21:24.621616 I | pkg/netutil: resolving example.com:2379 to 10.0.10.1:2379 2017-01-20 02:21:24.621639 I | pkg/netutil: resolving first.com:2379 to 10.0.11.1:2379 2017-01-20 02:21:24.621648 I | pkg/netutil: resolving second.com:2380 to 10.0.11.2:2380 2017-01-20 02:21:24.621672 I | pkg/netutil: resolving second.com:2380 to 10.0.11.2:2380 2017-01-20 02:21:24.621681 I | pkg/netutil: resolving first.com:2379 to 10.0.11.1:2379 --- FAIL: TestGetDefaultInterface (0.00s) routes_linux_test.go:24: default host not supported on linux_ppc64le FAIL coverage: 57.0% of statements FAIL github.com/coreos/etcd/pkg/netutil 3.006s

complete test log can be found @ https://gist.github.com/mkumatag/4ec4b1eabd3bdb0d91f24e39ebd4c7ca