petset replica fails to initialize

[bparees]

The extended test reveals that sometimes a slave member fails to initialize, in this case member "1" (of 0,1,2) fails to join the replica set because it can't contact itself:

Dec 18 05:38:55.488: INFO: Running 'oc logs --config=/tmp/extended-test-mongodb-petset-replica-jqdp4-qrh5o-user.kubeconfig --namespace=extended-test-mongodb-petset-replica-jqdp4-qrh5o mongodb-replicaset-1 --timestamps'
pod logs for 2016-12-18T10:37:43.981767000Z 2016-12-18T10:37:43.980+0000 I CONTROL  [initandlisten] MongoDB starting : pid=16 port=27017 dbpath=/var/lib/mongodb/data 64-bit host=mongodb-replicaset-1
2016-12-18T10:37:43.982068000Z 2016-12-18T10:37:43.980+0000 I CONTROL  [initandlisten] db version v3.2.6
2016-12-18T10:37:43.982308000Z 2016-12-18T10:37:43.980+0000 I CONTROL  [initandlisten] git version: 05552b562c7a0b3143a729aaa0838e558dc49b25
2016-12-18T10:37:43.982538000Z 2016-12-18T10:37:43.980+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
2016-12-18T10:37:43.982805000Z 2016-12-18T10:37:43.980+0000 I CONTROL  [initandlisten] allocator: tcmalloc
2016-12-18T10:37:43.983041000Z 2016-12-18T10:37:43.980+0000 I CONTROL  [initandlisten] modules: none
2016-12-18T10:37:43.983276000Z 2016-12-18T10:37:43.980+0000 I CONTROL  [initandlisten] build environment:
2016-12-18T10:37:43.983512000Z 2016-12-18T10:37:43.980+0000 I CONTROL  [initandlisten]     distarch: x86_64
2016-12-18T10:37:43.983760000Z 2016-12-18T10:37:43.980+0000 I CONTROL  [initandlisten]     target_arch: x86_64
2016-12-18T10:37:43.983997000Z 2016-12-18T10:37:43.981+0000 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net: { http: { enabled: false }, port: 27017 }, replication: { oplogSizeMB: 64, replSet: "rs0" }, security: { keyFile: "/var/lib/mongodb/keyfile" }, storage: { dbPath: "/var/lib/mongodb/data" }, systemLog: { quiet: true } }
2016-12-18T10:37:43.990110000Z 2016-12-18T10:37:43.989+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=3G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2016-12-18T10:37:44.052474000Z 2016-12-18T10:37:44.051+0000 I CONTROL  [initandlisten] 
2016-12-18T10:37:44.052789000Z 2016-12-18T10:37:44.051+0000 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2016-12-18T10:37:44.053041000Z 2016-12-18T10:37:44.051+0000 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2016-12-18T10:37:44.053288000Z 2016-12-18T10:37:44.051+0000 I CONTROL  [initandlisten] 
2016-12-18T10:37:44.053518000Z 2016-12-18T10:37:44.051+0000 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2016-12-18T10:37:44.053782000Z 2016-12-18T10:37:44.051+0000 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2016-12-18T10:37:44.055259000Z 2016-12-18T10:37:44.054+0000 I CONTROL  [initandlisten] 
2016-12-18T10:37:44.062866000Z 2016-12-18T10:37:44.062+0000 I REPL     [initandlisten] Did not find local voted for document at startup;  NoMatchingDocument: Did not find replica set lastVote document in local.replset.election
2016-12-18T10:37:44.067225000Z 2016-12-18T10:37:44.064+0000 I REPL     [initandlisten] Did not find local replica set configuration document at startup;  NoMatchingDocument: Did not find replica set configuration document in local.system.replset
2016-12-18T10:37:44.068022000Z 2016-12-18T10:37:44.067+0000 I FTDC     [initandlisten] Initializing full-time diagnostic data capture with directory '/var/lib/mongodb/data/diagnostic.data'
2016-12-18T10:37:44.073139000Z 2016-12-18T10:37:44.069+0000 I NETWORK  [HostnameCanonicalizationWorker] Starting hostname canonicalization worker
2016-12-18T10:37:44.077176000Z 2016-12-18T10:37:44.075+0000 I NETWORK  [initandlisten] waiting for connections on port 27017
2016-12-18T10:37:44.116189000Z => [Sun Dec 18 10:37:44] Waiting for local MongoDB to accept connections ...
2016-12-18T10:37:44.201977000Z 2016-12-18T10:37:44.201+0000 I ACCESS   [conn1] note: no users configured in admin.system.users, allowing localhost access
2016-12-18T10:37:44.207463000Z => [Sun Dec 18 10:37:44] Adding mongodb-replicaset-1.mongodb-replicaset.extended-test-mongodb-petset-replica-jqdp4-qrh5o.svc.cluster.local to replica set ...
2016-12-18T10:37:44.289860000Z 2016-12-18T10:37:44.289+0000 I NETWORK  [thread1] Starting new replica set monitor for rs0/mongodb-replicaset-0.mongodb-replicaset.extended-test-mongodb-petset-replica-jqdp4-qrh5o.svc.cluster.local:27017
2016-12-18T10:37:44.290694000Z 2016-12-18T10:37:44.290+0000 I NETWORK  [ReplicaSetMonitorWatcher] starting
2016-12-18T10:37:44.828167000Z {
2016-12-18T10:37:44.828441000Z 	"ok" : 0,
2016-12-18T10:37:44.828688000Z 	"errmsg" : "Quorum check failed because not enough voting nodes responded; required 2 but only the following 1 voting nodes responded: mongodb-replicaset-0.mongodb-replicaset.extended-test-mongodb-petset-replica-jqdp4-qrh5o.svc.cluster.local:27017; the following nodes did not respond affirmatively: mongodb-replicaset-1.mongodb-replicaset.extended-test-mongodb-petset-replica-jqdp4-qrh5o.svc.cluster.local:27017 failed with HostUnreachable",
2016-12-18T10:37:44.828954000Z 	"code" : 74
2016-12-18T10:37:44.829205000Z }
2016-12-18T10:37:44.887793000Z => [Sun Dec 18 10:37:44] ERROR: couldn't add host to replica set!

Guessing there is a race condition here, need to review the petset initialization logic to see if it's reasonable for it to expect to contact itself, and if so, why it can't at this point.

[bparees]

fixed by #217