KVM: lb-cluster test failed case

[hansode]

Problem

Some lb-cluster tests failed at the same point:

15:21:16 Running ./t.lb.tcp.node.multi.balancing.sh
...
15:25:57 test_balance_algorithm_leastconn
15:25:57 ---
15:25:57 :id: lb-ajtpa3fc
15:25:57 :uuid: lb-ajtpa3fc
15:25:57 :account_id: a-shpoolxx
15:25:57 :instance_id: i-99d5nd02
15:25:57 :instance_protocol: tcp
15:25:57 :instance_port: 80
15:25:57 :balance_algorithm: leastconn
15:25:57 :cookie_name: shunit2cookie
15:25:57 :description: ''
15:25:57 :private_key: ''
15:25:57 :public_key: ''
15:25:57 :created_at: 2015-04-28 06:22:55.000000000 Z
15:25:57 :updated_at: 2015-04-28 06:25:54.000000000 Z
15:25:57 :deleted_at: 
15:25:57 :display_name: 
15:25:57 :allow_list:
15:25:57 - 0.0.0.0
15:25:57 :httpchk_path: ''
15:25:57 :state: running
15:25:57 :status: online
15:25:57 :target_vifs:
15:25:57 - :network_vif_id: vif-5d320ri8
15:25:57   :instance_id: i-ij9d26f3
15:25:57   :display_name: ''
15:25:57   :fallback_mode: 'off'
15:25:57 - :network_vif_id: vif-e4z0gh21
15:25:57   :instance_id: i-b2uyj7xd
15:25:57   :display_name: ''
15:25:57   :fallback_mode: 'off'
15:25:57 - :network_vif_id: vif-g5qgwlzu
15:25:57   :instance_id: i-cwj8jrqz
15:25:57   :display_name: ''
15:25:57   :fallback_mode: 'off'
15:25:57 :vif:
15:25:57 - :vif_id: vif-8pck3h01
15:25:57   :vif_index: 0
15:25:57   :network_id: nw-demo1
15:25:57   :ipv4:
15:25:57     :address: 10.0.2.103
15:25:57     :nat_address: 
15:25:57 - :vif_id: vif-26z41hk6
15:25:57   :vif_index: 1
15:25:57   :network_id: nw-demo8
15:25:57   :ipv4:
15:25:57     :address: 10.1.0.10
15:25:57     :nat_address: 
15:25:57 :inbounds:
15:25:57 - :port: 8080
15:25:57   :protocol: tcp
15:26:00 expected_instance_uuid: i-cwj8jrqz
15:26:00 1. actual_instance_uuid: i-cwj8jrqz
15:26:00 2. actual_instance_uuid: i-ij9d26f3
15:26:00 ASSERT:should be the same uuid expected:<i-cwj8jrqz> but was:<i-ij9d26f3>
15:26:00 3. actual_instance_uuid: i-cwj8jrqz
15:26:00 4. actual_instance_uuid: i-cwj8jrqz
15:26:00 5. actual_instance_uuid: i-b2uyj7xd
15:26:00 ASSERT:should be the same uuid expected:<i-cwj8jrqz> but was:<i-b2uyj7xd>
15:26:00 ./../../../shunit2: line 775: 28611 Killed                  curl -fsSkL http://${load_balancer_ipaddr}:${port}/cgi-bin/sleep.cgi > /dev/null 2>&1
15:26:00 ./../../../shunit2: line 775: 28612 Killed                  curl -fsSkL http://${load_balancer_ipaddr}:${port}/cgi-bin/sleep.cgi > /dev/null 2>&1
15:26:00 ---
15:26:00 - i-cwj8jrqz

Solution Candicates

• https://github.com/axsh/wakame-vdc/compare/fix-haproxy-updater
  • fix/improbe upstart system job for haproxy_updater
    • prevent from restarting the job too fast
    • decrease io-load
  • build/release new lb image
• https://github.com/axsh/wakame-vdc/compare/fix-lb-cluster-testing
  • set a proper sleep time
  • no need so far

[hansode]

1. replaced lb image with new image fix-haproxy-updater installed.
2. adjusted slave scheduler on jenkins-ci.