[Flaky] Check test_check_node_statistics_can_be_retrieved

[ballot-scality]

The build https://eve.devsca.com/github/scality/metalk8s/#/builders/7/builds/1184 resulted in a flaky test.

Here is the traceback

tox -e tests -- post -s --run-services
 in dir /home/eve/worker/openstack-prod-metalk8s-backend-0/build (timeout 1200 secs) (maxTime 14400 secs)
 watching logfiles {}
 argv: tox -e tests -- post -s --run-services
 environment:
  ANSIBLE_INVENTORY=eve/tests/single-node/inventory/hosts
  HOME=/home/eve
  LANG=en_US.utf8
  LOGNAME=eve
  MAIL=/var/mail/eve
  PATH=/sbin:/bin:/usr/sbin:/usr/bin
  PWD=/home/eve/worker/openstack-prod-metalk8s-backend-0/build
  SHELL=/bin/bash
  SUDO_COMMAND=/bin/buildbot-worker start /home/eve/worker
  SUDO_GID=0
  SUDO_UID=0
  SUDO_USER=root
  TERM=unknown
  USER=eve
  USERNAME=eve
 using PTY: True
tests installed: adal==1.1.0,ansible==2.6.0,ansible-lint==3.4.23,ansible-modules-hashivault==3.9.7,anyconfig==0.9.7,arrow==0.12.1,asn1crypto==0.24.0,atomicwrites==1.2.1,attrs==18.2.0,backports.functools-lru-cache==1.5,bcrypt==3.1.4,binaryornot==0.4.4,cachetools==2.1.0,Cerberus==1.2,certifi==2018.8.24,cffi==1.11.5,chardet==3.0.4,click==6.7,click-completion==0.3.1,colorama==0.3.9,configparser==3.5.0,cookiecutter==1.6.0,cryptography==2.3.1,enum34==1.1.6,fasteners==0.14.1,flake8==3.5.0,funcsigs==1.0.2,future==0.16.0,git-url-parse==1.1.0,glob2==0.6,google-auth==1.5.1,hvac==0.6.4,idna==2.7,ipaddress==1.0.22,Jinja2==2.10,jinja2-time==0.2.0,kubernetes==7.0.0,Mako==1.0.7,MarkupSafe==1.0,mccabe==0.6.1,molecule==2.18.1,monotonic==1.5,more-itertools==4.3.0,netaddr==0.7.19,oauthlib==2.1.0,paramiko==2.4.2,parse==1.8.4,parse-type==0.4.2,pathspec==0.5.9,pbr==4.1.0,pexpect==4.6.0,pluggy==0.7.1,poyo==0.4.1,psutil==5.4.6,ptyprocess==0.6.0,py==1.6.0,pyasn1==0.4.4,pyasn1-modules==0.2.2,pycodestyle==2.3.1,pycparser==2.19,pyflakes==1.6.0,PyJWT==1.6.4,PyNaCl==1.3.0,pytest==3.6.4,pytest-bdd==2.21.0,pytest-services==1.3.0,python-dateutil==2.7.3,python-gilt==1.2.1,PyYAML==3.13,requests==2.19.1,requests-oauthlib==1.0.0,rsa==4.0,sh==1.12.14,six==1.11.0,subprocess32==3.5.2,tabulate==0.8.2,testinfra==1.14.1,tree-format==0.1.2,urllib3==1.23,websocket-client==0.53.0,whichcraft==0.4.1,yamllint==1.11.1
tests run-test-pre: PYTHONHASHSEED='1362643316'
tests runtests: commands[0] | pytest tests/post -s --run-services
============================= test session starts ==============================
platform linux -- Python 3.6.6, pytest-3.6.4, py-1.6.0, pluggy-0.7.1
rootdir: /home/eve/worker/openstack-prod-metalk8s-backend-0/build, inifile:
plugins: testinfra-1.14.1, services-1.3.0, bdd-2.21.0
collecting 0 items                                                             
collecting 1 item                                                              
collecting 2 items                                                             
collecting 3 items                                                             
collecting 4 items                                                             
collected 4 items                                                              
tests/post/test_kubectl.py .                                                   
tests/post/test_metrics_server.py F                                            
tests/post/test_prometheus.py Starting to serve on 127.0.0.1:8001
.                                                
tests/post/test_storage.py persistentvolumeclaim/testclaim created
pod/test-pv created
pod "test-pv" deleted
persistentvolumeclaim "testclaim" deleted
PLAY [kube-node:kube-master] ***************************************************
TASK [reclaim_local_storage : get list of PV from api server] ******************
ok: [ci-metalk8s -> ci-metalk8s]
TASK [reclaim_local_storage : get all released pv indexed by server] ***********
ok: [ci-metalk8s]
TASK [reclaim_local_storage : debug list of released pv] ***********************
ok: [ci-metalk8s] => {
    "released_pv": {
        "ci-metalk8s": [
            {
                "metadata": {
                    "annotations": {
                        "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"kind\":\"PersistentVolume\",\"metadata\":{\"annotations\":{},\"labels\":{\"scality.com/metalk8s_fstype\":\"ext4\",\"scality.com/metalk8s_node\":\"ci-metalk8s\",\"scality.com/metalk8s_vg\":\"vg_metalk8s\"},\"name\":\"vg-metalk8s-278031fb-b6cb-4da5-a516-15e8ded0771d\",\"namespace\":\"\"},\"spec\":{\"accessModes\":[\"ReadWriteOnce\"],\"capacity\":{\"storage\":\"2Gi\"},\"local\":{\"path\":\"/mnt/vg_metalk8s/278031fb-b6cb-4da5-a516-15e8ded0771d\"},\"nodeAffinity\":{\"required\":{\"nodeSelectorTerms\":[{\"matchExpressions\":[{\"key\":\"kubernetes.io/hostname\",\"operator\":\"In\",\"values\":[\"ci-metalk8s\"]}]}]}},\"persistentVolumeReclaimPolicy\":\"Retain\",\"storageClassName\":\"local-lvm\"}}\n",
                        "pv.kubernetes.io/bound-by-controller": "yes"
                    },
                    "creationTimestamp": "2018-10-15T09:20:01Z",
                    "finalizers": [
                        "kubernetes.io/pv-protection"
                    ],
                    "labels": {
                        "scality.com/metalk8s_fstype": "ext4",
                        "scality.com/metalk8s_node": "ci-metalk8s",
                        "scality.com/metalk8s_vg": "vg_metalk8s"
                    },
                    "name": "vg-metalk8s-278031fb-b6cb-4da5-a516-15e8ded0771d",
                    "resourceVersion": "3987",
                    "selfLink": "/api/v1/persistentvolumes/vg-metalk8s-278031fb-b6cb-4da5-a516-15e8ded0771d",
                    "uid": "7e93827f-d05b-11e8-89c1-fa163e656fef"
                },
                "spec": {
                    "accessModes": [
                        "ReadWriteOnce"
                    ],
                    "capacity": {
                        "storage": "2Gi"
                    },
                    "claimRef": {
                        "apiVersion": "v1",
                        "kind": "PersistentVolumeClaim",
                        "name": "testclaim",
                        "namespace": "default",
                        "resourceVersion": "3948",
                        "uid": "5dd45d43-d05e-11e8-89c1-fa163e656fef"
                    },
                    "local": {
                        "path": "/mnt/vg_metalk8s/278031fb-b6cb-4da5-a516-15e8ded0771d"
                    },
                    "nodeAffinity": {
                        "required": {
                            "nodeSelectorTerms": [
                                {
                                    "matchExpressions": [
                                        {
                                            "key": "kubernetes.io/hostname",
                                            "operator": "In",
                                            "values": [
                                                "ci-metalk8s"
                                            ]
                                        }
                                    ]
                                }
                            ]
                        }
                    },
                    "persistentVolumeReclaimPolicy": "Retain",
                    "storageClassName": "local-lvm"
                },
                "status": {
                    "phase": "Released"
                }
            }
        ]
    }
}
TASK [reclaim_local_storage : delete content of released pv] *******************
changed: [ci-metalk8s] => (item=ci-metalk8s-/mnt/vg_metalk8s/278031fb-b6cb-4da5-a516-15e8ded0771d)
TASK [reclaim_local_storage : delete pv] ***************************************
ok: [ci-metalk8s -> ci-metalk8s] => (item=vg-metalk8s-278031fb-b6cb-4da5-a516-15e8ded0771d)
TASK [reclaim_local_storage : recreate pv] *************************************
ok: [ci-metalk8s -> ci-metalk8s] => (item=vg-metalk8s-278031fb-b6cb-4da5-a516-15e8ded0771d)
PLAY RECAP *********************************************************************
ci-metalk8s                : ok=6    changed=1    unreachable=0    failed=0   
.make: *** [shell] Terminated
=================================== FAILURES ===================================
_________________ test_check_node_statistics_can_be_retrieved __________________
request = <FixtureRequest for <Function 'test_check_node_statistics_can_be_retrieved'>>
    assert response_code == 200
>   assert response_headers['content-type'] == 'application/json'
tests/post/test_metrics_server.py:41: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.tox/tests/lib/python3.6/site-packages/pytest_bdd/scenario.py:195: in _execute_scenario
    _execute_step_function(request, scenario, step, step_func)
.tox/tests/lib/python3.6/site-packages/pytest_bdd/scenario.py:137: in _execute_step_function
    step_func(**kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
request = <FixtureRequest for <Function 'test_check_node_statistics_can_be_retrieved'>>
inventory_obj = <utils.helper.Inventory object at 0x7f01b4e87da0>
group_name = 'k8s-cluster'
    @pytest_bdd.then(pytest_bdd.parsers.parse(
        'I should count as many nodes as {group_name} hosts'))
    def node_count_match(request, inventory_obj, group_name):
>       assert len(request.raw_response['items']) == \
            len(inventory_obj.get_groups_dict()[group_name])
E       AssertionError: assert 0 == 1
E        +  where 0 = len([])
E        +  and   1 = len(['ci-metalk8s'])
tests/post/test_metrics_server.py:51: AssertionError
===================== 1 failed, 3 passed in 101.28 seconds =====================
ERROR: InvocationError for command '/home/eve/worker/openstack-prod-metalk8s-backend-0/build/.tox/tests/bin/pytest tests/post -s --run-services' (exited with code 1)
___________________________________ summary ____________________________________
ERROR:   tests: commands failed
program finished with exit code 1
elapsedTime=104.153685

[thomasdanan]

metalk8s 1.X flaky test, won't fix, closing. the issue