[swss test/vs] there is no processes in docker vs

[tim-rj]

Description

Steps to reproduce the issue:

sshao@sshao:~/work$ docker run --privileged -id --name sw debian bash
sshao@sshao:~/work$ sudo ./create_vnet.sh sw
test
sw
Seting up servers
Bring sw-srv0eth0 up
Bring eth1 up in the virtual switch docker
Bring sw-srv1eth0 up
Bring eth2 up in the virtual switch docker
Bring sw-srv2eth0 up
Bring eth3 up in the virtual switch docker
Bring sw-srv3eth0 up
Bring eth4 up in the virtual switch docker
Bring sw-srv4eth0 up
Bring eth5 up in the virtual switch docker
Bring sw-srv5eth0 up
Bring eth6 up in the virtual switch docker
Bring sw-srv6eth0 up
Bring eth7 up in the virtual switch docker
Bring sw-srv7eth0 up
Bring eth8 up in the virtual switch docker
Bring sw-srv8eth0 up
Bring eth9 up in the virtual switch docker
Bring sw-srv9eth0 up
Bring eth10 up in the virtual switch docker
Bring sw-srv10eth0 up
Bring eth11 up in the virtual switch docker
Bring sw-srv11eth0 up
Bring eth12 up in the virtual switch docker
Bring sw-srv12eth0 up
Bring eth13 up in the virtual switch docker
Bring sw-srv13eth0 up
Bring eth14 up in the virtual switch docker
Bring sw-srv14eth0 up
Bring eth15 up in the virtual switch docker
Bring sw-srv15eth0 up
Bring eth16 up in the virtual switch docker
Bring sw-srv16eth0 up
Bring eth17 up in the virtual switch docker
Bring sw-srv17eth0 up
Bring eth18 up in the virtual switch docker
Bring sw-srv18eth0 up
Bring eth19 up in the virtual switch docker
Bring sw-srv19eth0 up
Bring eth20 up in the virtual switch docker
Bring sw-srv20eth0 up
Bring eth21 up in the virtual switch docker
Bring sw-srv21eth0 up
Bring eth22 up in the virtual switch docker
Bring sw-srv22eth0 up
Bring eth23 up in the virtual switch docker
Bring sw-srv23eth0 up
Bring eth24 up in the virtual switch docker
Bring sw-srv24eth0 up
Bring eth25 up in the virtual switch docker
Bring sw-srv25eth0 up
Bring eth26 up in the virtual switch docker
Bring sw-srv26eth0 up
Bring eth27 up in the virtual switch docker
Bring sw-srv27eth0 up
Bring eth28 up in the virtual switch docker
Bring sw-srv28eth0 up
Bring eth29 up in the virtual switch docker
Bring sw-srv29eth0 up
Bring eth30 up in the virtual switch docker
Bring sw-srv30eth0 up
Bring eth31 up in the virtual switch docker
Bring sw-srv31eth0 up
Bring eth32 up in the virtual switch docker
sshao@sshao:~/work$
sshao@sshao:~/work$ docker run --privileged -v /var/run/redis-vs/sw:/var/run/redis --network container:sw -d --name vs docker-sonic-vs
7502caea1470082504d60cfdcb2ff87eb84f89b68136fd7fc9d8f25ebfd67629
sshao@sshao:~/work$

sshao@sshao:~/work$
sshao@sshao:~/work$ docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES
7502caea1470        docker-sonic-vs     "/usr/bin/supervisord"   38 seconds ago      Up 37 seconds                           vs
0b2f4f7df62d        debian              "bash"                   2 hours ago         Up 3 minutes                            sw

Describe the results you received:
no processes up, all test casees failed:

sshao@sshao:~/work/tests$ sudo pytest -s -v --dvsname=vs test_setro.py::TestSetRo::test_SetReadOnlyAttribute
=========================================================================================== test session starts ===========================================================================================
platform linux2 -- Python 2.7.17, pytest-3.3.0, py-1.9.0, pluggy-0.6.0 -- /usr/bin/python
cachedir: .cache
rootdir: /home/sshao/work/tests, inifile:
collected 1 item

test_setro.py::TestSetRo::test_SetReadOnlyAttribute shaoshouping test name= vs
ERROR                                                                                                                                           [100%]

================================================================================================= ERRORS ==================================================================================================
__________________________________________________________________________ ERROR at setup of TestSetRo.test_SetReadOnlyAttribute __________________________________________________________________________

request = <SubRequest 'dvs' for <Function 'test_SetReadOnlyAttribute'>>

    @pytest.yield_fixture(scope="module")
    def dvs(request):
        name = request.config.getoption("--dvsname")
        keeptb = request.config.getoption("--keeptb")
        imgname = request.config.getoption("--imgname")
        fakeplatform = getattr(request.module, "DVS_FAKE_PLATFORM", None)
>       dvs = DockerVirtualSwitch(name, imgname, keeptb, fakeplatform)

conftest.py:827:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
conftest.py:233: in __init__
    self.check_ready()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <conftest.DockerVirtualSwitch object at 0x7fded7707dd0>, timeout = 30

    def check_ready(self, timeout=30):
        '''check if all processes in the dvs is ready'''

        re_space = re.compile('\s+')
        process_status = {}
        ready = False
        started = 0
        while True:
            # get process status
            res = self.ctn.exec_run("supervisorctl status")
            try:
                out = res.output
            except AttributeError:
                out = res
            for l in out.split('\n'):
                fds = re_space.split(l)
                if len(fds) < 2:
                    continue
                process_status[fds[0]] = fds[1]

            # check if all processes are running
            ready = True
            for pname in self.alld:
                try:
                    if process_status[pname] != "RUNNING":
                        ready = False
                except KeyError:
                    ready = False

            # check if start.sh exited
            if process_status["start.sh"] != "EXITED":
                ready = False

            if ready == True:
                break

            started += 1
            if started > timeout:
>               raise ValueError(out)
E               ValueError: arp_update                       STOPPED   Not started
E               bgpd                             STOPPED   Not started
E               buffermgrd                       STOPPED   Not started
E               fpmsyncd                         STOPPED   Not started
E               intfmgrd                         STOPPED   Not started
E               natmgrd                          STOPPED   Not started
E               natsyncd                         STOPPED   Not started
E               nbrmgrd                          STOPPED   Not started
E               neighsyncd                       STOPPED   Not started
E               orchagent                        STOPPED   Not started
E               portmgrd                         STOPPED   Not started
E               portsyncd                        STOPPED   Not started
E               redis-server                     STOPPED   Not started
E               restore_neighbors                STOPPED   Not started
E               rsyslogd                         STOPPED   Not started
E               sflowmgrd                        STOPPED   Not started
E               start.sh                         FATAL     Exited too quickly (process log may have details)
E               staticd                          STOPPED   Not started
E               syncd                            STOPPED   Not started
E               teammgrd                         STOPPED   Not started
E               teamsyncd                        STOPPED   Not started
E               vlanmgrd                         STOPPED   Not started
E               vrfmgrd                          STOPPED   Not started
E               vxlanmgrd                        STOPPED   Not started
E               zebra                            STOPPED   Not started

conftest.py:288: ValueError
------------------------------------------------------------------------------------------- Captured log setup --------------------------------------------------------------------------------------------
config.py                   21 DEBUG    Trying paths: ['/home/sshao/.docker/config.json', '/home/sshao/.dockercfg']
config.py                   28 DEBUG    No config file found
config.py                   21 DEBUG    Trying paths: ['/home/sshao/.docker/config.json', '/home/sshao/.dockercfg']
config.py                   28 DEBUG    No config file found
connectionpool.py          396 DEBUG    http://localhost:None "GET /v1.35/containers/json?all=0&limit=-1&trunc_cmd=0&size=0 HTTP/1.1" 200 1411
connectionpool.py          396 DEBUG    http://localhost:None "GET /v1.35/containers/7502caea1470082504d60cfdcb2ff87eb84f89b68136fd7fc9d8f25ebfd67629/json HTTP/1.1" 200 None
connectionpool.py          396 DEBUG    http://localhost:None "GET /v1.35/containers/0b2f4f7df62d88369185c6c303c1405bdf05eb7d00583b953f3db343636e432d/json HTTP/1.1" 200 None

sshao@sshao:~/work$ docker exec -it vs bash
root@0b2f4f7df62d:/# ps -ef
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 07:21 ?        00:00:00 /usr/bin/python /usr/bin/supervisord
root        13     0  0 07:22 pts/0    00:00:00 bash
root        18    13  0 07:22 pts/0    00:00:00 ps -ef
root@0b2f4f7df62d:/# exit

Describe the results you expected:

All test casees pass.

Additional information you deem important (e.g. issue happens only occasionally):

**Output of `show version`:**

```
(paste your output here)
```

**Attach debug file `sudo generate_dump`:**

```
(paste your output here)
```

[tim-rj]

@lguohan this issue relates to swss vs testing for gcov sonic-net/SONiC#642, help move on, thanks!

[tim-rj]

add one infomation may help, the processes will up if run below command to start vs contaniner , but the tests cases will failed due to "docker inspect --format '{{.HostConfig.NetworkMode}}' vs" return none.

sudo docker run --privileged --net=none -it --entrypoint /bin/bash --name=vs -d -v $PWD/switch1:/sonic -v $PWD/switch1/host:/host  docker-sonic-vs

[daall]

@prsunny is this the problem you ran into a couple weeks ago?

[prsunny]

@daall, I don't think I faced this issue.