[Blocking issue] Physical machine console related test failed due to ipmiconsole process became zombie.

[alice-suse]

Description:
Beijing side are now doing physical machine(ipmi) tests via openqa. I know that coolo helped jerry with the host installation and made it succeed. But I am afraid that the code was not completely put into official git repo. Because when I checkout the official openqa git code and made the same try, the installation failed.
Root cause is that the [ipmiconsole] process which was started when func do_start_vm calls func start_serial_grab, became zombie process soon after it was created. And the serial0 file of the ipmi worker pool has errors.

Severity:
Serious and blocking all physical machine tests in openqa

Logs:
http://147.2.212.158/tests/17

alice-openqa:/var/lib/openqa/pool/2 # cat vars.json
{
"ARCH" : "x86_64",
"ASSETDIR" : "/var/lib/openqa/share/factory",
"BACKEND" : "ipmi",
"BETA" : "1",
"CASEDIR" : "/var/lib/openqa/share/tests/sle-12-SP2",
"DESKTOP" : "gnome",
"DISTRI" : "sle",
"DVD" : 1,
"FLAVOR" : "Server-DVD",
"GNOME" : 1,
"HASLICENSE" : 1,
"HOST" : "localhost",
"HOST_IMG_URL" : "loader/sles-12-sp2-alpha2-x86_64-linux console=ttyS1,115200 console=tty initrd=loader/sles-12-sp2-alpha2-x86_64-initrd install=http://147.2.207.1/dist/install/SLP/SLE-12-SP2-Server-LATEST/x86_64/DVD1/",
"INSTLANG" : "en_US",
"IPMI_HOSTNAME" : "147.2.208.124",
"IPMI_PASSWORD" : "ADMIN",
"IPMI_USER" : "ADMIN",
"ISO_MAXSIZE" : "4700372992",
"JOBTOKEN" : "5PXmiZoxigNiCswP",
"MACHINE" : "64bit-ipmi",
"MAX_JOB_TIME" : "32000",
"NAME" : "00000015-sle-12-SP2-Server-DVD-x86_64-prj1_guest_installation_on_sles_12_sp2_kvm",
"NOAUTOLOGIN" : 1,
"NOIMAGES" : 1,
"OPENQA_HOSTNAME" : "localhost",
"OPENQA_URL" : "localhost",
"PACKAGETOINSTALL" : "x3270",
"PATTERNS" : "base,minimal,apparmor,32bit,help,gnome,x,print,wbem,kvm,file,kvmserve",
"PRODUCTDIR" : "/var/lib/openqa/share/tests/sle-12-SP2/products/sle",
"QA_SERVER_REPO" : "http://dist.nue.suse.com/ibs/QA:/Head/SLE-12-SP2/",
"QA_VIRTTEST_GI" : "1",
"QEMUPORT" : "20022",
"SHUTDOWN_NEEDS_AUTH" : 1,
"TEST" : "prj1_guest_installation_on_sles_12_sp2_kvm",
"VERSION" : "12-SP2",
"VNC" : "92",
"WALLPAPER" : "/usr/share/wallpapers/SLEdefault/contents/images/1280x1024.jpg",
"WORKER_CLASS" : "64bit-ipmi",
"WORKER_HOSTNAME" : "147.2.212.158",
"WORKER_ID" : "4",
"WORKER_INSTANCE" : "2"
}
alice-openqa:/var/lib/openqa/pool/2 # lsls
If 'lsls' is not a typo you can use command-not-found to lookup the package that contains it, like this:
cnf lsls
alice-openqa:/var/lib/openqa/pool/2 # ls
autoinst-log.txt backend.run job.json .locked os-autoinst.pid qemuscreenshot serial0 testresults tmp vars.json video.ogv
alice-openqa:/var/lib/openqa/pool/2 # cat serial0
Config File Error: configuration file cannot be opened
alice-openqa:/var/lib/openqa/pool/2 # cat /etc/openqa/workers.ini
[1]
BACKEND = qemu

[2]
WORKER_CLASS=64bit-ipmi
IPMI_HOSTNAME=147.2.208.124
IPMI_PASSWORD=ADMIN
IPMI_USER=ADMIN
MAX_JOB_TIME=32000
WORKER_HOSTNAME=147.2.212.158

[3]
BACKEND = qemu
alice-openqa:/usr/lib/os-autoinst/backend # pstree -pal 29342
worker,29342 /usr/share/openqa/script/worker --instance 2
└─isotovideo,29563 -w /usr/bin/isotovideo -d
├─(ipmiconsole,29577)
├─videoencoder,29567 /var/lib/openqa/pool/2/video.ogv
│ └─{videoencoder},29569
├─{isotovideo},29565
├─{isotovideo},29566
└─{isotovideo},29578
alice-openqa:/usr/lib/os-autoinst/backend # ps aux | grep 29577
_openqa+ 29577 0.0 0.0 0 0 ? Z 17:14 0:00 [ipmiconsole]
root 29911 0.0 0.0 10492 932 pts/4 S+ 17:34 0:00 grep --color=auto 29577
alice-openqa:/usr/lib/os-autoinst/backend #

[coolo]

As I explained Jerry, he needs to remind everyone using IPMI to shutdown apparmor

Apr 29 17:14:34 alice-openqa kernel: type=1400 audit(1461921273.999:64): apparmor="DENIED" operation="open" parent=29566 profile="/usr/share/openqa/script/worker" name="/etc/freeipmi/freeipmi.conf" pid=29577 comm="ipmiconso

IPMI and other backends just go too deep to make apparmor protection worth it.