functional tests are failing on 'shell to vm' #399

nvgoldin · 2016-12-21T10:28:35Z

The CirrOS VM doesn't get an IP, logging in with lago console and executing /etc/init.d/S40network restart resolves the issue. The only change done to automation was afec1ac, could it be that dnsmasq is not yet configured when we run lago start(now with KVM enabled it is probably done faster than before)? @david-caro - any ideas?

The text was updated successfully, but these errors were encountered:

mykaul · 2016-12-21T10:40:23Z

I would say that this is exactly why we need to make some order between SSH and root passwords, between password-based and public-key based authentication. We need to see that such a VM:

Does have a root password (for ovirt-system-tests use case)
Does have a SSH password (which happens of course to be the root password) - for Lago to connect to it.
Does not have a SSH public key (we do not inject to it a public key - note that ovirt-system-tests might in the future via cloud-init

Currently, there's quite a bit of a mixup. It's very similar to the situation with Hosted-Engine.

gbenhaim · 2016-12-22T15:50:05Z

We've already experienced this issue before: #318
The interface is shown in the vm, but for some reason ifup/ifdown can't recognize it. I don't think that this issue relates to ssh but also don't have a good explanation why it relates to the boot menu.

mykaul · 2016-12-22T16:03:52Z

On Thu, Dec 22, 2016 at 5:50 PM, Gal Ben Haim ***@***.***> wrote: We've already experienced this issue before: #318 <#318> The interface is shown in the vm, but for some reason ifup/ifdown can't recognize it. I don't think that this issue relates to ssh but also don't have a good explanation why it relates to the boot menu.

Reminds me of an infamous issue in OpenStack - the network (and specifically, the DHCP services) are not fast enough to be up and running before the VM is already requesting the network. Y.

…

— You are receiving this because you commented. Reply to this email directly, view it on GitHub <#399 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEcJ6cLM54qtXgQ8J6OCYiRdfSsvW8bKks5rKpwugaJpZM4LSy0t> .

gbenhaim · 2016-12-23T11:03:08Z

As I see it we have two options:

Enable the boot menu.
Search for another image to use in the functional tests.

@mykaul @nvgoldin @ifireball @david-caro
Thoughts ?

mykaul · 2016-12-23T12:20:36Z

I've opened a bug on libvirt on the issue. Y.

…

On Dec 23, 2016 1:03 PM, "Gal Ben Haim" ***@***.***> wrote: As I see it we have two options: 1. Enable the boot menu. 2. Search for another image to use in the functional tests. @mykaul <https://github.com/mykaul> @nvgoldin <https://github.com/nvgoldin> @ifireball <https://github.com/ifireball> @david-caro <https://github.com/david-caro> Thoughts ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#399 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEcJ6TQb2K9IZJs2PRK-KtrfRRytQPe0ks5rK6psgaJpZM4LSy0t> .

nvgoldin · 2016-12-25T08:41:25Z

I looked at dnsmasq logs:

Dec 25 07:54:57 ng-lt dnsmasq-dhcp[4624]: DHCP, IP range 192.168.201.100 -- 192.168.201.254, lease time 1h
Dec 25 07:54:57 ng-lt dnsmasq-dhcp[4624]: DHCPv6, IP range fd8f:1391:3a82:5e0d::c0a8:c964 -- fd8f:1391:3a82:5e0d::c0a8:c9fe, lease time 1h
Dec 25 07:54:57 ng-lt dnsmasq-dhcp[4624]: router advertisement on fd8f:1391:3a82:5e0d::
Dec 25 07:54:57 ng-lt dnsmasq-dhcp[4624]: IPv6 router advertisement enabled
Dec 25 07:54:57 ng-lt dnsmasq-dhcp[4624]: DHCP, sockets bound exclusively to interface 71b4-785d519
Dec 25 07:54:57 ng-lt dnsmasq[4624]: reading /etc/resolv.conf
Dec 25 07:54:57 ng-lt dnsmasq[4624]: using nameserver 10.35.255.14#53
Dec 25 07:54:57 ng-lt dnsmasq[4624]: using nameserver 10.38.5.26#53
Dec 25 07:54:57 ng-lt dnsmasq[4624]: using nameserver 10.200.0.246#53
Dec 25 07:54:57 ng-lt dnsmasq[4624]: using nameserver 10.192.206.245#53
Dec 25 07:54:57 ng-lt dnsmasq[4624]: using nameserver 10.200.0.245#53
Dec 25 07:54:57 ng-lt dnsmasq[4624]: read /etc/hosts - 3 addresses
Dec 25 07:54:57 ng-lt dnsmasq[4624]: read /var/lib/libvirt/dnsmasq/71b4-785d5195a1.addnhosts - 4 addresses
Dec 25 07:54:57 ng-lt dnsmasq-dhcp[4624]: read /var/lib/libvirt/dnsmasq/71b4-785d5195a1.hostsfile
Dec 25 07:54:57 ng-lt dnsmasq-dhcp[4624]: duplicate IP address 192.168.201.3 (lago_functional_tests_vm01) in dhcp-config directive
Dec 25 07:54:57 ng-lt dnsmasq-dhcp[4624]: duplicate IP address 192.168.201.2 (lago_functional_tests_vm02) in dhcp-config directive
Dec 25 07:54:57 ng-lt dnsmasq-dhcp[4624]: duplicate IP address fd8f:1391:3a82:5e0d::c0a8:c903 (lago_functional_tests_vm01) in dhcp-config directive
Dec 25 07:54:57 ng-lt dnsmasq-dhcp[4624]: duplicate IP address fd8f:1391:3a82:5e0d::c0a8:c902 (lago_functional_tests_vm02) in dhcp-config directive
Dec 25 07:55:13 ng-lt dnsmasq-dhcp[4624]: RTR-ADVERT(71b4-785d519) fd8f:1391:3a82:5e0d::
Dec 25 07:55:27 ng-lt dnsmasq-dhcp[4624]: RTR-ADVERT(71b4-785d519) fd8f:1391:3a82:5e0d::
Dec 25 07:55:33 ng-lt dnsmasq-dhcp[4624]: RTR-ADVERT(71b4-785d519) fd8f:1391:3a82:5e0d::

and after the reboot:

Dec 25 07:55:44 ng-lt dnsmasq-dhcp[4624]: DHCPDISCOVER(71b4-785d519) 54:52:c0:a8:c9:02
Dec 25 07:55:44 ng-lt dnsmasq-dhcp[4624]: DHCPOFFER(71b4-785d519) 192.168.201.2 54:52:c0:a8:c9:02
Dec 25 07:55:44 ng-lt dnsmasq-dhcp[4624]: DHCPREQUEST(71b4-785d519) 192.168.201.2 54:52:c0:a8:c9:02
Dec 25 07:55:44 ng-lt dnsmasq-dhcp[4624]: DHCPACK(71b4-785d519) 192.168.201.2 54:52:c0:a8:c9:02 lago_functional_tests_vm02

So I'm not sure yet why it is complaining about duplicate definitions(and we try to add 4 hosts), and also why rebooting the CirrOS VM resolves the issue. Also nothing this doesn't always happen.

mykaul · 2016-12-25T08:53:31Z

We need to ensure Lago is not creating those definitions - that it is not racy. I think the networks are not prepared in parallel (but worth checking).
I've opened a bug on libvirt that it seems to me that libvirt's dnsmasq config is sometimes not ready in time.

nvgoldin · 2016-12-25T13:30:57Z

@mykaul - will check. also need to check if the networkCreateXML call to libvirt is blocking, in the sense that it returns when completed.

for reference another failure: http://jenkins.ovirt.org/job/lago_master_check-merged-fc25-x86_64/3/console

mykaul · 2016-12-25T13:38:06Z

On Sun, Dec 25, 2016 at 3:30 PM, Nadav Goldin ***@***.***> wrote: @mykaul <https://github.com/mykaul> - will check. also need to check if the networkCreateXML call to libvirt is blocking, in the sense that it returns when completed.

perhaps we should ask isActive() on it?

for reference another failure: http://jenkins.ovirt.org/job/ lago_master_check-merged-fc25-x86_64/3/console

Notice that I've found #402 and fixed it in #404 (I hope). Perhaps it can explain some of the issues.

…

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#399 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEcJ6SFFatca_b3pL-nJVQOSW9M9P7ATks5rLnASgaJpZM4LSy0t> .

nvgoldin · 2016-12-25T14:00:47Z

perhaps we should ask isActive() on it?\

exactly what I'm checking now..

is #404 ok to merge? we can try merging and see how others go.

mykaul · 2016-12-25T15:11:39Z

On Sun, Dec 25, 2016 at 4:00 PM, Nadav Goldin ***@***.***> wrote: perhaps we should ask isActive() on it? exactly what I'm checking now.. is #404 <#404> ok to merge? we can try merging and see how others go.

I sincerely hope so ;-)

…

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#399 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEcJ6RJRSc_ym-ezFcXBexH-D5lVgpWiks5rLncPgaJpZM4LSy0t> .

nvgoldin · 2016-12-26T05:16:08Z

might have been resolved by #405, lets keep this ticket open for a while to see if it happens again in the following weeks.

nvgoldin · 2016-12-26T12:43:46Z

Unfortunately another failure:
http://jenkins.ovirt.org/job/lago_master_check-merged-fc25-x86_64/13/console
again restarting the network on the CirrOS VM resolved the issue. The difference here is that it was running on a bare-metal slave, which probably made things faster. So could be that the fix in #405, just gives more waiting time, but does not fix the issue.

nvgoldin · 2017-01-04T13:09:11Z

Moved the jobs back to VMs only for now.

nvgoldin · 2017-01-05T10:30:02Z

Finally got some true logs from the cirros vm on bootup:

Sending discover...

Sending discover...
Usage: /sbin/cirros-dhcpc <up|down>
No lease, failing
WARN: /etc/rc3.d/S40-network failed

For now I'm going to try add a work-around inside the image itself.

lago functional tests fail when they run "too quickly", specifically this happens on bare-metals. So bringing them back to VMs for now. bug-url: lago-project/lago#399 Change-Id: I8c08145a4209e56e9e06b339fb853720314eb7b7 Signed-off-by: Nadav Goldin <ngoldin@redhat.com>

mykaul · 2017-01-05T16:22:31Z

Did you open a bug on cirros?

…

On Jan 5, 2017 12:30 PM, "Nadav Goldin" ***@***.***> wrote: Finally got some true logs from the cirros vm on bootup: Sending discover... Sending discover... Usage: /sbin/cirros-dhcpc <up|down> No lease, failing WARN: /etc/rc3.d/S40-network failed For now I'm going to try add a work-around inside the image itself. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#399 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEcJ6VzEOU_op79MU1loYRCciKI3E3NFks5rPMYrgaJpZM4LSy0t> .

nvgoldin · 2017-01-05T17:57:47Z

Did you open a bug on cirros?

Not sure if this is cirros bug, I need some more work to debug this on the libvirt/dnsmasq, I would suspect them for as I didn't see those dhcp request get to dnsmasq at all(only after restarting the network/dhcp).

nvgoldin · 2017-01-08T06:59:17Z

libvirt bug: https://bugzilla.redhat.com/show_bug.cgi?id=1411025

nvgoldin · 2017-04-24T07:44:15Z

Safe to say this was fixed with the cirros workaround., haven't seen it for a while.

nvgoldin added the tests label Dec 21, 2016

nvgoldin mentioned this issue Jan 4, 2017

Make the package build on RHEL/CentOS #397

Merged

nvgoldin mentioned this issue Jan 5, 2017

tests: update cirros image to start cirros-dhcpc explicitly #419

Closed

nvgoldin mentioned this issue Jan 8, 2017

workaround for cirros images not getting ip on boot #421

Merged

nvgoldin closed this as completed Apr 24, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

functional tests are failing on 'shell to vm' #399

functional tests are failing on 'shell to vm' #399

nvgoldin commented Dec 21, 2016

mykaul commented Dec 21, 2016

gbenhaim commented Dec 22, 2016

mykaul commented Dec 22, 2016 via email

gbenhaim commented Dec 23, 2016

mykaul commented Dec 23, 2016 via email

nvgoldin commented Dec 25, 2016 •

edited

mykaul commented Dec 25, 2016

nvgoldin commented Dec 25, 2016

mykaul commented Dec 25, 2016 via email

nvgoldin commented Dec 25, 2016 •

edited

mykaul commented Dec 25, 2016 via email

nvgoldin commented Dec 26, 2016

nvgoldin commented Dec 26, 2016

nvgoldin commented Jan 4, 2017

nvgoldin commented Jan 5, 2017

mykaul commented Jan 5, 2017 via email

nvgoldin commented Jan 5, 2017

nvgoldin commented Jan 8, 2017

nvgoldin commented Apr 24, 2017

functional tests are failing on 'shell to vm' #399

functional tests are failing on 'shell to vm' #399

Comments

nvgoldin commented Dec 21, 2016

mykaul commented Dec 21, 2016

gbenhaim commented Dec 22, 2016

mykaul commented Dec 22, 2016 via email

gbenhaim commented Dec 23, 2016

mykaul commented Dec 23, 2016 via email

nvgoldin commented Dec 25, 2016 • edited

mykaul commented Dec 25, 2016

nvgoldin commented Dec 25, 2016

mykaul commented Dec 25, 2016 via email

nvgoldin commented Dec 25, 2016 • edited

mykaul commented Dec 25, 2016 via email

nvgoldin commented Dec 26, 2016

nvgoldin commented Dec 26, 2016

nvgoldin commented Jan 4, 2017

nvgoldin commented Jan 5, 2017

mykaul commented Jan 5, 2017 via email

nvgoldin commented Jan 5, 2017

nvgoldin commented Jan 8, 2017

nvgoldin commented Apr 24, 2017

nvgoldin commented Dec 25, 2016 •

edited

nvgoldin commented Dec 25, 2016 •

edited