dnsmasq fails often in captive-portal (4-server-options) on multiple OS's

[holta]

Ubuntu 16.04 (intermittent) and Debian 10 (every time for @deldesir) are not the only ones.

On some distros it happens every time, whereas on others it's intermittent ("./iiab-install --reinstall" can sometimes work to overcome this failure in 4-server-options).

In all cases, iiab-install fails here in Stage 4 (4-server-options) when dnsmasq fails to start:
https://github.com/iiab/iiab/blob/master/roles/captive-portal/tasks/main.yml#L141-L145

Sometimes ./iiab-network ("./runrole network") fails in apparently the same way — with dnsmasq strangely unable to start. Confoundingly, "systemctl start dnsmasq" seems to run fine [certainly on Ubuntu 16.04] when run manually at the command-line !

Thank you @deldesir @jvonau @georgejhunt for continuing to monitor this pattern towards identifying the root cause — so we can put an end to this in coming weeks if not days!

Refs: #1182 #1364 #1387 and others

Possibly Unrelated: @jvonau asks if 127.0.0.53 in Ubuntu 18.04's /etc/resolv.conf has anything to do with this? https://github.com/iiab/iiab/blob/master/roles/sugarizer/tasks/main.yml#L181-L182

[holta]

If we do not identify the root cause in coming days, we should consider a workaround so that IIAB 6.7 isn't stillborn/sabotaged across multiple OS's/distros (-:

[deldesir]

For now installation goes without it when captive_portal_install: False and captive_portale_enabled: False un /etc/iiab/local_vars.yml.

"systemctl start dnsmasq" stills fails even when run manually [on Debian 10, unlike Ubuntu 16.04].

I'll wait installation to finish without captive portal and after I'll play with tasks of captive-portal role.

[holta]

Thanks @deldesir! Please let us know if roles/network succeeds during the last stage of iiab-install, and when run manually as ./iiab-network (or "./runrole network") — in all cases making sure you're in /opt/iiab/iiab

(Or possibly dnsmasq has a serious glitch on Debian 10 for the moment?)

[deldesir]

Not dnsmasq fault nor captive-portal.
netstat -lnp | grep ":53 " showed connmand took the port. Now dnsmasq start after I removed connman.

Now I am rerunning iiab-install --reinstall with captive portal set to true for both install and enabled.

[holta]

ConnMan is an internet connection manager for embedded devices

Appropriately named ;)

How did you remove connman ?

Why is this installed as part of your Debian 10, do you know?

[holta]

Why is this installed as part of your Debian 10, do you know?

Clarif: @deldesir installed this (connman package) no worries, now we know!

Larger question about IIAB fragility in general — as we try to make dnsmasq resilient across distros, what error reporting (e.g. checking what daemon might already be using port 53?) might best communicate different ways that dnsmasq fails across different distros?

[georgejhunt]

I think we should change the default to off for captive-portal on all hardware that is not rpi. Our instructions could then advise the user to do the IIAB install, until it is successful. And then configure the wifi, until it works. Finally, they can "./runrole captive-portal", and have most of the dependencies sorted out.

…

On Sat, Feb 9, 2019 at 5:20 AM A Holt ***@***.***> wrote: Ubuntu 16.04 and Debian 10 are not the only ones. On some distros it happens every time, whereas on others it's intermittent ("./iiab-install --reinstall" can *sometimes* work to overcome this failure in 4-server-options). In all cases, IIAB stops installing here, when dnsmasq fails to start: https://github.com/iiab/iiab/blob/master/roles/captive-portal/tasks/main.yml#L141-L145 Thank you @deldesir <https://github.com/deldesir> @jvonau <https://github.com/jvonau> @georgejhunt <https://github.com/georgejhunt> for continuing to monitor this pattern towards identifying the root cause — so we can put an end to this in coming weeks if not days! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1469>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB04HP5ezMsFNjfln2VCNMyM-aJWR64Qks5vLssGgaJpZM4ayicT> .

[holta]

I think we should change the default to off for captive-portal on all hardware that is not rpi.

@georgejhunt how/where do you suggest this happen?

A couple related points:

1. Right now captive-portal is True/True in MEDIUM-sized and BIG-sized local_vars.yml and we should keep stuff like that, to expand Captive Portal testing wherever possible!
2. Folks in India (that @m-anish works with) very much want Captive Portal to work on Ubuntu 18.04 — is extending what we have on Raspbian to other distro(s) likely very hard?

[holta]

@georgejhunt asks a thoughtful question: if we do suppress dnsmasq glitches within 4-server-options, does this really help if they'll (presumably) just reappear later during the roles/network stage?

[georgejhunt]

I installed debian 10 on a test NUC. After fixing the lack of a proper entry in /etc/release, it ran to completion on medium without error. So no information relating to dnsmasq was gained

…

On Sun, Feb 10, 2019 at 8:50 PM A Holt ***@***.***> wrote: @georgejhunt <https://github.com/georgejhunt> asks a thoughtful question: if we do suppress dnsmasq glitches within 4-server-options, does this really help if they'll (presumably) just reappear later during the roles/network stage? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1469 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB04HDxar_4OK7TxguvvvEOnh1hGgEYzks5vMPaigaJpZM4ayicT> .

[holta]

While similar in some ways, is this unrelated to #1452 in the end?

[holta]

@jvonau this error no longer occurs on Debian 10 Buster according to @floydianslips at #1387

I also no longer see it on Ubuntu 16.04 (if we really still care about that, but FYI!)

Should we declare victory and close this?

[holta]

This still happens on Ubuntu 16.04, but that is now more than 3 years old and it really is time for IIAB implementers to use more modern OS's.