http://box/kolibri fails with "502 Bad Gateway" (especially when Kolibri 0.14.2 upgraded to 0.14.3 ?) [kolibri.service systemd timeout issue on boot]

[holta]

I'm opening a separate ticket so we can track this better.

The problem is not yet well understood, but has occurred on both "RaspiOS with desktop" 2020-08-20 and "Ubuntu Desktop" 20.04.1 — each running recent IIAB 7.2 pre-releases from Aug/Sept. Thanks much to @fisherbryan as we diagnose this.

@jvonau summarizes our suspicions so far:

502's are service not running or dns like errors, I suspect something like #1932 related, kolibri starts before the database or slow to start with channels present.

502's are either dns related or the backend served by nginx failing to start before the timeout, if you see a 502 error please capture the output of sudo systemctl status kolibri and sudo journalctl -b 0 -u kolibri

I suspect something like #1932 related, kolibri starts before the database or slow to start with channels present.
Additional Ref: #2318

Possibly related:

• Migrate IIAB to kolibri-server for RPi performance? #1646 "Migrate IIAB to kolibri-server for RPi performance?"
• Who can help test IIAB 7.2 on Ubuntu 20.04+, Debian 10.10+, Linux Mint 20 & Raspberry Pi ? [Kolibri 502 Bad Gateway?] #1737 "Who can help test IIAB 7.2 on Ubuntu 20.04+, Debian 10.6, Linux Mint 20 & Raspberry Pi ? [Kolibri 502 Bad Gateway?]"
• Kolibri 0.12.8 fails to start on RPi 3 B+ as part of IIAB 7.0 #1932 "Kolibri 0.12.8 fails to start on RPi 3 B+ as part of IIAB 7.0"
• Kolibri's new 0.12.9 apt/deb package fails to install #2045 "Kolibri's new 0.12.9 apt/deb package fails to install"
• Is Kolibri shutting down cleanly when IIAB powers off? #2318 "Is Kolibri shutting down cleanly when IIAB powers off?"

[holta]

CLARIF: the problem can temporarily be worked around by reinstalling Kolibri (0.14.2 or?) 0.14.3 as follows:

cd /opt/iiab/iiab
sudo ./runrole --reinstall kolibri

But then the problem would seem to recur...after the next reboot?? :/

[tim-moody]

is the gateway uwsgi? could be conflicts. do an install with only kolibri

[jvonau]

CLARIF: the problem can temporarily be worked around by reinstalling Kolibri (0.14.2 or?) 0.14.3 as follows:

cd /opt/iiab/iiab
sudo ./runrole --reinstall kolibri

But then the problem would seem to recur...after the next reboot?? :/

I don't see how re-installing fixes anything, in the log run just would make the service restart in the current boot process, but to narrow this down try without '--reinstall' that would be the equivalent of sudo systemctl restart kolibri sudo systemctl restart nginx,

If the 502 error returns on the next boot grab the requested info above to check the startup times of the kolibri service. Once this info it obtained restart with sudo systemctl restart kolibri and grab the requested info again and attach both dumps to this ticket.

[jvonau]

Ref: #1545

[jvonau]

I don't see how re-installing fixes anything

what if captive portal or admin cons content with kolibri over uwsgi config? reinstall could revert to kolibri's liking.

Don't think captive_portal comes into play

captiveportal_enabled: False

Kolibri itself does not use uwsgi the kolibri-server package does, kolibri-server is omitted and supply the nginx config file in place of it. We omit kolibri-server because our use of kolibri predates the release of kolibri-server when the role was introduced and we used apache at the time.

Think the easiest to rule out rewrites would be to test access between via the menu, going direct to http:/ /yourIP/kolibri and direct to http://yourIP:kolibriPORT. The last one really should not be allowed, kolibri should be bound to 127.0.0.1 only to force the use of the nginx proxy similar to use of a socket in kolibri-server.

[holta]

1. I found an old RaspiOS microSD card that has this Kolibri 502 error after every boot (or almost every boot).

2. systemctl status kolibri shows this soon after boot:

root@box:~# systemctl status kolibri
● kolibri.service - Kolibri
   Loaded: loaded (/etc/systemd/system/kolibri.service; enabled; vendor preset: enabled)
   Active: failed (Result: timeout) since Sat 2020-10-03 11:11:49 EDT; 2min 24s ago
  Process: 335 ExecStart=/usr/bin/kolibri start (code=killed, signal=TERM)

Oct 03 11:10:04 box.lan systemd[1]: Starting Kolibri...
Oct 03 11:10:06 box.lan kolibri[335]: WARNING:root:No C Extensions available for this platform.
Oct 03 11:11:49 box.lan systemd[1]: kolibri.service: Start operation timed out. Terminating.
Oct 03 11:11:49 box.lan systemd[1]: kolibri.service: Control process exited, code=killed, status=15/TERM
Oct 03 11:11:49 box.lan systemd[1]: kolibri.service: Failed with result 'timeout'.
Oct 03 11:11:49 box.lan systemd[1]: Failed to start Kolibri.

3. systemctl restart kolibri takes almost a full minute to come back to the prompt (that doesn't seem right, on an RPi 4 with no load at all) but fyi that does make the problem go away — i.e. http://box/kolibri loads properly after that.

4. I discovered kolibri.service was taking 9 minutes to start on this RPi 4 with 1GB RAM, so I added TimeoutStartSec=1000 to the [Service] section of /etc/systemd/system/kolibri.service and the problem went away. @jvonau & others might have other suggestions considering Is Kolibri shutting down cleanly when IIAB powers off? #2318 and similar?

5. The 502 Bad Gateway error message is not at all helpful during those first 9 minutes after boot.

   If patched as above, it would be far more helpful if IIAB could somehow warn students (who visit http://box/kolibri) to try again in a few minutes.

[jvonau]

That 502 error is served up by nginx, find the file used and make a template to replace it.

[holta]

@jredrejo thanks for your help on #2045. I've web-posted 2 Kolibri log files in case they help shed light:

• /library/kolibri/logs/daemon.txt
• /library/kolibri/logs/kolibri.txt

Noting that #2555 ("502 Bad Gateway") is probably now solved by PR #2556 (or so we hope!)

[holta]

Just FYI the RPi 4 with 1GB RAM contained these 2 Kolibri Channels from https://catalog.learningequality.org :

• How to get started with Kolibri (Version 29, which is 374 MB)
• CK-12 Spanish (Version 4, which is 84 MB)