MovieGEEKs web app is down at 2023/Jan/02 (Mon)

[9health]

Description

• Somehow the MovieGEEKs web app is not running after 02/Jan/2023 13:40:08

Initial Log

[02/Jan/2023 12:25:19] "GET / HTTP/1.1" 200 24513
[02/Jan/2023 12:25:19] code 505, message Invalid HTTP version (2.0)

[02/Jan/2023 12:25:19] "PRI * HTTP/2.0" 505 -
Not Found: /favicon.ico

[02/Jan/2023 12:25:20] "GET /favicon.ico HTTP/1.1" 404 2531
ensured id:  80173445036

[02/Jan/2023 12:47:50] "GET / HTTP/1.1" 200 24513
Not Found: /www.google.com:443

[02/Jan/2023 12:47:59] "CONNECT www.google.com:443 HTTP/1.1" 404 1666
Not Found: /cloudflare.com:443

[02/Jan/2023 13:04:20] "CONNECT cloudflare.com:443 HTTP/1.1" 404 1670
Invalid HTTP_HOST header: ''. The domain name provided is not valid according to RFC 1034/1035.
Bad Request: /manager/html

[02/Jan/2023 13:40:08] "GET /manager/html HTTP/1.1" 400 62540

(venv_3.6.9) ninehealth@LinuxVM15:~/work/moviegeek$                              

Output

• Explain why the MovieGEEKs web app is down.

[9health]

Check Python processes

ninehealth@LinuxVM15:~$ ps -ef  | grep python

root       983     1  0  2022 ?        00:00:00 /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers
root       984     1  0  2022 ?        00:00:00 /usr/bin/python3 -u /usr/sbin/waagent -daemon
root      1019     1  0  2022 ?        00:00:00 /usr/bin/python3 /usr/share/unattended-upgrades/unattended-upgrade-shutdown --wait-for-signal
root      1035   984  0  2022 ?        00:20:27 python3 -u bin/WALinuxAgent-2.9.0.4-py2.7.egg -run-exthandlers
ninehea+ 29952 29933  0 05:04 pts/0    00:00:00 grep --color=auto python

Kill Python processes. The author thinks that it causes the MovieGEEK web app down =.=

kill -9 983 984 1019 1035

After check Python processes again

ninehealth@LinuxVM15:~$ ps -ef  | grep python

root     29987     1  5 05:09 ?        00:00:00 /usr/bin/python3 -u /usr/sbin/waagent -daemon
root     30020 29987 24 05:09 ?        00:00:00 python3 -u bin/WALinuxAgent-2.9.0.4-py2.7.egg -run-exthandlers
ninehea+ 30072 29933  0 05:09 pts/0    00:00:00 grep --color=auto python

Check screen sessions

ninehealth@LinuxVM15:~$ screen -list

There are screens on:
	11853.hai	(12/24/22 07:46:43)	(Detached)
	6353.quang	(12/23/22 22:19:40)	(Detached)
2 Sockets in /run/screen/S-ninehealth.

ninehealth@LinuxVM15:~$ screen -r hai

[02/Jan/2023 12:47:59] "CONNECT www.google.com:443 HTTP/1.1" 404 1666
Not Found: /cloudflare.com:443

[02/Jan/2023 13:04:20] "CONNECT cloudflare.com:443 HTTP/1.1" 404 1670
Invalid HTTP_HOST header: ''. The domain name provided is not valid according to RFC 1034/1035.
Bad Request: /manager/html

[02/Jan/2023 13:40:08] "GET /manager/html HTTP/1.1" 400 62540

(venv_3.6.9) ninehealth@LinuxVM15:~/work/moviegeek$                              

=> The Python processes are created by Python virtual environment!!!

That explains why it is created again after being forcibly stoped!

[9health]

Check CPU usage

  System information as of Wed Jan  4 05:12:25 +07 2023

  System load:  0.0               Processes:           118
  Usage of /:   7.0% of 61.84GB   Users logged in:     0
  Memory usage: 34%               IP address for eth0: 10.0.0.4
  Swap usage:   0%

=> Not much CPU is used

Quang doesn't run a Python web server!!!

Check ninehealth user's processes

ninehealth@LinuxVM15:~$ ps -ef | grep nine

ninehea+  1386     1  0  2022 ?        00:00:00 /lib/systemd/systemd --user
ninehea+  1388  1386  0  2022 ?        00:00:00 (sd-pam)
ninehea+  6353     1  0  2022 ?        00:00:09 SCREEN
ninehea+  6354  6353  0  2022 pts/6    00:00:02 /bin/bash
ninehea+  6357  6353  0  2022 pts/7    00:00:00 /bin/bash
ninehea+  6364  6353  0  2022 pts/8    00:00:00 /bin/bash
ninehea+  6370  6353  0  2022 pts/9    00:00:00 /bin/bash
ninehea+ 11853     1  0  2022 ?        00:00:07 SCREEN -t hai
ninehea+ 11857 11853  0  2022 pts/2    00:00:00 /bin/bash
ninehea+ 11862 11853  0  2022 pts/3    00:00:00 /bin/bash
ninehea+ 11873 11853  0  2022 pts/4    00:00:00 /bin/bash
ninehea+ 25018  6357  0 Jan03 pts/7    00:00:00 sqlite3 db.sqlite3
ninehea+ 25212  6354  0 Jan03 pts/6    00:00:00 vim models.py
ninehea+ 25561 11853  0  2022 pts/1    00:00:00 /bin/bash
ninehea+ 26233 11853  0  2022 pts/5    00:00:00 /bin/bash
root     30108  1025  0 05:12 ?        00:00:00 sshd: ninehealth [priv]
ninehea+ 30174 30108  0 05:12 ?        00:00:00 sshd: ninehealth@pts/0
ninehea+ 30175 30174  0 05:12 pts/0    00:00:00 -bash
ninehea+ 30220 30175  0 05:14 pts/0    00:00:00 ps -ef
ninehea+ 30221 30175  0 05:14 pts/0    00:00:00 grep --color=auto nine

Check last modified related files

ninehealth@LinuxVM15:~$ ll -alhrt

drwxrwxr-x  7 ninehealth ninehealth 4.0K Dec 29 07:32 work/
-rw-rw-r--  1 ninehealth ninehealth 296K Dec 29 17:28 screen_log_hai_20221229.txt
-rw-rw-r--  1 ninehealth ninehealth  234 Dec 30 10:38 .wget-hsts
drwxrwxr-x  5 ninehealth ninehealth 4.0K Jan  2 20:50 .vscode-server/
drwx------  4 ninehealth ninehealth 4.0K Jan  2 20:51 .cache/
-rw-------  1 ninehealth ninehealth  15K Jan  3 21:11 .sqlite_history
-rw-------  1 ninehealth ninehealth  28K Jan  3 21:34 .viminfo
drwxrwxr-x  2 ninehealth ninehealth 4.0K Jan  3 22:02 .vim/
-rw-------  1 ninehealth ninehealth  14K Jan  4 05:12 .bash_history
-rw-------  1 ninehealth ninehealth   66 Jan  4 05:13 .lesshst
-rw-rw-r--  1 ninehealth ninehealth 380K Jan  4 05:15 screen_log_hai_20230104.txt
drwxr-xr-x 11 ninehealth ninehealth 4.0K Jan  4 05:15 ./

ninehealth@LinuxVM15:~$ ls -alhrt work/moviegeek

drwxrwxr-x  3 ninehealth ninehealth 4.0K Dec 19 11:55 prs_project
drwxrwxr-x  3 ninehealth ninehealth 4.0K Dec 27 21:25 builder
-rw-r--r--  1 ninehealth ninehealth  61M Dec 27 21:25 db_02_implicit_ratings.sqlite3
-rw-r--r--  1 ninehealth ninehealth  61M Dec 27 21:25 db_01_populate.sqlite3
lrwxrwxrwx  1 ninehealth ninehealth   15 Dec 27 21:36 db.sqlite3 -> db_real.sqlite3
drwxrwxr-x  7 ninehealth ninehealth 4.0K Dec 29 07:32 ..
-rw-r--r--  1 ninehealth ninehealth  55M Jan  2 19:47 db_real.sqlite3
drwxrwxr-x 16 ninehealth ninehealth 4.0K Jan  2 19:47 .

=> Not much information.
Maybe Quang accessed db_real.sqlite3 at Jan/02, 19:47.
Server down time is 02/Jan/2023 13:40:08

Summary

It can't be found the root cause why the web server is down.

The investigation can be temporarily paused here.

Log file will be uploaded to shared OneDrive folder and checked again later in the future!!!