Add 4.x logs and backup info, small tweaks elsewhere (#958)

* Add 4.x logs and backup info, small tweaks elsewhere per #831

* Update content/en/apps/guides/hosting/4.x/backups.md

Co-authored-by: Gareth Bowen <gareth@medic.org>

* remove stopping couch, add encryption per feedback

* Update content/en/apps/guides/hosting/4.x/backups.md

Co-authored-by: Gareth Bowen <gareth@medic.org>

---------

Co-authored-by: Gareth Bowen <gareth@medic.org>

content/en/apps/guides/debugging/obtaining-logs.md

@@ -1,6 +1,6 @@
 ---
-title: "Obtaining Logs"
-linkTitle: "Obtaining Logs"
+title: "Obtaining Browser and Phone Logs"
+linkTitle: "Browser and Phone Logs"
 weight:
 description: >
   How to obtain Android and browser client logs

content/en/apps/guides/hosting/4.x/backups.md

@@ -0,0 +1,117 @@
+---
+title: "Backups in CHT 4.x"
+linkTitle: "Backups"
+weight: 330
+description: >
+    Which data to backup when hosting the CHT 4.x
+---
+{{% alert title="Note" %}}
+This guide is about backups in CHT 4.x - there's the [self hosted guide for 3.x]({{< relref "apps/guides/hosting/3.x/self-hosting#backup" >}}) which includes backups for 3.x.
+{{% /alert %}}
+
+## Introduction 
+
+As CHT 4.x uses a container per service, the only data that needs to be backed up is:
+
+* CouchDB database
+* Docker Compose and `.env` files 
+* TLS certificates
+
+This is because Docker containers are inherently stateless so all the important binaries are already stored in [CHT's Docker images](https://gallery.ecr.aws/s5s3h4s7/).  Docker Compose files, including the `.env` file, store all of your deployment's configuration.  Finally, the TLS certificates should be backed up to reduce recovery time.
+
+How to backup each of these three pieces of data is covered below.
+
+Therefore, you do **not** need to back up the docker images for:
+
+* nginx
+* sentinel
+* api
+* haproxy
+* couchdb
+* healthcheck
+* upgrade-service
+
+## Assumptions
+
+This guide assumes you have an Ubuntu server running CHT 4.x in Docker as described in our [Self Hosting in CHT 4.x - Single CouchDB Node]({{< relref "apps/guides/hosting/4.x/self-hosting-single-node" >}}) guide. If you run `docker ps --format '{{.Names}}'` you should see something like this:
+
+```
+cht_nginx_1
+cht_sentinel_1
+cht_api_1
+cht_haproxy_1
+cht_healthcheck_1
+cht_couchdb_1
+upgrade-service-cht-upgrade-service-1
+```
+
+If you run `docker volume ls` you should see something like this:
+
+```
+DRIVER    VOLUME NAME
+local     cht_cht-credentials
+local     cht_cht-ssl
+```
+
+**Note** - In the volumes listed above, there is no volume for CouchDB data.  This is because the compose file declares this as a [bind mount](https://docs.docker.com/storage/bind-mounts/). Bind mounts use the host file system directly and do not show up in `docker volume ls` calls.  It's therefore assumed your CouchDB data location is declared in `/home/ubuntu/cht/upgrade-service/.env` which sets it with `COUCHDB_DATA=/home/ubuntu/cht/couchdb`.
+
+You should have SSH access to the server with `root` access.
+
+### Backup software
+
+It's assumed you are using which ever tool you're familiar with which might include [rsync](https://rsync.samba.org/examples.html), [borg](https://borgbackup.readthedocs.io/en/stable/), [duplicity](https://duplicity.gitlab.io/) or other solution.  The locations of the backups should follow the 3-2-1 rule:
+
+> There should be at least 3 copies of the data, stored on 2 different types of storage media, and one copy should be kept offsite, in a remote location. _- [Wikipedia](https://en.wikipedia.org/wiki/Backup)_
+
+Duplicity has the handy benefit of offering built in encryption using [GPG](https://gnupg.org/). Consider using this if you don't have an existing solution for encrypted backups. 
+
+## CouchDB
+
+{{% alert title="Note" %}}
+CouchDB backups, by necessity, will have PII and PHI.  They should be safely stored to prevent unauthorized access including encrypting backups. 
+{{% /alert %}}
+
+Assuming your CouchDB is stored in `/home/ubuntu/cht/couchdb`, you should use these steps to back it up:
+
+1. While you don't need to stop CouchDB to back it up, ensure you follow best practices to back it up. See the [CouchDB site](https://docs.couchdb.org/en/stable/maintenance/backups.html) for more info. Note that Medic recommends NOT using replication for backup.
+2. It is strongly recommended you encrypt your backups given the sensitivity of the contents. Do this now before copying the backup files to their long term location.
+3. Backup the CouchDB files using the [software specified above]({{< relref "#backup-software" >}})
+
+
+## Docker Compose files
+
+{{% alert title="Note" %}}
+The `.env` file contains cleartext passwords.  It should be safely stored to prevent unauthorized access.
+{{% /alert %}}
+
+All compose files, and the corresponding `.env` file, are in these three locations:
+
+* /home/ubuntu/cht/compose/*.yml
+* /home/ubuntu/cht/upgrade-service/*.yml
+* /home/ubuntu/cht/upgrade-service/.env
+
+While all three of these are trivial to recreate by downloading them again, they may change over time so should be archived with your CouchDB data.  Further, when there's been a critical failure of a production CHT instance, you want to be sure to make the restore process as speedy as possible.
+
+As all of these files are only read when Docker first loads a service, you can simply copy them whenever you want without stopping any of the CHT services.  They should be copied with the same frequency and put in the same location as the CouchDB data using the [backup software specified above]({{< relref "#backup-software" >}}).
+
+
+## TLS certificates
+
+{{% alert title="Note" %}}
+The `.key` file is the private key for TLS certificate.  It should be safely stored to prevent unauthorized access.
+{{% /alert %}}
+
+Like the compose files, the TLS certificate files can easily be regenerated or re-downloaded from your Certificate Authority, like Let's Encrypt for example. However, you want to have a backup of the at the ready to ease the restore process.
+
+1. Copy the cert and key files from the nginx container:
+
+   ```shell
+   docker cp cht_nginx_1:/etc/nginx/private/key.pem .
+   docker cp cht_nginx_1:/etc/nginx/private/cert.pem .
+   ```
+2. Back the up to the same location and frequency as the CouchDB data using the [backup software specified above]({{< relref "#backup-software" >}}).
+
+
+## Testing backups
+
+A backup that isn't tested, is effectively not a backup.  For a backup to be successful, a complete restore from all locations in the 3-2-1 plan need to be fully tested and documented as to how a restore works.  The more practiced and better documented the restore process, the less downtime a production CHT instance will have after data loss.

content/en/apps/guides/hosting/4.x/logs.md

@@ -0,0 +1,58 @@
+---
+title: "Viewing server logs in CHT 4.x"
+linkTitle: "Logs"
+weight: 300
+description: >
+    What to do when you need to find server side errors in CHT 4.x
+---
+
+CHT 4.x has the following services running via Docker and each can have its logs queried:
+
+* nginx
+* sentinel
+* api
+* haproxy
+* couchdb
+* healthcheck
+* upgrade-service
+
+First, find the actual names of the containers with the  `docker ps --format '{{.Names}}'`  command which should show something like this:
+
+```
+cht_nginx_1
+cht_sentinel_1
+cht_api_1
+cht_haproxy_1
+cht_healthcheck_1
+cht_couchdb_1
+upgrade-service-cht-upgrade-service-1
+```
+
+You can then use the `docker logs` command to view the logs of any given container.  For example, if we call `docker logs cht_nginx_1` it will show ALL the logs from that container.  To show only the last 5 lines, you can use the `--tail` flag to specify the number of lines like this `docker logs cht_nginx_1 --tail 5`.  The result will look like this:
+
+```
+10.131.161.1 - - [15/Feb/2023:21:08:35 +0000] "GET /medic/_changes?feed=longpoll&heartbeat=10000&since=115-g1AAAAH5eJyF0LENwjAQBVCLRIAEFBTMgESBCA0lrACJBzgnRXSKoKJmClaAxEswRZbIDCTHZ4GzXPzCT-d_rowx8zIqzDK_3fOycKdkf9jucJIKVyMybmVtxhQp6E-s23jfME00B-LdUWQIOBBxmJkG3gXJHHtfM8WaA2ncQ6RnGmsOZLjG5mLtk2mmSKAUCPHGSkxT3dZAiK_IR28A1AMhzta2wbko2iJe3ndMC92iaIfAzwrTmn8as5aY&limit=25 HTTP/1.1" 499 0 "https://10-131-161-159.my.local-ip.co/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/109.0"
+10.131.161.1 - - [15/Feb/2023:21:08:35 +0000] "GET /medic-user-medic-meta/_changes?include_docs=true&feed=longpoll&heartbeat=10000&since=13-g1AAAAH5eJyF0LENwkAMhWETKGkoWIICERpKWAESD3BOiugUQUXNFKwAiZdgiiyRGUjMY4FYV_zFfbJ8VxPRspqXtCpu96Iq5ZTuD9sdTlrjKgkka-Y8htkE-hOWjWrrOBCVo9noOBATzMwcB5JLVG0cB9LKw2xwHMh4XdCF-TktgTIg5I0nubYBQr5mnxiSaTsAIWfmzp2LRTvkpdq7Fov2CH7WYOMP5CCWMg&limit=25 HTTP/1.1" 499 0 "https://10-131-161-159.my.local-ip.co/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/109.0"
+10.131.161.1 - - [15/Feb/2023:21:08:35 +0000] "GET /fontawesome-webfont.woff2 HTTP/1.1" 304 0 "https://10-131-161-159.my.local-ip.co/styles.css" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/109.0"
+10.131.161.1 - - [15/Feb/2023:21:08:35 +0000] "GET /fonts/NotoSans-Bold.ttf HTTP/1.1" 304 0 "https://10-131-161-159.my.local-ip.co/styles.css" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/109.0"
+10.131.161.1 - - [15/Feb/2023:21:08:35 +0000] "GET /fonts/NotoSans-Regular.ttf HTTP/1.1" 200 221787 "https://10-131-161-159.my.local-ip.co/styles.css" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/109.0
+```
+
+Sometimes you may want to search the logs for a specific string. To search, use the pipe (`|`) and `grep` commands to do this.  Here we search for all the times HA Proxy thought CouchDB wasn't reachable (`DOWN`) with this call `docker logs cht_haproxy_1 2>&1 | grep 'DOWN'`:
+
+```
+<145>Feb 15 20:52:06 haproxy[25]: Server couchdb-servers/couchdb is DOWN, reason: Layer7 wrong status, code: 0, info: "via agent : down", check duration: 208ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
+[WARNING] 045/205206 (25) : Server couchdb-servers/couchdb is DOWN, reason: Layer7 wrong status, code: 0, info: "via agent : down", check duration: 208ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
+[WARNING] 045/205601 (25) : Server couchdb-servers/couchdb is DOWN, reason: Layer7 wrong status, code: 0, info: "via agent : down", check duration: 207ms. 0 active and 0 backup servers left. 5 sessions active, 0 requeued, 0 remaining in queue.
+<145>Feb 15 20:56:01 haproxy[25]: Server couchdb-servers/couchdb is DOWN, reason: Layer7 wrong status, code: 0, info: "via agent : down", check duration: 207ms. 0 active and 0 backup servers left. 5 sessions active, 0 requeued, 0 remaining in queue.
+```
+
+If you want to watch the logs for a specific container in real time, you can use the `--follow` flag.  This command would watch the requests come into API in realtime: `docker logs cht_api_1 --follow`.  It's nice to couple this with the `--tail` command so you only see the last 5 lines of the existing logs before watching for new lines with `docker logs cht_api_1 --follow --tail 5` which would show this:
+
+```
+RES d17d71f5-2dcb-4ebb-bb0e-7874b3000570 10.131.161.1 - GET /medic/_design/medic-client/_view/reports_by_subject?keys=%5B%22557e79b8-2d99-4bd1-a4d6-a44491d483d8%22%5D HTTP/1.0 200 - 12.452 ms
+RES e43c5d7f-4e32-433a-a96d-ef991f4298a3 10.131.161.1 - GET /medic/_design/medic/_view/doc_summaries_by_id?keys=%5B%22557e79b8-2d99-4bd1-a4d6-a44491d483d8%22%5D HTTP/1.0 200 - 31.226 ms
+REQ c656ecc7-e6af-4564-ad63-2cab2c42844a 10.131.161.1 - GET /medic/_all_docs?include_docs=true&startkey=%22target~2023-02~557e79b8-2d99-4bd1-a4d6-a44491d483d8~%22&endkey=%22target~2023-02~557e79b8-2d99-4bd1-a4d6-a44491d483d8~%EF%BF%B0%22 HTTP/1.0
+RES c656ecc7-e6af-4564-ad63-2cab2c42844a 10.131.161.1 - GET /medic/_all_docs?include_docs=true&startkey=%22target~2023-02~557e79b8-2d99-4bd1-a4d6-a44491d483d8~%22&endkey=%22target~2023-02~557e79b8-2d99-4bd1-a4d6-a44491d483d8~%EF%BF%B0%22 HTTP/1.0 200 - 11.153 ms
+2023-02-15 21:54:49 DEBUG: Checking for a configured outgoing message service 
+```

content/en/apps/guides/hosting/4.x/self-hosting-multiple-nodes.md

@@ -1,7 +1,7 @@
 ---
 title: "Self Hosting in CHT 4.x - Multiple CouchDB Nodes"
 linkTitle: "Self Hosting - Multiple Nodes"
-weight: 2
+weight: 20
 description: >
   Hosting the CHT on self run infrastructure with horizontally scaled CouchDB nodes
 ---

content/en/apps/guides/hosting/4.x/self-hosting-single-node.md

@@ -1,7 +1,7 @@
 ---
 title: "Self Hosting in CHT 4.x - Single CouchDB Node"
 linkTitle: "Self Hosting - Single Node"
-weight: 2
+weight: 10
 description: >
    Self Hosting in CHT 4.x - Single CouchDB Node
 ---