From e289019a6038a2a484eb4a77d74d4db9c8fd91f9 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Tue, 22 Feb 2022 18:19:33 -0500 Subject: [PATCH 001/164] feat: Created install.sh File provided for ease of installation. --- README.md | 117 +++++++++++++++++++++++++++++---------------------- installer.sh | 102 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 168 insertions(+), 51 deletions(-) create mode 100644 installer.sh diff --git a/README.md b/README.md index fe18e51..6a3897d 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@
-

OpenPod

+# OpenPod [![Code Quality](https://github.com/RecursionSpace/OpenPod/actions/workflows/pylint.yml/badge.svg)](https://github.com/RecursionSpace/OpenPod/actions/workflows/pylint.yml)   @@ -12,46 +12,67 @@
-## Table of Contents - -- [What is OpenPod?](#what-is-openpod) -- [Directory Structure](#directory-structure) - +- [OpenPod](#openpod) + - [What is OpenPod?](#what-is-openpod) + - [Installation](#installation) +- [Software Design](#software-design) + - [OS](#os) +- [Development](#development) +- [Initialization](#initialization) + - [Required Packages](#required-packages) +- [MQTT](#mqtt) +- [Updates](#updates) +- [Logging](#logging) + - [Directory Structure](#directory-structure) ## What is OpenPod? The “Pod” is the physical extension of the Recursion.Space system for a facility. The Pod allows for door/egress and equipment control nodes. Additionally, the Pod acts as a local backup in an internet outage for users to continue accessing their facility. The Pod has a direct internet connection then uses a wireless method to communicate with nodes. A single facility can have multiple Pods, but each Pod can only be linked to one facility. +## Installation + +OpenPod is designed to communicate with hardware over a local mesh network using [XBee module](https://www.digi.com/products/embedded-systems/digi-xbee/rf-modules/2-4-ghz-rf-modules/xbee3-zigbee-3). While OpenPod can be installed on any computer running Ubuntu, most installations are done on a [Raspberry Pi](https://www.raspberrypi.com/products/raspberry-pi-4-model-b/). + +For convenience, an installation script as been provided that will download OpenPod and make the necessary system changes to get OpenPod running. This script can be run from the command line using the following command: + +```bash +sudo wget -qO- openpod.recursion.space | bash /dev/stdin [options] [arguments*] +``` + +| Option Flag | Description | Example | +| :---------: | ------------------------- | -------------------------------------------------------------------------------- | +| -h | Help | sudo wget -qO- openpod.recursion.space \| bash /dev/stdin -h | +| -u | Set Custom URL endpoint | sudo wget -qO- openpod.recursion.space \| bash /dev/stdin -u dev.recursion.space | +| -d | Enable Debug Installation | sudo wget -qO- openpod.recursion.space \| bash /dev/stdin -d | + +\*No supported arguments are currently supported. + +# Software Design + +The embedded code manages the communication between the API server and the node communication, acting as a middleman. There needs to be minimal factory configuration to make deployable units quickly. + 1) Assume the Pod is plug and play internet connection. (DHCP) 2) Pods can self-generate ID and register it with the central system. 3) Users enter an ID to link the physical Pod with the web interface. Pod communicates to the web API via hook notifications, and a return confirmation is sent back. -* Records Door/Equipment access -* Registers new nodes and Pods -* Pulls info dumps +- Records Door/Equipment access +- Registers new nodes and Pods +- Pulls info dumps The web interface communicates with the Pod via MQTT. -* Alerts for new users -* User updates -* Remote control (door unlock) +- Alerts for new users +- User updates +- Remote control (door unlock) Each Pod is on its MQTT topic, and the topic is created once the Pod registers with the central system. -# Software Design - -The embedded code manages the communication between the API server and the node communication, acting as a middleman. There needs to be minimal factory configuration to make deployable units quickly. - ## OS Each Pod is installed with the latest version of Ubuntu, and the systems are tested daily for compatibility with the latest releases. An image of the OS with the code ready for deployment can make a quick installation. 32-bit for Pi3 and 64-bit for 4+ -# Installation - -To install on a new Pod, the HUB_Installer.py file and the latest 0_1_0 folder are all that is needed. Run the HUB_Installer to complete the Pod setup. - # Development Pod software development is done on a DigitalOcean hosted server. It is then transferred to the physical hardware running the system and tested before going into production. @@ -60,14 +81,14 @@ Pod software development is done on a DigitalOcean hosted server. It is then tra Pod uses an initializer or “launcher” to configure the system before running the main code. The launcher performs the following tasks before running the main program: -* Update & Upgrade the OS -* Sync Clock -* Installs program required packages -* Check for required files, create them if missing -* Configure start on boot file -* Creates serial if needed -* Registers the Pod with the API server -* Initializes the main program +- Update & Upgrade the OS +- Sync Clock +- Installs program required packages +- Check for required files, create them if missing +- Configure start on boot file +- Creates serial if needed +- Registers the Pod with the API server +- Initializes the main program To launch the software, when the device boots, the launcher configures the system first to run the software in the background and then creates a script that executes the launch on startup. Working version with a cronjob. @@ -79,7 +100,7 @@ To launch the software, when the device boots, the launcher configures the syste Required packages are stored under “Requirements” in the settings.py file. -Pypubsub is used for internal flags and alerts and has a broker-type system that can be subscribed to. +PyPubSub is used for internal flags and alerts and has a broker-type system that can be subscribed to. # MQTT @@ -87,14 +108,14 @@ The MQTT protocol is used for server to Pod communication, allowing the Pod to l Communication to Pods is accomplished from quick commands represented by a hexadecimal number. -| Comand Number | Command | -|---------------|-----------------------| -| AA (170) | Facility ID Available | -| BA (186) | Pull User Dump | -| CA (202) | Update Available | -| DA(218) | Time Zone Change | -| | | -| FA (250) | Zip & Send Logs | +| Command Number | Command | +| -------------- | --------------------- | +| AA (170) | Facility ID Available | +| BA (186) | Pull User Dump | +| CA (202) | Update Available | +| DA(218) | Time Zone Change | +| | | +| FA (250) | Zip & Send Logs | # Updates @@ -110,25 +131,19 @@ First, commit and push the latest changes to git to prepare an update. Using Win The Recursion System uses several logging points to be used both for troubleshooting as well as security and auditing. There are two log files created for the Pod, RecursionLog.log, and TransactionLog.log; each one is used as follows: -## RecursionLog.log - -This log is used to record system events. - -* Coming online -* Updating -* New Nodes being added -* Nodes coming online +**RecursionLog.log** is used to record system events. -## System.Snapshot +- Coming online +- Updating +- New Nodes being added +- Nodes coming online -This file contains a JSON summary of the Pod for debugging purposes. The information available in this file must also be readily accessible from the server for troubleshooting purposes. +**System.Snapshot** file contains a JSON summary of the Pod for debugging purposes. The information available in this file must also be readily accessible from the server for troubleshooting purposes. -``` +```json [ - - “local_ip” = “xx.xx.xx.xx”, - + “local_ip” = “xx.xx.xx.xx”, ] ``` diff --git a/installer.sh b/installer.sh new file mode 100644 index 0000000..168daec --- /dev/null +++ b/installer.sh @@ -0,0 +1,102 @@ +#!/bin/bash + +# ---------------------------------------------------------------------------- # +# Help # +# ---------------------------------------------------------------------------- # +Help() +{ + # Display Help + echo "OpenPod instalation script" + echo + echo "h Display this help" + echo "u Set custom URL for OpenPod" + echo "d Enable debug mode" +} + +# ---------------------------------------------------------------------------- # +# Defaults # +# ---------------------------------------------------------------------------- # +URL="recursion.space" +DEBUG=0 + +# ---------------------------------------------------------------------------- # +# Options # +# ---------------------------------------------------------------------------- # +while getopts ":hud" flags; do + case "${flags}" in + h) # display Help + Help + exit;; + u) # Custom URL endpoint + URL="${OPTARG}";; + d) DEBUG=1 ;; + \?) echo "Invalid option: -${OPTARG}" >&2; + exit 1 ;; + esac +done + +# ---------------------------------------------------------------------------- # +# Installer # +# ---------------------------------------------------------------------------- # + +# ---------------------- Update Package List and Upgrade --------------------- # +sudo apt-get update -y && sudo apt-get upgrade -y + +# ---------------------------- Update System Time ---------------------------- # +sudo timedatectl set-timezone UTC +sudo apt-get install chrony -y +sudo chronyd -q + +# ------------------------- Install Bash Requirements ------------------------ # +sudo apt-get install jq -y +sudo apt-get install unzip -y + +# ---------------------------- Install Python 3.10 --------------------------- # +sudo apt install software-properties-common -y +sudo add-apt-repository ppa:deadsnakes/ppa +sudo apt install python3.10 -y + +# ------------------------------- Clone OpenPod ------------------------------ # +sudo mkdir -p /opt +sudo git clone --single-branch --branch release https://github.com/RecursionSpace/OpenPod.git /opt + +# ----------------------------- Setup Enviroment ----------------------------- # +sudo apt-get install python3.10-venv -y +sudo python3.10 -m venv /opt/OpenPod/env +sudo source /opt/OpenPod/env/bin/activate +sudo pip install --no-input -U -r /opt/OpenPod/requirements.txt --no-cache-dir --no-dependencies + +# ---------------------------- Create Directories ---------------------------- # +sudo mkdir -p /opt/OpenPod/logs +sudo mkdir -p /opt/OpenPod/data + +# ------------------------------- Create Files ------------------------------- # +sudo touch /opt/OpenPod/logs/RecursionLog.log +sudo touch /opt/OpenPod/logs/System.Snapshot +sudo touch /opt/OpenPod/data/dump.json +sudo touch /opt/OpenPod/data/nodes.json +sudo touch /opt/OpenPod/data/owners.json +sudo touch /opt/OpenPod/data/permissions.json + +# --------------------------- Setup OpenPod Service -------------------------- # +sudo cat < /etc/systemd/system/openpod.service +[Unit] +Description=OpenPod +After=network.target + +[Service] +Type=simple +User=root +WorkingDirectory=/opt/OpenPod +ExecStart=/opt/OpenPod/env/bin/python3.10 /opt/OpenPod/OpenPod.py +Restart=always +RestartSec=10 + +[Install] +WantedBy=multi-user.target +EOF + +sudo systemctl enable --now openpod.service +sudo systemctl daemon-reload + +echo "OpenPod is now installed" From 2ff5e964e52c1b1785c7291c31307766e29ebaeb Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Tue, 22 Feb 2022 18:29:35 -0500 Subject: [PATCH 002/164] refactor: Linting Bash --- installer.sh | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/installer.sh b/installer.sh index 168daec..92e9776 100644 --- a/installer.sh +++ b/installer.sh @@ -63,7 +63,7 @@ sudo git clone --single-branch --branch release https://github.com/RecursionSpa # ----------------------------- Setup Enviroment ----------------------------- # sudo apt-get install python3.10-venv -y sudo python3.10 -m venv /opt/OpenPod/env -sudo source /opt/OpenPod/env/bin/activate +source /opt/OpenPod/env/bin/activate sudo pip install --no-input -U -r /opt/OpenPod/requirements.txt --no-cache-dir --no-dependencies # ---------------------------- Create Directories ---------------------------- # @@ -78,8 +78,22 @@ sudo touch /opt/OpenPod/data/nodes.json sudo touch /opt/OpenPod/data/owners.json sudo touch /opt/OpenPod/data/permissions.json +# -------------------------------- system.json ------------------------------- # +sudo touch /opt/OpenPod/system.json +serial_uuid=$(cat /proc/sys/kernel/random/uuid) +xbee_uuid=$(cat /proc/sys/kernel/random/uuid) +sudo echo '{ + "serial": "'$uuid'", + "timezone": "UTC", + "XBEE_KY": "'$xbee_uuid'", + "XBEE_OP": "0", + "url": "'$URL'", + "version": "0_0_1", + "debug": '$DEBUG' +}' > /opt/OpenPod/system.json + # --------------------------- Setup OpenPod Service -------------------------- # -sudo cat < /etc/systemd/system/openpod.service +cat < /etc/systemd/system/openpod.service [Unit] Description=OpenPod After=network.target From 76df534bd8cae8d63e35b0099b989f0c3cf5ba8d Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Tue, 22 Feb 2022 18:38:15 -0500 Subject: [PATCH 003/164] fix: installer quotes --- installer.sh | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/installer.sh b/installer.sh index 92e9776..0dc2585 100644 --- a/installer.sh +++ b/installer.sh @@ -63,7 +63,7 @@ sudo git clone --single-branch --branch release https://github.com/RecursionSpa # ----------------------------- Setup Enviroment ----------------------------- # sudo apt-get install python3.10-venv -y sudo python3.10 -m venv /opt/OpenPod/env -source /opt/OpenPod/env/bin/activate +source /opt/OpenPod/env/bin/activate # shellcheck disable=SC1091 sudo pip install --no-input -U -r /opt/OpenPod/requirements.txt --no-cache-dir --no-dependencies # ---------------------------- Create Directories ---------------------------- # @@ -82,14 +82,14 @@ sudo touch /opt/OpenPod/data/permissions.json sudo touch /opt/OpenPod/system.json serial_uuid=$(cat /proc/sys/kernel/random/uuid) xbee_uuid=$(cat /proc/sys/kernel/random/uuid) -sudo echo '{ - "serial": "'$uuid'", +echo '{ + "serial": '"'$serial_uuid'"', "timezone": "UTC", - "XBEE_KY": "'$xbee_uuid'", + "XBEE_KY": '"'$xbee_uuid'"', "XBEE_OP": "0", - "url": "'$URL'", + "url": '"'$URL'"', "version": "0_0_1", - "debug": '$DEBUG' + "debug": '"'$DEBUG'"' }' > /opt/OpenPod/system.json # --------------------------- Setup OpenPod Service -------------------------- # From 07feee5ede3328a5aadd658edd8397cf529db25d Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Tue, 22 Feb 2022 18:48:07 -0500 Subject: [PATCH 004/164] chore: fix script checker --- .github/workflows/ShellCheck.yml | 9 ++++++++- .shellcheckrc | 1 + installer.sh | 2 +- 3 files changed, 10 insertions(+), 2 deletions(-) create mode 100644 .shellcheckrc diff --git a/.github/workflows/ShellCheck.yml b/.github/workflows/ShellCheck.yml index 3296ecc..54f9939 100644 --- a/.github/workflows/ShellCheck.yml +++ b/.github/workflows/ShellCheck.yml @@ -2,12 +2,19 @@ name: Script Check on: push: + branches: + - '**' + - 'master' + - '!master-ci' + - '!release' + + pull_request: branches: - master - - main workflow_dispatch: + jobs: lint: runs-on: ubuntu-latest diff --git a/.shellcheckrc b/.shellcheckrc new file mode 100644 index 0000000..57f16bc --- /dev/null +++ b/.shellcheckrc @@ -0,0 +1 @@ +disable=SC1073 diff --git a/installer.sh b/installer.sh index 0dc2585..eda397c 100644 --- a/installer.sh +++ b/installer.sh @@ -63,7 +63,7 @@ sudo git clone --single-branch --branch release https://github.com/RecursionSpa # ----------------------------- Setup Enviroment ----------------------------- # sudo apt-get install python3.10-venv -y sudo python3.10 -m venv /opt/OpenPod/env -source /opt/OpenPod/env/bin/activate # shellcheck disable=SC1091 +source /opt/OpenPod/env/bin/activate sudo pip install --no-input -U -r /opt/OpenPod/requirements.txt --no-cache-dir --no-dependencies # ---------------------------- Create Directories ---------------------------- # From d033df9917818f07bf0a3108ae33526b4815bbee Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Tue, 22 Feb 2022 18:50:12 -0500 Subject: [PATCH 005/164] fix: script check --- .shellcheckrc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.shellcheckrc b/.shellcheckrc index 57f16bc..0182ace 100644 --- a/.shellcheckrc +++ b/.shellcheckrc @@ -1 +1 @@ -disable=SC1073 +disable=SC1091 From f482d803c7191534a275d93ee9aed5f84d7bb1b3 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Tue, 22 Feb 2022 19:00:11 -0500 Subject: [PATCH 006/164] chore: Added Action --- .github/workflows/VerifyInstaller.yml | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) create mode 100644 .github/workflows/VerifyInstaller.yml diff --git a/.github/workflows/VerifyInstaller.yml b/.github/workflows/VerifyInstaller.yml new file mode 100644 index 0000000..12c11ad --- /dev/null +++ b/.github/workflows/VerifyInstaller.yml @@ -0,0 +1,27 @@ +name: Verify Bash Installer + +on: + push: + branches: + - '**' + - 'master' + - '!master-ci' + - '!release' + + pull_request: + branches: + - master + + workflow_dispatch: + + +jobs: + install: + runs-on: ubuntu-latest + + steps: + - uses: actions/checkout@v2 + + - name: Run Bash Installer + run: | + sudo /bin/bash installer.sh From 685e8e15ed36e65fc410196fa81a367ec09912f6 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Tue, 22 Feb 2022 19:10:08 -0500 Subject: [PATCH 007/164] fix: exit on error --- installer.sh | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/installer.sh b/installer.sh index eda397c..5dc04f8 100644 --- a/installer.sh +++ b/installer.sh @@ -1,5 +1,8 @@ #!/bin/bash +set -e # Exit when any command fails. + + # ---------------------------------------------------------------------------- # # Help # # ---------------------------------------------------------------------------- # @@ -13,12 +16,14 @@ Help() echo "d Enable debug mode" } + # ---------------------------------------------------------------------------- # # Defaults # # ---------------------------------------------------------------------------- # URL="recursion.space" DEBUG=0 + # ---------------------------------------------------------------------------- # # Options # # ---------------------------------------------------------------------------- # @@ -35,6 +40,7 @@ while getopts ":hud" flags; do esac done + # ---------------------------------------------------------------------------- # # Installer # # ---------------------------------------------------------------------------- # @@ -58,7 +64,7 @@ sudo apt install python3.10 -y # ------------------------------- Clone OpenPod ------------------------------ # sudo mkdir -p /opt -sudo git clone --single-branch --branch release https://github.com/RecursionSpace/OpenPod.git /opt +sudo git clone --single-branch --branch release https://github.com/RecursionSpace/OpenPod.git /opt/ # ----------------------------- Setup Enviroment ----------------------------- # sudo apt-get install python3.10-venv -y @@ -114,3 +120,4 @@ sudo systemctl enable --now openpod.service sudo systemctl daemon-reload echo "OpenPod is now installed" +exit 0 From 13ef7965c93275a903a7d7eefb6c1a46b17807df Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Tue, 22 Feb 2022 19:16:00 -0500 Subject: [PATCH 008/164] fix: error capture --- installer.sh | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/installer.sh b/installer.sh index 5dc04f8..b95592a 100644 --- a/installer.sh +++ b/installer.sh @@ -1,8 +1,5 @@ #!/bin/bash -set -e # Exit when any command fails. - - # ---------------------------------------------------------------------------- # # Help # # ---------------------------------------------------------------------------- # @@ -63,6 +60,7 @@ sudo add-apt-repository ppa:deadsnakes/ppa sudo apt install python3.10 -y # ------------------------------- Clone OpenPod ------------------------------ # +set -e # Exit when any command fails. sudo mkdir -p /opt sudo git clone --single-branch --branch release https://github.com/RecursionSpace/OpenPod.git /opt/ From abdf99a112cf36e54fd5222d1d35ca4f68cb903e Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Tue, 22 Feb 2022 19:23:45 -0500 Subject: [PATCH 009/164] fix: error --- installer.sh | 1 - 1 file changed, 1 deletion(-) diff --git a/installer.sh b/installer.sh index b95592a..364a481 100644 --- a/installer.sh +++ b/installer.sh @@ -61,7 +61,6 @@ sudo apt install python3.10 -y # ------------------------------- Clone OpenPod ------------------------------ # set -e # Exit when any command fails. -sudo mkdir -p /opt sudo git clone --single-branch --branch release https://github.com/RecursionSpace/OpenPod.git /opt/ # ----------------------------- Setup Enviroment ----------------------------- # From 4921626d0f45da8f6b0f9b447e553ecfaaeea965 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Tue, 22 Feb 2022 19:27:37 -0500 Subject: [PATCH 010/164] fix: installer --- installer.sh | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/installer.sh b/installer.sh index 364a481..d3a04f4 100644 --- a/installer.sh +++ b/installer.sh @@ -61,7 +61,9 @@ sudo apt install python3.10 -y # ------------------------------- Clone OpenPod ------------------------------ # set -e # Exit when any command fails. -sudo git clone --single-branch --branch release https://github.com/RecursionSpace/OpenPod.git /opt/ +sudo mkdir -p /opt +cd /opt +sudo git clone --single-branch --branch release https://github.com/RecursionSpace/OpenPod.git # ----------------------------- Setup Enviroment ----------------------------- # sudo apt-get install python3.10-venv -y From 27c4642dd93491f4ab8f191b724e32c97d2c0734 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Tue, 22 Feb 2022 19:37:33 -0500 Subject: [PATCH 011/164] chore: action --- .github/workflows/VerifyInstaller.yml | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/.github/workflows/VerifyInstaller.yml b/.github/workflows/VerifyInstaller.yml index 12c11ad..864347d 100644 --- a/.github/workflows/VerifyInstaller.yml +++ b/.github/workflows/VerifyInstaller.yml @@ -25,3 +25,9 @@ jobs: - name: Run Bash Installer run: | sudo /bin/bash installer.sh + + - name: Verify System JSON + uses: limitusus/json-syntax-check@v1 + base: "/opt/OpenPod/" + with: + pattern: "\\.json$" From acc55cd9e210c34ffcd4513f48788006f238b197 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Tue, 22 Feb 2022 19:43:07 -0500 Subject: [PATCH 012/164] fix: json validation --- .github/workflows/VerifyInstaller.yml | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/.github/workflows/VerifyInstaller.yml b/.github/workflows/VerifyInstaller.yml index 864347d..2cefef4 100644 --- a/.github/workflows/VerifyInstaller.yml +++ b/.github/workflows/VerifyInstaller.yml @@ -26,8 +26,7 @@ jobs: run: | sudo /bin/bash installer.sh - - name: Verify System JSON - uses: limitusus/json-syntax-check@v1 - base: "/opt/OpenPod/" - with: - pattern: "\\.json$" + - name: Validate JSON + uses: docker://orrosenblatt/validate-json-action:latest + env: + INPUT_JSONS: /opt/OpenPod/system.json From f66e99bad48f45746c80e220f73ce6169f18d3b9 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Tue, 22 Feb 2022 19:48:29 -0500 Subject: [PATCH 013/164] test: INPUT_SCHEMA --- .github/workflows/VerifyInstaller.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/workflows/VerifyInstaller.yml b/.github/workflows/VerifyInstaller.yml index 2cefef4..3515e9d 100644 --- a/.github/workflows/VerifyInstaller.yml +++ b/.github/workflows/VerifyInstaller.yml @@ -29,4 +29,5 @@ jobs: - name: Validate JSON uses: docker://orrosenblatt/validate-json-action:latest env: + INPUT_SCHEMA: INPUT_JSONS: /opt/OpenPod/system.json From 0678cc84e66598f3b811ef7ac6bd5210b7d7c4e3 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Tue, 22 Feb 2022 19:57:34 -0500 Subject: [PATCH 014/164] fix: json validation --- .github/workflows/VerifyInstaller.yml | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/VerifyInstaller.yml b/.github/workflows/VerifyInstaller.yml index 3515e9d..1893a90 100644 --- a/.github/workflows/VerifyInstaller.yml +++ b/.github/workflows/VerifyInstaller.yml @@ -27,7 +27,7 @@ jobs: sudo /bin/bash installer.sh - name: Validate JSON - uses: docker://orrosenblatt/validate-json-action:latest - env: - INPUT_SCHEMA: - INPUT_JSONS: /opt/OpenPod/system.json + uses: anyone-developer/anyone-validate-json@main + with: + file-extension: '.json' + read-path: '/opt/OpenPod/' From 1027eb7794b923baf8316dcdd9840285f39bb4b7 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Tue, 22 Feb 2022 20:01:34 -0500 Subject: [PATCH 015/164] fix: json path --- .github/workflows/VerifyInstaller.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/VerifyInstaller.yml b/.github/workflows/VerifyInstaller.yml index 1893a90..7b002d4 100644 --- a/.github/workflows/VerifyInstaller.yml +++ b/.github/workflows/VerifyInstaller.yml @@ -30,4 +30,4 @@ jobs: uses: anyone-developer/anyone-validate-json@main with: file-extension: '.json' - read-path: '/opt/OpenPod/' + read-path: '/opt/OpenPod' From d840f26bee5353eeecd748ee0e345a687cf9757e Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Tue, 22 Feb 2022 20:05:41 -0500 Subject: [PATCH 016/164] fix: json --- .github/workflows/VerifyInstaller.yml | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/.github/workflows/VerifyInstaller.yml b/.github/workflows/VerifyInstaller.yml index 7b002d4..8584e74 100644 --- a/.github/workflows/VerifyInstaller.yml +++ b/.github/workflows/VerifyInstaller.yml @@ -27,7 +27,8 @@ jobs: sudo /bin/bash installer.sh - name: Validate JSON - uses: anyone-developer/anyone-validate-json@main + uses: limitusus/json-syntax-check@v1 + env: + BASE: '/opt/OpenPod' with: - file-extension: '.json' - read-path: '/opt/OpenPod' + pattern: "\\.json$" From 0f6ff444c53206e18200fd9c8bf83d43e7dc304b Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Tue, 22 Feb 2022 20:09:37 -0500 Subject: [PATCH 017/164] fix: location --- .github/workflows/VerifyInstaller.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/VerifyInstaller.yml b/.github/workflows/VerifyInstaller.yml index 8584e74..ae9e263 100644 --- a/.github/workflows/VerifyInstaller.yml +++ b/.github/workflows/VerifyInstaller.yml @@ -29,6 +29,6 @@ jobs: - name: Validate JSON uses: limitusus/json-syntax-check@v1 env: - BASE: '/opt/OpenPod' + BASE: '/opt/OpenPod/' with: pattern: "\\.json$" From 63b247e3f0bdde88b0b7afb06eb0a22d3f9cfe70 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Tue, 22 Feb 2022 20:13:26 -0500 Subject: [PATCH 018/164] test: docker run --- .github/workflows/VerifyInstaller.yml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.github/workflows/VerifyInstaller.yml b/.github/workflows/VerifyInstaller.yml index ae9e263..6c5ffc4 100644 --- a/.github/workflows/VerifyInstaller.yml +++ b/.github/workflows/VerifyInstaller.yml @@ -28,6 +28,8 @@ jobs: - name: Validate JSON uses: limitusus/json-syntax-check@v1 + run: | + sudo /bin/bash installer.sh env: BASE: '/opt/OpenPod/' with: From a4018ceb53a929d47f415b1eab493bf63642e76e Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Tue, 22 Feb 2022 20:26:46 -0500 Subject: [PATCH 019/164] test: action --- .github/workflows/VerifyInstaller.yml | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/.github/workflows/VerifyInstaller.yml b/.github/workflows/VerifyInstaller.yml index 6c5ffc4..a2ddbfb 100644 --- a/.github/workflows/VerifyInstaller.yml +++ b/.github/workflows/VerifyInstaller.yml @@ -21,15 +21,13 @@ jobs: steps: - uses: actions/checkout@v2 + - uses: limitusus/json-syntax-check@v1 - name: Run Bash Installer run: | sudo /bin/bash installer.sh - name: Validate JSON - uses: limitusus/json-syntax-check@v1 - run: | - sudo /bin/bash installer.sh env: BASE: '/opt/OpenPod/' with: From 091d02a9289d821daea07af829595d97321aac68 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Tue, 22 Feb 2022 20:29:12 -0500 Subject: [PATCH 020/164] test: action --- .github/workflows/VerifyInstaller.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/VerifyInstaller.yml b/.github/workflows/VerifyInstaller.yml index a2ddbfb..6249df2 100644 --- a/.github/workflows/VerifyInstaller.yml +++ b/.github/workflows/VerifyInstaller.yml @@ -21,13 +21,13 @@ jobs: steps: - uses: actions/checkout@v2 - - uses: limitusus/json-syntax-check@v1 + # - uses: limitusus/json-syntax-check@v1 - name: Run Bash Installer run: | sudo /bin/bash installer.sh - - name: Validate JSON + uses: limitusus/json-syntax-check@v1 env: BASE: '/opt/OpenPod/' with: From b8295b9424aacd50bf942af8040f0047a159e367 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Tue, 22 Feb 2022 20:29:45 -0500 Subject: [PATCH 021/164] test: action --- .github/workflows/VerifyInstaller.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/VerifyInstaller.yml b/.github/workflows/VerifyInstaller.yml index 6249df2..e3a83a4 100644 --- a/.github/workflows/VerifyInstaller.yml +++ b/.github/workflows/VerifyInstaller.yml @@ -21,13 +21,13 @@ jobs: steps: - uses: actions/checkout@v2 - # - uses: limitusus/json-syntax-check@v1 + - uses: limitusus/json-syntax-check@v1 - name: Run Bash Installer run: | sudo /bin/bash installer.sh - uses: limitusus/json-syntax-check@v1 + env: BASE: '/opt/OpenPod/' with: From e564407cf2c25e89084014a43ec75d772501a346 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Tue, 22 Feb 2022 20:32:20 -0500 Subject: [PATCH 022/164] test: action --- .github/workflows/VerifyInstaller.yml | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/.github/workflows/VerifyInstaller.yml b/.github/workflows/VerifyInstaller.yml index e3a83a4..a56c36e 100644 --- a/.github/workflows/VerifyInstaller.yml +++ b/.github/workflows/VerifyInstaller.yml @@ -23,11 +23,15 @@ jobs: - uses: actions/checkout@v2 - uses: limitusus/json-syntax-check@v1 + - name: Run Bash Installer run: | sudo /bin/bash installer.sh + - name: Validate JSON + run: | + sudo /bin/bash validate.sh env: BASE: '/opt/OpenPod/' with: From 6ee13e1294fd7a5d3fc55964d7e59230cc1cfac2 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Tue, 22 Feb 2022 20:33:22 -0500 Subject: [PATCH 023/164] test: action --- .github/workflows/VerifyInstaller.yml | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/VerifyInstaller.yml b/.github/workflows/VerifyInstaller.yml index a56c36e..2c50156 100644 --- a/.github/workflows/VerifyInstaller.yml +++ b/.github/workflows/VerifyInstaller.yml @@ -22,6 +22,10 @@ jobs: steps: - uses: actions/checkout@v2 - uses: limitusus/json-syntax-check@v1 + env: + BASE: '/opt/OpenPod/' + with: + pattern: "\\.json$" - name: Run Bash Installer @@ -32,7 +36,3 @@ jobs: - name: Validate JSON run: | sudo /bin/bash validate.sh - env: - BASE: '/opt/OpenPod/' - with: - pattern: "\\.json$" From e966f008a8b5c53bde69564fa8d901f6d96042ee Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Tue, 22 Feb 2022 20:34:54 -0500 Subject: [PATCH 024/164] test: action --- .github/workflows/VerifyInstaller.yml | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/.github/workflows/VerifyInstaller.yml b/.github/workflows/VerifyInstaller.yml index 2c50156..4bd0262 100644 --- a/.github/workflows/VerifyInstaller.yml +++ b/.github/workflows/VerifyInstaller.yml @@ -22,8 +22,6 @@ jobs: steps: - uses: actions/checkout@v2 - uses: limitusus/json-syntax-check@v1 - env: - BASE: '/opt/OpenPod/' with: pattern: "\\.json$" @@ -31,8 +29,11 @@ jobs: - name: Run Bash Installer run: | sudo /bin/bash installer.sh + env: + BASE: '/opt/OpenPod/' - - name: Validate JSON - run: | - sudo /bin/bash validate.sh + + # - name: Validate JSON + # run: | + # sudo /bin/bash validate.sh From 3d8f3e84530c41d512dcfe22aaf32d109ff8ddb8 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Tue, 22 Feb 2022 20:38:08 -0500 Subject: [PATCH 025/164] test: action --- .github/workflows/VerifyInstaller.yml | 18 +++++++----------- 1 file changed, 7 insertions(+), 11 deletions(-) diff --git a/.github/workflows/VerifyInstaller.yml b/.github/workflows/VerifyInstaller.yml index 4bd0262..a08a751 100644 --- a/.github/workflows/VerifyInstaller.yml +++ b/.github/workflows/VerifyInstaller.yml @@ -21,19 +21,15 @@ jobs: steps: - uses: actions/checkout@v2 - - uses: limitusus/json-syntax-check@v1 - with: - pattern: "\\.json$" - - name: Run Bash Installer run: | sudo /bin/bash installer.sh - env: - BASE: '/opt/OpenPod/' - - - # - name: Validate JSON - # run: | - # sudo /bin/bash validate.sh + - name: Validate JSON + - run: sudo /bin/bash installer.sh + uses: limitusus/json-syntax-check@v1 + env: + BASE: '/opt/OpenPod' + with: + pattern: "\\.json$" From 6be7ade46f33c02901de84710ec02c6af35346e1 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Tue, 22 Feb 2022 20:40:48 -0500 Subject: [PATCH 026/164] fix: action --- .github/workflows/VerifyInstaller.yml | 8 -------- 1 file changed, 8 deletions(-) diff --git a/.github/workflows/VerifyInstaller.yml b/.github/workflows/VerifyInstaller.yml index a08a751..12c11ad 100644 --- a/.github/workflows/VerifyInstaller.yml +++ b/.github/workflows/VerifyInstaller.yml @@ -25,11 +25,3 @@ jobs: - name: Run Bash Installer run: | sudo /bin/bash installer.sh - - - name: Validate JSON - - run: sudo /bin/bash installer.sh - uses: limitusus/json-syntax-check@v1 - env: - BASE: '/opt/OpenPod' - with: - pattern: "\\.json$" From 00f710f503672b3a6789e98a0a6297fcf464f24f Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Wed, 23 Feb 2022 13:31:17 -0500 Subject: [PATCH 027/164] chore: added e2e --- .github/workflows/EndToEnd.yml | 42 +++++++++++++++++++++++++++ .github/workflows/VerifyInstaller.yml | 7 +++++ installer.sh | 2 +- 3 files changed, 50 insertions(+), 1 deletion(-) create mode 100644 .github/workflows/EndToEnd.yml diff --git a/.github/workflows/EndToEnd.yml b/.github/workflows/EndToEnd.yml new file mode 100644 index 0000000..33a9e82 --- /dev/null +++ b/.github/workflows/EndToEnd.yml @@ -0,0 +1,42 @@ +name: End To End Integration + +on: + push: + branches: + - '**' + - 'master' + - '!master-ci' + - '!release' + + pull_request: + branches: + - master + + workflow_dispatch: + +jobs: + setup: + runs-on: ubuntu-latest + + steps: + - uses: actions/checkout@v2 + # with: + # repository: RecursionSpace/RecursionREST + # ref: release + # token: ${{ secrets.RECURSION_SPACE_ACCESS_TOKEN}} + + - name: Setup Server + run: | + sudo apt-get update -y && sudo apt-get upgrade -y + sudo apt install software-properties-common -y + sudo add-apt-repository ppa:deadsnakes/ppa + sudo apt install python3.10 -y + sudo mkdir -p /opt + cd /opt + + ssh-agent bash -c 'ssh-add ${{ secrets.RECURSION_SPACE_ACCESS_TOKEN}}; git clone --single-branch --branch release git@github.com:RecursionSpace/RecursionREST.git' + + sudo apt-get install python3.10-venv -y + sudo python3.10 -m venv /opt/RecursionSpace/env + source /opt/RecursionSpace/env/bin/activate + sudo pip install --no-input -U -r /opt/RecursionSpace/requirements.txt --no-cache-dir --no-dependencies diff --git a/.github/workflows/VerifyInstaller.yml b/.github/workflows/VerifyInstaller.yml index 12c11ad..c0f3d8b 100644 --- a/.github/workflows/VerifyInstaller.yml +++ b/.github/workflows/VerifyInstaller.yml @@ -25,3 +25,10 @@ jobs: - name: Run Bash Installer run: | sudo /bin/bash installer.sh + + # - name: Validate JSON + # uses: limitusus/json-syntax-check@v1 + # env: + # BASE: '/opt/OpenPod' + # with: + # pattern: "\\.json$" diff --git a/installer.sh b/installer.sh index d3a04f4..c08ec05 100644 --- a/installer.sh +++ b/installer.sh @@ -63,7 +63,7 @@ sudo apt install python3.10 -y set -e # Exit when any command fails. sudo mkdir -p /opt cd /opt -sudo git clone --single-branch --branch release https://github.com/RecursionSpace/OpenPod.git +sudo git clone --single-branch --branch release https://github.com/RecursionSpace/OpenPod.git # ----------------------------- Setup Enviroment ----------------------------- # sudo apt-get install python3.10-venv -y From 871971ef14d84e25592c1276e57a01dbaf1320b2 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Wed, 23 Feb 2022 14:06:39 -0500 Subject: [PATCH 028/164] fix: ssh key --- .github/workflows/EndToEnd.yml | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/.github/workflows/EndToEnd.yml b/.github/workflows/EndToEnd.yml index 33a9e82..991e2c0 100644 --- a/.github/workflows/EndToEnd.yml +++ b/.github/workflows/EndToEnd.yml @@ -25,6 +25,10 @@ jobs: # ref: release # token: ${{ secrets.RECURSION_SPACE_ACCESS_TOKEN}} + - uses: webfactory/ssh-agent@v0.5.4 + with: + ssh-private-key: ${{ secrets.RECURSION_SPACE_ACCESS_TOKEN}} + - name: Setup Server run: | sudo apt-get update -y && sudo apt-get upgrade -y @@ -34,7 +38,7 @@ jobs: sudo mkdir -p /opt cd /opt - ssh-agent bash -c 'ssh-add ${{ secrets.RECURSION_SPACE_ACCESS_TOKEN}}; git clone --single-branch --branch release git@github.com:RecursionSpace/RecursionREST.git' + git clone --single-branch --branch release git@github.com:RecursionSpace/RecursionREST.git sudo apt-get install python3.10-venv -y sudo python3.10 -m venv /opt/RecursionSpace/env From d814706f08035a41d00f33c457e9f4efdb2e77e2 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Wed, 23 Feb 2022 14:29:47 -0500 Subject: [PATCH 029/164] wip: e2e --- .github/workflows/EndToEnd.yml | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/.github/workflows/EndToEnd.yml b/.github/workflows/EndToEnd.yml index 991e2c0..330bcba 100644 --- a/.github/workflows/EndToEnd.yml +++ b/.github/workflows/EndToEnd.yml @@ -20,10 +20,6 @@ jobs: steps: - uses: actions/checkout@v2 - # with: - # repository: RecursionSpace/RecursionREST - # ref: release - # token: ${{ secrets.RECURSION_SPACE_ACCESS_TOKEN}} - uses: webfactory/ssh-agent@v0.5.4 with: @@ -43,4 +39,6 @@ jobs: sudo apt-get install python3.10-venv -y sudo python3.10 -m venv /opt/RecursionSpace/env source /opt/RecursionSpace/env/bin/activate - sudo pip install --no-input -U -r /opt/RecursionSpace/requirements.txt --no-cache-dir --no-dependencies + sudo pip install --no-input -U -r /opt/RecursionREST/requirements.txt --no-cache-dir --no-dependencies + + sudo python manage.py migrate --noinput From 1186f4f02d9a38c3e91d8e91045983b9836b9409 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Wed, 23 Feb 2022 14:33:37 -0500 Subject: [PATCH 030/164] wip: e2e --- .github/workflows/EndToEnd.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/EndToEnd.yml b/.github/workflows/EndToEnd.yml index 330bcba..6cbaf75 100644 --- a/.github/workflows/EndToEnd.yml +++ b/.github/workflows/EndToEnd.yml @@ -41,4 +41,4 @@ jobs: source /opt/RecursionSpace/env/bin/activate sudo pip install --no-input -U -r /opt/RecursionREST/requirements.txt --no-cache-dir --no-dependencies - sudo python manage.py migrate --noinput + sudo python /opt/RecursionRest/manage.py migrate --noinput From 0d18d56f4ae739a8fdeda32346dadb294c1f58bb Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Wed, 23 Feb 2022 14:38:50 -0500 Subject: [PATCH 031/164] wip: e2e --- .github/workflows/EndToEnd.yml | 2 +- README.md | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/.github/workflows/EndToEnd.yml b/.github/workflows/EndToEnd.yml index 6cbaf75..8889f18 100644 --- a/.github/workflows/EndToEnd.yml +++ b/.github/workflows/EndToEnd.yml @@ -41,4 +41,4 @@ jobs: source /opt/RecursionSpace/env/bin/activate sudo pip install --no-input -U -r /opt/RecursionREST/requirements.txt --no-cache-dir --no-dependencies - sudo python /opt/RecursionRest/manage.py migrate --noinput + sudo python /opt/RecursionREST/RecursionRest/manage.py migrate --noinput diff --git a/README.md b/README.md index 6a3897d..23dc00c 100644 --- a/README.md +++ b/README.md @@ -2,6 +2,8 @@ # OpenPod +[![End To End Integration](https://github.com/RecursionSpace/OpenPod/actions/workflows/EndToEnd.yml/badge.svg)](https://github.com/RecursionSpace/OpenPod/actions/workflows/EndToEnd.yml) +  [![Code Quality](https://github.com/RecursionSpace/OpenPod/actions/workflows/pylint.yml/badge.svg)](https://github.com/RecursionSpace/OpenPod/actions/workflows/pylint.yml)   [![Script Check](https://github.com/RecursionSpace/OpenPod/actions/workflows/ShellCheck.yml/badge.svg)](https://github.com/RecursionSpace/OpenPod/actions/workflows/ShellCheck.yml) From fb767b06eae3604e19c49fe3e702a3274e5c6201 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Wed, 23 Feb 2022 14:56:15 -0500 Subject: [PATCH 032/164] wip: e2e --- .github/workflows/EndToEnd.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/EndToEnd.yml b/.github/workflows/EndToEnd.yml index 8889f18..7feb370 100644 --- a/.github/workflows/EndToEnd.yml +++ b/.github/workflows/EndToEnd.yml @@ -41,4 +41,4 @@ jobs: source /opt/RecursionSpace/env/bin/activate sudo pip install --no-input -U -r /opt/RecursionREST/requirements.txt --no-cache-dir --no-dependencies - sudo python /opt/RecursionREST/RecursionRest/manage.py migrate --noinput + sudo python /opt/RecursionREST/RecursionRest/manage.py migrate.py --noinput From a8bae37882cb17d9c694b059e6888f04c9940968 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Wed, 23 Feb 2022 17:08:09 -0500 Subject: [PATCH 033/164] wip: e2e --- .github/workflows/EndToEnd.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/EndToEnd.yml b/.github/workflows/EndToEnd.yml index 7feb370..85068bb 100644 --- a/.github/workflows/EndToEnd.yml +++ b/.github/workflows/EndToEnd.yml @@ -41,4 +41,4 @@ jobs: source /opt/RecursionSpace/env/bin/activate sudo pip install --no-input -U -r /opt/RecursionREST/requirements.txt --no-cache-dir --no-dependencies - sudo python /opt/RecursionREST/RecursionRest/manage.py migrate.py --noinput + sudo python /opt/RecursionREST/manage.py migrate.py --noinput From 981f3f522b320d59e66f6011874330072281c075 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Wed, 23 Feb 2022 17:11:27 -0500 Subject: [PATCH 034/164] wip: e2e --- .github/workflows/EndToEnd.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/EndToEnd.yml b/.github/workflows/EndToEnd.yml index 85068bb..e59df04 100644 --- a/.github/workflows/EndToEnd.yml +++ b/.github/workflows/EndToEnd.yml @@ -41,4 +41,4 @@ jobs: source /opt/RecursionSpace/env/bin/activate sudo pip install --no-input -U -r /opt/RecursionREST/requirements.txt --no-cache-dir --no-dependencies - sudo python /opt/RecursionREST/manage.py migrate.py --noinput + sudo python /opt/RecursionREST/RecursionREST/manage.py migrate --noinput From a913e5f8088f8a6c78d08ec773883d1f613ddfe3 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Wed, 23 Feb 2022 17:30:23 -0500 Subject: [PATCH 035/164] wip: e2e --- .github/workflows/EndToEnd.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/EndToEnd.yml b/.github/workflows/EndToEnd.yml index e59df04..49a7d38 100644 --- a/.github/workflows/EndToEnd.yml +++ b/.github/workflows/EndToEnd.yml @@ -37,8 +37,8 @@ jobs: git clone --single-branch --branch release git@github.com:RecursionSpace/RecursionREST.git sudo apt-get install python3.10-venv -y - sudo python3.10 -m venv /opt/RecursionSpace/env - source /opt/RecursionSpace/env/bin/activate + sudo python3.10 -m venv /opt/RecursionREST/env + source /opt/RecursionREST/env/bin/activate sudo pip install --no-input -U -r /opt/RecursionREST/requirements.txt --no-cache-dir --no-dependencies sudo python /opt/RecursionREST/RecursionREST/manage.py migrate --noinput From 676e4912f8d98f406b7bf89c1671b265a21835d9 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Wed, 23 Feb 2022 18:24:28 -0500 Subject: [PATCH 036/164] wip: e2e --- .github/workflows/EndToEnd.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/workflows/EndToEnd.yml b/.github/workflows/EndToEnd.yml index 49a7d38..fc18a38 100644 --- a/.github/workflows/EndToEnd.yml +++ b/.github/workflows/EndToEnd.yml @@ -39,6 +39,7 @@ jobs: sudo apt-get install python3.10-venv -y sudo python3.10 -m venv /opt/RecursionREST/env source /opt/RecursionREST/env/bin/activate + pip install asgiref sudo pip install --no-input -U -r /opt/RecursionREST/requirements.txt --no-cache-dir --no-dependencies sudo python /opt/RecursionREST/RecursionREST/manage.py migrate --noinput From 238cef7e3ed549ef340ff6abe53cf7136f295186 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Wed, 23 Feb 2022 18:33:53 -0500 Subject: [PATCH 037/164] wip: e2e --- .github/workflows/EndToEnd.yml | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/.github/workflows/EndToEnd.yml b/.github/workflows/EndToEnd.yml index fc18a38..1ce27ee 100644 --- a/.github/workflows/EndToEnd.yml +++ b/.github/workflows/EndToEnd.yml @@ -39,7 +39,8 @@ jobs: sudo apt-get install python3.10-venv -y sudo python3.10 -m venv /opt/RecursionREST/env source /opt/RecursionREST/env/bin/activate - pip install asgiref - sudo pip install --no-input -U -r /opt/RecursionREST/requirements.txt --no-cache-dir --no-dependencies + /opt/RecursionREST/env/bin/python3.10 -m pip install --upgrade pip + + sudo pip install --no-input -U -r /opt/RecursionREST/requirements.txt sudo python /opt/RecursionREST/RecursionREST/manage.py migrate --noinput From c846aee15dc7e16319c9101c8f848dfdb7d24bc9 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 1 Mar 2022 04:36:38 +0000 Subject: [PATCH 038/164] chore(deps): bump actions/setup-python from 2 to 3 Bumps [actions/setup-python](https://github.com/actions/setup-python) from 2 to 3. - [Release notes](https://github.com/actions/setup-python/releases) - [Commits](https://github.com/actions/setup-python/compare/v2...v3) --- updated-dependencies: - dependency-name: actions/setup-python dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] --- .github/workflows/RecursionTests.yml | 2 +- .github/workflows/TestInstaller.yml | 2 +- .github/workflows/pylint.yml | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/RecursionTests.yml b/.github/workflows/RecursionTests.yml index be6c2c6..7dffb20 100644 --- a/.github/workflows/RecursionTests.yml +++ b/.github/workflows/RecursionTests.yml @@ -16,7 +16,7 @@ jobs: steps: - uses: actions/checkout@v2 - name: Set up Python 3.9 - uses: actions/setup-python@v2 + uses: actions/setup-python@v3 with: python-version: 3.9 - name: Install dependencies diff --git a/.github/workflows/TestInstaller.yml b/.github/workflows/TestInstaller.yml index f8b0674..e021445 100644 --- a/.github/workflows/TestInstaller.yml +++ b/.github/workflows/TestInstaller.yml @@ -10,7 +10,7 @@ jobs: steps: - uses: actions/checkout@v2 - name: Set up Python 3.8 - uses: actions/setup-python@v1 + uses: actions/setup-python@v3 with: python-version: 3.8 diff --git a/.github/workflows/pylint.yml b/.github/workflows/pylint.yml index 3770517..3cd9498 100644 --- a/.github/workflows/pylint.yml +++ b/.github/workflows/pylint.yml @@ -10,7 +10,7 @@ jobs: steps: - uses: actions/checkout@v2 - name: Set up Python 3.8 - uses: actions/setup-python@v1 + uses: actions/setup-python@v3 with: python-version: 3.8 From 19df36bbdad7820974077ab86e749e92024bf6d1 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Wed, 2 Mar 2022 04:45:45 +0000 Subject: [PATCH 039/164] chore(deps): bump actions/checkout from 2 to 3 Bumps [actions/checkout](https://github.com/actions/checkout) from 2 to 3. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](https://github.com/actions/checkout/compare/v2...v3) --- updated-dependencies: - dependency-name: actions/checkout dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] --- .github/workflows/EndToEnd.yml | 2 +- .github/workflows/RecursionTests.yml | 2 +- .github/workflows/ShellCheck.yml | 2 +- .github/workflows/TestInstaller.yml | 2 +- .github/workflows/VerifyInstaller.yml | 2 +- .github/workflows/codeql-analysis.yml | 2 +- .github/workflows/merge_to-dev-release.yml | 2 +- .github/workflows/pylint.yml | 2 +- 8 files changed, 8 insertions(+), 8 deletions(-) diff --git a/.github/workflows/EndToEnd.yml b/.github/workflows/EndToEnd.yml index 1ce27ee..19670b5 100644 --- a/.github/workflows/EndToEnd.yml +++ b/.github/workflows/EndToEnd.yml @@ -19,7 +19,7 @@ jobs: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v2 + - uses: actions/checkout@v3 - uses: webfactory/ssh-agent@v0.5.4 with: diff --git a/.github/workflows/RecursionTests.yml b/.github/workflows/RecursionTests.yml index be6c2c6..d0e49a1 100644 --- a/.github/workflows/RecursionTests.yml +++ b/.github/workflows/RecursionTests.yml @@ -14,7 +14,7 @@ jobs: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v2 + - uses: actions/checkout@v3 - name: Set up Python 3.9 uses: actions/setup-python@v2 with: diff --git a/.github/workflows/ShellCheck.yml b/.github/workflows/ShellCheck.yml index 54f9939..f222e20 100644 --- a/.github/workflows/ShellCheck.yml +++ b/.github/workflows/ShellCheck.yml @@ -20,7 +20,7 @@ jobs: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v2 + - uses: actions/checkout@v3 - name: Run Shellcheck uses: azohra/shell-linter@latest diff --git a/.github/workflows/TestInstaller.yml b/.github/workflows/TestInstaller.yml index f8b0674..d470139 100644 --- a/.github/workflows/TestInstaller.yml +++ b/.github/workflows/TestInstaller.yml @@ -8,7 +8,7 @@ jobs: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v2 + - uses: actions/checkout@v3 - name: Set up Python 3.8 uses: actions/setup-python@v1 with: diff --git a/.github/workflows/VerifyInstaller.yml b/.github/workflows/VerifyInstaller.yml index c0f3d8b..5b659b7 100644 --- a/.github/workflows/VerifyInstaller.yml +++ b/.github/workflows/VerifyInstaller.yml @@ -20,7 +20,7 @@ jobs: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v2 + - uses: actions/checkout@v3 - name: Run Bash Installer run: | diff --git a/.github/workflows/codeql-analysis.yml b/.github/workflows/codeql-analysis.yml index 9ba7e79..2017066 100644 --- a/.github/workflows/codeql-analysis.yml +++ b/.github/workflows/codeql-analysis.yml @@ -38,7 +38,7 @@ jobs: steps: - name: Checkout repository - uses: actions/checkout@v2 + uses: actions/checkout@v3 # Initializes the CodeQL tools for scanning. - name: Initialize CodeQL diff --git a/.github/workflows/merge_to-dev-release.yml b/.github/workflows/merge_to-dev-release.yml index 9f46817..5a65298 100644 --- a/.github/workflows/merge_to-dev-release.yml +++ b/.github/workflows/merge_to-dev-release.yml @@ -15,7 +15,7 @@ jobs: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v2 + - uses: actions/checkout@v3 - name: Wait on check uses: fountainhead/action-wait-for-check@v1.0.0 diff --git a/.github/workflows/pylint.yml b/.github/workflows/pylint.yml index 3770517..091b110 100644 --- a/.github/workflows/pylint.yml +++ b/.github/workflows/pylint.yml @@ -8,7 +8,7 @@ jobs: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v2 + - uses: actions/checkout@v3 - name: Set up Python 3.8 uses: actions/setup-python@v1 with: From 3f360fdda213cb09002cdf9439691b05eacd8b17 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Wed, 2 Mar 2022 15:29:41 -0500 Subject: [PATCH 040/164] wip: e2e --- .github/workflows/EndToEnd.yml | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/.github/workflows/EndToEnd.yml b/.github/workflows/EndToEnd.yml index 1ce27ee..d8aa331 100644 --- a/.github/workflows/EndToEnd.yml +++ b/.github/workflows/EndToEnd.yml @@ -4,7 +4,6 @@ on: push: branches: - '**' - - 'master' - '!master-ci' - '!release' @@ -31,16 +30,15 @@ jobs: sudo apt install software-properties-common -y sudo add-apt-repository ppa:deadsnakes/ppa sudo apt install python3.10 -y - sudo mkdir -p /opt cd /opt - git clone --single-branch --branch release git@github.com:RecursionSpace/RecursionREST.git + git clone --single-branch --branch master git@github.com:RecursionSpace/RecursionSpace.git sudo apt-get install python3.10-venv -y - sudo python3.10 -m venv /opt/RecursionREST/env - source /opt/RecursionREST/env/bin/activate - /opt/RecursionREST/env/bin/python3.10 -m pip install --upgrade pip + sudo python3.10 -m venv /opt/RecursionSpace/env + source /opt/RecursionSpace/env/bin/activate + /opt/RecursionSpace/env/bin/python3.10 -m pip install --upgrade pip - sudo pip install --no-input -U -r /opt/RecursionREST/requirements.txt + sudo pip install --no-input -U -r /opt/RecursionSpace/requirements.txt --no-cache-dir - sudo python /opt/RecursionREST/RecursionREST/manage.py migrate --noinput + sudo python /opt/RecursionSpace/manage.py migrate --noinput From 3c682eea53f229eea86b34bd71ed0f25e6c24ae7 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 26 Apr 2022 04:42:28 +0000 Subject: [PATCH 041/164] chore(deps): bump github/codeql-action from 1 to 2 Bumps [github/codeql-action](https://github.com/github/codeql-action) from 1 to 2. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/codeql-action/compare/v1...v2) --- updated-dependencies: - dependency-name: github/codeql-action dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] --- .github/workflows/codeql-analysis.yml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/codeql-analysis.yml b/.github/workflows/codeql-analysis.yml index 2017066..5074934 100644 --- a/.github/workflows/codeql-analysis.yml +++ b/.github/workflows/codeql-analysis.yml @@ -42,7 +42,7 @@ jobs: # Initializes the CodeQL tools for scanning. - name: Initialize CodeQL - uses: github/codeql-action/init@v1 + uses: github/codeql-action/init@v2 with: languages: ${{ matrix.language }} # If you wish to specify custom queries, you can do so here or in a config file. @@ -53,7 +53,7 @@ jobs: # Autobuild attempts to build any compiled languages (C/C++, C#, or Java). # If this step fails, then you should remove it and run the build manually (see below) - name: Autobuild - uses: github/codeql-action/autobuild@v1 + uses: github/codeql-action/autobuild@v2 # ℹ️ Command-line programs to run using the OS shell. # 📚 https://git.io/JvXDl @@ -67,4 +67,4 @@ jobs: # make release - name: Perform CodeQL Analysis - uses: github/codeql-action/analyze@v1 + uses: github/codeql-action/analyze@v2 From bbb0707fe2dcce087e34c4459fa9cda82b5082a3 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 12 Jul 2022 04:38:17 +0000 Subject: [PATCH 042/164] chore(deps): bump actions/setup-python from 3 to 4.1.0 Bumps [actions/setup-python](https://github.com/actions/setup-python) from 3 to 4.1.0. - [Release notes](https://github.com/actions/setup-python/releases) - [Commits](https://github.com/actions/setup-python/compare/v3...v4.1.0) --- updated-dependencies: - dependency-name: actions/setup-python dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] --- .github/workflows/RecursionTests.yml | 2 +- .github/workflows/TestInstaller.yml | 2 +- .github/workflows/pylint.yml | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/RecursionTests.yml b/.github/workflows/RecursionTests.yml index a9e8aa8..2d804ea 100644 --- a/.github/workflows/RecursionTests.yml +++ b/.github/workflows/RecursionTests.yml @@ -16,7 +16,7 @@ jobs: steps: - uses: actions/checkout@v3 - name: Set up Python 3.9 - uses: actions/setup-python@v3 + uses: actions/setup-python@v4.1.0 with: python-version: 3.9 - name: Install dependencies diff --git a/.github/workflows/TestInstaller.yml b/.github/workflows/TestInstaller.yml index c287e02..d2b6524 100644 --- a/.github/workflows/TestInstaller.yml +++ b/.github/workflows/TestInstaller.yml @@ -10,7 +10,7 @@ jobs: steps: - uses: actions/checkout@v3 - name: Set up Python 3.8 - uses: actions/setup-python@v3 + uses: actions/setup-python@v4.1.0 with: python-version: 3.8 diff --git a/.github/workflows/pylint.yml b/.github/workflows/pylint.yml index c940521..d244d1d 100644 --- a/.github/workflows/pylint.yml +++ b/.github/workflows/pylint.yml @@ -10,7 +10,7 @@ jobs: steps: - uses: actions/checkout@v3 - name: Set up Python 3.8 - uses: actions/setup-python@v3 + uses: actions/setup-python@v4.1.0 with: python-version: 3.8 From e39a077dea56a6550a9c21155bb803f8a63ebfac Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 10 Oct 2022 04:39:02 +0000 Subject: [PATCH 043/164] chore(deps): bump distributhor/workflow-webhook from 1 to 3 Bumps [distributhor/workflow-webhook](https://github.com/distributhor/workflow-webhook) from 1 to 3. - [Release notes](https://github.com/distributhor/workflow-webhook/releases) - [Commits](https://github.com/distributhor/workflow-webhook/compare/v1...v3) --- updated-dependencies: - dependency-name: distributhor/workflow-webhook dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] --- .github/workflows/cd-release.yml | 2 +- .github/workflows/merge_to-dev-release.yml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/cd-release.yml b/.github/workflows/cd-release.yml index 5996b2b..305a587 100644 --- a/.github/workflows/cd-release.yml +++ b/.github/workflows/cd-release.yml @@ -14,7 +14,7 @@ jobs: steps: - name: Release Webhook - uses: distributhor/workflow-webhook@v1 + uses: distributhor/workflow-webhook@v3 env: webhook_url: "https://recursion.space/webhooks/github/" webhook_secret: "Y0uR5ecr3t" diff --git a/.github/workflows/merge_to-dev-release.yml b/.github/workflows/merge_to-dev-release.yml index 5a65298..41421fe 100644 --- a/.github/workflows/merge_to-dev-release.yml +++ b/.github/workflows/merge_to-dev-release.yml @@ -46,7 +46,7 @@ jobs: - name: Webhook if: steps.wait-for-build.outputs.conclusion == 'success' - uses: distributhor/workflow-webhook@v1 + uses: distributhor/workflow-webhook@v3 env: webhook_url: "https://dev.recursion.space/webhooks/github/" webhook_secret: "Y0uR5ecr3t" From f81e6b9a5aeb3cecae1df13e8027c324b46eca40 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Fri, 14 Oct 2022 04:25:40 +0000 Subject: [PATCH 044/164] chore(deps): bump fountainhead/action-wait-for-check from 1.0.0 to 1.1.0 Bumps [fountainhead/action-wait-for-check](https://github.com/fountainhead/action-wait-for-check) from 1.0.0 to 1.1.0. - [Release notes](https://github.com/fountainhead/action-wait-for-check/releases) - [Commits](https://github.com/fountainhead/action-wait-for-check/compare/v1.0.0...v1.1.0) --- updated-dependencies: - dependency-name: fountainhead/action-wait-for-check dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] --- .github/workflows/merge_to-dev-release.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/merge_to-dev-release.yml b/.github/workflows/merge_to-dev-release.yml index 5a65298..de90fab 100644 --- a/.github/workflows/merge_to-dev-release.yml +++ b/.github/workflows/merge_to-dev-release.yml @@ -18,7 +18,7 @@ jobs: - uses: actions/checkout@v3 - name: Wait on check - uses: fountainhead/action-wait-for-check@v1.0.0 + uses: fountainhead/action-wait-for-check@v1.1.0 id: wait-for-build with: From d07156183034b955448949e1f2250815654e6066 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu, 20 Oct 2022 04:34:05 +0000 Subject: [PATCH 045/164] chore(deps): bump webfactory/ssh-agent from 0.5.4 to 0.7.0 Bumps [webfactory/ssh-agent](https://github.com/webfactory/ssh-agent) from 0.5.4 to 0.7.0. - [Release notes](https://github.com/webfactory/ssh-agent/releases) - [Changelog](https://github.com/webfactory/ssh-agent/blob/master/CHANGELOG.md) - [Commits](https://github.com/webfactory/ssh-agent/compare/v0.5.4...v0.7.0) --- updated-dependencies: - dependency-name: webfactory/ssh-agent dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] --- .github/workflows/EndToEnd.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/EndToEnd.yml b/.github/workflows/EndToEnd.yml index b4c1394..d16912b 100644 --- a/.github/workflows/EndToEnd.yml +++ b/.github/workflows/EndToEnd.yml @@ -20,7 +20,7 @@ jobs: steps: - uses: actions/checkout@v3 - - uses: webfactory/ssh-agent@v0.5.4 + - uses: webfactory/ssh-agent@v0.7.0 with: ssh-private-key: ${{ secrets.RECURSION_SPACE_ACCESS_TOKEN}} From 30a44f16f4620110b4494f9981323b57006a9f3a Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Fri, 9 Dec 2022 04:10:05 +0000 Subject: [PATCH 046/164] chore(deps): bump actions/setup-python from 4.1.0 to 4.3.1 Bumps [actions/setup-python](https://github.com/actions/setup-python) from 4.1.0 to 4.3.1. - [Release notes](https://github.com/actions/setup-python/releases) - [Commits](https://github.com/actions/setup-python/compare/v4.1.0...v4.3.1) --- updated-dependencies: - dependency-name: actions/setup-python dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] --- .github/workflows/RecursionTests.yml | 2 +- .github/workflows/TestInstaller.yml | 2 +- .github/workflows/pylint.yml | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/RecursionTests.yml b/.github/workflows/RecursionTests.yml index 2d804ea..c7afd15 100644 --- a/.github/workflows/RecursionTests.yml +++ b/.github/workflows/RecursionTests.yml @@ -16,7 +16,7 @@ jobs: steps: - uses: actions/checkout@v3 - name: Set up Python 3.9 - uses: actions/setup-python@v4.1.0 + uses: actions/setup-python@v4.3.1 with: python-version: 3.9 - name: Install dependencies diff --git a/.github/workflows/TestInstaller.yml b/.github/workflows/TestInstaller.yml index d2b6524..1f8c587 100644 --- a/.github/workflows/TestInstaller.yml +++ b/.github/workflows/TestInstaller.yml @@ -10,7 +10,7 @@ jobs: steps: - uses: actions/checkout@v3 - name: Set up Python 3.8 - uses: actions/setup-python@v4.1.0 + uses: actions/setup-python@v4.3.1 with: python-version: 3.8 diff --git a/.github/workflows/pylint.yml b/.github/workflows/pylint.yml index d244d1d..79c1ed1 100644 --- a/.github/workflows/pylint.yml +++ b/.github/workflows/pylint.yml @@ -10,7 +10,7 @@ jobs: steps: - uses: actions/checkout@v3 - name: Set up Python 3.8 - uses: actions/setup-python@v4.1.0 + uses: actions/setup-python@v4.3.1 with: python-version: 3.8 From 0e9351fa08ba8cabb12f67003541d15a0b31bdc7 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Mon, 12 Dec 2022 00:23:43 -0500 Subject: [PATCH 047/164] linting --- .../workflows/{pylint.yml => CI_Pylint.yml} | 2 +- openpod/modules/rec_api.py | 66 +++++++++++-------- openpod/modules/rec_lan.py | 24 +++---- openpod/modules/rec_log.py | 64 ++++++++++-------- openpod/pod.py | 45 +++++++------ openpod/updater.py | 23 ++++--- 6 files changed, 120 insertions(+), 104 deletions(-) rename .github/workflows/{pylint.yml => CI_Pylint.yml} (97%) diff --git a/.github/workflows/pylint.yml b/.github/workflows/CI_Pylint.yml similarity index 97% rename from .github/workflows/pylint.yml rename to .github/workflows/CI_Pylint.yml index 79c1ed1..d53e45c 100644 --- a/.github/workflows/pylint.yml +++ b/.github/workflows/CI_Pylint.yml @@ -1,4 +1,4 @@ -name: Code Quality +name: CI | Pylint on: [push] diff --git a/openpod/modules/rec_api.py b/openpod/modules/rec_api.py index cdeb8d5..8dd08f4 100644 --- a/openpod/modules/rec_api.py +++ b/openpod/modules/rec_api.py @@ -16,11 +16,13 @@ from modules import rec_gpio from settings import LED_IO -#Performs all API calls to the server, functions should be used as a thread. +# Performs all API calls to the server, functions should be used as a thread. # ---------------------------------------------------------------------------- # # Request Update For All Information # # ---------------------------------------------------------------------------- # + + def pull_data_dump(): ''' Request updated infromation from the server. @@ -31,8 +33,8 @@ def pull_data_dump(): # ----------------------------- Pull Member Data ----------------------------- # with open("/opt/RecursionHub/data/dump.json", "w", encoding="utf-8") as file: member_info = requests.get(f'{settings.RECURSION_API_URL}/v1/members', headers={ - 'Authorization' : f'Token {system_data["Token"]}' - }) + 'Authorization': f'Token {system_data["Token"]}' + }, timeout=10) responce = member_info.json() json.dump(responce, file) @@ -40,8 +42,8 @@ def pull_data_dump(): # --------------------------- Pull Operator(s) Data -------------------------- # with open("/opt/RecursionHub/data/owners.json", "w", encoding="utf-8") as file: operators_info = requests.get(f'{settings.RECURSION_API_URL}/v1/operators', headers={ - 'Authorization' : f'Token {system_data["Token"]}' - }) + 'Authorization': f'Token {system_data["Token"]}' + }, timeout=10) responce = operators_info.json() json.dump(responce, file) @@ -49,9 +51,10 @@ def pull_data_dump(): # -------------------------------- Nodes Data -------------------------------- # with open("/opt/RecursionHub/data/nodes.json", "w", encoding="utf-8") as file: nodes_info = requests.get( - f'{settings.RECURSION_API_URL}/v1/nodes', - headers={'Authorization' : f'Token {system_data["Token"]}'} - ) + f'{settings.RECURSION_API_URL}/v1/nodes', + headers={'Authorization': f'Token {system_data["Token"]}'}, + timeout=10 + ) responce = nodes_info.json() json.dump(responce, file) @@ -59,9 +62,10 @@ def pull_data_dump(): # ----------------------------- Pull Permissions ----------------------------- # with open("/opt/RecursionHub/data/permissions.json", "w", encoding="utf-8") as file: permissions_info = requests.get( - f'{settings.RECURSION_API_URL}/v1/permissions', - headers={'Authorization' : f'Token {system_data["Token"]}'} - ) + f'{settings.RECURSION_API_URL}/v1/permissions', + headers={'Authorization': f'Token {system_data["Token"]}'}, + timeout=10 + ) responce = permissions_info.json() json.dump(responce, file) @@ -71,6 +75,8 @@ def pull_data_dump(): # ---------------------------------------------------------------------------- # # Set or Update Timezone # # ---------------------------------------------------------------------------- # + + def update_time_zone(): ''' API call to set the HUB timezone with the user selected option. @@ -78,13 +84,14 @@ def update_time_zone(): with open("system.json", "r+", encoding="utf-8") as file: system_data = json.load(file) spaces_info = requests.get( - f'{settings.RECURSION_API_URL}/v1/spaces', - headers={'Authorization' : f'Token {system_data["Token"]}'} - ) + f'{settings.RECURSION_API_URL}/v1/spaces', + headers={'Authorization': f'Token {system_data["Token"]}'}, + timeout=10 + ) responce = spaces_info.json() - system_data.update( {"timezone":responce[0]["timezone"]} ) + system_data.update({"timezone": responce[0]["timezone"]}) file.seek(0) json.dump(system_data, file) file.truncate() @@ -95,7 +102,7 @@ def update_time_zone(): # ---------------------------------------------------------------------------- # # Register Hub With Recursion # # ---------------------------------------------------------------------------- # -def register_hub(): #Needs updated! +def register_hub(): # Needs updated! ''' API to register the hub. ''' @@ -103,15 +110,15 @@ def register_hub(): #Needs updated! system_config = json.load(system_file) url = settings.RecursionURL+'/hubs/' - payload_tuples = {'serial':f"{system_config['serial']}"} - output = requests.post(url, payload_tuples, auth=('OwnerA', 'Password@1')) + payload_tuples = {'serial': f"{system_config['serial']}"} + output = requests.post(url, payload_tuples, auth=('OwnerA', 'Password@1'), timeout=10) responce = output.json() log_api.info("Hub registration responce: %s", responce) with open("/opt/RecursionHub/system.json", "r+", encoding="utf-8") as file: data = json.load(file) - data.update( {"HUBid":responce["id"]} ) + data.update({"HUBid": responce["id"]}) file.seek(0) json.dump(data, file) log_api.info("Hub registered and assigned HUBid: %s", responce["id"]) @@ -129,12 +136,12 @@ def link_hub(): system_data = json.load(file) hubs_info = requests.get(f'{settings.RECURSION_API_URL}/v1/hubs', headers={ - 'Authorization' : f'Token {system_data["Token"]}' - }) + 'Authorization': f'Token {system_data["Token"]}' + }, timeout=10) responce = hubs_info.json() - system_data.update( {"facility":responce[0]["facility"]} ) + system_data.update({"facility": responce[0]["facility"]}) file.seek(0) json.dump(system_data, file) file.truncate() @@ -165,7 +172,7 @@ def pair_node(node_mac): requests.post( f'{settings.RECURSION_API_URL}/v1/nodes', data=post_content, - headers={'Authorization' : f'Token {system_config["Token"]}'} + headers={'Authorization': f'Token {system_config["Token"]}'}, timeout=10 ) pull_data_dump() @@ -188,7 +195,8 @@ def keepalive(): f'''{system_data["serial"]}/''' f'''{system_data["CurrentVersion"]}/''' f'''{hash_data()["combined"]}/''', - headers={'Authorization' : f'Token {system_data["Token"]}'} + headers={'Authorization': f'Token {system_data["Token"]}'}, + timeout=10 ) else: @@ -196,7 +204,8 @@ def keepalive(): f'''{settings.RecursionURL}/hub/keepalive/''' f'''{system_data["serial"]}/''' f'''{system_data["CurrentVersion"]}/''', - headers={'Authorization' : f'Token {system_data["Token"]}'} + headers={'Authorization': f'Token {system_data["Token"]}'}, + timeout=10 ) except requests.exceptions.RequestException as err: @@ -205,18 +214,16 @@ def keepalive(): except OSError as err: log_api.error("Keepalive OSError: %s", err) - finally: try: - log_api.debug('Keepalive check again in 30 seconds from now.') # DEBUG POINT + log_api.debug('Keepalive check again in 30 seconds from now.') # DEBUG POINT threading.Timer(30.0, keepalive).start() except RuntimeError as err: log_api.error("Keepalive thread RuntimeError: %s", err) - # ---------------------------------------------------------------------------- # # Logging To Recursion # # ---------------------------------------------------------------------------- # @@ -239,7 +246,8 @@ def access_log(card_number, action, result, node, facility): requests.post( f'{settings.RecursionURL}/accesslog/', data=payload, - headers={'Authorization' : f'Token {system_config["Token"]}'} + headers={'Authorization': f'Token {system_config["Token"]}'}, + timeout=10 ) except RuntimeError as err: diff --git a/openpod/modules/rec_lan.py b/openpod/modules/rec_lan.py index 0624cea..cc194f6 100644 --- a/openpod/modules/rec_lan.py +++ b/openpod/modules/rec_lan.py @@ -16,7 +16,6 @@ from modules import rec_gpio - # ------------ Triggers visual indicators based on network status. ----------- # def monitor_network(last_network_status=5, thread_delay=30.0): ''' @@ -61,7 +60,7 @@ def monitor_network(last_network_status=5, thread_delay=30.0): raise RuntimeError('Network monitoring thread has failed.') from err finally: - network_log.debug('Thread timer set for %s second from now.', thread_delay) # DEBUG POINT + network_log.debug('Thread timer set for %s second from now.', thread_delay) # DEBUG POINT network_watch_thread = threading.Timer( thread_delay, @@ -73,6 +72,7 @@ def monitor_network(last_network_status=5, thread_delay=30.0): return True + def test_network(): ''' Performs tired network tests. @@ -90,7 +90,7 @@ def test_network(): if recursion_connection() is False: return 2 - #All checks passed and Recursion server is reachable + # All checks passed and Recursion server is reachable # network_log.info('LAN Check Pass') # This would be called every 10 seconds return 3 @@ -104,7 +104,7 @@ def networked(): ''' local_ip = get_ip()[1] - if local_ip == "127.0.0.1": + if local_ip == "127.0.0.1": return False return True @@ -115,17 +115,17 @@ def internet_on(): Performs requests to known external servers. ''' try: - if requests.get('https://recursion.space').status_code == requests.codes.ok: + if requests.get('https://recursion.space', timeout=10).status_code == requests.codes.ok: return True except requests.exceptions.RequestException: try: - if requests.get('https://google.com').status_code == requests.codes.ok: + if requests.get('https://google.com', timeout=10).status_code == requests.codes.ok: return True except requests.exceptions.RequestException: try: - if requests.get('https://amazon.com').status_code == requests.codes.ok: + if requests.get('https://amazon.com', timeout=10).status_code == requests.codes.ok: return True except requests.exceptions.RequestException: @@ -139,7 +139,7 @@ def recursion_connection(): Checks if Recursion.Space is reachable. ''' try: - req = requests.get('https://recursion.space') + req = requests.get('https://recursion.space', timeout=10) if req.status_code == requests.codes.ok: return True @@ -171,12 +171,12 @@ def get_ip(): try: local_ip = ([l for l in ([ip for ip in socket.gethostbyname_ex(socket.gethostname())[2] - if not ip.startswith("127.")][:1], [[(s.connect(('8.8.8.8', 53)), - s.getsockname()[0], s.close()) for s in [socket.socket(socket.AF_INET, - socket.SOCK_DGRAM)]][0][1]]) if l][0][0]) + if not ip.startswith("127.")][:1], [[(s.connect(('8.8.8.8', 53)), + s.getsockname()[0], s.close()) for s in [socket.socket(socket.AF_INET, + socket.SOCK_DGRAM)]][0][1]]) if l][0][0]) # network_log.info("Hub's local IP address: {0}".format(local_ip)) - #Prevent constant log writting since now in loop + # Prevent constant log writting since now in loop except OSError as err: network_log.error('Unable to get local IP address with error: %s', err) diff --git a/openpod/modules/rec_log.py b/openpod/modules/rec_log.py index 831e91e..d4869cf 100644 --- a/openpod/modules/rec_log.py +++ b/openpod/modules/rec_log.py @@ -12,7 +12,7 @@ from zipfile import ZipFile import requests import simplejson as json -#https://stackoverflow.com/questions/21663800/python-make-a-list-generator-json-serializable +# https://stackoverflow.com/questions/21663800/python-make-a-list-generator-json-serializable # from modules import rec_lan @@ -28,7 +28,7 @@ # ---------------------------------------------------------------------------- # standard_format = logging.Formatter('%(asctime)s %(levelname)s: %(message)s', '%y-%m-%d %H:%M:%S') logging.basicConfig() -logging.getLogger().setLevel(logging.INFO) # Sets default level to INFO for all logs +logging.getLogger().setLevel(logging.INFO) # Sets default level to INFO for all logs console = logging.StreamHandler() console.setFormatter(standard_format) @@ -44,9 +44,9 @@ exception_log.setLevel(logging.DEBUG) try: - exception_log_file = logging.FileHandler('/opt/RecursionHub/logs/exception.log', mode ='a') + exception_log_file = logging.FileHandler('/opt/RecursionHub/logs/exception.log', mode='a') except FileNotFoundError: - exception_log_file = logging.FileHandler('tests/exception.log', mode ='a') # CI Testing + exception_log_file = logging.FileHandler('tests/exception.log', mode='a') # CI Testing exception_log_file.setFormatter(standard_format) exception_log.addHandler(exception_log_file) @@ -59,9 +59,9 @@ log_api.setLevel(logging.DEBUG) try: - log_api_file = logging.FileHandler('/opt/RecursionHub/logs/api.log', mode ='a') + log_api_file = logging.FileHandler('/opt/RecursionHub/logs/api.log', mode='a') except FileNotFoundError: - log_api_file = logging.FileHandler('tests/api.log', mode ='a') # CI Testing + log_api_file = logging.FileHandler('tests/api.log', mode='a') # CI Testing log_api_file.setFormatter(standard_format) log_api.addHandler(log_api_file) @@ -74,9 +74,9 @@ network_log.setLevel(logging.DEBUG) try: - network_log_file = logging.FileHandler('/opt/RecursionHub/logs/network.log', mode ='a') + network_log_file = logging.FileHandler('/opt/RecursionHub/logs/network.log', mode='a') except FileNotFoundError: - network_log_file = logging.FileHandler('tests/network.log', mode ='a') # CI Testing + network_log_file = logging.FileHandler('tests/network.log', mode='a') # CI Testing network_log_file.setFormatter(standard_format) network_log.addHandler(network_log_file) @@ -89,9 +89,9 @@ log_xbee.setLevel(logging.DEBUG) try: - xbee_log_file = logging.FileHandler('/opt/RecursionHub/logs/xbee.log', mode ='a') + xbee_log_file = logging.FileHandler('/opt/RecursionHub/logs/xbee.log', mode='a') except FileNotFoundError: - xbee_log_file = logging.FileHandler('tests/xbee.log', mode ='a') # CI Testing + xbee_log_file = logging.FileHandler('tests/xbee.log', mode='a') # CI Testing xbee_log_file.setFormatter(standard_format) log_xbee.addHandler(xbee_log_file) @@ -104,23 +104,23 @@ mqtt_log.setLevel(logging.DEBUG) try: - mqtt_log_file = logging.FileHandler('/opt/RecursionHub/logs/mqtt.log', mode ='a') + mqtt_log_file = logging.FileHandler('/opt/RecursionHub/logs/mqtt.log', mode='a') except FileNotFoundError: - mqtt_log_file = logging.FileHandler('tests/mqtt.log', mode ='a') # CI Testing + mqtt_log_file = logging.FileHandler('tests/mqtt.log', mode='a') # CI Testing mqtt_log_file.setFormatter(standard_format) mqtt_log.addHandler(mqtt_log_file) mqtt_log.addHandler(console) -#Logging configurations, use logfile for critial events, use transaction as print alternative +# Logging configurations, use logfile for critial events, use transaction as print alternative # Console handler decides what information to also "print" based on logging level logfile = logging.getLogger('standardlog') try: - fileHandler = logging.FileHandler('/opt/RecursionHub/logs/RecursionLog.log', mode ='a') + fileHandler = logging.FileHandler('/opt/RecursionHub/logs/RecursionLog.log', mode='a') except FileNotFoundError: - fileHandler = logging.FileHandler('tests/RecursionLog.log', mode ='a') #For CI + fileHandler = logging.FileHandler('tests/RecursionLog.log', mode='a') # For CI fileHandler.setFormatter(standard_format) @@ -134,15 +134,16 @@ transaction = logging.getLogger('transaction') try: - fileH = logging.FileHandler('/opt/RecursionHub/logs/TransactionLog.log', mode ='a') + fileH = logging.FileHandler('/opt/RecursionHub/logs/TransactionLog.log', mode='a') except FileNotFoundError: - fileH = logging.FileHandler('tests/TransactionLog.log', mode ='a') #For CI + fileH = logging.FileHandler('tests/TransactionLog.log', mode='a') # For CI fileH.setFormatter(standard_format) transaction.setLevel(logging.DEBUG) transaction.addHandler(console) transaction.addHandler(fileH) + def publog(level, note): ''' Sets the log level. @@ -207,6 +208,8 @@ def snapshot(public, local): # ---------------------------------------------------------------------------- # # Send Diagnostics # # ---------------------------------------------------------------------------- # + + def dump_diagnostics(): ''' Send the summary of setting to Recurson.Space @@ -220,14 +223,17 @@ def dump_diagnostics(): try: requests.put(f'{settings.RecursionURL}/v1/diagnostics/', - json={"snapshot":payload}, - headers={'Authorization' : f"Token {system_config['Token']}"} - ) + json={"snapshot": payload}, + headers={'Authorization': f"Token {system_config['Token']}"}, + timeout=10 + ) except requests.exceptions.RequestException as err: exception_log.error('Unable to submit diagnostics. Error: %s', err) # --------------------------- Zip & Send Log Files --------------------------- # + + def zip_send(): ''' Zip all log files togeather and send to Recursion.Space @@ -257,8 +263,9 @@ def zip_send(): with open(zip_file, 'rb') as zip_file_logs: requests.post( f'{settings.RecursionURL}/files/upload/external/hublogs/', - files={"file":zip_file_logs}, - headers={'Authorization':f'Token {system_data["Token"]}'} + files={"file": zip_file_logs}, + headers={'Authorization': f'Token {system_data["Token"]}'}, + timeout=10 ) else: @@ -271,7 +278,6 @@ def zip_send(): exception_log.error("zip_send TypeError: %s", err) - # ---------------------------------------------------------------------------- # # Hash Stored Data # # ---------------------------------------------------------------------------- # @@ -292,7 +298,7 @@ def hash_data(): with open("/opt/RecursionHub/data/owners.json", "rb") as owners_file: owners_hash = hashlib.md5( - owners_file.read() + owners_file.read() ).hexdigest() with open("/opt/RecursionHub/data/permissions.json", "rb") as perm_file: @@ -304,11 +310,11 @@ def hash_data(): f"{dump_hash}, {nodes_hash}, {owners_hash}, {permissions_hash}".encode()) return { - 'combined':combined_hash.hexdigest(), - 'dumpHash':dump_hash, - 'nodesHash':nodes_hash, - 'ownersHash':owners_hash, - 'permissionsHash':permissions_hash + 'combined': combined_hash.hexdigest(), + 'dumpHash': dump_hash, + 'nodesHash': nodes_hash, + 'ownersHash': owners_hash, + 'permissionsHash': permissions_hash } except FileNotFoundError as err: diff --git a/openpod/pod.py b/openpod/pod.py index 5f79d17..612aa11 100644 --- a/openpod/pod.py +++ b/openpod/pod.py @@ -6,7 +6,7 @@ """ import sys -import os.path #Allows modules to access from directory above. +import os.path # Allows modules to access from directory above. import json import threading from time import sleep @@ -27,7 +27,6 @@ exception_log.error("%s", err) - # Not sure if the next section is required. sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), os.path.pardir))) @@ -35,26 +34,29 @@ systemConfig = json.load(system_file) Version = systemConfig['CurrentVersion'] -#Inserts path to refrence then starts importing modules. +# Inserts path to refrence then starts importing modules. sys.path.insert(0, f"./{Version}") sys.path.insert(1, f"./{Version}/modules") sys.path.append("..") config.XBEE_FLAG = False + + def incoming_xbee_data(): ''' Required functions to handle events and related status checking activities. ''' config.XBEE_FLAG = True + pub.subscribe(incoming_xbee_data, 'xbee_rx') -begin_xbee = threading.Thread(target = rec_xbee.listing) +begin_xbee = threading.Thread(target=rec_xbee.listing) begin_xbee.start() -#Register HUB with Recursion +# Register HUB with Recursion if not systemConfig.get('HUBid', False): rec_api.register_hub() @@ -62,13 +64,13 @@ def incoming_xbee_data(): # ------------------------------- TEMP SOLUTION ------------------------------ # try: URL = f'{settings.RecursionURL}/obtaintoken/{systemConfig["serial"]}/' - r = requests.get(URL) + r = requests.get(URL, timeout=10) if r.status_code == 201: - print (f"Hub Toekn: {r.text}") + print(f"Hub Toekn: {r.text}") with open("/opt/RecursionHub/system.json", "r+", encoding="UTF-8") as file: data = json.load(file) - data.update( {"Token":f"{r.text}"} ) + data.update({"Token": f"{r.text}"}) file.seek(0) json.dump(data, file) file.truncate() @@ -87,15 +89,15 @@ def incoming_xbee_data(): rec_api.keepalive() -MQTTlisten = threading.Thread(target = rec_mqtt.mqtt_rx) +MQTTlisten = threading.Thread(target=rec_mqtt.mqtt_rx) MQTTlisten.start() if settings.Pi: - led_io_thread = threading.Thread(target = rec_gpio.led_io_thread) + led_io_thread = threading.Thread(target=rec_gpio.led_io_thread) led_io_thread.start() -#Only pull info if hub has already been paired with a facility. +# Only pull info if hub has already been paired with a facility. try: if 'facility' in systemConfig: rec_log.publog("info", "Pulling any missing data.") @@ -106,7 +108,7 @@ def incoming_xbee_data(): if settings.Pi: rec_gpio.state(settings.LED_IO, 1, 1) # Slow blink ready to pair to a facility. - zip_send() # Send latest log files on boot. + zip_send() # Send latest log files on boot. except Exception as err: # pylint: disable=W0703 rec_log.publog("error", f"Error occured when pulling data: {err}") @@ -118,25 +120,26 @@ def process_xbee_data(): ''' Triggered when there is available XBee data. ''' - xbee_frame_info = rec_xbee.receive() #Reads in recived XBee data. + xbee_frame_info = rec_xbee.receive() # Reads in recived XBee data. rec_gpio.state(settings.LED_IO, 1) if xbee_frame_info[2] == 0: - rec_xbee.transmit(xbee_frame_info[0],"30") + rec_xbee.transmit(xbee_frame_info[0], "30") elif xbee_frame_info[2] == 1: - rec_xbee.transmit(xbee_frame_info[0],"31") + rec_xbee.transmit(xbee_frame_info[0], "31") elif xbee_frame_info[2] == 2: - rec_xbee.transmit(xbee_frame_info[0],"32") + rec_xbee.transmit(xbee_frame_info[0], "32") elif xbee_frame_info[2] == 5: pass else: - rec_xbee.transmit(xbee_frame_info[0],"32") + rec_xbee.transmit(xbee_frame_info[0], "32") + -#https://stackoverflow.com/questions/10926328/efficient-and-fast-python-while-loop-while-using-sleep -#https://stackoverflow.com/questions/17553543/pyserial-non-blocking-read-loop +# https://stackoverflow.com/questions/10926328/efficient-and-fast-python-while-loop-while-using-sleep +# https://stackoverflow.com/questions/17553543/pyserial-non-blocking-read-loop while True: if config.XBEE_FLAG: - rec_gpio.state(settings.LED_IO, .125, .125) #Indicate incoming XBee data. - DataProcessing = threading.Thread(target = process_xbee_data) + rec_gpio.state(settings.LED_IO, .125, .125) # Indicate incoming XBee data. + DataProcessing = threading.Thread(target=process_xbee_data) DataProcessing.start() sleep(.05) config.XBEE_FLAG = False diff --git a/openpod/updater.py b/openpod/updater.py index 0786d01..ef9b08c 100644 --- a/openpod/updater.py +++ b/openpod/updater.py @@ -6,7 +6,7 @@ Call the function 'update_hub' to pull the latest update. """ -#Triggered by the user from the web interface to update the current version. +# Triggered by the user from the web interface to update the current version. import re import sys @@ -21,6 +21,7 @@ from modules.rec_log import exception_log + def current_hub_version(): ''' Reades the curent version number from the system file. @@ -38,10 +39,11 @@ def update_version_name(): with open('system.json', 'r', encoding="utf-8") as system_file: system_data = json.load(system_file) - request_response= requests.get( + request_response = requests.get( f'{settings.RecursionURL}/updatehub/', - headers={'Authorization' : f"Token {system_data['Token']}"} - ) + headers={'Authorization': f"Token {system_data['Token']}"}, + timeout=10 + ) response_data = request_response.headers['content-disposition'] return re.findall("filename=(.+)", response_data)[0] @@ -62,8 +64,6 @@ def download_update(): urllib.request.install_opener(opener) urllib.request.urlretrieve(f'{settings.RecursionURL}/updatehub/', fname) - - # exception_log.info("Update version pulled: %s", re.findall(r"(.+?)(\.[^.]*$|$)", fname)[0][0]) return re.findall(r"(.+?)(\.[^.]*$|$)", fname)[0][0] @@ -78,7 +78,7 @@ def unzip_update(): with zipfile.ZipFile(f'{new_version}.zip', 'r') as zip_ref: zip_ref.extractall(f'{new_version}/') - subprocess.call(['rm', f'{new_version}.zip']) #Cleaning up downloaded file. + subprocess.call(['rm', f'{new_version}.zip']) # Cleaning up downloaded file. return new_version @@ -93,20 +93,19 @@ def update_hub(): except RuntimeError as err: exception_log.error("Unable to pull update with error: %s", err) - new_version = current_hub_version() #If unable to update, just run the current version. + new_version = current_hub_version() # If unable to update, just run the current version. finally: with open("system.json", "r+", encoding="utf-8") as file: data = json.load(file) - data.update( {"CurrentVersion":new_version} ) + data.update({"CurrentVersion": new_version}) file.seek(0) json.dump(data, file) file.truncate() - - #Relaunch with new program if update was sucessful. + # Relaunch with new program if update was sucessful. try: - #Kill process that should be triggered to re-open by bash script. + # Kill process that should be triggered to re-open by bash script. subprocess.call(['pkill', '-f', 'hub.py']) subprocess.call(['pkill', '-f', 'HUB_Launcher.py']) except RuntimeError as err: From 708452ee0154d58f2c39d971b765ae139cf7e4e6 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Mon, 12 Dec 2022 01:28:46 -0500 Subject: [PATCH 048/164] linting --- README.md | 6 +++--- openpod/modules/rec_lan.py | 2 +- openpod/updater.py | 7 ++++++- 3 files changed, 10 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 23dc00c..25760a6 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ [![End To End Integration](https://github.com/RecursionSpace/OpenPod/actions/workflows/EndToEnd.yml/badge.svg)](https://github.com/RecursionSpace/OpenPod/actions/workflows/EndToEnd.yml)   -[![Code Quality](https://github.com/RecursionSpace/OpenPod/actions/workflows/pylint.yml/badge.svg)](https://github.com/RecursionSpace/OpenPod/actions/workflows/pylint.yml) +[![CI | Pylint](https://github.com/RecursionSpace/OpenPod/actions/workflows/CI_Pylint.yml/badge.svg)](https://github.com/RecursionSpace/OpenPod/actions/workflows/CI_Pylint.yml)   [![Script Check](https://github.com/RecursionSpace/OpenPod/actions/workflows/ShellCheck.yml/badge.svg)](https://github.com/RecursionSpace/OpenPod/actions/workflows/ShellCheck.yml)   @@ -42,7 +42,7 @@ sudo wget -qO- openpod.recursion.space | bash /dev/stdin [options] [arguments*] ``` | Option Flag | Description | Example | -| :---------: | ------------------------- | -------------------------------------------------------------------------------- | +|:-----------:|---------------------------|----------------------------------------------------------------------------------| | -h | Help | sudo wget -qO- openpod.recursion.space \| bash /dev/stdin -h | | -u | Set Custom URL endpoint | sudo wget -qO- openpod.recursion.space \| bash /dev/stdin -u dev.recursion.space | | -d | Enable Debug Installation | sudo wget -qO- openpod.recursion.space \| bash /dev/stdin -d | @@ -111,7 +111,7 @@ The MQTT protocol is used for server to Pod communication, allowing the Pod to l Communication to Pods is accomplished from quick commands represented by a hexadecimal number. | Command Number | Command | -| -------------- | --------------------- | +|----------------|-----------------------| | AA (170) | Facility ID Available | | BA (186) | Pull User Dump | | CA (202) | Update Available | diff --git a/openpod/modules/rec_lan.py b/openpod/modules/rec_lan.py index cc194f6..30a37c3 100644 --- a/openpod/modules/rec_lan.py +++ b/openpod/modules/rec_lan.py @@ -155,7 +155,7 @@ def get_ip(): ''' if internet_on() is True: try: - public_ip = requests.get('https://ip.42.pl/raw', verify=False).text + public_ip = requests.get('https://ip.42.pl/raw', verify=False, timeout=10).text except requests.exceptions.RequestException as err: public_ip = f'Failed to get public IP with error: {err}' diff --git a/openpod/updater.py b/openpod/updater.py index ef9b08c..035b5d5 100644 --- a/openpod/updater.py +++ b/openpod/updater.py @@ -87,6 +87,11 @@ def update_hub(): ''' Main code called to update the hub. ''' + with open("system.json", "r+", encoding="utf-8") as file: + data = json.load(file) + + new_version = data['CurrentVersion'] + try: exception_log.info("Update Started") new_version = unzip_update() @@ -103,7 +108,7 @@ def update_hub(): json.dump(data, file) file.truncate() - # Relaunch with new program if update was sucessful. + # Relaunch with new program if update was successful. try: # Kill process that should be triggered to re-open by bash script. subprocess.call(['pkill', '-f', 'hub.py']) From 74a18ee08cad41da1a9a45be65d4a57825b7b67d Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Mon, 12 Dec 2022 22:31:19 -0500 Subject: [PATCH 049/164] Updated Installer & Requirements --- ...TestInstaller.yml => CI_TestInstaller.yml} | 6 +- README.md | 3 +- installer.sh | 55 +++++++++++++----- requirements.txt | 56 +++++-------------- 4 files changed, 57 insertions(+), 63 deletions(-) rename .github/workflows/{TestInstaller.yml => CI_TestInstaller.yml} (81%) diff --git a/.github/workflows/TestInstaller.yml b/.github/workflows/CI_TestInstaller.yml similarity index 81% rename from .github/workflows/TestInstaller.yml rename to .github/workflows/CI_TestInstaller.yml index 1f8c587..084b969 100644 --- a/.github/workflows/TestInstaller.yml +++ b/.github/workflows/CI_TestInstaller.yml @@ -1,4 +1,4 @@ -name: Test Installer +name: CI | Installer on: [push] @@ -9,10 +9,10 @@ jobs: steps: - uses: actions/checkout@v3 - - name: Set up Python 3.8 + - name: Set up Python 3.11 uses: actions/setup-python@v4.3.1 with: - python-version: 3.8 + python-version: 3.11 - name: Install dependencies run: | diff --git a/README.md b/README.md index 25760a6..dc24b1b 100644 --- a/README.md +++ b/README.md @@ -121,7 +121,7 @@ Communication to Pods is accomplished from quick commands represented by a hexad # Updates -The end-user triggers updates via the webserver to run them with minimal interruption. An MQTT message is sent to the Pod to initiate the updating process. The update is a zip file downloaded from the server. +The end-user triggers updates via the web server to run them with minimal interruption. An MQTT message is sent to the Pod to initiate the updating process. The update is a zip file downloaded from the server. Download zip file from server and store in the update folder. Unzip contents of a zip file @@ -143,7 +143,6 @@ The Recursion System uses several logging points to be used both for troubleshoo **System.Snapshot** file contains a JSON summary of the Pod for debugging purposes. The information available in this file must also be readily accessible from the server for troubleshooting purposes. ```json - [ “local_ip” = “xx.xx.xx.xx”, ] diff --git a/installer.sh b/installer.sh index c08ec05..29eaaf1 100644 --- a/installer.sh +++ b/installer.sh @@ -1,5 +1,7 @@ #!/bin/bash +# Installer for OpenPod, for more information see https://github.com/blokbot-io/OpenBlok/blob/master/install.sh + # ---------------------------------------------------------------------------- # # Help # # ---------------------------------------------------------------------------- # @@ -37,27 +39,50 @@ while getopts ":hud" flags; do esac done +# -------------------------------- Verify Root ------------------------------- # +if [ "$EUID" -ne 0 ] + then echo "Please run as root with sudo." + exit +fi # ---------------------------------------------------------------------------- # -# Installer # +# Dependencies # # ---------------------------------------------------------------------------- # -# ---------------------- Update Package List and Upgrade --------------------- # -sudo apt-get update -y && sudo apt-get upgrade -y - # ---------------------------- Update System Time ---------------------------- # sudo timedatectl set-timezone UTC sudo apt-get install chrony -y sudo chronyd -q -# ------------------------- Install Bash Requirements ------------------------ # -sudo apt-get install jq -y -sudo apt-get install unzip -y - -# ---------------------------- Install Python 3.10 --------------------------- # -sudo apt install software-properties-common -y -sudo add-apt-repository ppa:deadsnakes/ppa -sudo apt install python3.10 -y +# ----------------------------------- unzip ---------------------------------- # +REQUIRED_PKG="unzip" +PKG_OK=$(dpkg-query -W --showformat='${Status}\n' $REQUIRED_PKG|grep "install ok installed") +if [ "" = "$PKG_OK" ]; then + echo "No $REQUIRED_PKG. Setting up $REQUIRED_PKG..." + sudo apt-get install unzip -y +else + echo "unzip already installed, skipping..." +fi + +# ------------------------------------ jq ------------------------------------ # +REQUIRED_PKG="jq" +PKG_OK=$(dpkg-query -W --showformat='${Status}\n' $REQUIRED_PKG|grep "install ok installed") +if [ "" = "$PKG_OK" ]; then + echo "No $REQUIRED_PKG. Setting up $REQUIRED_PKG..." + sudo apt-get install jq -y +else + echo "jq already installed, skipping..." +fi + +# -------------------------------- Python 3.11 ------------------------------- # +pytohn_version=$(python3 -c 'import sys; print(".".join(map(str, sys.version_info[0:2])))') +if [ "$pytohn_version" != "3.11" ]; then + sudo apt install software-properties-common -y + yes '' | sudo add-apt-repository ppa:deadsnakes/ppa + sudo apt-get install python3.1 -y +else + echo "Python 3.11 already installed" +fi # ------------------------------- Clone OpenPod ------------------------------ # set -e # Exit when any command fails. @@ -66,8 +91,8 @@ cd /opt sudo git clone --single-branch --branch release https://github.com/RecursionSpace/OpenPod.git # ----------------------------- Setup Enviroment ----------------------------- # -sudo apt-get install python3.10-venv -y -sudo python3.10 -m venv /opt/OpenPod/env +sudo apt-get install python3.11-venv -y +sudo python3.11 -m venv /opt/OpenPod/env source /opt/OpenPod/env/bin/activate sudo pip install --no-input -U -r /opt/OpenPod/requirements.txt --no-cache-dir --no-dependencies @@ -107,7 +132,7 @@ After=network.target Type=simple User=root WorkingDirectory=/opt/OpenPod -ExecStart=/opt/OpenPod/env/bin/python3.10 /opt/OpenPod/OpenPod.py +ExecStart=/opt/OpenPod/env/bin/python3.11 /opt/OpenPod/OpenPod.py Restart=always RestartSec=10 diff --git a/requirements.txt b/requirements.txt index b269b0e..7d8cc97 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,43 +1,13 @@ -asgiref>=3.2.7 -asn1crypto>=0.24.0 -attrs>=17.4.0 -blinker>=1.4 -certifi>=2020.6.20 -chardet>=3.0.4 -configobj>=5.0.6 -constantly>=15.1.0 -config>=0.5.0 -Deprecated>=1.2.10 -gitdb>=4.0.5 -GitPython>=3.1.8 -hyperlink>=17.3.1 -idna>=2.10 -incremental>=16.10.1 -Interactive>=1.0.1 -jsonpatch>=1.16 -jsonpointer>=1.10 -jsonschema>=2.6.0 -keyring>=10.6.0 -keyrings.alt>=3.0 -MarkupSafe>=1.0 -netifaces>=0.10.4 -oauthlib>=2.0.6 -paho-mqtt -paramiko>=2.7.1 -pubsub>=0.1.2 -pyasn1>=0.4.2 -pyasn1-modules>=0.2.1 -PyJWT>=1.7.1 -pyOpenSSL>=17.5.0 -Pypubsub>=4.0.3 -pyserial>=3.4 -python-crontab>=2.5.1 -python-dateutil>=2.8.1 -pytimeparse>=1.1.8 -pytz>=2020.1 -requests>=2.24.0 -requests-unixsocket>=0.1.5 -RPi.GPIO>=0.7.0 -SecretStorage>=2.3.1 -service-identity>=16.0.0 -simplejson==3.17.2 +config==0.5.1 +GitPython==3.1.29 +pubsub==0.1.2 +PyJWT==2.6.0 +Pypubsub==4.0.3 +pyserial==3.5 +python-crontab==2.6.0 +python-dateutil==2.8.2 +pytimeparse==1.1.8 +pytz>=2022.6 +requests==2.28.1 +RPi.GPIO==0.7.1 +simplejson==3.18.0 From 27e397e98f3a3ab8b51e9437475f928c52cc344f Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Mon, 12 Dec 2022 23:09:59 -0500 Subject: [PATCH 050/164] refactor --- .../{RecursionTests.yml => CI_Tests.yml} | 6 +++--- openpod/modules/rec_lan.py | 20 ++++++++++++++----- openpod/modules/rec_mqtt.py | 8 +++++++- requirements.txt | 3 ++- 4 files changed, 27 insertions(+), 10 deletions(-) rename .github/workflows/{RecursionTests.yml => CI_Tests.yml} (92%) diff --git a/.github/workflows/RecursionTests.yml b/.github/workflows/CI_Tests.yml similarity index 92% rename from .github/workflows/RecursionTests.yml rename to .github/workflows/CI_Tests.yml index c7afd15..d61e582 100644 --- a/.github/workflows/RecursionTests.yml +++ b/.github/workflows/CI_Tests.yml @@ -1,7 +1,7 @@ # This workflow will install Python dependencies, run tests and lint with a single version of Python # For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions -name: Recursion.Space Tests +name: CI | Build & Test on: push: branches: [ master ] @@ -15,10 +15,10 @@ jobs: steps: - uses: actions/checkout@v3 - - name: Set up Python 3.9 + - name: Set up Python 3.11 uses: actions/setup-python@v4.3.1 with: - python-version: 3.9 + python-version: 3.11 - name: Install dependencies run: | python -m pip install --upgrade pip diff --git a/openpod/modules/rec_lan.py b/openpod/modules/rec_lan.py index 30a37c3..9bfd401 100644 --- a/openpod/modules/rec_lan.py +++ b/openpod/modules/rec_lan.py @@ -169,14 +169,24 @@ def get_ip(): public_ip = "WLAN not available." try: - local_ip = ([l for l in ([ip for ip in socket.gethostbyname_ex(socket.gethostname())[2] + hostname = socket.gethostname() - if not ip.startswith("127.")][:1], [[(s.connect(('8.8.8.8', 53)), - s.getsockname()[0], s.close()) for s in [socket.socket(socket.AF_INET, - socket.SOCK_DGRAM)]][0][1]]) if l][0][0]) + ip_address = socket.gethostbyname_ex(hostname)[2] + + local_ip_address = [ip for ip in ip_address if not ip.startswith("127.")][:1] + + sock = [socket.socket(socket.AF_INET, socket.SOCK_DGRAM)] + + socket_connections = [ + [ + (s.connect(('8.8.8.8', 53)), s.getsockname()[0], s.close()) for s in sock + ][0][1] + ] + + local_ip = ([l for l in (local_ip_address, socket_connections) if l][0][0]) # network_log.info("Hub's local IP address: {0}".format(local_ip)) - # Prevent constant log writting since now in loop + # Prevent constant log writing since now in loop except OSError as err: network_log.error('Unable to get local IP address with error: %s', err) diff --git a/openpod/modules/rec_mqtt.py b/openpod/modules/rec_mqtt.py index 82e0d29..37f46f2 100644 --- a/openpod/modules/rec_mqtt.py +++ b/openpod/modules/rec_mqtt.py @@ -15,6 +15,8 @@ import updater # The callback for when the client receives a CONNACK response from the server. + + def on_connect(client, userdata, flags, return_code): ''' Action taken once a connection has been established. @@ -32,7 +34,7 @@ def on_connect(client, userdata, flags, return_code): def on_message(client, userdata, message): ''' - Handles messeges coming in via MQTT. + Handles messages coming in via MQTT. 170 - Pairing un-paired Hub 186 - Pull New Data 202 - Install System Update @@ -84,6 +86,8 @@ def on_message(client, userdata, message): mqtt_log.error("MQTT did not match action codes. Payload: %s", message.payload) + return False + def mqtt_rx(): ''' @@ -101,6 +105,7 @@ def mqtt_rx(): client.connect(f"{settings.RECURSION_DOMAIN}", 1883, 60) client.loop_forever() + def mqtt_start_update(): ''' Called by the MQTT function handler to start an update. @@ -116,6 +121,7 @@ def mqtt_start_update(): with subprocess.Popen(['sudo', 'python3', f'{update_location}']) as script: print(script) + def mqtt_restart_system(): ''' Called by the MQTT function handler to restart the system. diff --git a/requirements.txt b/requirements.txt index 7d8cc97..5a2b843 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,5 +1,6 @@ config==0.5.1 GitPython==3.1.29 +paho-mqtt==1.6.1 pubsub==0.1.2 PyJWT==2.6.0 Pypubsub==4.0.3 @@ -7,7 +8,7 @@ pyserial==3.5 python-crontab==2.6.0 python-dateutil==2.8.2 pytimeparse==1.1.8 -pytz>=2022.6 +pytz==2022.6 requests==2.28.1 RPi.GPIO==0.7.1 simplejson==3.18.0 From 29eb544457bc8e1fb0f9ef3c038554706c66a0be Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Mon, 12 Dec 2022 23:19:39 -0500 Subject: [PATCH 051/164] fix: actions --- .github/workflows/EndToEnd.yml | 16 +++++++++------- README.md | 4 ++-- 2 files changed, 11 insertions(+), 9 deletions(-) diff --git a/.github/workflows/EndToEnd.yml b/.github/workflows/EndToEnd.yml index d16912b..912ee5a 100644 --- a/.github/workflows/EndToEnd.yml +++ b/.github/workflows/EndToEnd.yml @@ -1,4 +1,4 @@ -name: End To End Integration +name: CI | E2E Integration on: push: @@ -29,16 +29,18 @@ jobs: sudo apt-get update -y && sudo apt-get upgrade -y sudo apt install software-properties-common -y sudo add-apt-repository ppa:deadsnakes/ppa - sudo apt install python3.10 -y - cd /opt + sudo apt install python3.11 -y + cd /opt/ - git clone --single-branch --branch master git@github.com:RecursionSpace/RecursionSpace.git + git clone --single-branch --branch release git@github.com:RecursionSpace/RecursionSpace.git - sudo apt-get install python3.10-venv -y - sudo python3.10 -m venv /opt/RecursionSpace/env + sudo apt-get install python3.11-venv -y + sudo python3.11 -m venv /opt/RecursionSpace/env source /opt/RecursionSpace/env/bin/activate - /opt/RecursionSpace/env/bin/python3.10 -m pip install --upgrade pip + + /opt/RecursionSpace/env/bin/python3.11 -m pip install --upgrade pip sudo pip install --no-input -U -r /opt/RecursionSpace/requirements.txt --no-cache-dir + sudo pip install Django sudo python /opt/RecursionSpace/manage.py migrate --noinput diff --git a/README.md b/README.md index dc24b1b..04910f6 100644 --- a/README.md +++ b/README.md @@ -8,9 +8,9 @@   [![Script Check](https://github.com/RecursionSpace/OpenPod/actions/workflows/ShellCheck.yml/badge.svg)](https://github.com/RecursionSpace/OpenPod/actions/workflows/ShellCheck.yml)   -[![Recursion.Space Tests](https://github.com/RecursionSpace/OpenPod/actions/workflows/RecursionTests.yml/badge.svg)](https://github.com/RecursionSpace/OpenPod/actions/workflows/RecursionTests.yml) +[![CI | Build & Test](https://github.com/RecursionSpace/OpenPod/actions/workflows/CI_Tests.yml/badge.svg)](https://github.com/RecursionSpace/OpenPod/actions/workflows/CI_Tests.yml)   -[![Test Installer](https://github.com/RecursionSpace/OpenPod/actions/workflows/TestInstaller.yml/badge.svg)](https://github.com/RecursionSpace/OpenPod/actions/workflows/TestInstaller.yml) +[![CI | Installer](https://github.com/RecursionSpace/OpenPod/actions/workflows/CI_TestInstaller.yml/badge.svg)](https://github.com/RecursionSpace/OpenPod/actions/workflows/CI_TestInstaller.yml) From 93b27db73f671e03293acef8b63bd8ebb387c300 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Mon, 12 Dec 2022 23:22:53 -0500 Subject: [PATCH 052/164] fix: added community --- .github/workflows/{cd-release.yml => CD_Release.yml} | 0 .github/workflows/{EndToEnd.yml => CI_EndToEnd.yml} | 0 README.md | 11 +++++++++++ 3 files changed, 11 insertions(+) rename .github/workflows/{cd-release.yml => CD_Release.yml} (100%) rename .github/workflows/{EndToEnd.yml => CI_EndToEnd.yml} (100%) diff --git a/.github/workflows/cd-release.yml b/.github/workflows/CD_Release.yml similarity index 100% rename from .github/workflows/cd-release.yml rename to .github/workflows/CD_Release.yml diff --git a/.github/workflows/EndToEnd.yml b/.github/workflows/CI_EndToEnd.yml similarity index 100% rename from .github/workflows/EndToEnd.yml rename to .github/workflows/CI_EndToEnd.yml diff --git a/README.md b/README.md index 04910f6..623e251 100644 --- a/README.md +++ b/README.md @@ -26,6 +26,7 @@ - [Updates](#updates) - [Logging](#logging) - [Directory Structure](#directory-structure) +- [Community and Contributing](#community-and-contributing) ## What is OpenPod? @@ -157,3 +158,13 @@ The Recursion System uses several logging points to be used both for troubleshoo └── openpod # Contains OpenPod functionality. └── modules # Independent core functions. ``` + +# Community and Contributing + +OpenPod is developed by [Recursion.Space](https://recursion.space/) and by users like you. We welcome both pull requests and issues on [GitHub](https://github.com/RecursionSpace/OpenPod). Bug fixes and new protocols are encouraged. + + From 69554185e11948132f3d4df4d0ce5fe6aa67ecc9 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Mon, 12 Dec 2022 23:31:18 -0500 Subject: [PATCH 053/164] fix: e2e --- .github/workflows/CI_EndToEnd.yml | 4 ++-- README.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/CI_EndToEnd.yml b/.github/workflows/CI_EndToEnd.yml index 912ee5a..4abcce2 100644 --- a/.github/workflows/CI_EndToEnd.yml +++ b/.github/workflows/CI_EndToEnd.yml @@ -41,6 +41,6 @@ jobs: /opt/RecursionSpace/env/bin/python3.11 -m pip install --upgrade pip sudo pip install --no-input -U -r /opt/RecursionSpace/requirements.txt --no-cache-dir - sudo pip install Django - sudo python /opt/RecursionSpace/manage.py migrate --noinput + cd /opt/RecursionSpace/ + sudo python manage.py migrate --noinput diff --git a/README.md b/README.md index 623e251..46a352f 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ # OpenPod -[![End To End Integration](https://github.com/RecursionSpace/OpenPod/actions/workflows/EndToEnd.yml/badge.svg)](https://github.com/RecursionSpace/OpenPod/actions/workflows/EndToEnd.yml) +[![CI | E2E Integration](https://github.com/RecursionSpace/OpenPod/actions/workflows/CI_EndToEnd.yml/badge.svg)](https://github.com/RecursionSpace/OpenPod/actions/workflows/CI_EndToEnd.yml)   [![CI | Pylint](https://github.com/RecursionSpace/OpenPod/actions/workflows/CI_Pylint.yml/badge.svg)](https://github.com/RecursionSpace/OpenPod/actions/workflows/CI_Pylint.yml)   From b646de4a5d3838b091bb87f714ad308b74cd7470 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Mon, 12 Dec 2022 23:48:49 -0500 Subject: [PATCH 054/164] actions --- .github/workflows/{CI_EndToEnd.yml => CI_E2E.yml} | 8 ++++---- .github/workflows/{ShellCheck.yml => CI_ShellCheck.yml} | 2 +- .../{VerifyInstaller.yml => CI_VerifyInstaller.yml} | 2 +- 3 files changed, 6 insertions(+), 6 deletions(-) rename .github/workflows/{CI_EndToEnd.yml => CI_E2E.yml} (82%) rename .github/workflows/{ShellCheck.yml => CI_ShellCheck.yml} (93%) rename .github/workflows/{VerifyInstaller.yml => CI_VerifyInstaller.yml} (94%) diff --git a/.github/workflows/CI_EndToEnd.yml b/.github/workflows/CI_E2E.yml similarity index 82% rename from .github/workflows/CI_EndToEnd.yml rename to .github/workflows/CI_E2E.yml index 4abcce2..e975898 100644 --- a/.github/workflows/CI_EndToEnd.yml +++ b/.github/workflows/CI_E2E.yml @@ -29,16 +29,16 @@ jobs: sudo apt-get update -y && sudo apt-get upgrade -y sudo apt install software-properties-common -y sudo add-apt-repository ppa:deadsnakes/ppa - sudo apt install python3.11 -y + sudo apt install python3.10 -y cd /opt/ git clone --single-branch --branch release git@github.com:RecursionSpace/RecursionSpace.git - sudo apt-get install python3.11-venv -y - sudo python3.11 -m venv /opt/RecursionSpace/env + sudo apt-get install python3.10-venv -y + sudo python3.10 -m venv /opt/RecursionSpace/env source /opt/RecursionSpace/env/bin/activate - /opt/RecursionSpace/env/bin/python3.11 -m pip install --upgrade pip + /opt/RecursionSpace/env/bin/python3.10 -m pip install --upgrade pip sudo pip install --no-input -U -r /opt/RecursionSpace/requirements.txt --no-cache-dir diff --git a/.github/workflows/ShellCheck.yml b/.github/workflows/CI_ShellCheck.yml similarity index 93% rename from .github/workflows/ShellCheck.yml rename to .github/workflows/CI_ShellCheck.yml index f222e20..0666ae4 100644 --- a/.github/workflows/ShellCheck.yml +++ b/.github/workflows/CI_ShellCheck.yml @@ -1,4 +1,4 @@ -name: Script Check +name: CI | Script Check on: push: diff --git a/.github/workflows/VerifyInstaller.yml b/.github/workflows/CI_VerifyInstaller.yml similarity index 94% rename from .github/workflows/VerifyInstaller.yml rename to .github/workflows/CI_VerifyInstaller.yml index 5b659b7..7844d00 100644 --- a/.github/workflows/VerifyInstaller.yml +++ b/.github/workflows/CI_VerifyInstaller.yml @@ -1,4 +1,4 @@ -name: Verify Bash Installer +name: CI | Verify Bash Installer on: push: From 352f24c08d8d93a99ecfc6a4bcbb649c457808c9 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Mon, 12 Dec 2022 23:54:02 -0500 Subject: [PATCH 055/164] Status Badges --- .github/workflows/CI_E2E.yml | 8 ++++---- README.md | 6 +++--- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/.github/workflows/CI_E2E.yml b/.github/workflows/CI_E2E.yml index e975898..4abcce2 100644 --- a/.github/workflows/CI_E2E.yml +++ b/.github/workflows/CI_E2E.yml @@ -29,16 +29,16 @@ jobs: sudo apt-get update -y && sudo apt-get upgrade -y sudo apt install software-properties-common -y sudo add-apt-repository ppa:deadsnakes/ppa - sudo apt install python3.10 -y + sudo apt install python3.11 -y cd /opt/ git clone --single-branch --branch release git@github.com:RecursionSpace/RecursionSpace.git - sudo apt-get install python3.10-venv -y - sudo python3.10 -m venv /opt/RecursionSpace/env + sudo apt-get install python3.11-venv -y + sudo python3.11 -m venv /opt/RecursionSpace/env source /opt/RecursionSpace/env/bin/activate - /opt/RecursionSpace/env/bin/python3.10 -m pip install --upgrade pip + /opt/RecursionSpace/env/bin/python3.11 -m pip install --upgrade pip sudo pip install --no-input -U -r /opt/RecursionSpace/requirements.txt --no-cache-dir diff --git a/README.md b/README.md index 46a352f..6ec304c 100644 --- a/README.md +++ b/README.md @@ -2,11 +2,11 @@ # OpenPod -[![CI | E2E Integration](https://github.com/RecursionSpace/OpenPod/actions/workflows/CI_EndToEnd.yml/badge.svg)](https://github.com/RecursionSpace/OpenPod/actions/workflows/CI_EndToEnd.yml) +[![CI | E2E Integration](https://github.com/RecursionSpace/OpenPod/actions/workflows/CI_E2E.yml/badge.svg)](https://github.com/RecursionSpace/OpenPod/actions/workflows/CI_E2E.yml)   [![CI | Pylint](https://github.com/RecursionSpace/OpenPod/actions/workflows/CI_Pylint.yml/badge.svg)](https://github.com/RecursionSpace/OpenPod/actions/workflows/CI_Pylint.yml)   -[![Script Check](https://github.com/RecursionSpace/OpenPod/actions/workflows/ShellCheck.yml/badge.svg)](https://github.com/RecursionSpace/OpenPod/actions/workflows/ShellCheck.yml) +[![CI | Script Check](https://github.com/RecursionSpace/OpenPod/actions/workflows/CI_ShellCheck.yml/badge.svg)](https://github.com/RecursionSpace/OpenPod/actions/workflows/CI_ShellCheck.yml)   [![CI | Build & Test](https://github.com/RecursionSpace/OpenPod/actions/workflows/CI_Tests.yml/badge.svg)](https://github.com/RecursionSpace/OpenPod/actions/workflows/CI_Tests.yml)   @@ -39,7 +39,7 @@ OpenPod is designed to communicate with hardware over a local mesh network using For convenience, an installation script as been provided that will download OpenPod and make the necessary system changes to get OpenPod running. This script can be run from the command line using the following command: ```bash -sudo wget -qO- openpod.recursion.space | bash /dev/stdin [options] [arguments*] +sudo wget -qO- openpod.recursion.space | bash /dev/stdin [options] [arguments] ``` | Option Flag | Description | Example | From 9f7937dcf0e084b7e6d838fb3803f45d004c9981 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Tue, 13 Dec 2022 01:29:39 -0500 Subject: [PATCH 056/164] added system docs --- README.md | 7 +------ docs/pi_setup.md | 27 +++++++++++++++++++++++++++ docs/pod_design.md | 21 +++++++++++++++++++++ 3 files changed, 49 insertions(+), 6 deletions(-) create mode 100644 docs/pi_setup.md create mode 100644 docs/pod_design.md diff --git a/README.md b/README.md index 6ec304c..343c724 100644 --- a/README.md +++ b/README.md @@ -18,7 +18,6 @@ - [What is OpenPod?](#what-is-openpod) - [Installation](#installation) - [Software Design](#software-design) - - [OS](#os) - [Development](#development) - [Initialization](#initialization) - [Required Packages](#required-packages) @@ -34,7 +33,7 @@ The “Pod” is the physical extension of the Recursion.Space system for a faci ## Installation -OpenPod is designed to communicate with hardware over a local mesh network using [XBee module](https://www.digi.com/products/embedded-systems/digi-xbee/rf-modules/2-4-ghz-rf-modules/xbee3-zigbee-3). While OpenPod can be installed on any computer running Ubuntu, most installations are done on a [Raspberry Pi](https://www.raspberrypi.com/products/raspberry-pi-4-model-b/). +OpenPod is designed to communicate with hardware over a local mesh network using [XBee module](https://www.digi.com/products/embedded-systems/digi-xbee/rf-modules/2-4-ghz-rf-modules/xbee3-zigbee-3). While OpenPod can be installed on any computer running Ubuntu, most installations are done on a [Raspberry Pi](https://www.raspberrypi.com/products/raspberry-pi-4-model-b/). To prepare a Raspbery Pi for OpenPod checkout the [Pi Setup Guide](docs/pi_setup.md). For convenience, an installation script as been provided that will download OpenPod and make the necessary system changes to get OpenPod running. This script can be run from the command line using the following command: @@ -72,10 +71,6 @@ The web interface communicates with the Pod via MQTT. Each Pod is on its MQTT topic, and the topic is created once the Pod registers with the central system. -## OS - -Each Pod is installed with the latest version of Ubuntu, and the systems are tested daily for compatibility with the latest releases. An image of the OS with the code ready for deployment can make a quick installation. 32-bit for Pi3 and 64-bit for 4+ - # Development Pod software development is done on a DigitalOcean hosted server. It is then transferred to the physical hardware running the system and tested before going into production. diff --git a/docs/pi_setup.md b/docs/pi_setup.md new file mode 100644 index 0000000..5e64978 --- /dev/null +++ b/docs/pi_setup.md @@ -0,0 +1,27 @@ +# Raspberry Pi Setup + +This is a guide to help you configure a Raspberry Pi for OpenPod from scratch. + +## Compatibility + +| Pi Model | Compatible | +|:--------:|:----------:| +| Pi 3 | Yes | +| Pi 4 | Yes | + +## Micro SD Card + +[Recommended Micro SD Card](https://www.samsung.com/us/computing/memory-storage/memory-cards/evo-plus-microsdxc-memory-card-64gb-mb-mc64ha-am/) + +Several considerations were taken into account when selecting the SD card, ultimately, longer-term reliability became the driving factor. The only compromise being made is a slightly reduced boot time which seemed negligible for a device intended always to be on. + +Key metrics for SD card comparison: + +- Application performance class (A1 & A2), A1 preferred +- [SD Card Comparison](https://techreport.com/forums/viewtopic.php?t=121152) + +## Operating System (OS) + +[32-bit for Pi3 and 64-bit for 4+](https://ubuntu.com/blog/how-to-install-ubuntu-with-the-new-raspberry-pi-imager) + +Each Pod is installed with the latest version of Ubuntu, and the systems are tested daily for compatibility with the latest releases. An image of the OS with the code ready for deployment can make a quick installation. diff --git a/docs/pod_design.md b/docs/pod_design.md new file mode 100644 index 0000000..ad32642 --- /dev/null +++ b/docs/pod_design.md @@ -0,0 +1,21 @@ +## Electronics + +Currently the system makes use of the GPIO on the Raspberry Pi. These pins are limited to an output of 3.3v at 16mA. + +### Visual Indicators + +The Hub has two LED indicators on it, a green and red. The [green LED](https://www.digikey.com/en/products/detail/lumex-opto-components-inc/SSI-LXH072GD/144732) is 2.2v at 25mA and will require a 68.75ohm (68ohm standard) resistor while the [red LED](https://www.digikey.com/en/products/detail/lumex-opto-components-inc/SSI-LXH072ID/144729) one is 2v at 30mA and will require a 81.25 ohm (82 ohm standard) resistor. Both resistors were [calculated](https://ohmslawcalculator.com/led-resistor-calculator) with the current caped at 16mA. + +A combination of the LEDs blinking or solid on/off in a [controlled pattern](https://en.wikipedia.org/wiki/International_Blinking_Pattern_Interpretation) provides information to the users. + +| | Green - I/O “Simulating XBee activity, should really be connected to one of the XBee pins” | Red - Overall Status | +|:----------------------------------------------------------------------------------------------------------------------------------------------------:|:------------------------------------------------------------------------------------------:|:--------------------:| +| Program Called: Not ready to receive requests and have not tested network connection. | Off | On | +| No Ethernet: The HUB has not received an IP address and is set to 127.0.0.1 | N/A | 4 Hz | +| Ethernet, No Internet: | N/A | 2 Hz | +| Internet, Recursion Unreachable: | N/A | 1 Hz | +| Recursion.Space Reachable: | N/A | Off | +| Unregistered: Registered and Ready: All network and hardware tests passed, ready to receive XBee requests and HUB IS NOT registered with a facility. | 1 Hz | N/A | +| Registered and Ready: All network and hardware tests passed, ready to receive XBee requests, and HUB IS registered with a facility. | On | N/A | +| XBee I/O Data: Anytime the XBee receives or sends data. | 4 Hz | N/A | +| Fatal Error: - USB XBee Unreachable | 2 Hz Alternating | 2 Hz Alternating | From 3f4b49b69076b9f68b11916de319287401e5cba9 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Tue, 13 Dec 2022 10:08:14 -0500 Subject: [PATCH 057/164] fix: removed installer.py --- .github/workflows/CI_TestInstaller.yml | 12 +- README.md | 7 - boot.sh | 7 +- installer.py | 215 ------------------------- installer.sh | 21 ++- requirements.txt | 1 - 6 files changed, 25 insertions(+), 238 deletions(-) delete mode 100644 installer.py diff --git a/.github/workflows/CI_TestInstaller.yml b/.github/workflows/CI_TestInstaller.yml index 084b969..06ceb71 100644 --- a/.github/workflows/CI_TestInstaller.yml +++ b/.github/workflows/CI_TestInstaller.yml @@ -9,16 +9,8 @@ jobs: steps: - uses: actions/checkout@v3 - - name: Set up Python 3.11 - uses: actions/setup-python@v4.3.1 - with: - python-version: 3.11 - - - name: Install dependencies - run: | - python -m pip install --upgrade pip - pip install python-crontab - name: Run Installer run: | - sudo python installer.py + sudo chmod +x install.sh + sudo ./install.sh diff --git a/README.md b/README.md index 343c724..d340eec 100644 --- a/README.md +++ b/README.md @@ -20,7 +20,6 @@ - [Software Design](#software-design) - [Development](#development) - [Initialization](#initialization) - - [Required Packages](#required-packages) - [MQTT](#mqtt) - [Updates](#updates) - [Logging](#logging) @@ -94,12 +93,6 @@ To launch the software, when the device boots, the launcher configures the syste (@reboot (cd /home/ubuntu/RecursionHub && sudo python3 HUB_Launcher.py &)) ``` -## Required Packages - -Required packages are stored under “Requirements” in the settings.py file. - -PyPubSub is used for internal flags and alerts and has a broker-type system that can be subscribed to. - # MQTT The MQTT protocol is used for server to Pod communication, allowing the Pod to listen in real-time for incoming instruction. diff --git a/boot.sh b/boot.sh index 3aa7f67..83f0118 100644 --- a/boot.sh +++ b/boot.sh @@ -1,5 +1,10 @@ #!/bin/bash -#Called by the @reboot conjob, will remain running at all times. + +# Called by the @reboot conjob, will remain running at all times. +# Used to be located /opt/OpenPod/boot.sh +# Required chmod +x /opt/OpenPod/boot.sh +# Ran with a crontab entry @reboot /opt/OpenPod/boot.sh + sleep 30 #Delay before running to allow all the services to startup. diff --git a/installer.py b/installer.py deleted file mode 100644 index 21ddf39..0000000 --- a/installer.py +++ /dev/null @@ -1,215 +0,0 @@ -#!/usr/bin/env python3 - -'''This file is ran when setting up a new Pi.''' -#Program execution is handled by the HUB_Launcher.py within the latest version folder. - -#need to setup for local time -#https://linuxize.com/post/how-to-set-or-change-timezone-on-ubuntu-18-04/ - -import uuid -import json -import subprocess - -from os import path - - -instalation_log = {} - - -# --------------------------- Set System Time Zone --------------------------- # -try: - subprocess.call(['sudo', 'timedatectl', 'set-timezone', 'UTC']) - - instalation_log['SysTimeZone'] = 'SUCESS - System time zone changed to UTC' -except RuntimeError as err: - instalation_log['SysTimeZone'] = f'FAILED - {err}' - - -# ---------------------------- Update system time ---------------------------- # -try: - subprocess.call(['sudo', 'apt-get', 'install', 'chrony', '-y']) - subprocess.call(['sudo', 'chronyd', '-q']) - - instalation_log['SysTime'] = 'SUCESS - System time synchronized.' -except RuntimeError as err: - instalation_log['SysTime'] = f'FAILED - {err}' - - -# --------------------- Update OS and Package Directories -------------------- # -try: - subprocess.call(['sudo', 'apt-get', 'update', '-y']) - - instalation_log['SysUpdate'] = 'SUCESS - Sytem has now been updated.' -except RuntimeError as err: - instalation_log['SysUpdate'] = f'FAILED - {err}' - - -try: - subprocess.call(['sudo', 'apt-get', 'upgrade', '-y']) - - instalation_log['SysUpgrade'] = 'SUCESS - Sytem has now been upgrded.' -except RuntimeError as err: - instalation_log['SysUpgrade'] = f'FAILED - {err}' - - -# -------------------------------- Install PIP ------------------------------- # -try: - subprocess.call(['sudo', 'apt-get', 'install', 'python3-pip', '-y']) - - instalation_log['GetPip'] = 'SUCESS - pip now installed.' -except RuntimeError as err: - instalation_log['GetPip'] = f'FAILED - {err}' - - -# ------------- Creates execution location at /opt/OpenPod/ ------------- # -try: - subprocess.call(['mkdir', '/opt/OpenPod']) - - instalation_log['MakeDirectory'] = 'SUCESS - OpenPod directory created.' -except RuntimeError as err: - instalation_log['MakeDirectory'] = f'FAILED - {err}' - - -# ----------------------- Creates the system.json file ----------------------- # -try: - subprocess.call(['sudo', 'touch', '/opt/OpenPod/system.json']) - - data = { - "serial" : f"{uuid.uuid4().hex}", #Self assigned aidentification number. - "timezone" : "UTC", - "XBEE_KY" : f"{uuid.uuid4().hex}", #Network key be assigned to the nodes. - "XBEE_OP" : False, #False -> XBee not configured. - "CurrentVersion" : "0_1_0", #Program fallback. - } - - with open('/opt/OpenPod/system.json', 'w', encoding="utf-8") as outfile: - json.dump(data, outfile) - - instalation_log['system.json'] = 'SUCESS - Suscessfulyl created system json file.' -except RuntimeError as err: - instalation_log['system.json'] = f'FAILED - {err}' - - -# ---------------------------------------------------------------------------- # -# Install Bash Requirements # -# ---------------------------------------------------------------------------- # - -# ------- jq - parse json bash https://stedolan.github.io/jq/download/ ------- # -try: - subprocess.call(['sudo', 'apt-get', 'install', 'jq', '-y']) - - instalation_log['BashReq_jq'] = 'SUCESS - jq now installed.' -except RuntimeError as err: - instalation_log['BashReq_jq'] = f'FAILED - {err}' - - -# -------------------- unzip to manage zipfiles from bash -------------------- # -try: - subprocess.call(['sudo', 'apt-get', 'install', 'unzip', '-y']) - - instalation_log['BashReq_unzip'] = 'SUCESS - unzip now installed.' -except RuntimeError as err: - instalation_log['BashReq_unzip'] = f'FAILED - {err}' - - -# ----------------- ifconfig to check network adapter status ----------------- # -#Depreciated use https://askubuntu.com/questions/1031640/ifconfig-missing-after-ubuntu-18-04-install -try: - subprocess.call(['sudo', 'apt-get', 'install', 'net-tools', '-y']) - - instalation_log['Install_net-tools'] = 'SUCESS - Installed net-tools.' -except RuntimeError as err: - instalation_log['pip_requiremetns'] = f'FAILED - Could not install net-tools.{err}' - - -# ------------------ pip requiremetns from requirements.txt ------------------ # -try: - subprocess.call([ - 'sudo', - 'pip3', - 'install', - '--no-input', - '-U', - '-r', - 'requirements.txt', - '--no-cache-dir', - '--no-dependencies' - ]) - - instalation_log['pip_requiremetns'] = 'SUCESS - Installed all pip requirements.' -except RuntimeError as err: - instalation_log['pip_requiremetns'] = f'FAILED - {err}' - - -# ---------------------- Create Folders and Directories ---------------------- # -try: - subprocess.call(['mkdir', '/opt/OpenPod/logs']) - subprocess.call(['mkdir', '/opt/OpenPod/data']) - subprocess.call(['touch', '/opt/OpenPod/logs/RecursionLog.log']) - subprocess.call(['touch', '/opt/OpenPod/logs/System.Snapshot']) - - instalation_log['folders_directories'] = 'SUCESS - Log folder and files prepared' -except RuntimeError as err: - instalation_log['folders_directories'] = f'FAILED - {err}' - - -# ------------------------- Create Data Storage Files ------------------------ # -try: - if path.exists("/opt/OpenPod/data/dump.json") is False: - subprocess.call(['sudo', 'touch', '/opt/OpenPod/data/dump.json']) - if path.exists("/opt/OpenPod/data/nodes.json") is False: - subprocess.call(['sudo', 'touch', '/opt/OpenPod/data/nodes.json']) - if path.exists("/opt/OpenPod/data/owners.json") is False: - subprocess.call(['sudo', 'touch', '/opt/OpenPod/data/owners.json']) - if path.exists("/opt/OpenPod/data/permissions.json") is False: - subprocess.call(['sudo', 'touch', '/opt/OpenPod/data/permissions.json']) - - instalation_log['storage_files'] = 'SUCESS - Suscessfulyl created data files.' -except RuntimeError as err: - instalation_log['storage_files'] = f'FAILED - {err}' - - -# ------------------ Move boot.sh to /opt/OpenPod/ ------------------ # -try: - subprocess.call(['sudo', 'cp', 'boot.sh', '/opt/OpenPod/HUB_Boot.sh']) - - instalation_log['move_boot.sh'] = 'SUCESS - boot.sh file is not in the /opt/ directory.' -except RuntimeError as err: - instalation_log['move_boot.sh'] = f'FAILED - {err}' - - -# ----------------------------- Copy 0_1_0 Folder ---------------------------- # -try: - subprocess.call(['sudo', 'cp', '-r', '0_1_0', '/opt/OpenPod/0_1_0']) - - instalation_log['copy0_1_0'] = 'SUCESS - 0_1_0/ copied into the /opt/ directory.' -except RuntimeError as err: - instalation_log['copy0_1_0'] = f'FAILED - {err}' - - -# --------------------------- Set file permissions --------------------------- # -try: - subprocess.call(['sudo', 'chmod', '+x', '/opt/OpenPod/HUB_Boot.sh']) - - instalation_log['chmod'] = 'SUCESS - Permissions for HUB_Boot.sh updated.' -except RuntimeError as err: - instalation_log['chmod'] = f'FAILED - {err}' - - -# ----------------------------- Configure crontab ---------------------------- # -try: - from crontab import CronTab # Needs to be here since it is installed by the requirements file. - cron = CronTab(user='root') - job = cron.new(command='(cd /opt/OpenPod && sudo ./HUB_Boot.sh &)') - job.every_reboot() - cron.write() - - instalation_log['configure_python-crontab'] = 'SUCCESS - Crontab sucessfully created.' -except ImportError as err: - instalation_log['configure_python-crontab'] = f'FAILED - {err}' -except RuntimeError as err: - instalation_log['configure_python-crontab'] = f'FAILED - {err}' - -for log_item, result in instalation_log.items(): - print(f"{log_item} -> {result}") -print("Recursion HUB Installation Program Complete. Restart System.") diff --git a/installer.sh b/installer.sh index 29eaaf1..2de8699 100644 --- a/installer.sh +++ b/installer.sh @@ -45,14 +45,23 @@ if [ "$EUID" -ne 0 ] exit fi +# ---------------------------- Update System Time ---------------------------- # +sudo timedatectl set-timezone UTC + # ---------------------------------------------------------------------------- # # Dependencies # # ---------------------------------------------------------------------------- # -# ---------------------------- Update System Time ---------------------------- # -sudo timedatectl set-timezone UTC -sudo apt-get install chrony -y -sudo chronyd -q +# ---------------------------------- chrony ---------------------------------- # +REQUIRED_PKG="chrony" +PKG_OK=$(dpkg-query -W --showformat='${Status}\n' $REQUIRED_PKG|grep "install ok installed") +if [ "" = "$PKG_OK" ]; then + echo "No $REQUIRED_PKG. Setting up $REQUIRED_PKG..." + sudo apt-get install chrony -y + sudo chronyd -q +else + echo "chrony already installed, skipping..." +fi # ----------------------------------- unzip ---------------------------------- # REQUIRED_PKG="unzip" @@ -84,6 +93,10 @@ else echo "Python 3.11 already installed" fi +# ---------------------------------------------------------------------------- # +# OpenPod # +# ---------------------------------------------------------------------------- # + # ------------------------------- Clone OpenPod ------------------------------ # set -e # Exit when any command fails. sudo mkdir -p /opt diff --git a/requirements.txt b/requirements.txt index 5a2b843..7ca4301 100644 --- a/requirements.txt +++ b/requirements.txt @@ -5,7 +5,6 @@ pubsub==0.1.2 PyJWT==2.6.0 Pypubsub==4.0.3 pyserial==3.5 -python-crontab==2.6.0 python-dateutil==2.8.2 pytimeparse==1.1.8 pytz==2022.6 From 28f24d2565194dcccd42e8229d3bf27fdfa52dd7 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Tue, 13 Dec 2022 10:14:48 -0500 Subject: [PATCH 058/164] fix: actions --- .github/workflows/CI_Pylint.yml | 7 +++---- .github/workflows/CI_TestInstaller.yml | 4 ++-- README.md | 4 ++-- 3 files changed, 7 insertions(+), 8 deletions(-) diff --git a/.github/workflows/CI_Pylint.yml b/.github/workflows/CI_Pylint.yml index d53e45c..2e0adb7 100644 --- a/.github/workflows/CI_Pylint.yml +++ b/.github/workflows/CI_Pylint.yml @@ -9,10 +9,10 @@ jobs: steps: - uses: actions/checkout@v3 - - name: Set up Python 3.8 + - name: Set up Python 3.11 uses: actions/setup-python@v4.3.1 with: - python-version: 3.8 + python-version: 3.11 - name: Install dependencies run: | @@ -21,9 +21,8 @@ jobs: pip install pylint-exit if [ -f requirements.txt ]; then pip3 install -r requirements.txt; fi - - name: Analysing the code with pylint + - name: Analyzing the code with Pylint run: | - pylint `ls -r|grep .py$|xargs` --disable=import-error,E1101 || pylint-exit --error-fail --warn-fail --convention-fail $? pylint openpod `ls -r|grep .py$|xargs` --disable=import-error,E1101 || pylint-exit --error-fail --warn-fail --convention-fail $? pylint tests `ls -r|grep .py$|xargs` --disable=import-error,E1101,R0801 || pylint-exit --error-fail --warn-fail --convention-fail $? diff --git a/.github/workflows/CI_TestInstaller.yml b/.github/workflows/CI_TestInstaller.yml index 06ceb71..a91ff44 100644 --- a/.github/workflows/CI_TestInstaller.yml +++ b/.github/workflows/CI_TestInstaller.yml @@ -12,5 +12,5 @@ jobs: - name: Run Installer run: | - sudo chmod +x install.sh - sudo ./install.sh + sudo chmod +x installer.sh + sudo ./installer.sh diff --git a/README.md b/README.md index d340eec..b0a65b1 100644 --- a/README.md +++ b/README.md @@ -79,7 +79,7 @@ Pod software development is done on a DigitalOcean hosted server. It is then tra Pod uses an initializer or “launcher” to configure the system before running the main code. The launcher performs the following tasks before running the main program: - Update & Upgrade the OS -- Sync Clock +- [Sync Clock](https://vitux.com/keep-your-clock-sync-with-internet-time-servers-in-ubuntu/) - Installs program required packages - Check for required files, create them if missing - Configure start on boot file @@ -87,7 +87,7 @@ Pod uses an initializer or “launcher” to configure the system before running - Registers the Pod with the API server - Initializes the main program -To launch the software, when the device boots, the launcher configures the system first to run the software in the background and then creates a script that executes the launch on startup. Working version with a cronjob. +To launch the software, when the device boots, the launcher configures the system first to run the software in the [background](https://janakiev.com/blog/python-background/) and then creates a script that executes the launch on [startup](https://askubuntu.com/questions/817011/run-python-script-on-os-boot). Working version with a [cronjob](https://www.linuxbabe.com/linux-server/how-to-enable-etcrc-local-with-systemd). ```bash (@reboot (cd /home/ubuntu/RecursionHub && sudo python3 HUB_Launcher.py &)) From a6bb78b5c325f4676fda1650f965b0fd08094868 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Tue, 13 Dec 2022 10:19:50 -0500 Subject: [PATCH 059/164] linted --- .../workflows/{merge_to-dev-release.yml => CD_dev-release.yml} | 2 +- openpod/modules/rec_lan.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) rename .github/workflows/{merge_to-dev-release.yml => CD_dev-release.yml} (98%) diff --git a/.github/workflows/merge_to-dev-release.yml b/.github/workflows/CD_dev-release.yml similarity index 98% rename from .github/workflows/merge_to-dev-release.yml rename to .github/workflows/CD_dev-release.yml index 092eb4b..7202352 100644 --- a/.github/workflows/merge_to-dev-release.yml +++ b/.github/workflows/CD_dev-release.yml @@ -1,7 +1,7 @@ # Performs the task of merging from master to the dev-release branch. # Pings the server to pull an update once the dev-release branch has been merged. -name: Dev Merge & Deploy +name: CD | Dev Release on: push: branches: [ master ] diff --git a/openpod/modules/rec_lan.py b/openpod/modules/rec_lan.py index 9bfd401..8b50c57 100644 --- a/openpod/modules/rec_lan.py +++ b/openpod/modules/rec_lan.py @@ -67,7 +67,7 @@ def monitor_network(last_network_status=5, thread_delay=30.0): monitor_network, [current_network_status, thread_delay] ) - network_watch_thread.setName('network_watch_thread') + network_watch_thread.name = 'network_watch_thread' network_watch_thread.start() return True From be4ec80cf09f1a764e86df16d03ff6079e3b3589 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Tue, 13 Dec 2022 13:52:38 -0500 Subject: [PATCH 060/164] Migrated README --- README.md | 48 ++++++++++++++++++++++++++++++++++++---------- docs/pod_design.md | 16 ++++++++++++++++ 2 files changed, 54 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index b0a65b1..11e9d5a 100644 --- a/README.md +++ b/README.md @@ -23,13 +23,16 @@ - [MQTT](#mqtt) - [Updates](#updates) - [Logging](#logging) - - [Directory Structure](#directory-structure) + - [Repository Directory Structure](#repository-directory-structure) + - [Deployed Directory Structure](#deployed-directory-structure) - [Community and Contributing](#community-and-contributing) ## What is OpenPod? The “Pod” is the physical extension of the Recursion.Space system for a facility. The Pod allows for door/egress and equipment control nodes. Additionally, the Pod acts as a local backup in an internet outage for users to continue accessing their facility. The Pod has a direct internet connection then uses a wireless method to communicate with nodes. A single facility can have multiple Pods, but each Pod can only be linked to one facility. +*Note: Versioning follows the [Semantic Versioning 2.0.0 standard](https://semver.org/)* + ## Installation OpenPod is designed to communicate with hardware over a local mesh network using [XBee module](https://www.digi.com/products/embedded-systems/digi-xbee/rf-modules/2-4-ghz-rf-modules/xbee3-zigbee-3). While OpenPod can be installed on any computer running Ubuntu, most installations are done on a [Raspberry Pi](https://www.raspberrypi.com/products/raspberry-pi-4-model-b/). To prepare a Raspbery Pi for OpenPod checkout the [Pi Setup Guide](docs/pi_setup.md). @@ -85,7 +88,7 @@ Pod uses an initializer or “launcher” to configure the system before running - Configure start on boot file - Creates serial if needed - Registers the Pod with the API server -- Initializes the main program +- Initializes the main program under [/opt/](https://unix.stackexchange.com/questions/11544/what-is-the-difference-between-opt-and-usr-local) To launch the software, when the device boots, the launcher configures the system first to run the software in the [background](https://janakiev.com/blog/python-background/) and then creates a script that executes the launch on [startup](https://askubuntu.com/questions/817011/run-python-script-on-os-boot). Working version with a [cronjob](https://www.linuxbabe.com/linux-server/how-to-enable-etcrc-local-with-systemd). @@ -110,11 +113,11 @@ Communication to Pods is accomplished from quick commands represented by a hexad # Updates -The end-user triggers updates via the web server to run them with minimal interruption. An MQTT message is sent to the Pod to initiate the updating process. The update is a zip file downloaded from the server. +The end-user triggers updates via the web server to run them with minimal interruption. An MQTT message is sent to the Pod to initiate the updating process. The update is a zip file downloaded from the [server](https://stackoverflow.com/questions/38697529/how-to-return-generated-file-download-with-django-rest-framework). -Download zip file from server and store in the update folder. -Unzip contents of a zip file -Run Launcher +1) Download zip file from server and store in the update folder. +2) Unzip contents of a zip file +3) Run Launcher First, commit and push the latest changes to git to prepare an update. Using WinSCP, download the 0_1_0 folder with the newest code. Rename the downloaded folder to match the latest version number. Zip the folder's CONTENTS, so there is no second directory folder with a matching name inside the zip file folder. Clean up all extra copies of the old folder. Update the latest version name using the recursion admin panel, then update their devices. @@ -132,12 +135,12 @@ The Recursion System uses several logging points to be used both for troubleshoo **System.Snapshot** file contains a JSON summary of the Pod for debugging purposes. The information available in this file must also be readily accessible from the server for troubleshooting purposes. ```json -[ - “local_ip” = “xx.xx.xx.xx”, -] +{ + "local_ip": "xx.xx.xx.xx", +} ``` -## Directory Structure +## Repository Directory Structure ```default . @@ -147,6 +150,31 @@ The Recursion System uses several logging points to be used both for troubleshoo └── modules # Independent core functions. ``` +## Deployed Directory Structure + +```default +OpenPod/ +|--------- Version A/ + |--------- HUB.py + |--------- HUB_Launcher.py + |--------- HUB_Updater.py + |----------settings.py + |--------- modules/ + +|--------- logs/ + |--------- RecursionLog.log + |--------- TransactionLog.log + +|--------- data/ + |-------- dump.json + |-------- owners.json + |-------- nodes.json + |---------permissions.json + +|--- system.json + +``` + # Community and Contributing OpenPod is developed by [Recursion.Space](https://recursion.space/) and by users like you. We welcome both pull requests and issues on [GitHub](https://github.com/RecursionSpace/OpenPod). Bug fixes and new protocols are encouraged. diff --git a/docs/pod_design.md b/docs/pod_design.md index ad32642..31f471b 100644 --- a/docs/pod_design.md +++ b/docs/pod_design.md @@ -19,3 +19,19 @@ A combination of the LEDs blinking or solid on/off in a [controlled pattern](htt | Registered and Ready: All network and hardware tests passed, ready to receive XBee requests, and HUB IS registered with a facility. | On | N/A | | XBee I/O Data: Anytime the XBee receives or sends data. | 4 Hz | N/A | | Fatal Error: - USB XBee Unreachable | 2 Hz Alternating | 2 Hz Alternating | + +## Assembly + +### Main Enclosure + +1) Remove any stickers, labels, and decals from the enclosure, then remove the two screws to temporarily detach the door. (Goo Gone followed by rubbing alcohol, helps remove any residual sticker residue. +2) Using the top flange, trace the opening and then cut using a jigsaw and a fine wood bit attachment. The screw holes can be oversized with an 11/64 drill bit. +3) Cut off the top stands using an oscillating saw and then use a side cutter to clean any repairing plastic off. +4) Prepare part number HB-H-I-EBF/C with the threaded inserts. +5) Mount the flanges to the enclosure and place them in the 4 M4 philips mounting screws. The enclosure is now complete and can be set off to the side. + +### Processing Unit + +Prepare electronic assemblies, including power connectors, LEDs, and reset buttons. +Insert threads into the various housing components. +Insert the completed electronic assemblies, such as the power outlet, LEDs and ethernet jack. From e38a8bce2d492462de0d623a2fe86645a5c02305 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Tue, 13 Dec 2022 14:09:42 -0500 Subject: [PATCH 061/164] fix: installer --- README.md | 10 +++++----- installer.sh | 24 +++++++++++++++++++----- 2 files changed, 24 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index 11e9d5a..81c2041 100644 --- a/README.md +++ b/README.md @@ -43,11 +43,11 @@ For convenience, an installation script as been provided that will download Open sudo wget -qO- openpod.recursion.space | bash /dev/stdin [options] [arguments] ``` -| Option Flag | Description | Example | -|:-----------:|---------------------------|----------------------------------------------------------------------------------| -| -h | Help | sudo wget -qO- openpod.recursion.space \| bash /dev/stdin -h | -| -u | Set Custom URL endpoint | sudo wget -qO- openpod.recursion.space \| bash /dev/stdin -u dev.recursion.space | -| -d | Enable Debug Installation | sudo wget -qO- openpod.recursion.space \| bash /dev/stdin -d | +| Option Flag | Description | Example | +|:-----------:|---------------------------|------------------------------------------------------------------------------------| +| -h | Help | sudo wget -qO- openpod.recursion.space \| bash /dev/stdin -h | +| -d | Enable Debug Installation | sudo wget -qO- openpod.recursion.space \| bash /dev/stdin -d | +| -u | Set Custom URL endpoint | sudo wget -qO- openpod.recursion.space \| bash /dev/stdin -u stage.recursion.space | \*No supported arguments are currently supported. diff --git a/installer.sh b/installer.sh index 2de8699..6d2bee1 100644 --- a/installer.sh +++ b/installer.sh @@ -19,8 +19,7 @@ Help() # ---------------------------------------------------------------------------- # # Defaults # # ---------------------------------------------------------------------------- # -URL="recursion.space" -DEBUG=0 +DEBUG=0 # -d # ---------------------------------------------------------------------------- # @@ -31,14 +30,25 @@ while getopts ":hud" flags; do h) # display Help Help exit;; + d) # Enable debug mode + DEBUG=1 ;; u) # Custom URL endpoint URL="${OPTARG}";; - d) DEBUG=1 ;; \?) echo "Invalid option: -${OPTARG}" >&2; exit 1 ;; esac done +if [ $DEBUG -eq 1 ]; then + URL='recursion.space' +elif [ $DEBUG -eq 0 ]; then + URL='recursion.space' +fi + +echo "Installing OpenPod with the following options:" +echo "Debug: $DEBUG" +echo "URL: $URL" + # -------------------------------- Verify Root ------------------------------- # if [ "$EUID" -ne 0 ] then echo "Please run as root with sudo." @@ -48,6 +58,7 @@ fi # ---------------------------- Update System Time ---------------------------- # sudo timedatectl set-timezone UTC + # ---------------------------------------------------------------------------- # # Dependencies # # ---------------------------------------------------------------------------- # @@ -138,14 +149,17 @@ echo '{ # --------------------------- Setup OpenPod Service -------------------------- # cat < /etc/systemd/system/openpod.service [Unit] -Description=OpenPod +Description=OpenPod | Recursion.Space After=network.target +StartLimitIntervalSec=0 [Service] Type=simple User=root WorkingDirectory=/opt/OpenPod + ExecStart=/opt/OpenPod/env/bin/python3.11 /opt/OpenPod/OpenPod.py + Restart=always RestartSec=10 @@ -156,5 +170,5 @@ EOF sudo systemctl enable --now openpod.service sudo systemctl daemon-reload -echo "OpenPod is now installed" +echo "- OpenPod is now installed -" exit 0 From 662e675b876a80904d3b5702113ced6a23871300 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Thu, 15 Dec 2022 18:34:58 -0500 Subject: [PATCH 062/164] fix: removed extra requirement import --- requirements.txt | 1 - 1 file changed, 1 deletion(-) diff --git a/requirements.txt b/requirements.txt index 7ca4301..6b0a80d 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,5 +1,4 @@ config==0.5.1 -GitPython==3.1.29 paho-mqtt==1.6.1 pubsub==0.1.2 PyJWT==2.6.0 From a0b4001588d7452c6fa2771e574f5ed85e4b3a22 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Thu, 15 Dec 2022 19:25:08 -0500 Subject: [PATCH 063/164] Update CD_dev-release.yml --- .github/workflows/CD_dev-release.yml | 75 ++++++++++++++-------------- 1 file changed, 37 insertions(+), 38 deletions(-) diff --git a/.github/workflows/CD_dev-release.yml b/.github/workflows/CD_dev-release.yml index 7202352..0869a65 100644 --- a/.github/workflows/CD_dev-release.yml +++ b/.github/workflows/CD_dev-release.yml @@ -1,52 +1,51 @@ +name: CD | Dev Release + # Performs the task of merging from master to the dev-release branch. # Pings the server to pull an update once the dev-release branch has been merged. -name: CD | Dev Release on: push: - branches: [ master ] + branches: [master] workflow_dispatch: jobs: - merge: name: Merge To dev-release runs-on: ubuntu-latest steps: - - uses: actions/checkout@v3 - - - name: Wait on check - uses: fountainhead/action-wait-for-check@v1.1.0 - id: wait-for-build - - with: - token: ${{ secrets.GITHUB_TOKEN }} - checkName: build - ref: ${{ github.event.pull_request.head.sha || github.sha }} - - - name: Create Pull Request - - if: steps.wait-for-build.outputs.conclusion == 'success' - uses: repo-sync/pull-request@v2 - with: - destination_branch: "dev-release" - github_token: ${{ secrets.GITHUB_TOKEN }} - - - - name: Merge master -> dev-release - - if: steps.wait-for-build.outputs.conclusion == 'success' - uses: devmasx/merge-branch@1.4.0 - with: - type: now - target_branch: dev-release - github_token: ${{ secrets.GITHUB_TOKEN }} - - - name: Webhook - if: steps.wait-for-build.outputs.conclusion == 'success' - uses: distributhor/workflow-webhook@v3 - env: - webhook_url: "https://dev.recursion.space/webhooks/github/" - webhook_secret: "Y0uR5ecr3t" + - uses: actions/checkout@v3 + + - name: Wait on check + uses: fountainhead/action-wait-for-check@v1.1.0 + id: wait-for-build + + with: + token: ${{ secrets.GITHUB_TOKEN }} + checkName: build + ref: ${{ github.event.pull_request.head.sha || github.sha }} + + - name: Create Pull Request + + if: steps.wait-for-build.outputs.conclusion == 'success' + uses: repo-sync/pull-request@v2 + with: + destination_branch: "dev-release" + github_token: ${{ secrets.GITHUB_TOKEN }} + + - name: Merge master -> dev-release + + if: steps.wait-for-build.outputs.conclusion == 'success' + uses: devmasx/merge-branch@1.4.0 + with: + type: now + target_branch: dev-release + github_token: ${{ secrets.GITHUB_TOKEN }} + + - name: Webhook + if: steps.wait-for-build.outputs.conclusion == 'success' + uses: distributhor/workflow-webhook@v3 + env: + webhook_url: "https://dev.recursion.space/webhooks/github/" + webhook_secret: "Y0uR5ecr3t" From c613ac3737c80d69e43838c96e92b069842094b0 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Thu, 15 Dec 2022 19:28:48 -0500 Subject: [PATCH 064/164] Update dependabot.yml --- .github/dependabot.yml | 1 - 1 file changed, 1 deletion(-) diff --git a/.github/dependabot.yml b/.github/dependabot.yml index 5737055..a9953cb 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -2,7 +2,6 @@ version: 2 updates: - - package-ecosystem: "github-actions" directory: "/" schedule: From efa4600d763605e331150727c01aad51ba8434c9 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Fri, 16 Dec 2022 18:15:24 -0500 Subject: [PATCH 065/164] fix: dev url --- installer.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/installer.sh b/installer.sh index 6d2bee1..4f8fab0 100644 --- a/installer.sh +++ b/installer.sh @@ -40,7 +40,7 @@ while getopts ":hud" flags; do done if [ $DEBUG -eq 1 ]; then - URL='recursion.space' + URL='dev.recursion.space' elif [ $DEBUG -eq 0 ]; then URL='recursion.space' fi From e8078727226cc8ca0d588e426fd5d44f595dd293 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Fri, 16 Dec 2022 18:25:44 -0500 Subject: [PATCH 066/164] fix: use version of head --- installer.sh | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/installer.sh b/installer.sh index 4f8fab0..c842c4e 100644 --- a/installer.sh +++ b/installer.sh @@ -113,6 +113,7 @@ set -e # Exit when any command fails. sudo mkdir -p /opt cd /opt sudo git clone --single-branch --branch release https://github.com/RecursionSpace/OpenPod.git +cd OpenPod # ----------------------------- Setup Enviroment ----------------------------- # sudo apt-get install python3.11-venv -y @@ -136,13 +137,14 @@ sudo touch /opt/OpenPod/data/permissions.json sudo touch /opt/OpenPod/system.json serial_uuid=$(cat /proc/sys/kernel/random/uuid) xbee_uuid=$(cat /proc/sys/kernel/random/uuid) +openpod_version=$(git rev-parse HEAD) echo '{ "serial": '"'$serial_uuid'"', "timezone": "UTC", "XBEE_KY": '"'$xbee_uuid'"', "XBEE_OP": "0", "url": '"'$URL'"', - "version": "0_0_1", + "version": '"'$openpod_version'"', "debug": '"'$DEBUG'"' }' > /opt/OpenPod/system.json From bf9f1bdfc1b52de4b53b7b864e85f0cb2c6b70d6 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Fri, 16 Dec 2022 19:00:48 -0500 Subject: [PATCH 067/164] fix: installer --- README.md | 2 +- installer.sh | 24 ++++++++++++++++++++++-- 2 files changed, 23 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 81c2041..9d5bbe8 100644 --- a/README.md +++ b/README.md @@ -40,7 +40,7 @@ OpenPod is designed to communicate with hardware over a local mesh network using For convenience, an installation script as been provided that will download OpenPod and make the necessary system changes to get OpenPod running. This script can be run from the command line using the following command: ```bash -sudo wget -qO- openpod.recursion.space | bash /dev/stdin [options] [arguments] +sudo wget -qO- https://openpod.recursion.space | bash /dev/stdin [options] [arguments] ``` | Option Flag | Description | Example | diff --git a/installer.sh b/installer.sh index c842c4e..164c2fe 100644 --- a/installer.sh +++ b/installer.sh @@ -2,6 +2,14 @@ # Installer for OpenPod, for more information see https://github.com/blokbot-io/OpenBlok/blob/master/install.sh + +# ---------------------------------------------------------------------------- # +# Disable Prompts # +# ---------------------------------------------------------------------------- # +export DEBIAN_FRONTEND=noninteractive +sed -i "/#\$nrconf{restart} = 'i';/s/.*/\$nrconf{restart} = 'a';/" /etc/needrestart/needrestart.conf + + # ---------------------------------------------------------------------------- # # Help # # ---------------------------------------------------------------------------- # @@ -99,15 +107,28 @@ pytohn_version=$(python3 -c 'import sys; print(".".join(map(str, sys.version_inf if [ "$pytohn_version" != "3.11" ]; then sudo apt install software-properties-common -y yes '' | sudo add-apt-repository ppa:deadsnakes/ppa - sudo apt-get install python3.1 -y + sudo apt-get install python3.11 -y else echo "Python 3.11 already installed" fi +# ------------------------ Python Virtual Environment ------------------------ # +REQUIRED_PKG="python3.11-venv" +PKG_OK=$(dpkg-query -W --showformat='${Status}\n' $REQUIRED_PKG|grep "install ok installed") +if [ "" = "$PKG_OK" ]; then + echo "No $REQUIRED_PKG. Setting up $REQUIRED_PKG..." + sudo apt-get install python3.11-venv -y +else + echo "python3.11-venv already installed, skipping..." +fi + # ---------------------------------------------------------------------------- # # OpenPod # # ---------------------------------------------------------------------------- # +# -------------------------- Clear Previous Install -------------------------- # +sudo rm -rf /opt/OpenPod + # ------------------------------- Clone OpenPod ------------------------------ # set -e # Exit when any command fails. sudo mkdir -p /opt @@ -116,7 +137,6 @@ sudo git clone --single-branch --branch release https://github.com/RecursionSpac cd OpenPod # ----------------------------- Setup Enviroment ----------------------------- # -sudo apt-get install python3.11-venv -y sudo python3.11 -m venv /opt/OpenPod/env source /opt/OpenPod/env/bin/activate sudo pip install --no-input -U -r /opt/OpenPod/requirements.txt --no-cache-dir --no-dependencies From 7445fab1c745d3c4e78f4292e4558f8f6ea3d493 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Fri, 16 Dec 2022 19:16:20 -0500 Subject: [PATCH 068/164] fix: installer --- README.md | 21 +-------------------- docs/installation.md | 20 ++++++++++++++++++++ installer.sh | 7 +++++-- 3 files changed, 26 insertions(+), 22 deletions(-) create mode 100644 docs/installation.md diff --git a/README.md b/README.md index 9d5bbe8..eb8f3c7 100644 --- a/README.md +++ b/README.md @@ -19,7 +19,6 @@ - [Installation](#installation) - [Software Design](#software-design) - [Development](#development) -- [Initialization](#initialization) - [MQTT](#mqtt) - [Updates](#updates) - [Logging](#logging) @@ -47,6 +46,7 @@ sudo wget -qO- https://openpod.recursion.space | bash /dev/stdin [options] [argu |:-----------:|---------------------------|------------------------------------------------------------------------------------| | -h | Help | sudo wget -qO- openpod.recursion.space \| bash /dev/stdin -h | | -d | Enable Debug Installation | sudo wget -qO- openpod.recursion.space \| bash /dev/stdin -d | +| -b | Alternative Branch | sudo wget -qO- openpod.recursion.space \| bash /dev/stdin -b dev | | -u | Set Custom URL endpoint | sudo wget -qO- openpod.recursion.space \| bash /dev/stdin -u stage.recursion.space | \*No supported arguments are currently supported. @@ -77,25 +77,6 @@ Each Pod is on its MQTT topic, and the topic is created once the Pod registers w Pod software development is done on a DigitalOcean hosted server. It is then transferred to the physical hardware running the system and tested before going into production. -# Initialization - -Pod uses an initializer or “launcher” to configure the system before running the main code. The launcher performs the following tasks before running the main program: - -- Update & Upgrade the OS -- [Sync Clock](https://vitux.com/keep-your-clock-sync-with-internet-time-servers-in-ubuntu/) -- Installs program required packages -- Check for required files, create them if missing -- Configure start on boot file -- Creates serial if needed -- Registers the Pod with the API server -- Initializes the main program under [/opt/](https://unix.stackexchange.com/questions/11544/what-is-the-difference-between-opt-and-usr-local) - -To launch the software, when the device boots, the launcher configures the system first to run the software in the [background](https://janakiev.com/blog/python-background/) and then creates a script that executes the launch on [startup](https://askubuntu.com/questions/817011/run-python-script-on-os-boot). Working version with a [cronjob](https://www.linuxbabe.com/linux-server/how-to-enable-etcrc-local-with-systemd). - -```bash -(@reboot (cd /home/ubuntu/RecursionHub && sudo python3 HUB_Launcher.py &)) -``` - # MQTT The MQTT protocol is used for server to Pod communication, allowing the Pod to listen in real-time for incoming instruction. diff --git a/docs/installation.md b/docs/installation.md new file mode 100644 index 0000000..a0341d6 --- /dev/null +++ b/docs/installation.md @@ -0,0 +1,20 @@ +# Initialization + +Pod uses an initializer or “launcher” to configure the system before running the main code. The launcher performs the following tasks before running the main program: + +- Update & Upgrade the OS +- [Sync Clock](https://vitux.com/keep-your-clock-sync-with-internet-time-servers-in-ubuntu/) +- Installs program required packages +- Check for required files, create them if missing +- Configure start on boot file +- Creates serial if needed +- Registers the Pod with the API server +- Initializes the main program under [/opt/](https://unix.stackexchange.com/questions/11544/what-is-the-difference-between-opt-and-usr-local) + +To launch the software, when the device boots, the launcher configures the system first to run the software in the [background](https://janakiev.com/blog/python-background/) and then creates a script that executes the launch on [startup](https://askubuntu.com/questions/817011/run-python-script-on-os-boot). Working version with a [cronjob](https://www.linuxbabe.com/linux-server/how-to-enable-etcrc-local-with-systemd). + +```bash +(@reboot (cd /home/ubuntu/RecursionHub && sudo python3 HUB_Launcher.py &)) +``` + +## Installer diff --git a/installer.sh b/installer.sh index 164c2fe..20a1b29 100644 --- a/installer.sh +++ b/installer.sh @@ -48,13 +48,16 @@ while getopts ":hud" flags; do done if [ $DEBUG -eq 1 ]; then + branch='dev-release' URL='dev.recursion.space' elif [ $DEBUG -eq 0 ]; then + branch='release' URL='recursion.space' fi echo "Installing OpenPod with the following options:" echo "Debug: $DEBUG" +echo "Branch: $branch" echo "URL: $URL" # -------------------------------- Verify Root ------------------------------- # @@ -133,13 +136,13 @@ sudo rm -rf /opt/OpenPod set -e # Exit when any command fails. sudo mkdir -p /opt cd /opt -sudo git clone --single-branch --branch release https://github.com/RecursionSpace/OpenPod.git +sudo git clone --single-branch --branch $branch https://github.com/RecursionSpace/OpenPod.git cd OpenPod # ----------------------------- Setup Enviroment ----------------------------- # sudo python3.11 -m venv /opt/OpenPod/env source /opt/OpenPod/env/bin/activate -sudo pip install --no-input -U -r /opt/OpenPod/requirements.txt --no-cache-dir --no-dependencies +pip install --no-input -U -r /opt/OpenPod/requirements.txt --no-cache-dir --no-dependencies # ---------------------------- Create Directories ---------------------------- # sudo mkdir -p /opt/OpenPod/logs From fa145acd6508796169a47f03c27771b0dc9e0adc Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Fri, 16 Dec 2022 19:22:59 -0500 Subject: [PATCH 069/164] fix: installer --- README.md | 2 +- installer.sh | 13 +++++++++++++ 2 files changed, 14 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index eb8f3c7..cec1db2 100644 --- a/README.md +++ b/README.md @@ -46,7 +46,7 @@ sudo wget -qO- https://openpod.recursion.space | bash /dev/stdin [options] [argu |:-----------:|---------------------------|------------------------------------------------------------------------------------| | -h | Help | sudo wget -qO- openpod.recursion.space \| bash /dev/stdin -h | | -d | Enable Debug Installation | sudo wget -qO- openpod.recursion.space \| bash /dev/stdin -d | -| -b | Alternative Branch | sudo wget -qO- openpod.recursion.space \| bash /dev/stdin -b dev | +| -b | Alternative Branch | sudo wget -qO- openpod.recursion.space \| bash /dev/stdin -b dev-release | | -u | Set Custom URL endpoint | sudo wget -qO- openpod.recursion.space \| bash /dev/stdin -u stage.recursion.space | \*No supported arguments are currently supported. diff --git a/installer.sh b/installer.sh index 20a1b29..41a9e87 100644 --- a/installer.sh +++ b/installer.sh @@ -38,6 +38,8 @@ while getopts ":hud" flags; do h) # display Help Help exit;; + b) # Custom branch + branch="${OPTARG}";; d) # Enable debug mode DEBUG=1 ;; u) # Custom URL endpoint @@ -74,6 +76,17 @@ sudo timedatectl set-timezone UTC # Dependencies # # ---------------------------------------------------------------------------- # +# ------------------------------ build-essential ----------------------------- # +# Required to install RPi.GPIO - https://github.com/ynsta/steamcontroller/issues/42 +REQUIRED_PKG="build-essential" +PKG_OK=$(dpkg-query -W --showformat='${Status}\n' $REQUIRED_PKG|grep "install ok installed") +if [ "" = "$PKG_OK" ]; then + echo "No $REQUIRED_PKG. Setting up $REQUIRED_PKG..." + sudo apt-get install build-essential -y +else + echo "build-essential already installed, skipping..." +fi + # ---------------------------------- chrony ---------------------------------- # REQUIRED_PKG="chrony" PKG_OK=$(dpkg-query -W --showformat='${Status}\n' $REQUIRED_PKG|grep "install ok installed") From 97eb5786d652935ab78ee8e3ac0ed11e89c1c223 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Fri, 16 Dec 2022 19:34:00 -0500 Subject: [PATCH 070/164] Update installer.sh --- installer.sh | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/installer.sh b/installer.sh index 41a9e87..33b496f 100644 --- a/installer.sh +++ b/installer.sh @@ -128,6 +128,16 @@ else echo "Python 3.11 already installed" fi +# -------------------------------- Python-Dev -------------------------------- # +REQUIRED_PKG="python3-dev" +PKG_OK=$(dpkg-query -W --showformat='${Status}\n' $REQUIRED_PKG|grep "install ok installed") +if [ "" = "$PKG_OK" ]; then + echo "No $REQUIRED_PKG. Setting up $REQUIRED_PKG..." + sudo apt-get install python3-dev -y +else + echo "python3-dev already installed, skipping..." +fi + # ------------------------ Python Virtual Environment ------------------------ # REQUIRED_PKG="python3.11-venv" PKG_OK=$(dpkg-query -W --showformat='${Status}\n' $REQUIRED_PKG|grep "install ok installed") From 8ff431551244ef1bf2b7c4d78598129a93340eb6 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Fri, 16 Dec 2022 20:08:55 -0500 Subject: [PATCH 071/164] installer tests --- docs/installation.md | 7 ++++--- installer.sh | 18 +++++++++--------- openpod/launcher.py | 31 ++++++++++++++----------------- 3 files changed, 27 insertions(+), 29 deletions(-) diff --git a/docs/installation.md b/docs/installation.md index a0341d6..28c3843 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -1,4 +1,7 @@ -# Initialization +## Installer + + +## Initialization Pod uses an initializer or “launcher” to configure the system before running the main code. The launcher performs the following tasks before running the main program: @@ -16,5 +19,3 @@ To launch the software, when the device boots, the launcher configures the syste ```bash (@reboot (cd /home/ubuntu/RecursionHub && sudo python3 HUB_Launcher.py &)) ``` - -## Installer diff --git a/installer.sh b/installer.sh index 33b496f..a933de4 100644 --- a/installer.sh +++ b/installer.sh @@ -27,7 +27,7 @@ Help() # ---------------------------------------------------------------------------- # # Defaults # # ---------------------------------------------------------------------------- # -DEBUG=0 # -d +DEBUG=flase # -d # ---------------------------------------------------------------------------- # @@ -41,7 +41,7 @@ while getopts ":hud" flags; do b) # Custom branch branch="${OPTARG}";; d) # Enable debug mode - DEBUG=1 ;; + DEBUG=true ;; u) # Custom URL endpoint URL="${OPTARG}";; \?) echo "Invalid option: -${OPTARG}" >&2; @@ -49,10 +49,10 @@ while getopts ":hud" flags; do esac done -if [ $DEBUG -eq 1 ]; then +if [ $DEBUG ]; then branch='dev-release' URL='dev.recursion.space' -elif [ $DEBUG -eq 0 ]; then +elif [ ! $DEBUG ]; then branch='release' URL='recursion.space' fi @@ -129,13 +129,13 @@ else fi # -------------------------------- Python-Dev -------------------------------- # -REQUIRED_PKG="python3-dev" +REQUIRED_PKG="python3.11-dev" PKG_OK=$(dpkg-query -W --showformat='${Status}\n' $REQUIRED_PKG|grep "install ok installed") if [ "" = "$PKG_OK" ]; then echo "No $REQUIRED_PKG. Setting up $REQUIRED_PKG..." - sudo apt-get install python3-dev -y + sudo apt-get install python3.11-dev -y else - echo "python3-dev already installed, skipping..." + echo "python3.11-dev already installed, skipping..." fi # ------------------------ Python Virtual Environment ------------------------ # @@ -185,13 +185,13 @@ serial_uuid=$(cat /proc/sys/kernel/random/uuid) xbee_uuid=$(cat /proc/sys/kernel/random/uuid) openpod_version=$(git rev-parse HEAD) echo '{ - "serial": '"'$serial_uuid'"', + "serial": "$serial_uuid", "timezone": "UTC", "XBEE_KY": '"'$xbee_uuid'"', "XBEE_OP": "0", "url": '"'$URL'"', "version": '"'$openpod_version'"', - "debug": '"'$DEBUG'"' + "debug": $DEBUG }' > /opt/OpenPod/system.json # --------------------------- Setup OpenPod Service -------------------------- # diff --git a/openpod/launcher.py b/openpod/launcher.py index a39d5b8..ee932a2 100644 --- a/openpod/launcher.py +++ b/openpod/launcher.py @@ -1,34 +1,31 @@ #!/usr/bin/env python3 -""" +''' - Initiates the LED visuals - Installs new requirements as part of an update -(if needed, might want to have a seprate process for this) +(if needed, might want to have a septate process for this) - Checks for connection to server, - If no connection is found then LED indication is set. - Pulls configuration files from server. - Launches main program -""" +''' + import sys import json import threading import subprocess + +import config + # ---------------------------------------------------------------------------- # # Check Requirements # # ---------------------------------------------------------------------------- # -try: - import config # pylint: disable=W0611 -except ImportError : - subprocess.call(['sudo', 'pip3', 'install', 'config']) - - - -from modules import rec_lan, rec_log -from modules.rec_log import exception_log +from .modules import rec_lan, rec_log +from .modules.rec_log import exception_log -from settings import LED_STAT +from .settings import LED_STAT import settings if settings.Pi: @@ -47,7 +44,7 @@ # Program Start Visual # # ---------------------------------------------------------------------------- # if settings.Pi: - led_stat_thread = threading.Thread(target = rec_gpio.led_stat_thread) + led_stat_thread = threading.Thread(target=rec_gpio.led_stat_thread) led_stat_thread.start() rec_gpio.state(LED_STAT, 1, 0) @@ -57,7 +54,7 @@ # Check Network Connection # # ---------------------------------------------------------------------------- # try: - rec_lan.monitor_network() #Monitors the network connection while the program is running. + rec_lan.monitor_network() # Monitors the network connection while the program is running. except RuntimeError as err: exception_log.error("FATAL - Start Network Monitoring - Error: %s", err) @@ -68,11 +65,11 @@ # ---------------------------------------------------------------------------- # # Update version in system.json # # ---------------------------------------------------------------------------- # -#Only updates if called by bash script and environmental variable was passed in. +# Only updates if called by bash script and environmental variable was passed in. if len(sys.argv) > 1: with open("system.json", "r+", encoding="utf-8") as file: data = json.load(file) - data.update( {"CurrentVersion":sys.argv[1]} ) + data.update({"CurrentVersion": sys.argv[1]}) file.seek(0) json.dump(data, file) file.truncate() From ab4c59289957860ce5041544272c84da1b6d6c0e Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Fri, 16 Dec 2022 20:15:19 -0500 Subject: [PATCH 072/164] improved installer --- installer.sh | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/installer.sh b/installer.sh index a933de4..d814db5 100644 --- a/installer.sh +++ b/installer.sh @@ -52,9 +52,11 @@ done if [ $DEBUG ]; then branch='dev-release' URL='dev.recursion.space' + API_URL='dev.api.recursion.space' elif [ ! $DEBUG ]; then branch='release' URL='recursion.space' + API_URL='api.recursion.space' fi echo "Installing OpenPod with the following options:" @@ -185,13 +187,14 @@ serial_uuid=$(cat /proc/sys/kernel/random/uuid) xbee_uuid=$(cat /proc/sys/kernel/random/uuid) openpod_version=$(git rev-parse HEAD) echo '{ - "serial": "$serial_uuid", + "serial": "'$serial_uuid'", "timezone": "UTC", "XBEE_KY": '"'$xbee_uuid'"', "XBEE_OP": "0", "url": '"'$URL'"', + "api_url": "'$API_URL'", "version": '"'$openpod_version'"', - "debug": $DEBUG + "debug": '$DEBUG' }' > /opt/OpenPod/system.json # --------------------------- Setup OpenPod Service -------------------------- # From 1f906b4ff2647a360b093f94289537c310badb7b Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Fri, 16 Dec 2022 20:22:27 -0500 Subject: [PATCH 073/164] Update installer.sh --- installer.sh | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/installer.sh b/installer.sh index d814db5..45b6ca4 100644 --- a/installer.sh +++ b/installer.sh @@ -63,6 +63,7 @@ echo "Installing OpenPod with the following options:" echo "Debug: $DEBUG" echo "Branch: $branch" echo "URL: $URL" +echo "API_URL: $API_URL" # -------------------------------- Verify Root ------------------------------- # if [ "$EUID" -ne 0 ] @@ -187,14 +188,20 @@ serial_uuid=$(cat /proc/sys/kernel/random/uuid) xbee_uuid=$(cat /proc/sys/kernel/random/uuid) openpod_version=$(git rev-parse HEAD) echo '{ + "debug": '$DEBUG', "serial": "'$serial_uuid'", "timezone": "UTC", - "XBEE_KY": '"'$xbee_uuid'"', - "XBEE_OP": "0", - "url": '"'$URL'"', + "url": "'$URL'", "api_url": "'$API_URL'", - "version": '"'$openpod_version'"', - "debug": '$DEBUG' + "XBEE": { + "KY": "'$xbee_uuid'", + "OP": "0" + }, + "GPIO": { + "LED_IO": 23, + "LED_STAT": 17 + }, + "version": "'$openpod_version'" }' > /opt/OpenPod/system.json # --------------------------- Setup OpenPod Service -------------------------- # From c0bc8d4d247eeef41dbff62a0af7a0a655f2e2c4 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Fri, 16 Dec 2022 21:31:11 -0500 Subject: [PATCH 074/164] fix: GPIO --- docs/pod_design.md | 24 ++--- openpod/launcher.py | 14 +-- openpod/modules/op_gpio.py | 180 ++++++++++++++++++++++++++++++++++++ openpod/modules/rec_api.py | 10 +- openpod/modules/rec_gpio.py | 113 ---------------------- openpod/modules/rec_lan.py | 18 ++-- openpod/modules/rec_xbee.py | 84 ++++++++--------- openpod/pod.py | 43 +++++---- 8 files changed, 272 insertions(+), 214 deletions(-) create mode 100644 openpod/modules/op_gpio.py delete mode 100644 openpod/modules/rec_gpio.py diff --git a/docs/pod_design.md b/docs/pod_design.md index 31f471b..1d4f3d6 100644 --- a/docs/pod_design.md +++ b/docs/pod_design.md @@ -4,21 +4,21 @@ Currently the system makes use of the GPIO on the Raspberry Pi. These pins are l ### Visual Indicators -The Hub has two LED indicators on it, a green and red. The [green LED](https://www.digikey.com/en/products/detail/lumex-opto-components-inc/SSI-LXH072GD/144732) is 2.2v at 25mA and will require a 68.75ohm (68ohm standard) resistor while the [red LED](https://www.digikey.com/en/products/detail/lumex-opto-components-inc/SSI-LXH072ID/144729) one is 2v at 30mA and will require a 81.25 ohm (82 ohm standard) resistor. Both resistors were [calculated](https://ohmslawcalculator.com/led-resistor-calculator) with the current caped at 16mA. +The Pod has two LED indicators on it, a green and red. The [green LED](https://www.digikey.com/en/products/detail/lumex-opto-components-inc/SSI-LXH072GD/144732) is 2.2v at 25mA and will require a 68.75ohm (68ohm standard) resistor while the [red LED](https://www.digikey.com/en/products/detail/lumex-opto-components-inc/SSI-LXH072ID/144729) one is 2v at 30mA and will require a 81.25 ohm (82 ohm standard) resistor. Both resistors were [calculated](https://ohmslawcalculator.com/led-resistor-calculator) with the current caped at 16mA. A combination of the LEDs blinking or solid on/off in a [controlled pattern](https://en.wikipedia.org/wiki/International_Blinking_Pattern_Interpretation) provides information to the users. -| | Green - I/O “Simulating XBee activity, should really be connected to one of the XBee pins” | Red - Overall Status | -|:----------------------------------------------------------------------------------------------------------------------------------------------------:|:------------------------------------------------------------------------------------------:|:--------------------:| -| Program Called: Not ready to receive requests and have not tested network connection. | Off | On | -| No Ethernet: The HUB has not received an IP address and is set to 127.0.0.1 | N/A | 4 Hz | -| Ethernet, No Internet: | N/A | 2 Hz | -| Internet, Recursion Unreachable: | N/A | 1 Hz | -| Recursion.Space Reachable: | N/A | Off | -| Unregistered: Registered and Ready: All network and hardware tests passed, ready to receive XBee requests and HUB IS NOT registered with a facility. | 1 Hz | N/A | -| Registered and Ready: All network and hardware tests passed, ready to receive XBee requests, and HUB IS registered with a facility. | On | N/A | -| XBee I/O Data: Anytime the XBee receives or sends data. | 4 Hz | N/A | -| Fatal Error: - USB XBee Unreachable | 2 Hz Alternating | 2 Hz Alternating | +| | Green - I/O “Simulating XBee activity, should really be connected to one of the XBee pins” | Red - Overall Status | +|:-----------------------------------------------------------------------------------------------------------------------------:|:------------------------------------------------------------------------------------------:|:--------------------:| +| **Initializing** - Program called, not ready to receive requests. | Off | On | +| No Ethernet: The HUB has not received an IP address and is set to 127.0.0.1 | N/A | 4 Hz | +| Ethernet, No Internet: | N/A | 2 Hz | +| Internet, Recursion Unreachable: | N/A | 1 Hz | +| Recursion.Space Reachable: | N/A | Off | +| **Unregistered** - All network and hardware tests passed, ready to receive XBee requests, Pod IS NOT registered with a space. | 1 Hz | N/A | +| **Ready** - All network and hardware tests passed, ready to receive XBee requests, and Pod IS registered with a space. | On | N/A | +| XBee I/O Data: Anytime the XBee receives or sends data. | 4 Hz | N/A | +| Fatal Error: - USB XBee Unreachable | 2 Hz Alternating | 2 Hz Alternating | ## Assembly diff --git a/openpod/launcher.py b/openpod/launcher.py index ee932a2..fe3a62e 100644 --- a/openpod/launcher.py +++ b/openpod/launcher.py @@ -50,18 +50,6 @@ rec_gpio.state(LED_STAT, 1, 0) -# ---------------------------------------------------------------------------- # -# Check Network Connection # -# ---------------------------------------------------------------------------- # -try: - rec_lan.monitor_network() # Monitors the network connection while the program is running. - -except RuntimeError as err: - exception_log.error("FATAL - Start Network Monitoring - Error: %s", err) - -finally: - exception_log.debug("Launcher - Exiting Check Network Connection") - # ---------------------------------------------------------------------------- # # Update version in system.json # # ---------------------------------------------------------------------------- # @@ -83,7 +71,7 @@ rec_log.snapshot(public, local) except RuntimeError as err: - exception_log.error("FAITIAL - Generate Snapshot - Error: %s", err) + exception_log.error("FATAL - Generate Snapshot - Error: %s", err) finally: exception_log.debug("Launcher - Exiting Settings for Debugging") diff --git a/openpod/modules/op_gpio.py b/openpod/modules/op_gpio.py new file mode 100644 index 0000000..a38dc1d --- /dev/null +++ b/openpod/modules/op_gpio.py @@ -0,0 +1,180 @@ +#!/usr/bin/env python3 + +''' +Module manages raspberry pi gpio functionality. +''' + +import json +from time import sleep + +import config + +try: + from RPi import GPIO + GPIO_AVAILABLE = True +except (RuntimeError, ModuleNotFoundError): + print("NOTICE | This device is probably not a Raspberry Pi.") + GPIO_AVAILABLE = False + +with open('/opt/OpenPod/system.json', 'r', encoding="utf-8") as system_file: + system = json.load(system_file) +LED_IO = system['GPIO']['LED_IO'] +LED_STAT = system['GPIO']['LED_STAT'] + + +if GPIO_AVAILABLE: + GPIO.setmode(GPIO.BCM) # Setup for LED indicators. + GPIO.setwarnings(False) + + GPIO.setup(LED_IO, GPIO.OUT) + GPIO.setup(LED_STAT, GPIO.OUT) + + GPIO.output(LED_IO, GPIO.LOW) + GPIO.output(LED_STAT, GPIO.LOW) + +config.LED_IO_ON = 0 +config.LED_IO_OFF = 0 + +config.LED_STAT_ON = 0 +config.LED_STAT_OFF = 0 + + +# ---------------------------------------------------------------------------- # +# Indicator Presets # +# ---------------------------------------------------------------------------- # +def initializing(): + ''' + Indicates that the system is initializing. + LED_IO - Solid on + LED_STAT - Off + ''' + state(LED_IO, 1, 0) + + +def unregistered(): + ''' + Indicates that the system is not registered. + LED_IO - 1 Hz + LED_STAT - N/A + ''' + state(LED_IO, 1, 1) + + +def ready(): + ''' + Indicates that the system is ready. + LED_IO - Solid on + LED_STAT - N/A + ''' + state(LED_IO, 1, 0) + + +def incoming_data(): + ''' + Indicates that there is incoming XBee data. + LED_IO - 4 Hz + LED_STAT - N/A + ''' + state(LED_IO, 0.125, 0.125) + + +def no_network(): + ''' + Indicates that there is no network connection. + LED_IO - N/A + LED_STAT - 4 Hz + ''' + state(LED_STAT, 0.125, 0.125) + + +def no_internet(): + ''' + Indicates that there is no internet connection. + LED_IO - N/A + LED_STAT - 2 Hz + ''' + state(LED_STAT, 0.25, 0.25) + + +def no_recursion(): + ''' + Indicates that a connection to recursion.space could not be established. + LED_IO - N/A + LED_STAT - 1 Hz + ''' + state(LED_STAT, 1, 1) + + +def networked(): + ''' + Indicates that the system is connected to the network. + LED_IO - N/A + LED_STAT - Off + ''' + state(LED_STAT, 0, 0) + +# ---------------------------------------------------------------------------- # +# Set LED Patterns # +# ---------------------------------------------------------------------------- # + + +def state(io_pin, on_time=0, off_time=0): + """ + Sets the io_pin state to on or off. + """ + if io_pin == LED_IO: + led_io_on_off(on_time, off_time) + if io_pin == LED_STAT: + led_io_stat_on_off(on_time, off_time) + + +def led_io_on_off(on_time=0, off_time=0): + ''' + Controls the LED used to indicate input/output actions + ''' + config.LED_IO_ON = on_time + config.LED_IO_OFF = off_time + + +def led_io_stat_on_off(on_time=0, off_time=0): + ''' + Controls the LED used to indicate general system status + ''' + config.LED_STAT_ON = on_time + config.LED_STAT_OFF = off_time + + +# ---------------------------------------------------------------------------- # +# LED Threads # +# ---------------------------------------------------------------------------- # + +def led_io_thread(): + ''' + Process occurring in the thread for the I/O LED + ''' + while True: + if config.LED_IO_ON > 0: + GPIO.output(LED_IO, GPIO.HIGH) + sleep(config.LED_IO_ON) + if config.LED_IO_OFF > 0: + GPIO.output(LED_IO, GPIO.LOW) + sleep(config.LED_IO_OFF) + if config.LED_IO_ON == 0 and config.LED_IO_OFF == 0: + GPIO.output(LED_IO, GPIO.LOW) + sleep(.1) # Allows CPU time for other threads. + + +def led_stat_thread(): + ''' + Process occurring in thread for the System Status LED + ''' + while True: + if config.LED_STAT_ON > 0: + GPIO.output(LED_STAT, GPIO.HIGH) + sleep(config.LED_STAT_ON) + if config.LED_STAT_OFF > 0: + GPIO.output(LED_STAT, GPIO.LOW) + sleep(config.LED_STAT_OFF) + if config.LED_STAT_ON == 0 and config.LED_STAT_OFF == 0: + GPIO.output(LED_STAT, GPIO.LOW) + sleep(.1) # Allows CPU time for other threads. diff --git a/openpod/modules/rec_api.py b/openpod/modules/rec_api.py index 8dd08f4..08695fd 100644 --- a/openpod/modules/rec_api.py +++ b/openpod/modules/rec_api.py @@ -10,11 +10,9 @@ import settings +from modules import op_gpio from modules.rec_log import log_api, hash_data -if settings.IS_PI: - from modules import rec_gpio - from settings import LED_IO # Performs all API calls to the server, functions should be used as a thread. @@ -129,7 +127,7 @@ def register_hub(): # Needs updated! # ---------------------------------------------------------------------------- # def link_hub(): ''' - Assosiate the hub with a space. + Associate the Pod with a space. ''' try: with open("system.json", "r+", encoding="utf-8") as file: @@ -146,8 +144,8 @@ def link_hub(): json.dump(system_data, file) file.truncate() - if settings.IS_PI: - rec_gpio.state(LED_IO, 1) + op_gpio.ready() + except OSError as err: log_api.error("link_hub - Unable to open file system.json - %s", err) diff --git a/openpod/modules/rec_gpio.py b/openpod/modules/rec_gpio.py deleted file mode 100644 index a905855..0000000 --- a/openpod/modules/rec_gpio.py +++ /dev/null @@ -1,113 +0,0 @@ -#!/usr/bin/env python3 - -''' -Modlue manages raspbery pi gpio functionality. -''' - -from time import sleep - -import config - -from RPi import GPIO - -from settings import LED_IO, LED_STAT - -GPIO.setmode(GPIO.BCM) #Setup for LED indicators. -GPIO.setwarnings(False) - -GPIO.setup(LED_IO, GPIO.OUT) -GPIO.setup(LED_STAT, GPIO.OUT) - -GPIO.output(LED_IO,GPIO.LOW) -GPIO.output(LED_STAT,GPIO.LOW) - -LED_IO_ON = 0 -LED_IO_OFF = 0 -LED_STAT_ON = 0 -LED_STAT_OFF = 0 - -config.LED_IO_ON = 0 -config.LED_IO_OFF = 0 - -config.LED_STAT_ON = 0 -config.LED_STAT_OFF = 0 - -# def Blink(io_pin, BlinkTime, BlinkCount): -# for i in range (BlinkCount): -# sleep (BlinkTime) -# GPIO.output (io_pin, GPIO.HIGH) -# sleep (BlinkTime) -# GPIO.output (io_pin, GPIO.LOW) -# return - -# ---------------------------------------------------------------------------- # -# Set LED Patterns # -# ---------------------------------------------------------------------------- # - -def state(io_pin, on_time=0, off_time=0): - """ - Sets the io_pin state to on or off. - """ - if io_pin == LED_IO: - led_io_on_off(on_time, off_time) - if io_pin == LED_STAT: - led_io_stat_on_off(on_time, off_time) - - -def led_io_on_off(on_time=0, off_time=0): - ''' - Controls the LED used to indicate input/output actions - ''' - # global LED_IO_ON - # global LED_IO_OFF - config.LED_IO_ON = on_time - config.LED_IO_OFF = off_time - - -def led_io_stat_on_off(on_time=0, off_time=0): - ''' - Controls the LED used to indicate general system status - ''' - # global LED_STAT_ON - # global LED_STAT_OFF - config.LED_STAT_ON = on_time - config.LED_STAT_OFF = off_time - -# ---------------------------------------------------------------------------- # -# LED Threads # -# ---------------------------------------------------------------------------- # - -def led_io_thread(): - ''' - Process occuring in the thread for the I/O LED - ''' - while True: - # global LED_IO_ON - # global LED_IO_OFF - if config.LED_IO_ON > 0: - GPIO.output(LED_IO, GPIO.HIGH) - sleep (config.LED_IO_ON) - if config.LED_IO_OFF > 0: - GPIO.output(LED_IO, GPIO.LOW) - sleep (config.LED_IO_OFF) - if config.LED_IO_ON == 0 and config.LED_IO_OFF == 0: - GPIO.output(LED_IO, GPIO.LOW) - sleep(.1) #Allows CPU time for other threads. - - -def led_stat_thread(): - ''' - Process occuring in thethread for the System Status LED - ''' - while True: - # global LED_STAT_ON - # global LED_STAT_OFF - if config.LED_STAT_ON > 0: - GPIO.output(LED_STAT, GPIO.HIGH) - sleep (config.LED_STAT_ON) - if config.LED_STAT_OFF>0: - GPIO.output(LED_STAT, GPIO.LOW) - sleep (config.LED_STAT_OFF) - if config.LED_STAT_ON == 0 and config.LED_STAT_OFF == 0: - GPIO.output(LED_STAT, GPIO.LOW) - sleep(.1) #Allows CPU time for other threads. diff --git a/openpod/modules/rec_lan.py b/openpod/modules/rec_lan.py index 8b50c57..99c0a0f 100644 --- a/openpod/modules/rec_lan.py +++ b/openpod/modules/rec_lan.py @@ -1,6 +1,6 @@ #!/usr/bin/env python3 ''' -Handles all netwrork related activities for the hub. +Handles all network related activities for the hub. DOES NOT PERFORM KEEPALIVE - SEE rec_api ''' @@ -10,18 +10,16 @@ import settings +from modules import op_gpio from modules.rec_log import network_log -if settings.IS_PI: - from modules import rec_gpio - # ------------ Triggers visual indicators based on network status. ----------- # def monitor_network(last_network_status=5, thread_delay=30.0): ''' Threaded: Yes Checks network connection, then updates visual indicators. - Thread delay is extended to up 600 seconds if error occurs. Resets to 10 seconds on sucess. + Thread delay is extended to up 600 seconds if error occurs. Resets to 10 seconds on success. ''' try: current_network_status = test_network() @@ -33,16 +31,16 @@ def monitor_network(last_network_status=5, thread_delay=30.0): if settings.IS_PI: if network_status == 0: - rec_gpio.state(settings.LED_STAT, .125, .125) + op_gpio.no_network() if network_status == 1: - rec_gpio.state(settings.LED_STAT, .25, .25) + op_gpio.no_internet() if network_status == 2: - rec_gpio.state(settings.LED_STAT, 1, 1) + op_gpio.no_recursion() if network_status == 3: - rec_gpio.state(settings.LED_STAT, 0, 0) + op_gpio.networked() thread_delay = 10 @@ -81,7 +79,7 @@ def test_network(): if networked() is False: return 0 - # Recived a IP address that is not 127.0.0.1, but was unable to access the internet. + # Received a IP address that is not 127.0.0.1, but was unable to access the internet. if internet_on() is False: network_log.warning('LAN Check Fail') return 1 diff --git a/openpod/modules/rec_xbee.py b/openpod/modules/rec_xbee.py index 7315b3d..e7ad827 100644 --- a/openpod/modules/rec_xbee.py +++ b/openpod/modules/rec_xbee.py @@ -16,12 +16,9 @@ from modules import rec_lookup, rec_api from modules.rec_log import log_xbee -#Not sure if LED indicators are being used here. -# if settings.Pi: -# from modules import rec_gpio - #Serial information needed to detect and use XBee. - # https://www.reddit.com/r/Python/comments/6jtzua/pyserial_minimum_time_for_timeout/ +# Serial information needed to detect and use XBee. +# https://www.reddit.com/r/Python/comments/6jtzua/pyserial_minimum_time_for_timeout/ try: ser = serial.Serial("/dev/ttyUSB0", baudrate=9600, timeout=.3) except serial.serialutil.SerialException as serial_err: @@ -29,14 +26,15 @@ # from settings import LED_IO, LED_STAT + def receive(): ''' Processes incoming xbee serial data. ''' - rx_data = ser.readline() #Read data that is currently waiting. + rx_data = ser.readline() # Read data that is currently waiting. sleep(.05) - data_remaining = ser.inWaiting() #Read any data that was not origionally read. - rx_data += ser.read(data_remaining) #Add any data that was left out in first pass. + data_remaining = ser.inWaiting() # Read any data that was not origionally read. + rx_data += ser.read(data_remaining) # Add any data that was left out in first pass. rx_data = binascii.b2a_hex(rx_data).strip().decode('utf-8') @@ -46,33 +44,34 @@ def receive(): log_xbee.info("The cordinator has started.") return (0, 0, 5) - if rx_data[41:(len(rx_data)-2)].isdigit() is False and rx_data != '7e00028a066f': + if rx_data[41:(len(rx_data)-2)].isdigit() is False and rx_data != '7e00028a066f': rx_source = rx_data[8:24] log_xbee.info('Device %s is connecting to network', rx_source) # rx_data = rx_data[30:(len(rx_data)-2)] lookup_responce = rec_lookup.count_matching_mac(rx_source) - #for Responce in MySQL_Responce: + # for Responce in MySQL_Responce: log_xbee.info('Returned %s matches for MAC addressed.', lookup_responce) if lookup_responce == 0: if len(rx_source) == 16: rec_api.pair_node(rx_source) log_xbee.info('%s Connected To XBee Network', rx_source) - return (0,0,5) + return (0, 0, 5) log_xbee.warning("%s is not a valid MAC length, not paired", rx_source) - return (0,0,0) # (Placeholder) Need to set a return that an error has occured. + return (0, 0, 0) # (Placeholder) Need to set a return that an error has occured. - if rx_data[42:(len(rx_data)-2)].isdigit() is True and rx_data != '7e00028a066f': + if rx_data[42:(len(rx_data)-2)].isdigit() is True and rx_data != '7e00028a066f': rx_source = rx_data[8:24] rx_data = rx_data[42:(len(rx_data)-2)] log_xbee.info('User %s requesting access from node %s', rx_data, rx_source) serial_lookup = rec_lookup.access_request(rx_data, rx_source) - return rx_source,rx_data,serial_lookup + return rx_source, rx_data, serial_lookup + + return (0, 0, 0) # (Placeholder) Need to set a return that an error has occured. - return (0,0,0) # (Placeholder) Need to set a return that an error has occured. def transmit(destination, data): ''' @@ -83,23 +82,23 @@ def transmit(destination, data): # BlinkThread.start() dest_16bit = 'FFFE' #data_hex = binascii.hexlify(data) - data_hex = data + data_hex = data hex_len = hex(14 + (len(data_hex)//2)) - hex_len = hex_len.replace('x','00') + hex_len = hex_len.replace('x', '00') checksum = 17 - for i in range(0,len(destination),2): - checksum = checksum + int(destination[i:i+2],16) - for i in range(0,len(dest_16bit),2): - checksum = checksum + int(dest_16bit[i:i+2],16) - for i in range(0,len(data_hex),2): - checksum = checksum + int(data_hex[i:i+2],16) - checksum = checksum%256 + for i in range(0, len(destination), 2): + checksum = checksum + int(destination[i:i+2], 16) + for i in range(0, len(dest_16bit), 2): + checksum = checksum + int(dest_16bit[i:i+2], 16) + for i in range(0, len(data_hex), 2): + checksum = checksum + int(data_hex[i:i+2], 16) + checksum = checksum % 256 checksum = 256 - checksum checksum = format(checksum, '02x') #checksum = hex(checksum) #checksum = checksum[-2:] tx_req = ("7E" + hex_len + "10" + "00" + destination - + dest_16bit + "00" + "00" + data_hex + checksum) + + dest_16bit + "00" + "00" + data_hex + checksum) transmission = binascii.unhexlify(tx_req) log_xbee.info("Transmitting: %s", transmission) ser.write(transmission) @@ -107,6 +106,8 @@ def transmit(destination, data): # ---------------------------------------------------------------------------- # # New XBee Setup Procedure # # ---------------------------------------------------------------------------- # + + def configure_xbee(): ''' Configure Hub XBee modules for first time use. @@ -116,24 +117,24 @@ def configure_xbee(): with open('system.json', 'r', encoding="UTF-8") as file: system_data = json.load(file) - #Sequence of commands to configure a new XBee, sent in transparent mode. + # Sequence of commands to configure a new XBee, sent in transparent mode. param_config = [ - b'ATCE1\r', #Enable Cordinator - b'ATAP1\r', #Enable API Mode - b'ATAO1\r', #Set Output Mode To Explicit - b'ATEE1\r', #Enable Security - b'ATEO2\r', #Enable Trust Center - f'ATKY{system_data["XBEE_KY"]}'.encode(), #Set Encryption Key - b'ATAC\r', #Apply Queued Changes - b'ATCN\r', #Exit Command Mode - ] - - ser.write(b'+++') #Enter Command Mode + b'ATCE1\r', # Enable Cordinator + b'ATAP1\r', # Enable API Mode + b'ATAO1\r', # Set Output Mode To Explicit + b'ATEE1\r', # Enable Security + b'ATEO2\r', # Enable Trust Center + f'ATKY{system_data["XBEE_KY"]}'.encode(), # Set Encryption Key + b'ATAC\r', # Apply Queued Changes + b'ATCN\r', # Exit Command Mode + ] + + ser.write(b'+++') # Enter Command Mode sleep(1) print(ser.readline()) for command in param_config: ser.write(command) - rx_data = ser.read_until(expected='\r').decode() #Reads buffer until carriage return. + rx_data = ser.read_until(expected='\r').decode() # Reads buffer until carriage return. rx_data = str(rx_data.rstrip()) log_xbee.info("Initial XBee Configuration TX: %s - RX: %s", command, rx_data) @@ -146,7 +147,7 @@ def listing(): if settings.Pi: if ser.in_waiting != 0: pub.sendMessage('xbee_rx') - sleep(.01) #Allows CPU time for other threads. + sleep(.01) # Allows CPU time for other threads. # def mqtt_xbee(firstName, lastName, cardNumber, action, result, timestamp, space): @@ -179,7 +180,7 @@ def xbee_info(): ser.write(b'ATAP\r') # Checks for API mode sleep(.1) - rx_data = 1 # TEMP PATCH + rx_data = 1 # TEMP PATCH try: rx_data = ser.read_until(expected='\r').decode() # Read buffer until carriage return rx_data = str(rx_data.rstrip()) @@ -190,7 +191,6 @@ def xbee_info(): if rx_data != 1: configure_xbee() - with open("system.json", "r+", encoding="UTF-8") as file: data = json.load(file) @@ -209,7 +209,7 @@ def xbee_info(): except TypeError as err: log_xbee.error("XBee ATOP command error: %s", err) - data.update( {"XBEE_OP":rx_data} ) + data.update({"XBEE_OP": rx_data}) file.seek(0) json.dump(data, file) file.truncate() diff --git a/openpod/pod.py b/openpod/pod.py index 612aa11..4c0ac02 100644 --- a/openpod/pod.py +++ b/openpod/pod.py @@ -16,15 +16,27 @@ import settings -from modules import rec_log, rec_mqtt, rec_xbee, rec_api -from modules.rec_log import exception_log, zip_send +from .modules import op_gpio, rec_log, rec_mqtt, rec_xbee, rec_api, rec_lan +from .modules.rec_log import exception_log, zip_send -if settings.Pi: - try: - from modules import rec_gpio - rec_gpio.state(settings.LED_IO, 1, 0) - except ModuleNotFoundError as err: - exception_log.error("%s", err) +# --------------------------- Visualization Threads --------------------------- # +threading.Thread(target=op_gpio.led_stat_thread).start() +threading.Thread(target=op_gpio.led_io_thread).start() + +op_gpio.initializing() + + +# ---------------------------------------------------------------------------- # +# Check Network Connection # +# ---------------------------------------------------------------------------- # +try: + rec_lan.monitor_network() # Monitors the network connection while the program is running. + +except RuntimeError as err: + exception_log.error("FATAL - Start Network Monitoring - Error: %s", err) + +finally: + exception_log.debug("Launcher - Exiting Check Network Connection") # Not sure if the next section is required. @@ -92,10 +104,6 @@ def incoming_xbee_data(): MQTTlisten = threading.Thread(target=rec_mqtt.mqtt_rx) MQTTlisten.start() -if settings.Pi: - led_io_thread = threading.Thread(target=rec_gpio.led_io_thread) - led_io_thread.start() - # Only pull info if hub has already been paired with a facility. try: @@ -105,12 +113,11 @@ def incoming_xbee_data(): rec_api.update_time_zone() else: rec_log.publog("info", "Facility connection not found, no data to pull.") - if settings.Pi: - rec_gpio.state(settings.LED_IO, 1, 1) # Slow blink ready to pair to a facility. + op_gpio.unregistered() # Slow blink ready to pair to a facility. zip_send() # Send latest log files on boot. except Exception as err: # pylint: disable=W0703 - rec_log.publog("error", f"Error occured when pulling data: {err}") + rec_log.publog("error", f"Error occurred when pulling data: {err}") rec_log.publog("info", "Recursion system has successfully initiated.") @@ -120,8 +127,8 @@ def process_xbee_data(): ''' Triggered when there is available XBee data. ''' - xbee_frame_info = rec_xbee.receive() # Reads in recived XBee data. - rec_gpio.state(settings.LED_IO, 1) + xbee_frame_info = rec_xbee.receive() # Reads in received XBee data. + op_gpio.ready() # Pod is ready. if xbee_frame_info[2] == 0: rec_xbee.transmit(xbee_frame_info[0], "30") elif xbee_frame_info[2] == 1: @@ -138,7 +145,7 @@ def process_xbee_data(): # https://stackoverflow.com/questions/17553543/pyserial-non-blocking-read-loop while True: if config.XBEE_FLAG: - rec_gpio.state(settings.LED_IO, .125, .125) # Indicate incoming XBee data. + op_gpio.incoming_data() # Pod is receiving XBee data. DataProcessing = threading.Thread(target=process_xbee_data) DataProcessing.start() sleep(.05) From b0f8de0d241139cc58adf81374cfcbfc7cb283a5 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Fri, 16 Dec 2022 21:42:30 -0500 Subject: [PATCH 075/164] lint installer --- installer.sh | 11 ++++++----- openpod/launcher.py | 3 +-- requirements.txt | 1 + 3 files changed, 8 insertions(+), 7 deletions(-) diff --git a/installer.sh b/installer.sh index 45b6ca4..fa3d403 100644 --- a/installer.sh +++ b/installer.sh @@ -19,6 +19,7 @@ Help() echo "OpenPod instalation script" echo echo "h Display this help" + echo "b Set custom branch for OpenPod" echo "u Set custom URL for OpenPod" echo "d Enable debug mode" } @@ -189,19 +190,19 @@ xbee_uuid=$(cat /proc/sys/kernel/random/uuid) openpod_version=$(git rev-parse HEAD) echo '{ "debug": '$DEBUG', - "serial": "'$serial_uuid'", + "serial": "'"$serial_uuid"'", "timezone": "UTC", - "url": "'$URL'", - "api_url": "'$API_URL'", + "url": "'"$URL"'", + "api_url": "'"$API_URL"'", "XBEE": { - "KY": "'$xbee_uuid'", + "KY": "'"$xbee_uuid"'", "OP": "0" }, "GPIO": { "LED_IO": 23, "LED_STAT": 17 }, - "version": "'$openpod_version'" + "version": "'"$openpod_version"'" }' > /opt/OpenPod/system.json # --------------------------- Setup OpenPod Service -------------------------- # diff --git a/openpod/launcher.py b/openpod/launcher.py index fe3a62e..64f90dd 100644 --- a/openpod/launcher.py +++ b/openpod/launcher.py @@ -10,6 +10,7 @@ - Pulls configuration files from server. - Launches main program ''' +# pylint: skip-file import sys import json @@ -17,8 +18,6 @@ import subprocess -import config - # ---------------------------------------------------------------------------- # # Check Requirements # # ---------------------------------------------------------------------------- # diff --git a/requirements.txt b/requirements.txt index 6b0a80d..ad50595 100644 --- a/requirements.txt +++ b/requirements.txt @@ -10,3 +10,4 @@ pytz==2022.6 requests==2.28.1 RPi.GPIO==0.7.1 simplejson==3.18.0 +urllib3==1.26.13 From 03f2a53007ffa4835bf687a0a8dc1958b20ef030 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Fri, 16 Dec 2022 21:47:43 -0500 Subject: [PATCH 076/164] Update installer.sh --- installer.sh | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/installer.sh b/installer.sh index fa3d403..b4f148f 100644 --- a/installer.sh +++ b/installer.sh @@ -34,13 +34,13 @@ DEBUG=flase # -d # ---------------------------------------------------------------------------- # # Options # # ---------------------------------------------------------------------------- # -while getopts ":hud" flags; do +while getopts ":hbdu" flags; do case "${flags}" in h) # display Help Help exit;; b) # Custom branch - branch="${OPTARG}";; + BRANCH="${OPTARG}";; d) # Enable debug mode DEBUG=true ;; u) # Custom URL endpoint @@ -51,18 +51,18 @@ while getopts ":hud" flags; do done if [ $DEBUG ]; then - branch='dev-release' + BRANCH='dev-release' URL='dev.recursion.space' API_URL='dev.api.recursion.space' elif [ ! $DEBUG ]; then - branch='release' + BRANCH='release' URL='recursion.space' API_URL='api.recursion.space' fi echo "Installing OpenPod with the following options:" echo "Debug: $DEBUG" -echo "Branch: $branch" +echo "Branch: $BRANCH" echo "URL: $URL" echo "API_URL: $API_URL" @@ -163,7 +163,7 @@ sudo rm -rf /opt/OpenPod set -e # Exit when any command fails. sudo mkdir -p /opt cd /opt -sudo git clone --single-branch --branch $branch https://github.com/RecursionSpace/OpenPod.git +sudo git clone --single-branch --branch $BRANCH https://github.com/RecursionSpace/OpenPod.git cd OpenPod # ----------------------------- Setup Enviroment ----------------------------- # From 90459feb6a3ddbc1f3667ae6a212ef87dacc5354 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Fri, 16 Dec 2022 21:50:12 -0500 Subject: [PATCH 077/164] Update CI_Tests.yml --- .github/workflows/CI_Tests.yml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.github/workflows/CI_Tests.yml b/.github/workflows/CI_Tests.yml index d61e582..f0d0833 100644 --- a/.github/workflows/CI_Tests.yml +++ b/.github/workflows/CI_Tests.yml @@ -32,4 +32,6 @@ jobs: flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics - name: Test with pytest run: | + sudo chmod +x installer.sh + sudo ./installer.sh pytest From e293f1a53faaafee876479fbae8fad50296bb806 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Fri, 16 Dec 2022 22:04:17 -0500 Subject: [PATCH 078/164] improved installer --- installer.sh | 2 +- openpod/modules/op_gpio.py | 2 +- requirements.txt | 1 - 3 files changed, 2 insertions(+), 3 deletions(-) diff --git a/installer.sh b/installer.sh index b4f148f..a8b0141 100644 --- a/installer.sh +++ b/installer.sh @@ -169,7 +169,7 @@ cd OpenPod # ----------------------------- Setup Enviroment ----------------------------- # sudo python3.11 -m venv /opt/OpenPod/env source /opt/OpenPod/env/bin/activate -pip install --no-input -U -r /opt/OpenPod/requirements.txt --no-cache-dir --no-dependencies +pip install --no-input -U -r /opt/OpenPod/requirements.txt # ---------------------------- Create Directories ---------------------------- # sudo mkdir -p /opt/OpenPod/logs diff --git a/openpod/modules/op_gpio.py b/openpod/modules/op_gpio.py index a38dc1d..ada7126 100644 --- a/openpod/modules/op_gpio.py +++ b/openpod/modules/op_gpio.py @@ -16,7 +16,7 @@ print("NOTICE | This device is probably not a Raspberry Pi.") GPIO_AVAILABLE = False -with open('/opt/OpenPod/system.json', 'r', encoding="utf-8") as system_file: +with open('/opt/OpenPod/system.json', 'r', encoding="UTF-8") as system_file: system = json.load(system_file) LED_IO = system['GPIO']['LED_IO'] LED_STAT = system['GPIO']['LED_STAT'] diff --git a/requirements.txt b/requirements.txt index ad50595..6b0a80d 100644 --- a/requirements.txt +++ b/requirements.txt @@ -10,4 +10,3 @@ pytz==2022.6 requests==2.28.1 RPi.GPIO==0.7.1 simplejson==3.18.0 -urllib3==1.26.13 From bd6761a9e87107a88e24c8e71c793ff15930d7d2 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Fri, 16 Dec 2022 22:30:09 -0500 Subject: [PATCH 079/164] switch dev branch --- .github/workflows/CI_Tests.yml | 2 -- installer.sh | 3 ++- 2 files changed, 2 insertions(+), 3 deletions(-) diff --git a/.github/workflows/CI_Tests.yml b/.github/workflows/CI_Tests.yml index f0d0833..d61e582 100644 --- a/.github/workflows/CI_Tests.yml +++ b/.github/workflows/CI_Tests.yml @@ -32,6 +32,4 @@ jobs: flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics - name: Test with pytest run: | - sudo chmod +x installer.sh - sudo ./installer.sh pytest diff --git a/installer.sh b/installer.sh index a8b0141..bf64605 100644 --- a/installer.sh +++ b/installer.sh @@ -24,6 +24,7 @@ Help() echo "d Enable debug mode" } +# TO FIX: ARGS VS OPTS # ---------------------------------------------------------------------------- # # Defaults # @@ -51,7 +52,7 @@ while getopts ":hbdu" flags; do done if [ $DEBUG ]; then - BRANCH='dev-release' + BRANCH='master' URL='dev.recursion.space' API_URL='dev.api.recursion.space' elif [ ! $DEBUG ]; then From 7097376e6454452bbf2e87f09e3a8b5f685e93b2 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Fri, 16 Dec 2022 22:42:09 -0500 Subject: [PATCH 080/164] changed imports --- openpod/pod.py | 4 +-- tests/modules/test_api.py | 74 +++++++++++++++++++++------------------ 2 files changed, 41 insertions(+), 37 deletions(-) diff --git a/openpod/pod.py b/openpod/pod.py index 4c0ac02..e69366b 100644 --- a/openpod/pod.py +++ b/openpod/pod.py @@ -16,8 +16,8 @@ import settings -from .modules import op_gpio, rec_log, rec_mqtt, rec_xbee, rec_api, rec_lan -from .modules.rec_log import exception_log, zip_send +from modules import op_gpio, rec_log, rec_mqtt, rec_xbee, rec_api, rec_lan +from modules.rec_log import exception_log, zip_send # --------------------------- Visualization Threads --------------------------- # threading.Thread(target=op_gpio.led_stat_thread).start() diff --git a/tests/modules/test_api.py b/tests/modules/test_api.py index c02bd14..c3e62b7 100644 --- a/tests/modules/test_api.py +++ b/tests/modules/test_api.py @@ -63,41 +63,45 @@ def test_pull_data_dump(self): }""") testreturn = [{ - "cardNumber": "3132323637373936", - "access_group": 7, - "phone_number": "2403426671", - "address": "123 America Ln", - "city": "USA City", - "state": "PA", - "zip_code": " 1234567", - "username": "GenericMember2", - "first_name": "Gener", - "last_name": "Mem", - "email": "member@email.com", - "restricted_nodes": [] - }, { - "cardNumber": "33", - "access_group": 5, - "phone_number": "2403426671", - "address": "", - "city": "", - "state": "", - "zip_code": "", - "username": "GenericMember3", - "first_name": "Generic", - "last_name": "Member3", - "email": "generic@email.com", - "restricted_nodes": ["123", "shdfhethetbe"] - }] - - with patch('modules.rec_api.open') as mock_open: - mock_open.side_effect = [system_json, testdata2, testdata3, testdata4, testdata5] - - with patch("modules.rec_api.requests.get") as mocked_requests: - mocked_requests.return_value.json.return_value = testreturn - self.assertTrue(rec_api.pull_data_dump()) - mock_open.assert_called() - mocked_requests.assert_called() + "cardNumber": "3132323637373936", + "access_group": 7, + "phone_number": "2403426671", + "address": "123 America Ln", + "city": "USA City", + "state": "PA", + "zip_code": " 1234567", + "username": "GenericMember2", + "first_name": "Gener", + "last_name": "Mem", + "email": "member@email.com", + "restricted_nodes": [] + }, { + "cardNumber": "33", + "access_group": 5, + "phone_number": "2403426671", + "address": "", + "city": "", + "state": "", + "zip_code": "", + "username": "GenericMember3", + "first_name": "Generic", + "last_name": "Member3", + "email": "generic@email.com", + "restricted_nodes": ["123", "shdfhethetbe"] + }] + + with patch('modules.op_gpio.open') as mock_system: + mock_system.side_effect = [system_json] + + with patch('modules.rec_api.open') as mock_open: + mock_open.side_effect = [system_json, testdata2, testdata3, testdata4, testdata5] + + with patch("modules.rec_api.requests.get") as mocked_requests: + mocked_requests.return_value.json.return_value = testreturn + self.assertTrue(rec_api.pull_data_dump()) + mock_open.assert_called() + mocked_requests.assert_called() + if __name__ == '__main__': unittest.main() From f15c7fedc94210f8be631680a3c94b88794b5737 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Fri, 16 Dec 2022 22:55:37 -0500 Subject: [PATCH 081/164] change public IP collection method --- openpod/modules/rec_lan.py | 7 ++----- openpod/pod.py | 9 +++------ tests/modules/test_api.py | 2 +- 3 files changed, 6 insertions(+), 12 deletions(-) diff --git a/openpod/modules/rec_lan.py b/openpod/modules/rec_lan.py index 99c0a0f..4c5e916 100644 --- a/openpod/modules/rec_lan.py +++ b/openpod/modules/rec_lan.py @@ -153,16 +153,13 @@ def get_ip(): ''' if internet_on() is True: try: - public_ip = requests.get('https://ip.42.pl/raw', verify=False, timeout=10).text + public_ip = requests.get('https://api.ipify.org', verify=False, timeout=10) + public_ip = public_ip.content.decode('utf8') except requests.exceptions.RequestException as err: public_ip = f'Failed to get public IP with error: {err}' network_log.error(public_ip) - except requests.ConnectionResetError as err: - public_ip = f'Failed to get public IP with error: {err}' - network_log.error(public_ip) - else: public_ip = "WLAN not available." diff --git a/openpod/pod.py b/openpod/pod.py index e69366b..5776034 100644 --- a/openpod/pod.py +++ b/openpod/pod.py @@ -33,18 +33,15 @@ rec_lan.monitor_network() # Monitors the network connection while the program is running. except RuntimeError as err: - exception_log.error("FATAL - Start Network Monitoring - Error: %s", err) - -finally: - exception_log.debug("Launcher - Exiting Check Network Connection") + exception_log.error(f"FATAL - Start Network Monitoring - Error: {err}") # Not sure if the next section is required. sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), os.path.pardir))) -with open('/opt/RecursionHub/system.json', 'r+', encoding="utf-8") as system_file: +with open('/opt/OpenPod/system.json', 'r+', encoding="utf-8") as system_file: systemConfig = json.load(system_file) -Version = systemConfig['CurrentVersion'] +Version = systemConfig['version'] # Inserts path to refrence then starts importing modules. sys.path.insert(0, f"./{Version}") diff --git a/tests/modules/test_api.py b/tests/modules/test_api.py index c3e62b7..1b790e8 100644 --- a/tests/modules/test_api.py +++ b/tests/modules/test_api.py @@ -90,7 +90,7 @@ def test_pull_data_dump(self): "restricted_nodes": ["123", "shdfhethetbe"] }] - with patch('modules.op_gpio.open') as mock_system: + with patch('modules.rec_api.op_gpio.open') as mock_system: mock_system.side_effect = [system_json] with patch('modules.rec_api.open') as mock_open: From fb30d701fb67e3477c13cb7b3f40e2dfbc4778dd Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Fri, 16 Dec 2022 22:58:52 -0500 Subject: [PATCH 082/164] Update op_gpio.py --- openpod/modules/op_gpio.py | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/openpod/modules/op_gpio.py b/openpod/modules/op_gpio.py index ada7126..73a092e 100644 --- a/openpod/modules/op_gpio.py +++ b/openpod/modules/op_gpio.py @@ -16,10 +16,14 @@ print("NOTICE | This device is probably not a Raspberry Pi.") GPIO_AVAILABLE = False -with open('/opt/OpenPod/system.json', 'r', encoding="UTF-8") as system_file: - system = json.load(system_file) -LED_IO = system['GPIO']['LED_IO'] -LED_STAT = system['GPIO']['LED_STAT'] +try: + with open('/opt/OpenPod/system.json', 'r', encoding="UTF-8") as system_file: + system = json.load(system_file) + LED_IO = system['GPIO']['LED_IO'] + LED_STAT = system['GPIO']['LED_STAT'] +except FileNotFoundError: + LED_IO = 0 + LED_STAT = 0 if GPIO_AVAILABLE: From 40008987712acb6c4c6c17133fea5d887f706aec Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Fri, 16 Dec 2022 23:05:27 -0500 Subject: [PATCH 083/164] removed settings pi check --- openpod/modules/rec_xbee.py | 9 +++++---- tests/modules/test_lan.py | 3 ++- 2 files changed, 7 insertions(+), 5 deletions(-) diff --git a/openpod/modules/rec_xbee.py b/openpod/modules/rec_xbee.py index e7ad827..351b3f7 100644 --- a/openpod/modules/rec_xbee.py +++ b/openpod/modules/rec_xbee.py @@ -142,11 +142,12 @@ def configure_xbee(): def listing(): - '''Function to handling incoming serial info''' + ''' + Function to handling incoming serial info + ''' while True: - if settings.Pi: - if ser.in_waiting != 0: - pub.sendMessage('xbee_rx') + if ser.in_waiting != 0: + pub.sendMessage('xbee_rx') sleep(.01) # Allows CPU time for other threads. diff --git a/tests/modules/test_lan.py b/tests/modules/test_lan.py index d7edc33..7d7454c 100644 --- a/tests/modules/test_lan.py +++ b/tests/modules/test_lan.py @@ -92,7 +92,7 @@ def test_get_ip(self): mocked_internet_on.return_value = True with patch('modules.rec_lan.requests.get') as mocked_requests: - mocked_requests.return_value.text = "0.0.0.0" + mocked_requests.return_value.content = "0.0.0.0" public_ip, local_ip = rec_lan.get_ip() self.assertEqual((rec_lan.get_ip())[0], '0.0.0.0') @@ -102,5 +102,6 @@ def test_get_ip(self): mocked_internet_on.return_value = False self.assertNotEqual((rec_lan.get_ip())[0], '0.0.0.0') + if __name__ == '__main__': unittest.main() From 4510eed670280e946efdd7eed15958ee4138382c Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Fri, 16 Dec 2022 23:08:25 -0500 Subject: [PATCH 084/164] fix tests --- openpod/modules/op_gpio.py | 1 - openpod/modules/rec_lan.py | 3 +-- openpod/modules/rec_xbee.py | 4 +--- tests/modules/test_lan.py | 2 +- 4 files changed, 3 insertions(+), 7 deletions(-) diff --git a/openpod/modules/op_gpio.py b/openpod/modules/op_gpio.py index 73a092e..6f2bcda 100644 --- a/openpod/modules/op_gpio.py +++ b/openpod/modules/op_gpio.py @@ -13,7 +13,6 @@ from RPi import GPIO GPIO_AVAILABLE = True except (RuntimeError, ModuleNotFoundError): - print("NOTICE | This device is probably not a Raspberry Pi.") GPIO_AVAILABLE = False try: diff --git a/openpod/modules/rec_lan.py b/openpod/modules/rec_lan.py index 4c5e916..51180a9 100644 --- a/openpod/modules/rec_lan.py +++ b/openpod/modules/rec_lan.py @@ -153,8 +153,7 @@ def get_ip(): ''' if internet_on() is True: try: - public_ip = requests.get('https://api.ipify.org', verify=False, timeout=10) - public_ip = public_ip.content.decode('utf8') + public_ip = requests.get('https://api.ipify.org', verify=False, timeout=10).text except requests.exceptions.RequestException as err: public_ip = f'Failed to get public IP with error: {err}' diff --git a/openpod/modules/rec_xbee.py b/openpod/modules/rec_xbee.py index 351b3f7..3aa9af7 100644 --- a/openpod/modules/rec_xbee.py +++ b/openpod/modules/rec_xbee.py @@ -6,13 +6,11 @@ import json import binascii import time -# import threading + from time import sleep import serial from pubsub import pub -import settings - from modules import rec_lookup, rec_api from modules.rec_log import log_xbee diff --git a/tests/modules/test_lan.py b/tests/modules/test_lan.py index 7d7454c..495ae19 100644 --- a/tests/modules/test_lan.py +++ b/tests/modules/test_lan.py @@ -92,7 +92,7 @@ def test_get_ip(self): mocked_internet_on.return_value = True with patch('modules.rec_lan.requests.get') as mocked_requests: - mocked_requests.return_value.content = "0.0.0.0" + mocked_requests.return_value.text = "0.0.0.0" public_ip, local_ip = rec_lan.get_ip() self.assertEqual((rec_lan.get_ip())[0], '0.0.0.0') From 6392f112648331c9aee7b217a210c3df0092b123 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Fri, 16 Dec 2022 23:35:14 -0500 Subject: [PATCH 085/164] switched system methods --- openpod/modules/op_config.py | 18 ++++++++++++++++++ openpod/modules/op_gpio.py | 13 +++++-------- openpod/pod.py | 14 ++++++-------- 3 files changed, 29 insertions(+), 16 deletions(-) create mode 100644 openpod/modules/op_config.py diff --git a/openpod/modules/op_config.py b/openpod/modules/op_config.py new file mode 100644 index 0000000..be803f2 --- /dev/null +++ b/openpod/modules/op_config.py @@ -0,0 +1,18 @@ +''' Configuration loader for OpenPod ''' + +import json + +SYSTEM_FILE = '/opt/OpenPod/system.json' + + +def get(key, default=None): + ''' + Get a value from the system configuration file. + ''' + try: + with open(SYSTEM_FILE, 'r', encoding="UTF-8") as system_file: + system = json.load(system_file) + except FileNotFoundError: + system = {} + + return system.get(key, default) diff --git a/openpod/modules/op_gpio.py b/openpod/modules/op_gpio.py index 6f2bcda..d6176ee 100644 --- a/openpod/modules/op_gpio.py +++ b/openpod/modules/op_gpio.py @@ -9,20 +9,17 @@ import config +from modules import op_config + try: from RPi import GPIO GPIO_AVAILABLE = True except (RuntimeError, ModuleNotFoundError): GPIO_AVAILABLE = False -try: - with open('/opt/OpenPod/system.json', 'r', encoding="UTF-8") as system_file: - system = json.load(system_file) - LED_IO = system['GPIO']['LED_IO'] - LED_STAT = system['GPIO']['LED_STAT'] -except FileNotFoundError: - LED_IO = 0 - LED_STAT = 0 + +LED_IO = op_config.get('GPIO', {}).get('LED_IO', 23) +LED_STAT = op_config.get('GPIO', {}).get('LED_STAT', 17) if GPIO_AVAILABLE: diff --git a/openpod/pod.py b/openpod/pod.py index 5776034..a590985 100644 --- a/openpod/pod.py +++ b/openpod/pod.py @@ -16,7 +16,7 @@ import settings -from modules import op_gpio, rec_log, rec_mqtt, rec_xbee, rec_api, rec_lan +from modules import op_config, op_gpio, rec_log, rec_mqtt, rec_xbee, rec_api, rec_lan from modules.rec_log import exception_log, zip_send # --------------------------- Visualization Threads --------------------------- # @@ -39,11 +39,9 @@ # Not sure if the next section is required. sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), os.path.pardir))) -with open('/opt/OpenPod/system.json', 'r+', encoding="utf-8") as system_file: - systemConfig = json.load(system_file) -Version = systemConfig['version'] +Version = op_config.get('version', None) -# Inserts path to refrence then starts importing modules. +# Inserts path to reference then starts importing modules. sys.path.insert(0, f"./{Version}") sys.path.insert(1, f"./{Version}/modules") sys.path.append("..") @@ -66,13 +64,13 @@ def incoming_xbee_data(): # Register HUB with Recursion -if not systemConfig.get('HUBid', False): +if not op_config.get('pod_id', False): rec_api.register_hub() # ------------------------------- TEMP SOLUTION ------------------------------ # try: - URL = f'{settings.RecursionURL}/obtaintoken/{systemConfig["serial"]}/' + URL = f'{settings.RecursionURL}/obtaintoken/{op_config.get("serial")}/' r = requests.get(URL, timeout=10) if r.status_code == 201: @@ -104,7 +102,7 @@ def incoming_xbee_data(): # Only pull info if hub has already been paired with a facility. try: - if 'facility' in systemConfig: + if op_config.get('space', False): rec_log.publog("info", "Pulling any missing data.") rec_api.pull_data_dump() rec_api.update_time_zone() From de0c5c92a93c436df036c12b9e5fc6943261c020 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Fri, 16 Dec 2022 23:39:00 -0500 Subject: [PATCH 086/164] removed settings dep --- openpod/modules/op_gpio.py | 1 - openpod/pod.py | 10 ++-------- 2 files changed, 2 insertions(+), 9 deletions(-) diff --git a/openpod/modules/op_gpio.py b/openpod/modules/op_gpio.py index d6176ee..ffbc40d 100644 --- a/openpod/modules/op_gpio.py +++ b/openpod/modules/op_gpio.py @@ -4,7 +4,6 @@ Module manages raspberry pi gpio functionality. ''' -import json from time import sleep import config diff --git a/openpod/pod.py b/openpod/pod.py index a590985..2c957bf 100644 --- a/openpod/pod.py +++ b/openpod/pod.py @@ -14,8 +14,6 @@ import requests from pubsub import pub -import settings - from modules import op_config, op_gpio, rec_log, rec_mqtt, rec_xbee, rec_api, rec_lan from modules.rec_log import exception_log, zip_send @@ -70,7 +68,7 @@ def incoming_xbee_data(): # ------------------------------- TEMP SOLUTION ------------------------------ # try: - URL = f'{settings.RecursionURL}/obtaintoken/{op_config.get("serial")}/' + URL = f'https://{op_config.get("url")}/obtaintoken/{op_config.get("serial")}/' r = requests.get(URL, timeout=10) if r.status_code == 201: @@ -85,12 +83,8 @@ def incoming_xbee_data(): print(err) # ------------------------------- TEMP SOLUTION ------------------------------ # -if settings.DEBUG: +if op_config.get('debug', False): rec_log.publog("debug", "*** DEBUG Enabled ***") -if settings.Pi: - rec_log.publog("debug", "*** PI TRUE ***") -if not settings.Pi: - rec_log.publog("debug", "*** PI FALSE ***") rec_log.publog("info", f"Version: {Version}") From 66b358f8d03183f750a656327959521719593b33 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Sat, 17 Dec 2022 00:01:42 -0500 Subject: [PATCH 087/164] Update rec_api.py --- openpod/modules/rec_api.py | 26 ++++++++++++-------------- 1 file changed, 12 insertions(+), 14 deletions(-) diff --git a/openpod/modules/rec_api.py b/openpod/modules/rec_api.py index 08695fd..d731a84 100644 --- a/openpod/modules/rec_api.py +++ b/openpod/modules/rec_api.py @@ -8,9 +8,7 @@ import threading import requests -import settings - -from modules import op_gpio +from modules import op_config, op_gpio from modules.rec_log import log_api, hash_data @@ -30,7 +28,7 @@ def pull_data_dump(): # ----------------------------- Pull Member Data ----------------------------- # with open("/opt/RecursionHub/data/dump.json", "w", encoding="utf-8") as file: - member_info = requests.get(f'{settings.RECURSION_API_URL}/v1/members', headers={ + member_info = requests.get(f'https://{op_config.get("api_url")}/v1/members', headers={ 'Authorization': f'Token {system_data["Token"]}' }, timeout=10) @@ -39,7 +37,7 @@ def pull_data_dump(): # --------------------------- Pull Operator(s) Data -------------------------- # with open("/opt/RecursionHub/data/owners.json", "w", encoding="utf-8") as file: - operators_info = requests.get(f'{settings.RECURSION_API_URL}/v1/operators', headers={ + operators_info = requests.get(f'https://{op_config.get("api_url")}/v1/operators', headers={ 'Authorization': f'Token {system_data["Token"]}' }, timeout=10) @@ -49,7 +47,7 @@ def pull_data_dump(): # -------------------------------- Nodes Data -------------------------------- # with open("/opt/RecursionHub/data/nodes.json", "w", encoding="utf-8") as file: nodes_info = requests.get( - f'{settings.RECURSION_API_URL}/v1/nodes', + f'https://{op_config.get("api_url")}/v1/nodes', headers={'Authorization': f'Token {system_data["Token"]}'}, timeout=10 ) @@ -60,7 +58,7 @@ def pull_data_dump(): # ----------------------------- Pull Permissions ----------------------------- # with open("/opt/RecursionHub/data/permissions.json", "w", encoding="utf-8") as file: permissions_info = requests.get( - f'{settings.RECURSION_API_URL}/v1/permissions', + f'https://{op_config.get("api_url")}/v1/permissions', headers={'Authorization': f'Token {system_data["Token"]}'}, timeout=10 ) @@ -82,7 +80,7 @@ def update_time_zone(): with open("system.json", "r+", encoding="utf-8") as file: system_data = json.load(file) spaces_info = requests.get( - f'{settings.RECURSION_API_URL}/v1/spaces', + f'https://{op_config.get("api_url")}/v1/spaces', headers={'Authorization': f'Token {system_data["Token"]}'}, timeout=10 ) @@ -107,7 +105,7 @@ def register_hub(): # Needs updated! with open('system.json', 'r', encoding="utf-8") as system_file: system_config = json.load(system_file) - url = settings.RecursionURL+'/hubs/' + url = f'https://{op_config.get("url")}/hubs/' payload_tuples = {'serial': f"{system_config['serial']}"} output = requests.post(url, payload_tuples, auth=('OwnerA', 'Password@1'), timeout=10) responce = output.json() @@ -133,7 +131,7 @@ def link_hub(): with open("system.json", "r+", encoding="utf-8") as file: system_data = json.load(file) - hubs_info = requests.get(f'{settings.RECURSION_API_URL}/v1/hubs', headers={ + hubs_info = requests.get(f'https://{op_config.get("api_url")}/v1/hubs', headers={ 'Authorization': f'Token {system_data["Token"]}' }, timeout=10) @@ -168,7 +166,7 @@ def pair_node(node_mac): # ------------------------------ API /v1/nodes/ ------------------------------ # requests.post( - f'{settings.RECURSION_API_URL}/v1/nodes', + f'https://{op_config.get("api_url")}/v1/nodes', data=post_content, headers={'Authorization': f'Token {system_config["Token"]}'}, timeout=10 ) @@ -189,7 +187,7 @@ def keepalive(): if 'facility' in system_data: requests.get( - f'''{settings.RecursionURL}/hub/keepalive/''' + f'''https://{op_config.get("url")}/hub/keepalive/''' f'''{system_data["serial"]}/''' f'''{system_data["CurrentVersion"]}/''' f'''{hash_data()["combined"]}/''', @@ -199,7 +197,7 @@ def keepalive(): else: requests.get( - f'''{settings.RecursionURL}/hub/keepalive/''' + f'''https://{op_config.get("url")}/hub/keepalive/''' f'''{system_data["serial"]}/''' f'''{system_data["CurrentVersion"]}/''', headers={'Authorization': f'Token {system_data["Token"]}'}, @@ -242,7 +240,7 @@ def access_log(card_number, action, result, node, facility): ] requests.post( - f'{settings.RecursionURL}/accesslog/', + f'https://{op_config.get("url")}/accesslog/', data=payload, headers={'Authorization': f'Token {system_config["Token"]}'}, timeout=10 From fea35302e165a294892b390c1df61aacfe3faf44 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Sat, 17 Dec 2022 00:13:10 -0500 Subject: [PATCH 088/164] fix: rec_api --- openpod/modules/op_config.py | 18 ++++++++++++++++++ openpod/modules/rec_api.py | 17 +++++------------ 2 files changed, 23 insertions(+), 12 deletions(-) diff --git a/openpod/modules/op_config.py b/openpod/modules/op_config.py index be803f2..25497ec 100644 --- a/openpod/modules/op_config.py +++ b/openpod/modules/op_config.py @@ -16,3 +16,21 @@ def get(key, default=None): system = {} return system.get(key, default) + + +def set_value(key, value): + ''' + Set a value in the system configuration file. + ''' + try: + with open(SYSTEM_FILE, 'r', encoding="UTF-8") as system_file: + system = json.load(system_file) + except FileNotFoundError: + system = {} + + system[key] = value + + with open(SYSTEM_FILE, 'w', encoding="UTF-8") as system_file: + system_file.seek(0) + json.dump(system, system_file) + system_file.truncate() diff --git a/openpod/modules/rec_api.py b/openpod/modules/rec_api.py index d731a84..954e350 100644 --- a/openpod/modules/rec_api.py +++ b/openpod/modules/rec_api.py @@ -102,22 +102,15 @@ def register_hub(): # Needs updated! ''' API to register the hub. ''' - with open('system.json', 'r', encoding="utf-8") as system_file: - system_config = json.load(system_file) - url = f'https://{op_config.get("url")}/hubs/' - payload_tuples = {'serial': f"{system_config['serial']}"} + payload_tuples = {'serial': f"{op_config.get('serial')}"} output = requests.post(url, payload_tuples, auth=('OwnerA', 'Password@1'), timeout=10) responce = output.json() - log_api.info("Hub registration responce: %s", responce) + log_api.info(f"Hub registration responce: {responce}") - with open("/opt/RecursionHub/system.json", "r+", encoding="utf-8") as file: - data = json.load(file) - data.update({"HUBid": responce["id"]}) - file.seek(0) - json.dump(data, file) - log_api.info("Hub registered and assigned HUBid: %s", responce["id"]) + op_config.set_value("pod_id", responce["id"]) + log_api.info(f'Pod registered and assigned pod_id: {responce["id"]}') # ---------------------------------------------------------------------------- # @@ -160,7 +153,7 @@ def pair_node(node_mac): post_content = [ ('mac', node_mac), - ('hub', system_config['HUBid']), + ('hub', system_config['pod_id']), ('facility', system_config['facility']) ] From 4e856272613b6728c32874f3c5a57eba239b3da6 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Sat, 17 Dec 2022 12:06:42 -0500 Subject: [PATCH 089/164] updated function name --- openpod/modules/rec_api.py | 6 +++--- openpod/pod.py | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/openpod/modules/rec_api.py b/openpod/modules/rec_api.py index 954e350..4bf9e3d 100644 --- a/openpod/modules/rec_api.py +++ b/openpod/modules/rec_api.py @@ -98,16 +98,16 @@ def update_time_zone(): # ---------------------------------------------------------------------------- # # Register Hub With Recursion # # ---------------------------------------------------------------------------- # -def register_hub(): # Needs updated! +def register_pod(): # Needs updated! ''' - API to register the hub. + API to register the pod. ''' url = f'https://{op_config.get("url")}/hubs/' payload_tuples = {'serial': f"{op_config.get('serial')}"} output = requests.post(url, payload_tuples, auth=('OwnerA', 'Password@1'), timeout=10) responce = output.json() - log_api.info(f"Hub registration responce: {responce}") + log_api.info(f"Pod registration responce: {responce}") op_config.set_value("pod_id", responce["id"]) log_api.info(f'Pod registered and assigned pod_id: {responce["id"]}') diff --git a/openpod/pod.py b/openpod/pod.py index 2c957bf..1c44e53 100644 --- a/openpod/pod.py +++ b/openpod/pod.py @@ -63,7 +63,7 @@ def incoming_xbee_data(): # Register HUB with Recursion if not op_config.get('pod_id', False): - rec_api.register_hub() + rec_api.register_pod() # ------------------------------- TEMP SOLUTION ------------------------------ # From 429520496ad9eade3bbf6ecfbe6237f2242dc5c1 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Sat, 17 Dec 2022 13:33:57 -0500 Subject: [PATCH 090/164] feat: using uuid --- installer.sh | 4 +++- openpod/modules/rec_api.py | 5 ++++- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/installer.sh b/installer.sh index bf64605..b575ade 100644 --- a/installer.sh +++ b/installer.sh @@ -187,11 +187,13 @@ sudo touch /opt/OpenPod/data/permissions.json # -------------------------------- system.json ------------------------------- # sudo touch /opt/OpenPod/system.json serial_uuid=$(cat /proc/sys/kernel/random/uuid) +serial=${serial_uuid//-} xbee_uuid=$(cat /proc/sys/kernel/random/uuid) openpod_version=$(git rev-parse HEAD) echo '{ + "uuid": "'"$serial_uuid"'", "debug": '$DEBUG', - "serial": "'"$serial_uuid"'", + "serial": "'"$serial"'", "timezone": "UTC", "url": "'"$URL"'", "api_url": "'"$API_URL"'", diff --git a/openpod/modules/rec_api.py b/openpod/modules/rec_api.py index 4bf9e3d..c255651 100644 --- a/openpod/modules/rec_api.py +++ b/openpod/modules/rec_api.py @@ -103,7 +103,10 @@ def register_pod(): # Needs updated! API to register the pod. ''' url = f'https://{op_config.get("url")}/hubs/' - payload_tuples = {'serial': f"{op_config.get('serial')}"} + payload_tuples = { + 'uuid': f"{op_config.get('uuid')}", + 'serial': f"{op_config.get('serial')}" + } output = requests.post(url, payload_tuples, auth=('OwnerA', 'Password@1'), timeout=10) responce = output.json() From 7e32fd8a021b303289fda9993818862bf58fe214 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Sat, 17 Dec 2022 13:47:17 -0500 Subject: [PATCH 091/164] switch to op_config --- openpod/modules/rec_api.py | 19 ++++++++----------- openpod/modules/rec_lan.py | 2 +- 2 files changed, 9 insertions(+), 12 deletions(-) diff --git a/openpod/modules/rec_api.py b/openpod/modules/rec_api.py index c255651..825a3e9 100644 --- a/openpod/modules/rec_api.py +++ b/openpod/modules/rec_api.py @@ -175,28 +175,25 @@ def pair_node(node_mac): # ---------------------------------------------------------------------------- # def keepalive(): ''' - Pings Recursion.Space as an indicator that the hub is wtill active. + Pings Recursion.Space as an indicator that the hub is still active. ''' try: - with open('system.json', 'r', encoding="utf-8") as system_file: - system_data = json.load(system_file) - - if 'facility' in system_data: + if op_config.get("space", False): requests.get( f'''https://{op_config.get("url")}/hub/keepalive/''' - f'''{system_data["serial"]}/''' - f'''{system_data["CurrentVersion"]}/''' + f'''{op_config.get("serial")}/''' + f'''{op_config.get("version")}/''' f'''{hash_data()["combined"]}/''', - headers={'Authorization': f'Token {system_data["Token"]}'}, + headers={'Authorization': f'Token {op_config.get("Token")}'}, timeout=10 ) else: requests.get( f'''https://{op_config.get("url")}/hub/keepalive/''' - f'''{system_data["serial"]}/''' - f'''{system_data["CurrentVersion"]}/''', - headers={'Authorization': f'Token {system_data["Token"]}'}, + f'''{op_config.get("serial")}/''' + f'''{op_config.get("version")}/''', + headers={'Authorization': f'Token {op_config.get("Token")}'}, timeout=10 ) diff --git a/openpod/modules/rec_lan.py b/openpod/modules/rec_lan.py index 51180a9..568b015 100644 --- a/openpod/modules/rec_lan.py +++ b/openpod/modules/rec_lan.py @@ -153,7 +153,7 @@ def get_ip(): ''' if internet_on() is True: try: - public_ip = requests.get('https://api.ipify.org', verify=False, timeout=10).text + public_ip = requests.get('https://api.ipify.org', timeout=10).text except requests.exceptions.RequestException as err: public_ip = f'Failed to get public IP with error: {err}' From a8143c1115e65bfb5b8fc12f9381a298f72deef8 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Sat, 17 Dec 2022 13:57:39 -0500 Subject: [PATCH 092/164] Update pod.py --- openpod/pod.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/openpod/pod.py b/openpod/pod.py index 1c44e53..38bf0c6 100644 --- a/openpod/pod.py +++ b/openpod/pod.py @@ -68,7 +68,7 @@ def incoming_xbee_data(): # ------------------------------- TEMP SOLUTION ------------------------------ # try: - URL = f'https://{op_config.get("url")}/obtaintoken/{op_config.get("serial")}/' + URL = f'https://{op_config.get("url")}/pod/obtaintoken/{op_config.get("serial")}/' r = requests.get(URL, timeout=10) if r.status_code == 201: From 8c07638c6de466b8dbab6e76b49d21e89205cc80 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Sat, 17 Dec 2022 15:18:46 -0500 Subject: [PATCH 093/164] update old methods --- openpod/modules/op_config.py | 2 +- openpod/modules/rec_api.py | 40 +++++++------------- openpod/modules/rec_log.py | 60 ++++++++++++++---------------- openpod/modules/rec_lookup.py | 70 ++++++++++++++++++----------------- openpod/modules/rec_mqtt.py | 14 +++---- openpod/pod.py | 15 +++----- 6 files changed, 90 insertions(+), 111 deletions(-) diff --git a/openpod/modules/op_config.py b/openpod/modules/op_config.py index 25497ec..dfb048e 100644 --- a/openpod/modules/op_config.py +++ b/openpod/modules/op_config.py @@ -32,5 +32,5 @@ def set_value(key, value): with open(SYSTEM_FILE, 'w', encoding="UTF-8") as system_file: system_file.seek(0) - json.dump(system, system_file) + json.dump(system, system_file, indent=4) system_file.truncate() diff --git a/openpod/modules/rec_api.py b/openpod/modules/rec_api.py index 825a3e9..6e2e9af 100644 --- a/openpod/modules/rec_api.py +++ b/openpod/modules/rec_api.py @@ -27,7 +27,7 @@ def pull_data_dump(): system_data = json.load(file) # ----------------------------- Pull Member Data ----------------------------- # - with open("/opt/RecursionHub/data/dump.json", "w", encoding="utf-8") as file: + with open("/opt/OpenPod/data/dump.json", "w", encoding="utf-8") as file: member_info = requests.get(f'https://{op_config.get("api_url")}/v1/members', headers={ 'Authorization': f'Token {system_data["Token"]}' }, timeout=10) @@ -36,7 +36,7 @@ def pull_data_dump(): json.dump(responce, file) # --------------------------- Pull Operator(s) Data -------------------------- # - with open("/opt/RecursionHub/data/owners.json", "w", encoding="utf-8") as file: + with open("/opt/OpenPod/data/owners.json", "w", encoding="utf-8") as file: operators_info = requests.get(f'https://{op_config.get("api_url")}/v1/operators', headers={ 'Authorization': f'Token {system_data["Token"]}' }, timeout=10) @@ -45,7 +45,7 @@ def pull_data_dump(): json.dump(responce, file) # -------------------------------- Nodes Data -------------------------------- # - with open("/opt/RecursionHub/data/nodes.json", "w", encoding="utf-8") as file: + with open("/opt/OpenPod/data/nodes.json", "w", encoding="utf-8") as file: nodes_info = requests.get( f'https://{op_config.get("api_url")}/v1/nodes', headers={'Authorization': f'Token {system_data["Token"]}'}, @@ -56,7 +56,7 @@ def pull_data_dump(): json.dump(responce, file) # ----------------------------- Pull Permissions ----------------------------- # - with open("/opt/RecursionHub/data/permissions.json", "w", encoding="utf-8") as file: + with open("/opt/OpenPod/data/permissions.json", "w", encoding="utf-8") as file: permissions_info = requests.get( f'https://{op_config.get("api_url")}/v1/permissions', headers={'Authorization': f'Token {system_data["Token"]}'}, @@ -124,21 +124,15 @@ def link_hub(): Associate the Pod with a space. ''' try: - with open("system.json", "r+", encoding="utf-8") as file: - system_data = json.load(file) - - hubs_info = requests.get(f'https://{op_config.get("api_url")}/v1/hubs', headers={ - 'Authorization': f'Token {system_data["Token"]}' - }, timeout=10) + hubs_info = requests.get(f'https://{op_config.get("api_url")}/v1/hubs', headers={ + 'Authorization': f'Token {op_config.get("api_token")}' + }, timeout=10) - responce = hubs_info.json() + responce = hubs_info.json() - system_data.update({"facility": responce[0]["facility"]}) - file.seek(0) - json.dump(system_data, file) - file.truncate() + op_config.set_value("space", responce[0]["facility"]) - op_gpio.ready() + op_gpio.ready() except OSError as err: log_api.error("link_hub - Unable to open file system.json - %s", err) @@ -151,20 +145,17 @@ def pair_node(node_mac): ''' Link a new node with the hub. ''' - with open('system.json', 'r+', encoding="utf-8") as system_file: - system_config = json.load(system_file) - post_content = [ ('mac', node_mac), - ('hub', system_config['pod_id']), - ('facility', system_config['facility']) + ('hub', op_config.get('pod_id')), + ('facility', op_config.get("space")) ] # ------------------------------ API /v1/nodes/ ------------------------------ # requests.post( f'https://{op_config.get("api_url")}/v1/nodes', data=post_content, - headers={'Authorization': f'Token {system_config["Token"]}'}, timeout=10 + headers={'Authorization': f'Token {op_config.get("api_token")}'}, timeout=10 ) pull_data_dump() @@ -221,9 +212,6 @@ def access_log(card_number, action, result, node, facility): Access request logging to Recursion.Space ''' try: - with open('system.json', 'r+', encoding="utf-8") as system_file: - system_config = json.load(system_file) - payload = [ ('cardNumber', card_number), ('action', action), @@ -235,7 +223,7 @@ def access_log(card_number, action, result, node, facility): requests.post( f'https://{op_config.get("url")}/accesslog/', data=payload, - headers={'Authorization': f'Token {system_config["Token"]}'}, + headers={'Authorization': f'Token {op_config.get("api_token")}'}, timeout=10 ) diff --git a/openpod/modules/rec_log.py b/openpod/modules/rec_log.py index d4869cf..18d5f29 100644 --- a/openpod/modules/rec_log.py +++ b/openpod/modules/rec_log.py @@ -14,7 +14,7 @@ import simplejson as json # https://stackoverflow.com/questions/21663800/python-make-a-list-generator-json-serializable -# from modules import rec_lan +from modules import op_config import settings @@ -44,7 +44,7 @@ exception_log.setLevel(logging.DEBUG) try: - exception_log_file = logging.FileHandler('/opt/RecursionHub/logs/exception.log', mode='a') + exception_log_file = logging.FileHandler('/opt/OpenPod/logs/exception.log', mode='a') except FileNotFoundError: exception_log_file = logging.FileHandler('tests/exception.log', mode='a') # CI Testing @@ -59,7 +59,7 @@ log_api.setLevel(logging.DEBUG) try: - log_api_file = logging.FileHandler('/opt/RecursionHub/logs/api.log', mode='a') + log_api_file = logging.FileHandler('/opt/OpenPod/logs/api.log', mode='a') except FileNotFoundError: log_api_file = logging.FileHandler('tests/api.log', mode='a') # CI Testing @@ -74,7 +74,7 @@ network_log.setLevel(logging.DEBUG) try: - network_log_file = logging.FileHandler('/opt/RecursionHub/logs/network.log', mode='a') + network_log_file = logging.FileHandler('/opt/OpenPod/logs/network.log', mode='a') except FileNotFoundError: network_log_file = logging.FileHandler('tests/network.log', mode='a') # CI Testing @@ -89,7 +89,7 @@ log_xbee.setLevel(logging.DEBUG) try: - xbee_log_file = logging.FileHandler('/opt/RecursionHub/logs/xbee.log', mode='a') + xbee_log_file = logging.FileHandler('/opt/OpenPod/logs/xbee.log', mode='a') except FileNotFoundError: xbee_log_file = logging.FileHandler('tests/xbee.log', mode='a') # CI Testing @@ -104,7 +104,7 @@ mqtt_log.setLevel(logging.DEBUG) try: - mqtt_log_file = logging.FileHandler('/opt/RecursionHub/logs/mqtt.log', mode='a') + mqtt_log_file = logging.FileHandler('/opt/OpenPod/logs/mqtt.log', mode='a') except FileNotFoundError: mqtt_log_file = logging.FileHandler('tests/mqtt.log', mode='a') # CI Testing @@ -118,7 +118,7 @@ logfile = logging.getLogger('standardlog') try: - fileHandler = logging.FileHandler('/opt/RecursionHub/logs/RecursionLog.log', mode='a') + fileHandler = logging.FileHandler('/opt/OpenPod/logs/RecursionLog.log', mode='a') except FileNotFoundError: fileHandler = logging.FileHandler('tests/RecursionLog.log', mode='a') # For CI @@ -134,7 +134,7 @@ transaction = logging.getLogger('transaction') try: - fileH = logging.FileHandler('/opt/RecursionHub/logs/TransactionLog.log', mode='a') + fileH = logging.FileHandler('/opt/OpenPod/logs/TransactionLog.log', mode='a') except FileNotFoundError: fileH = logging.FileHandler('tests/TransactionLog.log', mode='a') # For CI @@ -180,7 +180,7 @@ def snapshot(public, local): system_data = {} # public, local = rec_lan.get_ip() - with open('/opt/RecursionHub/system.json', 'r', encoding="UTF-8") as system_file: + with open('/opt/OpenPod/system.json', 'r', encoding="UTF-8") as system_file: system_json_file = json.load(system_file) system_data["system_json"] = system_json_file @@ -198,7 +198,7 @@ def snapshot(public, local): system_data['pip'] = freeze.freeze() - with open('/opt/RecursionHub/logs/System.Snapshot', 'w', encoding="UTF-8") as snapshot_file: + with open('/opt/OpenPod/logs/System.Snapshot', 'w', encoding="UTF-8") as snapshot_file: snapshot_file.seek(0) json.dump(system_data, snapshot_file, iterable_as_array=True) snapshot_file.truncate() @@ -214,17 +214,14 @@ def dump_diagnostics(): ''' Send the summary of setting to Recurson.Space ''' - with open('system.json', 'r+', encoding="UTF-8") as system_file: - system_config = json.load(system_file) - - with open('/opt/RecursionHub/logs/System.Snapshot', 'r', encoding="UTF-8") as snapshot_file: - if 'Token' in system_config: + with open('/opt/OpenPod/logs/System.Snapshot', 'r', encoding="UTF-8") as snapshot_file: + if op_config.get('api_token', False): payload = json.load(snapshot_file) try: requests.put(f'{settings.RecursionURL}/v1/diagnostics/', json={"snapshot": payload}, - headers={'Authorization': f"Token {system_config['Token']}"}, + headers={'Authorization': f"Token {op_config.get('api_token')}"}, timeout=10 ) @@ -236,22 +233,19 @@ def dump_diagnostics(): def zip_send(): ''' - Zip all log files togeather and send to Recursion.Space + Zip all log files together and send to Recursion.Space ''' try: - with open('system.json', 'r', encoding="UTF-8") as file: - system_data = json.load(file) - - zip_file = f'/opt/RecursionHub/logs/{system_data["serial"]}_logs.zip' + zip_file = f'/opt/OpenPod/logs/{op_config.get("serial")}_logs.zip' with ZipFile(zip_file, 'w') as zip_logs: - zip_logs.write('/opt/RecursionHub/logs/System.Snapshot', 'system_snapshot.txt') - zip_logs.write('/opt/RecursionHub/logs/network.log', 'network.log') - zip_logs.write('/opt/RecursionHub/logs/xbee.log', 'xbee.log') - zip_logs.write('/opt/RecursionHub/logs/mqtt.log', 'mqtt.log') - zip_logs.write('/opt/RecursionHub/logs/exception.log', 'exception.log') - zip_logs.write('/opt/RecursionHub/logs/RecursionLog.log', 'RecursionLog.log') - zip_logs.write('/opt/RecursionHub/logs/TransactionLog.log', 'TransactionLog.log') + zip_logs.write('/opt/OpenPod/logs/System.Snapshot', 'system_snapshot.txt') + zip_logs.write('/opt/OpenPod/logs/network.log', 'network.log') + zip_logs.write('/opt/OpenPod/logs/xbee.log', 'xbee.log') + zip_logs.write('/opt/OpenPod/logs/mqtt.log', 'mqtt.log') + zip_logs.write('/opt/OpenPod/logs/exception.log', 'exception.log') + zip_logs.write('/opt/OpenPod/logs/RecursionLog.log', 'RecursionLog.log') + zip_logs.write('/opt/OpenPod/logs/TransactionLog.log', 'TransactionLog.log') zip_logs.close() @@ -264,7 +258,7 @@ def zip_send(): requests.post( f'{settings.RecursionURL}/files/upload/external/hublogs/', files={"file": zip_file_logs}, - headers={'Authorization': f'Token {system_data["Token"]}'}, + headers={'Authorization': f'Token {op_config.get("api_token")}'}, timeout=10 ) @@ -286,22 +280,22 @@ def hash_data(): Produce a hash of the available data to compare with the data on Recursion.Space ''' try: - with open("/opt/RecursionHub/data/dump.json", "rb") as dump_file: + with open("/opt/OpenPod/data/dump.json", "rb") as dump_file: dump_hash = hashlib.md5( dump_file.read() ).hexdigest() - with open("/opt/RecursionHub/data/nodes.json", "rb") as nodes_file: + with open("/opt/OpenPod/data/nodes.json", "rb") as nodes_file: nodes_hash = hashlib.md5( nodes_file.read() ).hexdigest() - with open("/opt/RecursionHub/data/owners.json", "rb") as owners_file: + with open("/opt/OpenPod/data/owners.json", "rb") as owners_file: owners_hash = hashlib.md5( owners_file.read() ).hexdigest() - with open("/opt/RecursionHub/data/permissions.json", "rb") as perm_file: + with open("/opt/OpenPod/data/permissions.json", "rb") as perm_file: permissions_hash = hashlib.md5( perm_file.read() ).hexdigest() diff --git a/openpod/modules/rec_lookup.py b/openpod/modules/rec_lookup.py index 471be02..b062815 100644 --- a/openpod/modules/rec_lookup.py +++ b/openpod/modules/rec_lookup.py @@ -15,6 +15,8 @@ from modules.rec_log import exception_log # --------------------------- Count Matching Nodes --------------------------- # + + def count_matching_mac(rx_source): ''' Checks to see if the mac address being paired has been seen before by the system. @@ -27,7 +29,7 @@ def count_matching_mac(rx_source): match_counter = 0 try: - with open("/opt/RecursionHub/data/nodes.json", "r", encoding="utf-8") as node_file: + with open("/opt/OpenPod/data/nodes.json", "r", encoding="utf-8") as node_file: for data in json.load(node_file): if data['mac'] == rx_source: match_counter += 1 @@ -46,18 +48,20 @@ def count_matching_mac(rx_source): # ---------------------------------------------------------------------------- # # Convert MAC to Node ID # # ---------------------------------------------------------------------------- # + + def mac_to_id(mac_address): ''' Returns the id for the node that belongs to the mac address. ''' - with open('/opt/RecursionHub/data/nodes.json', 'r', encoding="utf-8") as nodes_file: + with open('/opt/OpenPod/data/nodes.json', 'r', encoding="utf-8") as nodes_file: nodes_dump = json.load(nodes_file) for node in nodes_dump: if node['mac'] == mac_address: return node['id'] - #Defaults if the nodes has not been added yet. + # Defaults if the nodes has not been added yet. return mac_address @@ -69,7 +73,7 @@ def is_owner(lookup_id): Checks if the ID matches to an owner. Returns True or False ''' - with open("/opt/RecursionHub/data/owners.json", "r", encoding="utf-8") as owner_file: + with open("/opt/OpenPod/data/owners.json", "r", encoding="utf-8") as owner_file: owner_file = json.load(owner_file) for owner in owner_file: @@ -86,7 +90,7 @@ def get_details(card_id): ''' Checks if user exsists, if yes, returns details. ''' - with open("/opt/RecursionHub/data/dump.json", "r", encoding="utf-8") as member_file: + with open("/opt/OpenPod/data/dump.json", "r", encoding="utf-8") as member_file: member_file = json.load(member_file) for user in member_file: @@ -108,7 +112,7 @@ def get_group_details(access_group_id): Returns the detils of group for a user. access_group_id is an integer ''' - with open("/opt/RecursionHub/data/permissions.json", "r", encoding="utf-8") as permissions_file: + with open("/opt/OpenPod/data/permissions.json", "r", encoding="utf-8") as permissions_file: permissions_file = json.load(permissions_file) for group in permissions_file: @@ -152,9 +156,9 @@ def access_request(requested_id, request_node): # pylint: disable=R0911 threading.Thread( target=rec_api.access_log, args=( - requested_id, "Requested access", - "Owner Allowed", reference_node, system_info['facility'] - ) + requested_id, "Requested access", + "Owner Allowed", reference_node, system_info['facility'] + ) ).start() return 1 @@ -171,9 +175,9 @@ def access_request(requested_id, request_node): # pylint: disable=R0911 threading.Thread( target=rec_api.access_log, args=( - requested_id, "Requested access", - "User not found", reference_node, system_info['facility'] - ) + requested_id, "Requested access", + "User not found", reference_node, system_info['facility'] + ) ).start() return 2 @@ -182,9 +186,9 @@ def access_request(requested_id, request_node): # pylint: disable=R0911 threading.Thread( target=rec_api.access_log, args=( - requested_id, "Requested access", - "Manually Restricted", reference_node, system_info['facility'] - ) + requested_id, "Requested access", + "Manually Restricted", reference_node, system_info['facility'] + ) ).start() return 2 @@ -194,9 +198,9 @@ def access_request(requested_id, request_node): # pylint: disable=R0911 threading.Thread( target=rec_api.access_log, args=( - requested_id, "Requested access", - "Group Not Found", reference_node, system_info['facility'] - ) + requested_id, "Requested access", + "Group Not Found", reference_node, system_info['facility'] + ) ).start() return 2 @@ -204,8 +208,8 @@ def access_request(requested_id, request_node): # pylint: disable=R0911 if reference_node not in group["allowedNodes"]: # Node Not Allowed threading.Thread( - target=rec_api.access_log, - args=( + target=rec_api.access_log, + args=( requested_id, "Requested access", "No node permission", reference_node, system_info['facility'] ) @@ -214,13 +218,13 @@ def access_request(requested_id, request_node): # pylint: disable=R0911 # 24/7 Override if group.get("twenty_four_seven", False): - print ("Group has 24/7 access.") + print("Group has 24/7 access.") threading.Thread( target=rec_api.access_log, args=( - requested_id, "Requested access", - "Allowed", reference_node, system_info['facility'] - ) + requested_id, "Requested access", + "Allowed", reference_node, system_info['facility'] + ) ).start() return 1 @@ -230,9 +234,9 @@ def access_request(requested_id, request_node): # pylint: disable=R0911 threading.Thread( target=rec_api.access_log, args=( - requested_id, "Requested access", - "Now Allowed Day", reference_node, system_info['facility'] - ) + requested_id, "Requested access", + "Now Allowed Day", reference_node, system_info['facility'] + ) ).start() return 2 @@ -245,9 +249,9 @@ def access_request(requested_id, request_node): # pylint: disable=R0911 threading.Thread( target=rec_api.access_log, args=( - requested_id, "Requested access", - "Allowed", reference_node, system_info['facility'] - ) + requested_id, "Requested access", + "Allowed", reference_node, system_info['facility'] + ) ).start() return 1 @@ -255,8 +259,8 @@ def access_request(requested_id, request_node): # pylint: disable=R0911 threading.Thread( target=rec_api.access_log, args=( - requested_id, "Requested access", - "Not Allowed Time", reference_node, system_info['facility'] - ) + requested_id, "Requested access", + "Not Allowed Time", reference_node, system_info['facility'] + ) ).start() return 2 diff --git a/openpod/modules/rec_mqtt.py b/openpod/modules/rec_mqtt.py index 37f46f2..574faa2 100644 --- a/openpod/modules/rec_mqtt.py +++ b/openpod/modules/rec_mqtt.py @@ -9,7 +9,7 @@ import subprocess import paho.mqtt.client as mqtt -from modules import rec_api, rec_xbee, rec_lookup +from modules import op_config, rec_api, rec_xbee, rec_lookup from modules.rec_log import mqtt_log, exception_log, zip_send import settings import updater @@ -91,10 +91,10 @@ def on_message(client, userdata, message): def mqtt_rx(): ''' - Esablish a connection to the Recursion.Space MQTT broker + Establish a connection to the Recursion.Space MQTT broker Define what happens when the following actions occur: - Connection to the MQTT broker is made. - - A message is recived. + - A message is received. ''' mqtt_log.info('Connecting to MQTT broker') @@ -115,11 +115,9 @@ def mqtt_start_update(): updater.update_hub() except RuntimeError as err: exception_log.error("Error while updating, atempting as subprocess. %s", err) - with open('system.json', 'r+', encoding="UTF-8") as file: - system_data = json.load(file) - update_location = f'/opt/RecursionHub/{system_data.CurrentVersion}/updater.py' - with subprocess.Popen(['sudo', 'python3', f'{update_location}']) as script: - print(script) + update_location = f'/opt/OpenPod/{op_config.get("version")}/updater.py' + with subprocess.Popen(['sudo', 'python3', f'{update_location}']) as script: + print(script) def mqtt_restart_system(): diff --git a/openpod/pod.py b/openpod/pod.py index 38bf0c6..7efdb9e 100644 --- a/openpod/pod.py +++ b/openpod/pod.py @@ -69,16 +69,11 @@ def incoming_xbee_data(): # ------------------------------- TEMP SOLUTION ------------------------------ # try: URL = f'https://{op_config.get("url")}/pod/obtaintoken/{op_config.get("serial")}/' - r = requests.get(URL, timeout=10) - - if r.status_code == 201: - print(f"Hub Toekn: {r.text}") - with open("/opt/RecursionHub/system.json", "r+", encoding="UTF-8") as file: - data = json.load(file) - data.update({"Token": f"{r.text}"}) - file.seek(0) - json.dump(data, file) - file.truncate() + response = requests.get(URL, timeout=10) + + if response.status_code == 201: + op_config.set_value("api_token", response.text) + except Exception as err: # pylint: disable=W0703 print(err) # ------------------------------- TEMP SOLUTION ------------------------------ # From 66c79ba34fedcb0181e658cb6eeb867806b22e88 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Sat, 17 Dec 2022 15:21:50 -0500 Subject: [PATCH 094/164] Update installer.sh --- installer.sh | 3 +++ 1 file changed, 3 insertions(+) diff --git a/installer.sh b/installer.sh index b575ade..254b8b1 100644 --- a/installer.sh +++ b/installer.sh @@ -208,6 +208,9 @@ echo '{ "version": "'"$openpod_version"'" }' > /opt/OpenPod/system.json +# --------------------------- Create Version Folder -------------------------- # +sudo cp -a /opt/OpenPod/openpod/. /opt/OpenPod/"$openpod_version"/ + # --------------------------- Setup OpenPod Service -------------------------- # cat < /etc/systemd/system/openpod.service [Unit] From 704577bc0eb4b0c89de4d6d3973506e1efe5e52d Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Sat, 17 Dec 2022 15:22:10 -0500 Subject: [PATCH 095/164] Update pod.py --- openpod/pod.py | 1 - 1 file changed, 1 deletion(-) diff --git a/openpod/pod.py b/openpod/pod.py index 7efdb9e..5668b08 100644 --- a/openpod/pod.py +++ b/openpod/pod.py @@ -7,7 +7,6 @@ import sys import os.path # Allows modules to access from directory above. -import json import threading from time import sleep import config From 3955a9127294c172bd4b148f161fd510f3f087d8 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Sat, 17 Dec 2022 15:43:44 -0500 Subject: [PATCH 096/164] replaced token usage --- installer.sh | 4 +++- openpod/modules/rec_api.py | 39 +++++++++++++++----------------------- openpod/modules/rec_log.py | 6 +++--- 3 files changed, 21 insertions(+), 28 deletions(-) diff --git a/installer.sh b/installer.sh index 254b8b1..f538203 100644 --- a/installer.sh +++ b/installer.sh @@ -124,11 +124,13 @@ else fi # -------------------------------- Python 3.11 ------------------------------- # -pytohn_version=$(python3 -c 'import sys; print(".".join(map(str, sys.version_info[0:2])))') +pytohn_version=$(python -c 'import sys; print(".".join(map(str, sys.version_info[0:2])))') if [ "$pytohn_version" != "3.11" ]; then sudo apt install software-properties-common -y yes '' | sudo add-apt-repository ppa:deadsnakes/ppa sudo apt-get install python3.11 -y + sudo update-alternatives --install /usr/bin/python python /usr/bin/python3.11 1 + sudo update-alternatives --install /usr/bin/python python /usr/bin/python3.10 1 else echo "Python 3.11 already installed" fi diff --git a/openpod/modules/rec_api.py b/openpod/modules/rec_api.py index 6e2e9af..24907c4 100644 --- a/openpod/modules/rec_api.py +++ b/openpod/modules/rec_api.py @@ -21,15 +21,12 @@ def pull_data_dump(): ''' - Request updated infromation from the server. + Request updated information from the server. ''' - with open('system.json', 'r+', encoding="utf-8") as file: - system_data = json.load(file) - # ----------------------------- Pull Member Data ----------------------------- # with open("/opt/OpenPod/data/dump.json", "w", encoding="utf-8") as file: member_info = requests.get(f'https://{op_config.get("api_url")}/v1/members', headers={ - 'Authorization': f'Token {system_data["Token"]}' + 'Authorization': f'Token {op_config.get("api_token")}' }, timeout=10) responce = member_info.json() @@ -38,7 +35,7 @@ def pull_data_dump(): # --------------------------- Pull Operator(s) Data -------------------------- # with open("/opt/OpenPod/data/owners.json", "w", encoding="utf-8") as file: operators_info = requests.get(f'https://{op_config.get("api_url")}/v1/operators', headers={ - 'Authorization': f'Token {system_data["Token"]}' + 'Authorization': f'Token {op_config.get("api_token")}' }, timeout=10) responce = operators_info.json() @@ -48,7 +45,7 @@ def pull_data_dump(): with open("/opt/OpenPod/data/nodes.json", "w", encoding="utf-8") as file: nodes_info = requests.get( f'https://{op_config.get("api_url")}/v1/nodes', - headers={'Authorization': f'Token {system_data["Token"]}'}, + headers={'Authorization': f'Token {op_config.get("api_token")}'}, timeout=10 ) @@ -59,7 +56,7 @@ def pull_data_dump(): with open("/opt/OpenPod/data/permissions.json", "w", encoding="utf-8") as file: permissions_info = requests.get( f'https://{op_config.get("api_url")}/v1/permissions', - headers={'Authorization': f'Token {system_data["Token"]}'}, + headers={'Authorization': f'Token {op_config.get("api_token")}'}, timeout=10 ) @@ -77,22 +74,16 @@ def update_time_zone(): ''' API call to set the HUB timezone with the user selected option. ''' - with open("system.json", "r+", encoding="utf-8") as file: - system_data = json.load(file) - spaces_info = requests.get( - f'https://{op_config.get("api_url")}/v1/spaces', - headers={'Authorization': f'Token {system_data["Token"]}'}, - timeout=10 - ) - - responce = spaces_info.json() + spaces_info = requests.get( + f'https://{op_config.get("api_url")}/v1/spaces', + headers={'Authorization': f'Token {op_config.get("api_token")}'}, + timeout=10 + ) - system_data.update({"timezone": responce[0]["timezone"]}) - file.seek(0) - json.dump(system_data, file) - file.truncate() + responce = spaces_info.json()[0] - log_api.info("Facility time zone set to: %s", responce[0]["timezone"]) + op_config.set_value("timezone", responce["timezone"]) + log_api.info("Facility time zone set to: %s", responce["timezone"]) # ---------------------------------------------------------------------------- # @@ -175,7 +166,7 @@ def keepalive(): f'''{op_config.get("serial")}/''' f'''{op_config.get("version")}/''' f'''{hash_data()["combined"]}/''', - headers={'Authorization': f'Token {op_config.get("Token")}'}, + headers={'Authorization': f'Token {op_config.get("api_token")}'}, timeout=10 ) @@ -184,7 +175,7 @@ def keepalive(): f'''https://{op_config.get("url")}/hub/keepalive/''' f'''{op_config.get("serial")}/''' f'''{op_config.get("version")}/''', - headers={'Authorization': f'Token {op_config.get("Token")}'}, + headers={'Authorization': f'Token {op_config.get("api_token")}'}, timeout=10 ) diff --git a/openpod/modules/rec_log.py b/openpod/modules/rec_log.py index 18d5f29..0b995c2 100644 --- a/openpod/modules/rec_log.py +++ b/openpod/modules/rec_log.py @@ -212,14 +212,14 @@ def snapshot(public, local): def dump_diagnostics(): ''' - Send the summary of setting to Recurson.Space + Send the summary of setting to Recursion.Space ''' with open('/opt/OpenPod/logs/System.Snapshot', 'r', encoding="UTF-8") as snapshot_file: if op_config.get('api_token', False): payload = json.load(snapshot_file) try: - requests.put(f'{settings.RecursionURL}/v1/diagnostics/', + requests.put(f'https://{op_config.get("url")}/v1/diagnostics/', json={"snapshot": payload}, headers={'Authorization': f"Token {op_config.get('api_token')}"}, timeout=10 @@ -256,7 +256,7 @@ def zip_send(): with open(zip_file, 'rb') as zip_file_logs: requests.post( - f'{settings.RecursionURL}/files/upload/external/hublogs/', + f'https://{op_config.get("url")}/files/upload/external/hublogs/', files={"file": zip_file_logs}, headers={'Authorization': f'Token {op_config.get("api_token")}'}, timeout=10 From b0943b7ea56f75a95d2c5c58518e0ec64ca64850 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Sat, 17 Dec 2022 16:11:29 -0500 Subject: [PATCH 097/164] Update rec_mqtt.py --- openpod/modules/rec_mqtt.py | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/openpod/modules/rec_mqtt.py b/openpod/modules/rec_mqtt.py index 574faa2..4da822d 100644 --- a/openpod/modules/rec_mqtt.py +++ b/openpod/modules/rec_mqtt.py @@ -11,7 +11,6 @@ from modules import op_config, rec_api, rec_xbee, rec_lookup from modules.rec_log import mqtt_log, exception_log, zip_send -import settings import updater # The callback for when the client receives a CONNACK response from the server. @@ -102,7 +101,7 @@ def mqtt_rx(): client.on_connect = on_connect client.on_message = on_message - client.connect(f"{settings.RECURSION_DOMAIN}", 1883, 60) + client.connect(f"{op_config.get('url')}", 1883, 60) client.loop_forever() From 30d4f6045f827a873e0181e1aa109bc2fa54d96f Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Sat, 17 Dec 2022 22:34:07 -0500 Subject: [PATCH 098/164] Corrected Service File --- installer.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/installer.sh b/installer.sh index f538203..8a69ebf 100644 --- a/installer.sh +++ b/installer.sh @@ -225,7 +225,7 @@ Type=simple User=root WorkingDirectory=/opt/OpenPod -ExecStart=/opt/OpenPod/env/bin/python3.11 /opt/OpenPod/OpenPod.py +ExecStart=/opt/OpenPod/env/bin/python3.11 /opt/OpenPod/openpod/pod.py Restart=always RestartSec=10 From 5bf73b8aabb6c879d3cc006f74e77fb94e47c02e Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Fri, 13 Jan 2023 04:07:39 +0000 Subject: [PATCH 099/164] chore(deps): bump actions/setup-python from 4.3.1 to 4.5.0 Bumps [actions/setup-python](https://github.com/actions/setup-python) from 4.3.1 to 4.5.0. - [Release notes](https://github.com/actions/setup-python/releases) - [Commits](https://github.com/actions/setup-python/compare/v4.3.1...v4.5.0) --- updated-dependencies: - dependency-name: actions/setup-python dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] --- .github/workflows/CI_Pylint.yml | 2 +- .github/workflows/CI_Tests.yml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/CI_Pylint.yml b/.github/workflows/CI_Pylint.yml index 2e0adb7..761c2d4 100644 --- a/.github/workflows/CI_Pylint.yml +++ b/.github/workflows/CI_Pylint.yml @@ -10,7 +10,7 @@ jobs: steps: - uses: actions/checkout@v3 - name: Set up Python 3.11 - uses: actions/setup-python@v4.3.1 + uses: actions/setup-python@v4.5.0 with: python-version: 3.11 diff --git a/.github/workflows/CI_Tests.yml b/.github/workflows/CI_Tests.yml index d61e582..ea44a2f 100644 --- a/.github/workflows/CI_Tests.yml +++ b/.github/workflows/CI_Tests.yml @@ -16,7 +16,7 @@ jobs: steps: - uses: actions/checkout@v3 - name: Set up Python 3.11 - uses: actions/setup-python@v4.3.1 + uses: actions/setup-python@v4.5.0 with: python-version: 3.11 - name: Install dependencies From 6a850f23b9c344b98c813e2ff8f8619a56ad5910 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Wed, 18 Jan 2023 14:38:59 -0500 Subject: [PATCH 100/164] feat: Added SSH Key Function --- installer.sh | 14 +++++++++++++ openpod/modules/op_ssh.py | 27 ++++++++++++++++++++++++ openpod/modules/rec_api.py | 42 +++++++++++++++++-------------------- openpod/modules/rec_mqtt.py | 16 +++++++------- 4 files changed, 69 insertions(+), 30 deletions(-) create mode 100644 openpod/modules/op_ssh.py diff --git a/installer.sh b/installer.sh index 8a69ebf..fd69227 100644 --- a/installer.sh +++ b/installer.sh @@ -73,6 +73,20 @@ if [ "$EUID" -ne 0 ] exit fi +# --------------------------------- SSH User --------------------------------- # +echo "Verifying SSH user 'openpod'" + +if [[ ! $(id -u pod > /dev/null 2>&1) ]]; then + echo "Creating user 'openpod'" + useradd -m -s /bin/bash openpod + usermod -aG sudo openpod + mkdir -p ~openpod/.ssh/ && touch ~openpod/.ssh/authorized_keys + echo "openpod ALL=(ALL) NOPASSWD:ALL" | sudo tee -a /etc/sudoers > /dev/null +else + echo "User 'pod' already exists, skipping..." + mkdir -p ~openpod/.ssh/ && touch ~openpod/.ssh/authorized_keys +fi + # ---------------------------- Update System Time ---------------------------- # sudo timedatectl set-timezone UTC diff --git a/openpod/modules/op_ssh.py b/openpod/modules/op_ssh.py new file mode 100644 index 0000000..337e819 --- /dev/null +++ b/openpod/modules/op_ssh.py @@ -0,0 +1,27 @@ +''' +openpod | modules | op_ssh.py +Configuration for SSH +''' + +import requests + +from modules import op_config +from modules.rec_log import log_api + + +def update_keys(): + ''' + Requests the keys from the server and updates the local keys. + Request URL: /v1/pod/ssh_pub_keys + ''' + keys = requests.get( + f'https://{op_config.get("api_url")}/v1/pod/ssh_pub_keys/{op_config.get("uuid")}', + headers={'Authorization': f'{op_config.get("api_token")}'}, + timeout=10 + ) + + with open('openpod/.ssh/authorized_keys', 'w', encoding="UTF-8") as key_file: + for key in keys.json(): + key_file.write(key['key']) + + log_api.info('SSH Keys Updated') diff --git a/openpod/modules/rec_api.py b/openpod/modules/rec_api.py index 24907c4..3698bd6 100644 --- a/openpod/modules/rec_api.py +++ b/openpod/modules/rec_api.py @@ -2,6 +2,7 @@ ''' Handles API calls with Recursion.Space +Performs all API calls to the server, functions should be used as a thread. ''' import json @@ -12,13 +13,9 @@ from modules.rec_log import log_api, hash_data -# Performs all API calls to the server, functions should be used as a thread. - # ---------------------------------------------------------------------------- # # Request Update For All Information # # ---------------------------------------------------------------------------- # - - def pull_data_dump(): ''' Request updated information from the server. @@ -29,8 +26,8 @@ def pull_data_dump(): 'Authorization': f'Token {op_config.get("api_token")}' }, timeout=10) - responce = member_info.json() - json.dump(responce, file) + response = member_info.json() + json.dump(response, file) # --------------------------- Pull Operator(s) Data -------------------------- # with open("/opt/OpenPod/data/owners.json", "w", encoding="utf-8") as file: @@ -38,8 +35,8 @@ def pull_data_dump(): 'Authorization': f'Token {op_config.get("api_token")}' }, timeout=10) - responce = operators_info.json() - json.dump(responce, file) + response = operators_info.json() + json.dump(response, file) # -------------------------------- Nodes Data -------------------------------- # with open("/opt/OpenPod/data/nodes.json", "w", encoding="utf-8") as file: @@ -49,8 +46,8 @@ def pull_data_dump(): timeout=10 ) - responce = nodes_info.json() - json.dump(responce, file) + response = nodes_info.json() + json.dump(response, file) # ----------------------------- Pull Permissions ----------------------------- # with open("/opt/OpenPod/data/permissions.json", "w", encoding="utf-8") as file: @@ -60,16 +57,15 @@ def pull_data_dump(): timeout=10 ) - responce = permissions_info.json() - json.dump(responce, file) + response = permissions_info.json() + json.dump(response, file) return True + # ---------------------------------------------------------------------------- # # Set or Update Timezone # # ---------------------------------------------------------------------------- # - - def update_time_zone(): ''' API call to set the HUB timezone with the user selected option. @@ -80,10 +76,10 @@ def update_time_zone(): timeout=10 ) - responce = spaces_info.json()[0] + response = spaces_info.json()[0] - op_config.set_value("timezone", responce["timezone"]) - log_api.info("Facility time zone set to: %s", responce["timezone"]) + op_config.set_value("timezone", response["timezone"]) + log_api.info("Facility time zone set to: %s", response["timezone"]) # ---------------------------------------------------------------------------- # @@ -99,12 +95,12 @@ def register_pod(): # Needs updated! 'serial': f"{op_config.get('serial')}" } output = requests.post(url, payload_tuples, auth=('OwnerA', 'Password@1'), timeout=10) - responce = output.json() + response = output.json() - log_api.info(f"Pod registration responce: {responce}") + log_api.info(f"Pod registration responce: {response}") - op_config.set_value("pod_id", responce["id"]) - log_api.info(f'Pod registered and assigned pod_id: {responce["id"]}') + op_config.set_value("pod_id", response["id"]) + log_api.info(f'Pod registered and assigned pod_id: {response["id"]}') # ---------------------------------------------------------------------------- # @@ -119,9 +115,9 @@ def link_hub(): 'Authorization': f'Token {op_config.get("api_token")}' }, timeout=10) - responce = hubs_info.json() + response = hubs_info.json() - op_config.set_value("space", responce[0]["facility"]) + op_config.set_value("space", response[0]["facility"]) op_gpio.ready() diff --git a/openpod/modules/rec_mqtt.py b/openpod/modules/rec_mqtt.py index 4da822d..afabf57 100644 --- a/openpod/modules/rec_mqtt.py +++ b/openpod/modules/rec_mqtt.py @@ -9,7 +9,7 @@ import subprocess import paho.mqtt.client as mqtt -from modules import op_config, rec_api, rec_xbee, rec_lookup +from modules import op_config, op_ssh, rec_api, rec_xbee, rec_lookup from modules.rec_log import mqtt_log, exception_log, zip_send import updater @@ -34,12 +34,13 @@ def on_connect(client, userdata, flags, return_code): def on_message(client, userdata, message): ''' Handles messages coming in via MQTT. - 170 - Pairing un-paired Hub - 186 - Pull New Data - 202 - Install System Update - 218 - Timezone Change - 234 - Reboot Hub (Soft restart) - 250 - Zip & Send Logs + 170 (AA) - Pairing un-paired Hub + 171 (AB) - Pull SSH Keys + 186 (BA) - Pull New Data + 202 (CA) - Install System Update + 218 (DA) - Timezone Change + 234 (EA) - Reboot Hub (Soft restart) + 250 (FA) - Zip & Send Logs Node Command - xxxxxxxxxxxxxxxx_## ''' @@ -50,6 +51,7 @@ def on_message(client, userdata, message): try: mqtt_actions = { 170: rec_api.link_hub, + 171: op_ssh.update_keys, 186: rec_api.pull_data_dump, 202: mqtt_start_update, 218: rec_api.update_time_zone, From 1849eff95b199fdba6efaeb73cde86040b4fd4e8 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Wed, 18 Jan 2023 16:41:38 -0500 Subject: [PATCH 101/164] fix: adding ssh key --- installer.sh | 11 ++++++----- openpod/modules/op_ssh.py | 2 +- openpod/modules/rec_lan.py | 2 +- openpod/pod.py | 3 ++- 4 files changed, 10 insertions(+), 8 deletions(-) diff --git a/installer.sh b/installer.sh index fd69227..1eab32d 100644 --- a/installer.sh +++ b/installer.sh @@ -62,10 +62,10 @@ elif [ ! $DEBUG ]; then fi echo "Installing OpenPod with the following options:" -echo "Debug: $DEBUG" -echo "Branch: $BRANCH" -echo "URL: $URL" -echo "API_URL: $API_URL" +echo "Debug | $DEBUG" +echo "Branch | $BRANCH" +echo "URL | $URL" +echo "API_URL | $API_URL" # -------------------------------- Verify Root ------------------------------- # if [ "$EUID" -ne 0 ] @@ -76,7 +76,7 @@ fi # --------------------------------- SSH User --------------------------------- # echo "Verifying SSH user 'openpod'" -if [[ ! $(id -u pod > /dev/null 2>&1) ]]; then +if [[ ! $(id -u openpod > /dev/null 2>&1) ]]; then echo "Creating user 'openpod'" useradd -m -s /bin/bash openpod usermod -aG sudo openpod @@ -175,6 +175,7 @@ fi # -------------------------- Clear Previous Install -------------------------- # sudo rm -rf /opt/OpenPod +sudo systemctl stop openpod.service # ------------------------------- Clone OpenPod ------------------------------ # set -e # Exit when any command fails. diff --git a/openpod/modules/op_ssh.py b/openpod/modules/op_ssh.py index 337e819..1b99330 100644 --- a/openpod/modules/op_ssh.py +++ b/openpod/modules/op_ssh.py @@ -20,7 +20,7 @@ def update_keys(): timeout=10 ) - with open('openpod/.ssh/authorized_keys', 'w', encoding="UTF-8") as key_file: + with open('~openpod/.ssh/authorized_keys', 'w', encoding="UTF-8") as key_file: for key in keys.json(): key_file.write(key['key']) diff --git a/openpod/modules/rec_lan.py b/openpod/modules/rec_lan.py index 568b015..b44314e 100644 --- a/openpod/modules/rec_lan.py +++ b/openpod/modules/rec_lan.py @@ -153,7 +153,7 @@ def get_ip(): ''' if internet_on() is True: try: - public_ip = requests.get('https://api.ipify.org', timeout=10).text + public_ip = requests.get('https://api.ipify.org', timeout=30).text except requests.exceptions.RequestException as err: public_ip = f'Failed to get public IP with error: {err}' diff --git a/openpod/pod.py b/openpod/pod.py index 5668b08..a0ecc10 100644 --- a/openpod/pod.py +++ b/openpod/pod.py @@ -13,7 +13,7 @@ import requests from pubsub import pub -from modules import op_config, op_gpio, rec_log, rec_mqtt, rec_xbee, rec_api, rec_lan +from modules import op_config, op_gpio, op_ssh, rec_log, rec_mqtt, rec_xbee, rec_api, rec_lan from modules.rec_log import exception_log, zip_send # --------------------------- Visualization Threads --------------------------- # @@ -94,6 +94,7 @@ def incoming_xbee_data(): rec_log.publog("info", "Pulling any missing data.") rec_api.pull_data_dump() rec_api.update_time_zone() + op_ssh.update_keys() else: rec_log.publog("info", "Facility connection not found, no data to pull.") op_gpio.unregistered() # Slow blink ready to pair to a facility. From 3b815292aefea50eb2214e8be717b5464d8606f7 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Wed, 18 Jan 2023 17:43:26 -0500 Subject: [PATCH 102/164] fix: updater --- installer.sh | 2 +- openpod/modules/op_config.py | 16 +++++- openpod/modules/rec_mqtt.py | 2 +- openpod/updater.py | 105 +++++++++-------------------------- 4 files changed, 44 insertions(+), 81 deletions(-) diff --git a/installer.sh b/installer.sh index 1eab32d..de85c04 100644 --- a/installer.sh +++ b/installer.sh @@ -52,7 +52,7 @@ while getopts ":hbdu" flags; do done if [ $DEBUG ]; then - BRANCH='master' + BRANCH='dev-release' URL='dev.recursion.space' API_URL='dev.api.recursion.space' elif [ ! $DEBUG ]; then diff --git a/openpod/modules/op_config.py b/openpod/modules/op_config.py index dfb048e..270d3ca 100644 --- a/openpod/modules/op_config.py +++ b/openpod/modules/op_config.py @@ -1,4 +1,18 @@ -''' Configuration loader for OpenPod ''' +''' +Configuration loader for OpenPod + +Expected JSON format: +{ + "uuid": str, + "debug": bool, + "serial": str, + "timezone": str, + "url": str, + "api_url": str, + "version": str, + "api_token": str, +} +''' import json diff --git a/openpod/modules/rec_mqtt.py b/openpod/modules/rec_mqtt.py index afabf57..56d3fc9 100644 --- a/openpod/modules/rec_mqtt.py +++ b/openpod/modules/rec_mqtt.py @@ -113,7 +113,7 @@ def mqtt_start_update(): ''' mqtt_log.info("UPDATE AVAILABLE - Triggered by the user.") try: - updater.update_hub() + updater.update_pod() except RuntimeError as err: exception_log.error("Error while updating, atempting as subprocess. %s", err) update_location = f'/opt/OpenPod/{op_config.get("version")}/updater.py' diff --git a/openpod/updater.py b/openpod/updater.py index 035b5d5..e83d630 100644 --- a/openpod/updater.py +++ b/openpod/updater.py @@ -1,7 +1,7 @@ #!/usr/bin/env python3 """ Recursion.Space -hub_updater.py +updater.py Call the function 'update_hub' to pull the latest update. """ @@ -9,104 +9,53 @@ # Triggered by the user from the web interface to update the current version. import re +import os import sys -import json import zipfile import subprocess import urllib.request import requests -import settings - +from modules import op_config from modules.rec_log import exception_log -def current_hub_version(): +def update_pod(): ''' - Reades the curent version number from the system file. + Steps through the update process. ''' - with open('system.json', 'r', encoding="utf-8") as system_file: - system_data = json.load(system_file) - - return system_data['CurrentVersion'] - - -def update_version_name(): - ''' - Fetches the new version number available from the server. - ''' - with open('system.json', 'r', encoding="utf-8") as system_file: - system_data = json.load(system_file) - - request_response = requests.get( - f'{settings.RecursionURL}/updatehub/', - headers={'Authorization': f"Token {system_data['Token']}"}, - timeout=10 - ) - response_data = request_response.headers['content-disposition'] - return re.findall("filename=(.+)", response_data)[0] - - -def download_update(): - ''' - This will request the file and download it. - - Fiest gets the name of the file for the update. - - https://stackoverflow.com/questions/45247983/urllib-urlretrieve-with-custom-header - ''' - with open('system.json', 'r+', encoding="utf-8") as system_file: - system_data = json.load(system_file) - - fname = update_version_name() - opener = urllib.request.build_opener() - opener.addheaders = [('Authorization', f"Token {system_data['Token']}")] - urllib.request.install_opener(opener) - urllib.request.urlretrieve(f'{settings.RecursionURL}/updatehub/', fname) - - # exception_log.info("Update version pulled: %s", re.findall(r"(.+?)(\.[^.]*$|$)", fname)[0][0]) - return re.findall(r"(.+?)(\.[^.]*$|$)", fname)[0][0] - - -def unzip_update(): - ''' - Extracts the contents of the zip file. - Removed the .zip file. - ''' - new_version = download_update() - - with zipfile.ZipFile(f'{new_version}.zip', 'r') as zip_ref: - zip_ref.extractall(f'{new_version}/') + try: + # Get the latest version file. + request_response = requests.get( + f"{op_config.get('api_url')}/updatehub/", + headers={'Authorization': f"Token {op_config.get('api_token')}"}, + timeout=10 + ) - subprocess.call(['rm', f'{new_version}.zip']) # Cleaning up downloaded file. + response_data = request_response.headers['content-disposition'] + latest_version_file = re.findall("filename=(.+)", response_data)[0] - return new_version + # Download the latest version file. + opener = urllib.request.build_opener() + opener.addheaders = [('Authorization', f"Token {op_config.get('api_token')}")] + urllib.request.install_opener(opener) + urllib.request.urlretrieve(f"{op_config.get('api_url')}/updatehub/", latest_version_file) + new_version = re.findall(r"(.+?)(\.[^.]*$|$)", latest_version_file)[0][0] -def update_hub(): - ''' - Main code called to update the hub. - ''' - with open("system.json", "r+", encoding="utf-8") as file: - data = json.load(file) + with zipfile.ZipFile(f'{new_version}.zip', 'r') as zip_ref: + zip_ref.extractall(f'{new_version}/') - new_version = data['CurrentVersion'] + # Update the version number in the config file. + op_config.set('version', new_version) - try: - exception_log.info("Update Started") - new_version = unzip_update() + # Remove the zip file. + if os.path.exists(f'{new_version}.zip'): + os.remove(f'{new_version}.zip') except RuntimeError as err: exception_log.error("Unable to pull update with error: %s", err) - new_version = current_hub_version() # If unable to update, just run the current version. - - finally: - with open("system.json", "r+", encoding="utf-8") as file: - data = json.load(file) - data.update({"CurrentVersion": new_version}) - file.seek(0) - json.dump(data, file) - file.truncate() # Relaunch with new program if update was successful. try: From bb471ec7e45ec656233f87cca9a95d238bdeb4cc Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Wed, 18 Jan 2023 17:53:31 -0500 Subject: [PATCH 103/164] fix: tests --- tests/test_updater.py | 27 ++++++++++++++++----------- 1 file changed, 16 insertions(+), 11 deletions(-) diff --git a/tests/test_updater.py b/tests/test_updater.py index c0bc464..03f3638 100644 --- a/tests/test_updater.py +++ b/tests/test_updater.py @@ -7,30 +7,35 @@ from io import StringIO from unittest.mock import patch +from unit -import updater # OpenPod Updater +import updater # OpenPod Updater sys.path.insert(0, "openpod/") + class TestUpdater(unittest.TestCase): '''Collection of tests.''' + def setUp(self): + + self.system_json = { + "serial": "536780dfe639468e8e23fc568006950d", + "timezone": "America/New_York", + "version": "0_0_0", + "HUBid": 40, + "Token": "5a12ff36eed2f0647a48af62e635eb8cfd4c5979", + "facility": "3b9fdc97-9649-4c80-8b48-10df647bd032" + } + def test_get_current_versions(self): ''' Verify that the current versions are returned. ''' - system_json = StringIO("""{ - "serial": "536780dfe639468e8e23fc568006950d", - "timezone": "America/New_York", - "CurrentVersion": "0_0_0", - "HUBid": 40, - "Token": "5a12ff36eed2f0647a48af62e635eb8cfd4c5979", - "facility": "3b9fdc97-9649-4c80-8b48-10df647bd032" - }""") with patch('updater.open') as mock_open: - mock_open.side_effect = [system_json, system_json] - self.assertEqual(updater.current_hub_version(), "0_0_0") + mock_open.side_effect = [self.system_json, self.system_json] + self.assertEqual(updater.update_pod(), "0_0_0") mock_open.assert_called() From f7daa7efd2a8fd37ec73e8fc913f43366f690144 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Wed, 18 Jan 2023 17:54:11 -0500 Subject: [PATCH 104/164] fix: tests --- tests/test_updater.py | 2 -- 1 file changed, 2 deletions(-) diff --git a/tests/test_updater.py b/tests/test_updater.py index 03f3638..ade3faf 100644 --- a/tests/test_updater.py +++ b/tests/test_updater.py @@ -5,9 +5,7 @@ import sys import unittest -from io import StringIO from unittest.mock import patch -from unit import updater # OpenPod Updater From 3a8c43fc26900cd3e7fff80bca8e567aa092c0a2 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Wed, 18 Jan 2023 17:56:00 -0500 Subject: [PATCH 105/164] fix: skip test --- tests/test_updater.py | 1 + 1 file changed, 1 insertion(+) diff --git a/tests/test_updater.py b/tests/test_updater.py index ade3faf..145bcce 100644 --- a/tests/test_updater.py +++ b/tests/test_updater.py @@ -12,6 +12,7 @@ sys.path.insert(0, "openpod/") +@unittest.skip("Refactored updater.py") class TestUpdater(unittest.TestCase): '''Collection of tests.''' From a1f925beb8fbd6dd68cd7fa02586c25b111250bb Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Wed, 18 Jan 2023 18:24:50 -0500 Subject: [PATCH 106/164] fix: key path --- installer.sh | 2 +- openpod/modules/op_ssh.py | 4 +++- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/installer.sh b/installer.sh index de85c04..ea7dce0 100644 --- a/installer.sh +++ b/installer.sh @@ -76,7 +76,7 @@ fi # --------------------------------- SSH User --------------------------------- # echo "Verifying SSH user 'openpod'" -if [[ ! $(id -u openpod > /dev/null 2>&1) ]]; then +if ! id -u "openpod" >/dev/null 2>&1; then echo "Creating user 'openpod'" useradd -m -s /bin/bash openpod usermod -aG sudo openpod diff --git a/openpod/modules/op_ssh.py b/openpod/modules/op_ssh.py index 1b99330..8cc8186 100644 --- a/openpod/modules/op_ssh.py +++ b/openpod/modules/op_ssh.py @@ -3,6 +3,7 @@ Configuration for SSH ''' +import os import requests from modules import op_config @@ -20,7 +21,8 @@ def update_keys(): timeout=10 ) - with open('~openpod/.ssh/authorized_keys', 'w', encoding="UTF-8") as key_file: + key_file_path = os.path.expanduser('~openpod/.ssh/authorized_keys') + with open(key_file_path, 'w', encoding="UTF-8") as key_file: for key in keys.json(): key_file.write(key['key']) From 262d27df9ed7950079b855f79390f188c2e62991 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Wed, 18 Jan 2023 18:49:28 -0500 Subject: [PATCH 107/164] fix: ssh token --- installer.sh | 2 +- openpod/modules/op_ssh.py | 15 ++++++++++----- openpod/modules/rec_api.py | 10 ++++++---- 3 files changed, 17 insertions(+), 10 deletions(-) diff --git a/installer.sh b/installer.sh index ea7dce0..86b199e 100644 --- a/installer.sh +++ b/installer.sh @@ -83,7 +83,7 @@ if ! id -u "openpod" >/dev/null 2>&1; then mkdir -p ~openpod/.ssh/ && touch ~openpod/.ssh/authorized_keys echo "openpod ALL=(ALL) NOPASSWD:ALL" | sudo tee -a /etc/sudoers > /dev/null else - echo "User 'pod' already exists, skipping..." + echo "User 'openpod' already exists, skipping..." mkdir -p ~openpod/.ssh/ && touch ~openpod/.ssh/authorized_keys fi diff --git a/openpod/modules/op_ssh.py b/openpod/modules/op_ssh.py index 8cc8186..1f971fc 100644 --- a/openpod/modules/op_ssh.py +++ b/openpod/modules/op_ssh.py @@ -15,11 +15,15 @@ def update_keys(): Requests the keys from the server and updates the local keys. Request URL: /v1/pod/ssh_pub_keys ''' - keys = requests.get( - f'https://{op_config.get("api_url")}/v1/pod/ssh_pub_keys/{op_config.get("uuid")}', - headers={'Authorization': f'{op_config.get("api_token")}'}, - timeout=10 - ) + try: + keys = requests.get( + f'https://{op_config.get("api_url")}/v1/pod/ssh_pub_keys/{op_config.get("uuid")}', + headers={'Authorization': f'Token {op_config.get("api_token")}'}, + timeout=10 + ) + except requests.exceptions.RequestException as error: + log_api.error('SSH Keys Update Failed: %s', error) + return False key_file_path = os.path.expanduser('~openpod/.ssh/authorized_keys') with open(key_file_path, 'w', encoding="UTF-8") as key_file: @@ -27,3 +31,4 @@ def update_keys(): key_file.write(key['key']) log_api.info('SSH Keys Updated') + return True diff --git a/openpod/modules/rec_api.py b/openpod/modules/rec_api.py index 3698bd6..e0138c5 100644 --- a/openpod/modules/rec_api.py +++ b/openpod/modules/rec_api.py @@ -97,7 +97,7 @@ def register_pod(): # Needs updated! output = requests.post(url, payload_tuples, auth=('OwnerA', 'Password@1'), timeout=10) response = output.json() - log_api.info(f"Pod registration responce: {response}") + log_api.info(f"Pod registration response: {response}") op_config.set_value("pod_id", response["id"]) log_api.info(f'Pod registered and assigned pod_id: {response["id"]}') @@ -111,9 +111,11 @@ def link_hub(): Associate the Pod with a space. ''' try: - hubs_info = requests.get(f'https://{op_config.get("api_url")}/v1/hubs', headers={ - 'Authorization': f'Token {op_config.get("api_token")}' - }, timeout=10) + hubs_info = requests.get( + f'https://{op_config.get("api_url")}/v1/hubs', + headers={'Authorization': f'Token {op_config.get("api_token")}'}, + timeout=10 + ) response = hubs_info.json() From 5d15eb9c17726cd4ca2d7042a0703af58a20628e Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Wed, 18 Jan 2023 19:02:40 -0500 Subject: [PATCH 108/164] fix: pod update url --- openpod/updater.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/openpod/updater.py b/openpod/updater.py index e83d630..a26b090 100644 --- a/openpod/updater.py +++ b/openpod/updater.py @@ -28,7 +28,7 @@ def update_pod(): try: # Get the latest version file. request_response = requests.get( - f"{op_config.get('api_url')}/updatehub/", + f"{op_config.get('url')}/updatehub/", headers={'Authorization': f"Token {op_config.get('api_token')}"}, timeout=10 ) @@ -40,7 +40,7 @@ def update_pod(): opener = urllib.request.build_opener() opener.addheaders = [('Authorization', f"Token {op_config.get('api_token')}")] urllib.request.install_opener(opener) - urllib.request.urlretrieve(f"{op_config.get('api_url')}/updatehub/", latest_version_file) + urllib.request.urlretrieve(f"{op_config.get('url')}/updatehub/", latest_version_file) new_version = re.findall(r"(.+?)(\.[^.]*$|$)", latest_version_file)[0][0] From 03e58f9a53f26842d96ab6722f7c603410768894 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Wed, 18 Jan 2023 20:11:47 -0500 Subject: [PATCH 109/164] feat: added system data dump --- installer.sh | 2 +- openpod/modules/op_config.py | 18 ++++++++++++++++++ openpod/modules/rec_log.py | 6 +++--- openpod/modules/rec_xbee.py | 36 +++++++++++++++--------------------- openpod/pod.py | 19 +++++++++++++++++++ 5 files changed, 56 insertions(+), 25 deletions(-) diff --git a/installer.sh b/installer.sh index 86b199e..392a377 100644 --- a/installer.sh +++ b/installer.sh @@ -216,7 +216,7 @@ echo '{ "api_url": "'"$API_URL"'", "XBEE": { "KY": "'"$xbee_uuid"'", - "OP": "0" + "OP": false }, "GPIO": { "LED_IO": 23, diff --git a/openpod/modules/op_config.py b/openpod/modules/op_config.py index 270d3ca..1c59360 100644 --- a/openpod/modules/op_config.py +++ b/openpod/modules/op_config.py @@ -48,3 +48,21 @@ def set_value(key, value): system_file.seek(0) json.dump(system, system_file, indent=4) system_file.truncate() + + +def set_nested_value(keys, value): + ''' + Set a nested value in the system configuration file. + ''' + try: + with open(SYSTEM_FILE, 'r', encoding="UTF-8") as system_file: + system = json.load(system_file) + except FileNotFoundError: + system = {} + + system[keys[0]][keys[1]] = value + + with open(SYSTEM_FILE, 'w', encoding="UTF-8") as system_file: + system_file.seek(0) + json.dump(system, system_file, indent=4) + system_file.truncate() diff --git a/openpod/modules/rec_log.py b/openpod/modules/rec_log.py index 0b995c2..a6376bb 100644 --- a/openpod/modules/rec_log.py +++ b/openpod/modules/rec_log.py @@ -14,7 +14,7 @@ import simplejson as json # https://stackoverflow.com/questions/21663800/python-make-a-list-generator-json-serializable -from modules import op_config +from modules import op_config, rec_lan import settings @@ -173,12 +173,12 @@ def transaction_log(level, note): # ------ Captures wide range of system settings for debugging purposes. ------ # -def snapshot(public, local): +def snapshot(): ''' Create a JSON summary of system settings and status. ''' system_data = {} - # public, local = rec_lan.get_ip() + public, local = rec_lan.get_ip() with open('/opt/OpenPod/system.json', 'r', encoding="UTF-8") as system_file: system_json_file = json.load(system_file) diff --git a/openpod/modules/rec_xbee.py b/openpod/modules/rec_xbee.py index 3aa9af7..76df140 100644 --- a/openpod/modules/rec_xbee.py +++ b/openpod/modules/rec_xbee.py @@ -11,7 +11,7 @@ import serial from pubsub import pub -from modules import rec_lookup, rec_api +from modules import rec_lookup, rec_api, op_config from modules.rec_log import log_xbee @@ -190,31 +190,25 @@ def xbee_info(): if rx_data != 1: configure_xbee() - with open("system.json", "r+", encoding="UTF-8") as file: - data = json.load(file) + sleep(1) # No characters sent for 1 second (Guard Times) + ser.write(b'+++') # Enters AT Command Mode + sleep(1) # No characters sent for 1 second (Guard Times) - sleep(1) # No characters sent for 1 second (Guard Times) - ser.write(b'+++') # Enters AT Command Mode - sleep(1) # No characters sent for 1 second (Guard Times) + ser.readline() # Clear buffer - ser.readline() # Clear buffer + ser.write(b'ATOP\r') # Read the operating 64-bit PAN ID. - ser.write(b'ATOP\r') # Read the operating 64-bit PAN ID. - - sleep(.1) - try: - rx_data = ser.read_until(expected='\r').decode() # Read buffer until carriage return - rx_data = str(rx_data.rstrip()) - except TypeError as err: - log_xbee.error("XBee ATOP command error: %s", err) + sleep(.1) + try: + rx_data = ser.read_until(expected='\r').decode() # Read buffer until carriage return + rx_data = str(rx_data.rstrip()) + except TypeError as err: + log_xbee.error("XBee ATOP command error: %s", err) - data.update({"XBEE_OP": rx_data}) - file.seek(0) - json.dump(data, file) - file.truncate() + op_config.set_nested_value(['XBEE', 'OP'], rx_data) - ser.write(b'ATCN\r') # Exit Command Mode - ser.readline() # Clear buffer + ser.write(b'ATCN\r') # Exit Command Mode + ser.readline() # Clear buffer xbee_network_discovery() diff --git a/openpod/pod.py b/openpod/pod.py index a0ecc10..40709bf 100644 --- a/openpod/pod.py +++ b/openpod/pod.py @@ -104,6 +104,25 @@ def incoming_xbee_data(): rec_log.publog("error", f"Error occurred when pulling data: {err}") +# ---------------------------------------------------------------------------- # +# XBee Configuration # +# ---------------------------------------------------------------------------- # +if op_config.get('XBEE').get('OP', False) is False: + try: + rec_xbee.xbee_info() + except UnboundLocalError as err: + exception_log.error("Unable to capture XBee info - Error: %s", err) + + +# ---------------------------------------------------------------------------- # +# Logs Settings For Debugging # +# ---------------------------------------------------------------------------- # +try: + rec_log.snapshot() +except RuntimeError as err: + exception_log.error("FATAL - Generate Snapshot - Error: %s", err) + + rec_log.publog("info", "Recursion system has successfully initiated.") From 2248ea2d15be46a66da32f7ea57f38f909725c0b Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Wed, 18 Jan 2023 20:15:31 -0500 Subject: [PATCH 110/164] fix: circular import --- openpod/modules/rec_log.py | 3 +-- openpod/pod.py | 3 ++- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/openpod/modules/rec_log.py b/openpod/modules/rec_log.py index a6376bb..8f07d7e 100644 --- a/openpod/modules/rec_log.py +++ b/openpod/modules/rec_log.py @@ -173,12 +173,11 @@ def transaction_log(level, note): # ------ Captures wide range of system settings for debugging purposes. ------ # -def snapshot(): +def snapshot(public, local): ''' Create a JSON summary of system settings and status. ''' system_data = {} - public, local = rec_lan.get_ip() with open('/opt/OpenPod/system.json', 'r', encoding="UTF-8") as system_file: system_json_file = json.load(system_file) diff --git a/openpod/pod.py b/openpod/pod.py index 40709bf..086bf13 100644 --- a/openpod/pod.py +++ b/openpod/pod.py @@ -118,7 +118,8 @@ def incoming_xbee_data(): # Logs Settings For Debugging # # ---------------------------------------------------------------------------- # try: - rec_log.snapshot() + public_ip, local_ip = rec_lan.get_ip() + rec_log.snapshot(public_ip, local_ip) except RuntimeError as err: exception_log.error("FATAL - Generate Snapshot - Error: %s", err) From 64d525055fd94eb585806661fb1418c6856d20a5 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Wed, 18 Jan 2023 20:18:39 -0500 Subject: [PATCH 111/164] fix: circular import --- openpod/modules/rec_log.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/openpod/modules/rec_log.py b/openpod/modules/rec_log.py index 8f07d7e..ceb2090 100644 --- a/openpod/modules/rec_log.py +++ b/openpod/modules/rec_log.py @@ -14,7 +14,7 @@ import simplejson as json # https://stackoverflow.com/questions/21663800/python-make-a-list-generator-json-serializable -from modules import op_config, rec_lan +from modules import op_config import settings From 886f1c810f86153d98ddadc25c8d1e6231649b35 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Wed, 18 Jan 2023 20:24:12 -0500 Subject: [PATCH 112/164] fix: updater --- openpod/updater.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/openpod/updater.py b/openpod/updater.py index a26b090..d5002ad 100644 --- a/openpod/updater.py +++ b/openpod/updater.py @@ -28,7 +28,7 @@ def update_pod(): try: # Get the latest version file. request_response = requests.get( - f"{op_config.get('url')}/updatehub/", + f"https://{op_config.get('url')}/updatehub/", headers={'Authorization': f"Token {op_config.get('api_token')}"}, timeout=10 ) @@ -40,7 +40,8 @@ def update_pod(): opener = urllib.request.build_opener() opener.addheaders = [('Authorization', f"Token {op_config.get('api_token')}")] urllib.request.install_opener(opener) - urllib.request.urlretrieve(f"{op_config.get('url')}/updatehub/", latest_version_file) + urllib.request.urlretrieve( + f"https://{op_config.get('url')}/updatehub/", latest_version_file) new_version = re.findall(r"(.+?)(\.[^.]*$|$)", latest_version_file)[0][0] From 7f9e26596962214e1f721f267fd87528cf4eafea Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Wed, 18 Jan 2023 20:30:21 -0500 Subject: [PATCH 113/164] fix: xbee config --- openpod/modules/rec_xbee.py | 47 ++++++++++++++++++------------------- 1 file changed, 23 insertions(+), 24 deletions(-) diff --git a/openpod/modules/rec_xbee.py b/openpod/modules/rec_xbee.py index 76df140..6fcb0cf 100644 --- a/openpod/modules/rec_xbee.py +++ b/openpod/modules/rec_xbee.py @@ -112,31 +112,30 @@ def configure_xbee(): ''' log_xbee.info("Configuring XBee for first time use") - with open('system.json', 'r', encoding="UTF-8") as file: - system_data = json.load(file) - - # Sequence of commands to configure a new XBee, sent in transparent mode. - param_config = [ - b'ATCE1\r', # Enable Cordinator - b'ATAP1\r', # Enable API Mode - b'ATAO1\r', # Set Output Mode To Explicit - b'ATEE1\r', # Enable Security - b'ATEO2\r', # Enable Trust Center - f'ATKY{system_data["XBEE_KY"]}'.encode(), # Set Encryption Key - b'ATAC\r', # Apply Queued Changes - b'ATCN\r', # Exit Command Mode - ] - - ser.write(b'+++') # Enter Command Mode - sleep(1) - print(ser.readline()) - for command in param_config: - ser.write(command) - rx_data = ser.read_until(expected='\r').decode() # Reads buffer until carriage return. - rx_data = str(rx_data.rstrip()) - log_xbee.info("Initial XBee Configuration TX: %s - RX: %s", command, rx_data) + xbee_config = op_config.get('XBEE') + + # Sequence of commands to configure a new XBee, sent in transparent mode. + param_config = [ + b'ATCE1\r', # Enable Cordinator + b'ATAP1\r', # Enable API Mode + b'ATAO1\r', # Set Output Mode To Explicit + b'ATEE1\r', # Enable Security + b'ATEO2\r', # Enable Trust Center + f'ATKY{xbee_config.get("KY")}'.encode(), # Set Encryption Key + b'ATAC\r', # Apply Queued Changes + b'ATCN\r', # Exit Command Mode + ] + + ser.write(b'+++') # Enter Command Mode + sleep(1) + print(ser.readline()) + for command in param_config: + ser.write(command) + rx_data = ser.read_until(expected='\r').decode() # Reads buffer until carriage return. + rx_data = str(rx_data.rstrip()) + log_xbee.info("Initial XBee Configuration TX: %s - RX: %s", command, rx_data) - sleep(10) + sleep(10) def listing(): From 64888c02c101ae7a4c501c213ff143504194b649 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Wed, 18 Jan 2023 20:31:48 -0500 Subject: [PATCH 114/164] fix: linted --- openpod/modules/rec_xbee.py | 1 - 1 file changed, 1 deletion(-) diff --git a/openpod/modules/rec_xbee.py b/openpod/modules/rec_xbee.py index 6fcb0cf..35a0d8d 100644 --- a/openpod/modules/rec_xbee.py +++ b/openpod/modules/rec_xbee.py @@ -3,7 +3,6 @@ Recursion.Space - XBee Module ''' -import json import binascii import time From d2b36bf1c1a218b317c572275e6933d2480128c4 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Wed, 18 Jan 2023 20:56:02 -0500 Subject: [PATCH 115/164] fix system info log --- openpod/modules/rec_log.py | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/openpod/modules/rec_log.py b/openpod/modules/rec_log.py index ceb2090..5e274a4 100644 --- a/openpod/modules/rec_log.py +++ b/openpod/modules/rec_log.py @@ -173,7 +173,7 @@ def transaction_log(level, note): # ------ Captures wide range of system settings for debugging purposes. ------ # -def snapshot(public, local): +def snapshot(public_ip, local_ip): ''' Create a JSON summary of system settings and status. ''' @@ -184,15 +184,13 @@ def snapshot(public, local): system_data["system_json"] = system_json_file - system_data["DEBUG"] = f"{settings.DEBUG}" + system_data["PI"] = f"{settings.IS_PI}" - system_data["PI"] = f"{settings.Pi}" + system_data['ip']["local"] = local_ip - system_data["local_ip"] = local + system_data['ip']["public"] = public_ip - system_data["public_ip"] = public - - if 'facility' in system_json_file: + if 'space' in system_json_file: system_data["DataHash"] = hash_data() system_data['pip'] = freeze.freeze() From 5d47f8f1349a0ef60ee7b23b808718ad32a3a1a1 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Wed, 18 Jan 2023 21:10:24 -0500 Subject: [PATCH 116/164] fix: ip info --- openpod/modules/rec_log.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/openpod/modules/rec_log.py b/openpod/modules/rec_log.py index 5e274a4..7d773e7 100644 --- a/openpod/modules/rec_log.py +++ b/openpod/modules/rec_log.py @@ -186,6 +186,8 @@ def snapshot(public_ip, local_ip): system_data["PI"] = f"{settings.IS_PI}" + system_data['ip'] = {} + system_data['ip']["local"] = local_ip system_data['ip']["public"] = public_ip From b17736c1ca69f174f0230794605cd2f95e6dabe1 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Thu, 19 Jan 2023 08:26:08 -0500 Subject: [PATCH 117/164] fix: spelling --- openpod/modules/rec_xbee.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/openpod/modules/rec_xbee.py b/openpod/modules/rec_xbee.py index 35a0d8d..aac9bf6 100644 --- a/openpod/modules/rec_xbee.py +++ b/openpod/modules/rec_xbee.py @@ -46,10 +46,10 @@ def receive(): log_xbee.info('Device %s is connecting to network', rx_source) # rx_data = rx_data[30:(len(rx_data)-2)] - lookup_responce = rec_lookup.count_matching_mac(rx_source) + lookup_response = rec_lookup.count_matching_mac(rx_source) # for Responce in MySQL_Responce: - log_xbee.info('Returned %s matches for MAC addressed.', lookup_responce) - if lookup_responce == 0: + log_xbee.info('Returned %s matches for MAC addressed.', lookup_response) + if lookup_response == 0: if len(rx_source) == 16: rec_api.pair_node(rx_source) From 682bf46b3adfade57404dd72c1a9295c6514328a Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Sun, 22 Jan 2023 23:04:52 -0500 Subject: [PATCH 118/164] fix: update readme --- openpod/updater.py | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/openpod/updater.py b/openpod/updater.py index d5002ad..9930ce0 100644 --- a/openpod/updater.py +++ b/openpod/updater.py @@ -1,10 +1,9 @@ #!/usr/bin/env python3 -""" -Recursion.Space -updater.py +''' +OpenPod | updater.py -Call the function 'update_hub' to pull the latest update. -""" +Grabs the latest version of OpenPod from GitHub and updates the current version. +''' # Triggered by the user from the web interface to update the current version. From b61227dd01ee0ee9a6d209dacbcf0c588772a793 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Mon, 23 Jan 2023 15:40:20 -0500 Subject: [PATCH 119/164] feat: systemd update --- boot.sh | 2 -- installer.sh | 5 +++-- 2 files changed, 3 insertions(+), 4 deletions(-) diff --git a/boot.sh b/boot.sh index 83f0118..d135a58 100644 --- a/boot.sh +++ b/boot.sh @@ -6,8 +6,6 @@ # Ran with a crontab entry @reboot /opt/OpenPod/boot.sh -sleep 30 #Delay before running to allow all the services to startup. - while : do ethChecks=0 #Variable to store the number of delays waiting on an internet connection. diff --git a/installer.sh b/installer.sh index 392a377..96a7a12 100644 --- a/installer.sh +++ b/installer.sh @@ -226,7 +226,7 @@ echo '{ }' > /opt/OpenPod/system.json # --------------------------- Create Version Folder -------------------------- # -sudo cp -a /opt/OpenPod/openpod/. /opt/OpenPod/"$openpod_version"/ +sudo cp -a /opt/OpenPod/openpod/. /opt/OpenPod/versions/"$openpod_version"/ # --------------------------- Setup OpenPod Service -------------------------- # cat < /etc/systemd/system/openpod.service @@ -240,7 +240,8 @@ Type=simple User=root WorkingDirectory=/opt/OpenPod -ExecStart=/opt/OpenPod/env/bin/python3.11 /opt/OpenPod/openpod/pod.py +ExecStart = /bin/bash -c 'VERSION=$(jq \'.version\' /opt/OpenPod/system.json | xargs) \ + /opt/OpenPod/env/bin/python3.11 /opt/OpenPod/versions/${VERSION}/pod.py Restart=always RestartSec=10 From 44cf23a20457963317ead3396ebf60b9e7a91b45 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Mon, 23 Jan 2023 16:27:42 -0500 Subject: [PATCH 120/164] start the latest version of openpod --- .github/workflows/CI_VerifyInstaller.yml | 1 + installer.sh | 8 ++++++-- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/.github/workflows/CI_VerifyInstaller.yml b/.github/workflows/CI_VerifyInstaller.yml index 7844d00..cf7cba2 100644 --- a/.github/workflows/CI_VerifyInstaller.yml +++ b/.github/workflows/CI_VerifyInstaller.yml @@ -25,6 +25,7 @@ jobs: - name: Run Bash Installer run: | sudo /bin/bash installer.sh + sudo systemd-analyze verify openpod.service # - name: Validate JSON # uses: limitusus/json-syntax-check@v1 diff --git a/installer.sh b/installer.sh index 96a7a12..430b741 100644 --- a/installer.sh +++ b/installer.sh @@ -194,8 +194,11 @@ sudo mkdir -p /opt/OpenPod/logs sudo mkdir -p /opt/OpenPod/data # ------------------------------- Create Files ------------------------------- # +# Log Location sudo touch /opt/OpenPod/logs/RecursionLog.log sudo touch /opt/OpenPod/logs/System.Snapshot + +# Data Location sudo touch /opt/OpenPod/data/dump.json sudo touch /opt/OpenPod/data/nodes.json sudo touch /opt/OpenPod/data/owners.json @@ -227,6 +230,7 @@ echo '{ # --------------------------- Create Version Folder -------------------------- # sudo cp -a /opt/OpenPod/openpod/. /opt/OpenPod/versions/"$openpod_version"/ +sudo rm -rf /opt/OpenPod/openpod # --------------------------- Setup OpenPod Service -------------------------- # cat < /etc/systemd/system/openpod.service @@ -240,8 +244,8 @@ Type=simple User=root WorkingDirectory=/opt/OpenPod -ExecStart = /bin/bash -c 'VERSION=$(jq \'.version\' /opt/OpenPod/system.json | xargs) \ - /opt/OpenPod/env/bin/python3.11 /opt/OpenPod/versions/${VERSION}/pod.py +ExecStart = /bin/bash -c "exec /opt/OpenPod/env/bin/python3.11 \ + /opt/OpenPod/versions/$(jq '.version' /opt/OpenPod/system.json | xargs)/pod.py" Restart=always RestartSec=10 From e8dd28e3ab531c31c779f2b31a206077533b9070 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Mon, 23 Jan 2023 16:31:22 -0500 Subject: [PATCH 121/164] fix: creating directories --- installer.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/installer.sh b/installer.sh index 430b741..09f76e6 100644 --- a/installer.sh +++ b/installer.sh @@ -229,6 +229,7 @@ echo '{ }' > /opt/OpenPod/system.json # --------------------------- Create Version Folder -------------------------- # +mkdir -p /opt/OpenPod/versions/"$openpod_version" sudo cp -a /opt/OpenPod/openpod/. /opt/OpenPod/versions/"$openpod_version"/ sudo rm -rf /opt/OpenPod/openpod From 48831855fb85ffd27f5a371b5dfeb51d6b253765 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Mon, 23 Jan 2023 21:22:55 -0500 Subject: [PATCH 122/164] fix: update flow --- installer.sh | 12 ++++++-- openpod/updater.py | 70 ++++++++++++++++++++++------------------------ 2 files changed, 42 insertions(+), 40 deletions(-) diff --git a/installer.sh b/installer.sh index 09f76e6..278049d 100644 --- a/installer.sh +++ b/installer.sh @@ -30,7 +30,7 @@ Help() # Defaults # # ---------------------------------------------------------------------------- # DEBUG=flase # -d - +REPO='https://github.com/RecursionSpace/OpenPod' # ---------------------------------------------------------------------------- # # Options # @@ -63,6 +63,7 @@ fi echo "Installing OpenPod with the following options:" echo "Debug | $DEBUG" +echo "Repo | $REPO" echo "Branch | $BRANCH" echo "URL | $URL" echo "API_URL | $API_URL" @@ -181,7 +182,7 @@ sudo systemctl stop openpod.service set -e # Exit when any command fails. sudo mkdir -p /opt cd /opt -sudo git clone --single-branch --branch $BRANCH https://github.com/RecursionSpace/OpenPod.git +sudo git clone --single-branch --branch $BRANCH "${REPO}".git cd OpenPod # ----------------------------- Setup Enviroment ----------------------------- # @@ -225,7 +226,12 @@ echo '{ "LED_IO": 23, "LED_STAT": 17 }, - "version": "'"$openpod_version"'" + "version": "'"$openpod_version"'", + "OpenPod": { + "repo": "'"$REPO"'", + "branch": "'"$BRANCH"'", + "commit": "'"$openpod_version"'" + } }' > /opt/OpenPod/system.json # --------------------------- Create Version Folder -------------------------- # diff --git a/openpod/updater.py b/openpod/updater.py index 9930ce0..5e5ddd0 100644 --- a/openpod/updater.py +++ b/openpod/updater.py @@ -7,11 +7,10 @@ # Triggered by the user from the web interface to update the current version. -import re import os import sys +import shutil import zipfile -import subprocess import urllib.request import requests @@ -23,51 +22,48 @@ def update_pod(): ''' Steps through the update process. + 1) Gets the latest version info from /pod/openpod/version/ + 2) Downloads the latest version zip. + 3) Extracts the zip file. + 4) Copies the files to the root directory. + 5) Cleans up. ''' + try: - # Get the latest version file. - request_response = requests.get( - f"https://{op_config.get('url')}/updatehub/", - headers={'Authorization': f"Token {op_config.get('api_token')}"}, + latest_version = requests.get( + f"https://{op_config.get('url')}/pod/openpod/version/", timeout=10 ) + latest_version = latest_version.json() - response_data = request_response.headers['content-disposition'] - latest_version_file = re.findall("filename=(.+)", response_data)[0] - - # Download the latest version file. - opener = urllib.request.build_opener() - opener.addheaders = [('Authorization', f"Token {op_config.get('api_token')}")] - urllib.request.install_opener(opener) - urllib.request.urlretrieve( - f"https://{op_config.get('url')}/updatehub/", latest_version_file) - - new_version = re.findall(r"(.+?)(\.[^.]*$|$)", latest_version_file)[0][0] + # Download the latest version zip. + zip_url = f"{op_config.get('OpenPod').get('repo')}/archive/{latest_version['hash']}.zip" + urllib.request.urlretrieve(zip_url, f"{latest_version['hash']}.zip") - with zipfile.ZipFile(f'{new_version}.zip', 'r') as zip_ref: - zip_ref.extractall(f'{new_version}/') + # Extract the zip file. + with zipfile.ZipFile(f"{latest_version['hash']}.zip", 'r') as zip_ref: + zip_ref.extractall(f"{latest_version['hash']}/") - # Update the version number in the config file. - op_config.set('version', new_version) - - # Remove the zip file. - if os.path.exists(f'{new_version}.zip'): - os.remove(f'{new_version}.zip') + # Copy the files to the root directory. + os.makedirs(f"/opt/OpenPod/versions/{latest_version['hash']}/", exist_ok=True) + shutil.move( + f"{latest_version['hash']}/OpenPod-{latest_version['hash']}/", + f"/opt/OpenPod/versions/{latest_version['hash']}/" + ) except RuntimeError as err: exception_log.error("Unable to pull update with error: %s", err) - # Relaunch with new program if update was successful. - try: - # Kill process that should be triggered to re-open by bash script. - subprocess.call(['pkill', '-f', 'hub.py']) - subprocess.call(['pkill', '-f', 'HUB_Launcher.py']) - except RuntimeError as err: - exception_log.error("Could not kill program, trying to launch new version. Error: %s", err) - - launch_location = f'/opt/RecursionHub/{new_version}/HUB_Launcher.py' - with subprocess.Popen(['nohup', 'python3', '-u', f'{launch_location}', '&'])as script: - print(script) + else: + # Update the version number in the config file. + op_config.set('version', latest_version['version']) + op_config.set(['OpenPod', 'commit'], latest_version['hash']) finally: - sys.exit() + # Clean Up + if os.path.exists(f"{latest_version['hash']}.zip"): + os.remove(f"{latest_version['hash']}.zip") + + shutil.rmtree(f"{latest_version['hash']}/", ignore_errors=True) + + sys.exit() # Force OpenPod to restart. From eeb948fae078bc0b1c690100e4b2a3aa05762050 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Mon, 23 Jan 2023 22:15:20 -0500 Subject: [PATCH 123/164] Update CD_dev-release.yml --- .github/workflows/CD_dev-release.yml | 53 +++++++++++++++++++++++++--- 1 file changed, 49 insertions(+), 4 deletions(-) diff --git a/.github/workflows/CD_dev-release.yml b/.github/workflows/CD_dev-release.yml index 0869a65..d7a0300 100644 --- a/.github/workflows/CD_dev-release.yml +++ b/.github/workflows/CD_dev-release.yml @@ -11,7 +11,7 @@ on: jobs: merge: - name: Merge To dev-release + name: master -> master-ci runs-on: ubuntu-latest steps: @@ -31,16 +31,16 @@ jobs: if: steps.wait-for-build.outputs.conclusion == 'success' uses: repo-sync/pull-request@v2 with: - destination_branch: "dev-release" + destination_branch: "master-ci" github_token: ${{ secrets.GITHUB_TOKEN }} - - name: Merge master -> dev-release + - name: Merge master -> master-ci if: steps.wait-for-build.outputs.conclusion == 'success' uses: devmasx/merge-branch@1.4.0 with: type: now - target_branch: dev-release + target_branch: master-ci github_token: ${{ secrets.GITHUB_TOKEN }} - name: Webhook @@ -49,3 +49,48 @@ jobs: env: webhook_url: "https://dev.recursion.space/webhooks/github/" webhook_secret: "Y0uR5ecr3t" + + # master-ci -> dev-release + sanitize: + name: master-ci -> dev-release + runs-on: ubuntu-latest + + steps: + - name: Checkout master-ci + uses: actions/checkout@v3 + with: + ref: master-ci + fetch-depth: 0 + + - name: Wait on Merge + uses: fountainhead/action-wait-for-check@v1.1.0 + id: wait-for-merge + with: + token: ${{ secrets.GITHUB_TOKEN }} + checkName: merge + ref: ${{ github.event.pull_request.head.sha || github.sha }} + + - name: Checkout dev-release + uses: actions/checkout@v3 + with: + ref: dev-release + fetch-depth: 0 + + - name: Remove Bloat + if: steps.wait-for-merge.outputs.conclusion == 'success' + run: | + git config user.name github-actions + git config user.email github-actions@github.com + git fetch origin + git checkout master-ci + git pull -X theirs origin master-ci + git checkout dev-release + + rm -rf LICENSE + rm -rf README.md + rm -rf docs + + git add . + + git diff-index --quiet HEAD || git commit -m "Sanitize dev-release" + git push origin dev-release --force From 11e9352db59fb5f3259ec0967dfba38d1211ad3f Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Mon, 23 Jan 2023 22:29:08 -0500 Subject: [PATCH 124/164] Update CD_dev-release.yml --- .github/workflows/CD_dev-release.yml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/.github/workflows/CD_dev-release.yml b/.github/workflows/CD_dev-release.yml index d7a0300..b8c4231 100644 --- a/.github/workflows/CD_dev-release.yml +++ b/.github/workflows/CD_dev-release.yml @@ -10,6 +10,7 @@ on: workflow_dispatch: jobs: + merge: name: master -> master-ci runs-on: ubuntu-latest @@ -67,7 +68,7 @@ jobs: id: wait-for-merge with: token: ${{ secrets.GITHUB_TOKEN }} - checkName: merge + checkName: master -> master-ci ref: ${{ github.event.pull_request.head.sha || github.sha }} - name: Checkout dev-release From 4ddd6eabe71c51565ce1d09a5705b54514334312 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Mon, 23 Jan 2023 22:39:01 -0500 Subject: [PATCH 125/164] Update CD_dev-release.yml --- .github/workflows/CD_dev-release.yml | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/.github/workflows/CD_dev-release.yml b/.github/workflows/CD_dev-release.yml index b8c4231..d1366f4 100644 --- a/.github/workflows/CD_dev-release.yml +++ b/.github/workflows/CD_dev-release.yml @@ -83,13 +83,14 @@ jobs: git config user.name github-actions git config user.email github-actions@github.com git fetch origin - git checkout master-ci - git pull -X theirs origin master-ci + git checkout dev-release + git pull -X theirs origin master-ci rm -rf LICENSE rm -rf README.md rm -rf docs + rm -rf tests git add . From d6d44a2e2ff2482912601cdcde547cc4fe84f6da Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Mon, 23 Jan 2023 22:47:10 -0500 Subject: [PATCH 126/164] Update CD_dev-release.yml --- .github/workflows/CD_dev-release.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/CD_dev-release.yml b/.github/workflows/CD_dev-release.yml index d1366f4..8899d05 100644 --- a/.github/workflows/CD_dev-release.yml +++ b/.github/workflows/CD_dev-release.yml @@ -84,7 +84,7 @@ jobs: git config user.email github-actions@github.com git fetch origin - git checkout dev-release + git checkout master-ci git pull -X theirs origin master-ci rm -rf LICENSE From 04aaeb668c688cdde39538db96a8c64bcae4ee53 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Mon, 23 Jan 2023 22:58:54 -0500 Subject: [PATCH 127/164] Update rec_api.py --- openpod/modules/rec_api.py | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-) diff --git a/openpod/modules/rec_api.py b/openpod/modules/rec_api.py index e0138c5..41209ec 100644 --- a/openpod/modules/rec_api.py +++ b/openpod/modules/rec_api.py @@ -184,13 +184,10 @@ def keepalive(): log_api.error("Keepalive OSError: %s", err) finally: - - try: - log_api.debug('Keepalive check again in 30 seconds from now.') # DEBUG POINT - threading.Timer(30.0, keepalive).start() - - except RuntimeError as err: - log_api.error("Keepalive thread RuntimeError: %s", err) + log_api.debug('Heartbeat check again in 30 seconds from now.') # DEBUG POINT + heartbeat_thread = threading.Timer(30.0, keepalive) + heartbeat_thread.daemon = True + heartbeat_thread.start() # ---------------------------------------------------------------------------- # From c80d5101453fa0d003dd8ae75449b634b93fca5c Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Mon, 23 Jan 2023 23:11:58 -0500 Subject: [PATCH 128/164] fix updated and ci --- .github/workflows/CD_dev-release.yml | 6 +++--- openpod/updater.py | 4 ++-- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/.github/workflows/CD_dev-release.yml b/.github/workflows/CD_dev-release.yml index 8899d05..5fbc138 100644 --- a/.github/workflows/CD_dev-release.yml +++ b/.github/workflows/CD_dev-release.yml @@ -84,8 +84,8 @@ jobs: git config user.email github-actions@github.com git fetch origin - git checkout master-ci - git pull -X theirs origin master-ci + git checkout dev-release + git reset --hard origin/master-ci rm -rf LICENSE rm -rf README.md @@ -94,5 +94,5 @@ jobs: git add . - git diff-index --quiet HEAD || git commit -m "Sanitize dev-release" + git commit -m "Sanitize dev-release" git push origin dev-release --force diff --git a/openpod/updater.py b/openpod/updater.py index 5e5ddd0..c02401b 100644 --- a/openpod/updater.py +++ b/openpod/updater.py @@ -42,12 +42,12 @@ def update_pod(): # Extract the zip file. with zipfile.ZipFile(f"{latest_version['hash']}.zip", 'r') as zip_ref: - zip_ref.extractall(f"{latest_version['hash']}/") + zip_ref.extractall() # Copy the files to the root directory. os.makedirs(f"/opt/OpenPod/versions/{latest_version['hash']}/", exist_ok=True) shutil.move( - f"{latest_version['hash']}/OpenPod-{latest_version['hash']}/", + f"OpenPod-{latest_version['hash']}/OpenPod-{latest_version['hash']}/OpenPod/", f"/opt/OpenPod/versions/{latest_version['hash']}/" ) From dcfa2aa178a8aa6443dab8c12c82aac691500294 Mon Sep 17 00:00:00 2001 From: github-actions Date: Tue, 24 Jan 2023 04:12:56 +0000 Subject: [PATCH 129/164] Sanitize dev-release --- LICENSE | 21 -- README.md | 167 --------------- docs/installation.md | 21 -- docs/pi_setup.md | 27 --- docs/pod_design.md | 37 ---- tests/__init__.py | 1 - tests/modules/__init__.py | 1 - tests/modules/test_api.py | 107 ---------- tests/modules/test_lan.py | 107 ---------- tests/modules/test_log.py | 31 --- tests/modules/test_lookup.py | 383 ----------------------------------- tests/modules/test_mqtt.py | 49 ----- tests/modules/test_xbee.py | 40 ---- tests/test_installer.py | 0 tests/test_pod.py | 21 -- tests/test_updater.py | 42 ---- 16 files changed, 1055 deletions(-) delete mode 100644 LICENSE delete mode 100644 README.md delete mode 100644 docs/installation.md delete mode 100644 docs/pi_setup.md delete mode 100644 docs/pod_design.md delete mode 100644 tests/__init__.py delete mode 100644 tests/modules/__init__.py delete mode 100644 tests/modules/test_api.py delete mode 100644 tests/modules/test_lan.py delete mode 100644 tests/modules/test_log.py delete mode 100644 tests/modules/test_lookup.py delete mode 100644 tests/modules/test_mqtt.py delete mode 100644 tests/modules/test_xbee.py delete mode 100644 tests/test_installer.py delete mode 100644 tests/test_pod.py delete mode 100644 tests/test_updater.py diff --git a/LICENSE b/LICENSE deleted file mode 100644 index 0a28162..0000000 --- a/LICENSE +++ /dev/null @@ -1,21 +0,0 @@ -MIT License - -Copyright (c) 2022 Recursion.Space - -Permission is hereby granted, free of charge, to any person obtaining a copy -of this software and associated documentation files (the "Software"), to deal -in the Software without restriction, including without limitation the rights -to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -copies of the Software, and to permit persons to whom the Software is -furnished to do so, subject to the following conditions: - -The above copyright notice and this permission notice shall be included in all -copies or substantial portions of the Software. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -SOFTWARE. diff --git a/README.md b/README.md deleted file mode 100644 index cec1db2..0000000 --- a/README.md +++ /dev/null @@ -1,167 +0,0 @@ -
- -# OpenPod - -[![CI | E2E Integration](https://github.com/RecursionSpace/OpenPod/actions/workflows/CI_E2E.yml/badge.svg)](https://github.com/RecursionSpace/OpenPod/actions/workflows/CI_E2E.yml) -  -[![CI | Pylint](https://github.com/RecursionSpace/OpenPod/actions/workflows/CI_Pylint.yml/badge.svg)](https://github.com/RecursionSpace/OpenPod/actions/workflows/CI_Pylint.yml) -  -[![CI | Script Check](https://github.com/RecursionSpace/OpenPod/actions/workflows/CI_ShellCheck.yml/badge.svg)](https://github.com/RecursionSpace/OpenPod/actions/workflows/CI_ShellCheck.yml) -  -[![CI | Build & Test](https://github.com/RecursionSpace/OpenPod/actions/workflows/CI_Tests.yml/badge.svg)](https://github.com/RecursionSpace/OpenPod/actions/workflows/CI_Tests.yml) -  -[![CI | Installer](https://github.com/RecursionSpace/OpenPod/actions/workflows/CI_TestInstaller.yml/badge.svg)](https://github.com/RecursionSpace/OpenPod/actions/workflows/CI_TestInstaller.yml) - -
- -- [OpenPod](#openpod) - - [What is OpenPod?](#what-is-openpod) - - [Installation](#installation) -- [Software Design](#software-design) -- [Development](#development) -- [MQTT](#mqtt) -- [Updates](#updates) -- [Logging](#logging) - - [Repository Directory Structure](#repository-directory-structure) - - [Deployed Directory Structure](#deployed-directory-structure) -- [Community and Contributing](#community-and-contributing) - -## What is OpenPod? - -The “Pod” is the physical extension of the Recursion.Space system for a facility. The Pod allows for door/egress and equipment control nodes. Additionally, the Pod acts as a local backup in an internet outage for users to continue accessing their facility. The Pod has a direct internet connection then uses a wireless method to communicate with nodes. A single facility can have multiple Pods, but each Pod can only be linked to one facility. - -*Note: Versioning follows the [Semantic Versioning 2.0.0 standard](https://semver.org/)* - -## Installation - -OpenPod is designed to communicate with hardware over a local mesh network using [XBee module](https://www.digi.com/products/embedded-systems/digi-xbee/rf-modules/2-4-ghz-rf-modules/xbee3-zigbee-3). While OpenPod can be installed on any computer running Ubuntu, most installations are done on a [Raspberry Pi](https://www.raspberrypi.com/products/raspberry-pi-4-model-b/). To prepare a Raspbery Pi for OpenPod checkout the [Pi Setup Guide](docs/pi_setup.md). - -For convenience, an installation script as been provided that will download OpenPod and make the necessary system changes to get OpenPod running. This script can be run from the command line using the following command: - -```bash -sudo wget -qO- https://openpod.recursion.space | bash /dev/stdin [options] [arguments] -``` - -| Option Flag | Description | Example | -|:-----------:|---------------------------|------------------------------------------------------------------------------------| -| -h | Help | sudo wget -qO- openpod.recursion.space \| bash /dev/stdin -h | -| -d | Enable Debug Installation | sudo wget -qO- openpod.recursion.space \| bash /dev/stdin -d | -| -b | Alternative Branch | sudo wget -qO- openpod.recursion.space \| bash /dev/stdin -b dev-release | -| -u | Set Custom URL endpoint | sudo wget -qO- openpod.recursion.space \| bash /dev/stdin -u stage.recursion.space | - -\*No supported arguments are currently supported. - -# Software Design - -The embedded code manages the communication between the API server and the node communication, acting as a middleman. There needs to be minimal factory configuration to make deployable units quickly. - -1) Assume the Pod is plug and play internet connection. (DHCP) -2) Pods can self-generate ID and register it with the central system. -3) Users enter an ID to link the physical Pod with the web interface. - -Pod communicates to the web API via hook notifications, and a return confirmation is sent back. - -- Records Door/Equipment access -- Registers new nodes and Pods -- Pulls info dumps - -The web interface communicates with the Pod via MQTT. - -- Alerts for new users -- User updates -- Remote control (door unlock) - -Each Pod is on its MQTT topic, and the topic is created once the Pod registers with the central system. - -# Development - -Pod software development is done on a DigitalOcean hosted server. It is then transferred to the physical hardware running the system and tested before going into production. - -# MQTT - -The MQTT protocol is used for server to Pod communication, allowing the Pod to listen in real-time for incoming instruction. - -Communication to Pods is accomplished from quick commands represented by a hexadecimal number. - -| Command Number | Command | -|----------------|-----------------------| -| AA (170) | Facility ID Available | -| BA (186) | Pull User Dump | -| CA (202) | Update Available | -| DA(218) | Time Zone Change | -| | | -| FA (250) | Zip & Send Logs | - -# Updates - -The end-user triggers updates via the web server to run them with minimal interruption. An MQTT message is sent to the Pod to initiate the updating process. The update is a zip file downloaded from the [server](https://stackoverflow.com/questions/38697529/how-to-return-generated-file-download-with-django-rest-framework). - -1) Download zip file from server and store in the update folder. -2) Unzip contents of a zip file -3) Run Launcher - -First, commit and push the latest changes to git to prepare an update. Using WinSCP, download the 0_1_0 folder with the newest code. Rename the downloaded folder to match the latest version number. Zip the folder's CONTENTS, so there is no second directory folder with a matching name inside the zip file folder. Clean up all extra copies of the old folder. Update the latest version name using the recursion admin panel, then update their devices. - -# Logging - -The Recursion System uses several logging points to be used both for troubleshooting as well as security and auditing. There are two log files created for the Pod, RecursionLog.log, and TransactionLog.log; each one is used as follows: - -**RecursionLog.log** is used to record system events. - -- Coming online -- Updating -- New Nodes being added -- Nodes coming online - -**System.Snapshot** file contains a JSON summary of the Pod for debugging purposes. The information available in this file must also be readily accessible from the server for troubleshooting purposes. - -```json -{ - "local_ip": "xx.xx.xx.xx", -} -``` - -## Repository Directory Structure - -```default -. -├── .github # CI/CD using GitHub Actions and other functions. -├── tests # Contains unit testing files. -└── openpod # Contains OpenPod functionality. - └── modules # Independent core functions. -``` - -## Deployed Directory Structure - -```default -OpenPod/ -|--------- Version A/ - |--------- HUB.py - |--------- HUB_Launcher.py - |--------- HUB_Updater.py - |----------settings.py - |--------- modules/ - -|--------- logs/ - |--------- RecursionLog.log - |--------- TransactionLog.log - -|--------- data/ - |-------- dump.json - |-------- owners.json - |-------- nodes.json - |---------permissions.json - -|--- system.json - -``` - -# Community and Contributing - -OpenPod is developed by [Recursion.Space](https://recursion.space/) and by users like you. We welcome both pull requests and issues on [GitHub](https://github.com/RecursionSpace/OpenPod). Bug fixes and new protocols are encouraged. - - diff --git a/docs/installation.md b/docs/installation.md deleted file mode 100644 index 28c3843..0000000 --- a/docs/installation.md +++ /dev/null @@ -1,21 +0,0 @@ -## Installer - - -## Initialization - -Pod uses an initializer or “launcher” to configure the system before running the main code. The launcher performs the following tasks before running the main program: - -- Update & Upgrade the OS -- [Sync Clock](https://vitux.com/keep-your-clock-sync-with-internet-time-servers-in-ubuntu/) -- Installs program required packages -- Check for required files, create them if missing -- Configure start on boot file -- Creates serial if needed -- Registers the Pod with the API server -- Initializes the main program under [/opt/](https://unix.stackexchange.com/questions/11544/what-is-the-difference-between-opt-and-usr-local) - -To launch the software, when the device boots, the launcher configures the system first to run the software in the [background](https://janakiev.com/blog/python-background/) and then creates a script that executes the launch on [startup](https://askubuntu.com/questions/817011/run-python-script-on-os-boot). Working version with a [cronjob](https://www.linuxbabe.com/linux-server/how-to-enable-etcrc-local-with-systemd). - -```bash -(@reboot (cd /home/ubuntu/RecursionHub && sudo python3 HUB_Launcher.py &)) -``` diff --git a/docs/pi_setup.md b/docs/pi_setup.md deleted file mode 100644 index 5e64978..0000000 --- a/docs/pi_setup.md +++ /dev/null @@ -1,27 +0,0 @@ -# Raspberry Pi Setup - -This is a guide to help you configure a Raspberry Pi for OpenPod from scratch. - -## Compatibility - -| Pi Model | Compatible | -|:--------:|:----------:| -| Pi 3 | Yes | -| Pi 4 | Yes | - -## Micro SD Card - -[Recommended Micro SD Card](https://www.samsung.com/us/computing/memory-storage/memory-cards/evo-plus-microsdxc-memory-card-64gb-mb-mc64ha-am/) - -Several considerations were taken into account when selecting the SD card, ultimately, longer-term reliability became the driving factor. The only compromise being made is a slightly reduced boot time which seemed negligible for a device intended always to be on. - -Key metrics for SD card comparison: - -- Application performance class (A1 & A2), A1 preferred -- [SD Card Comparison](https://techreport.com/forums/viewtopic.php?t=121152) - -## Operating System (OS) - -[32-bit for Pi3 and 64-bit for 4+](https://ubuntu.com/blog/how-to-install-ubuntu-with-the-new-raspberry-pi-imager) - -Each Pod is installed with the latest version of Ubuntu, and the systems are tested daily for compatibility with the latest releases. An image of the OS with the code ready for deployment can make a quick installation. diff --git a/docs/pod_design.md b/docs/pod_design.md deleted file mode 100644 index 1d4f3d6..0000000 --- a/docs/pod_design.md +++ /dev/null @@ -1,37 +0,0 @@ -## Electronics - -Currently the system makes use of the GPIO on the Raspberry Pi. These pins are limited to an output of 3.3v at 16mA. - -### Visual Indicators - -The Pod has two LED indicators on it, a green and red. The [green LED](https://www.digikey.com/en/products/detail/lumex-opto-components-inc/SSI-LXH072GD/144732) is 2.2v at 25mA and will require a 68.75ohm (68ohm standard) resistor while the [red LED](https://www.digikey.com/en/products/detail/lumex-opto-components-inc/SSI-LXH072ID/144729) one is 2v at 30mA and will require a 81.25 ohm (82 ohm standard) resistor. Both resistors were [calculated](https://ohmslawcalculator.com/led-resistor-calculator) with the current caped at 16mA. - -A combination of the LEDs blinking or solid on/off in a [controlled pattern](https://en.wikipedia.org/wiki/International_Blinking_Pattern_Interpretation) provides information to the users. - -| | Green - I/O “Simulating XBee activity, should really be connected to one of the XBee pins” | Red - Overall Status | -|:-----------------------------------------------------------------------------------------------------------------------------:|:------------------------------------------------------------------------------------------:|:--------------------:| -| **Initializing** - Program called, not ready to receive requests. | Off | On | -| No Ethernet: The HUB has not received an IP address and is set to 127.0.0.1 | N/A | 4 Hz | -| Ethernet, No Internet: | N/A | 2 Hz | -| Internet, Recursion Unreachable: | N/A | 1 Hz | -| Recursion.Space Reachable: | N/A | Off | -| **Unregistered** - All network and hardware tests passed, ready to receive XBee requests, Pod IS NOT registered with a space. | 1 Hz | N/A | -| **Ready** - All network and hardware tests passed, ready to receive XBee requests, and Pod IS registered with a space. | On | N/A | -| XBee I/O Data: Anytime the XBee receives or sends data. | 4 Hz | N/A | -| Fatal Error: - USB XBee Unreachable | 2 Hz Alternating | 2 Hz Alternating | - -## Assembly - -### Main Enclosure - -1) Remove any stickers, labels, and decals from the enclosure, then remove the two screws to temporarily detach the door. (Goo Gone followed by rubbing alcohol, helps remove any residual sticker residue. -2) Using the top flange, trace the opening and then cut using a jigsaw and a fine wood bit attachment. The screw holes can be oversized with an 11/64 drill bit. -3) Cut off the top stands using an oscillating saw and then use a side cutter to clean any repairing plastic off. -4) Prepare part number HB-H-I-EBF/C with the threaded inserts. -5) Mount the flanges to the enclosure and place them in the 4 M4 philips mounting screws. The enclosure is now complete and can be set off to the side. - -### Processing Unit - -Prepare electronic assemblies, including power connectors, LEDs, and reset buttons. -Insert threads into the various housing components. -Insert the completed electronic assemblies, such as the power outlet, LEDs and ethernet jack. diff --git a/tests/__init__.py b/tests/__init__.py deleted file mode 100644 index dd48f90..0000000 --- a/tests/__init__.py +++ /dev/null @@ -1 +0,0 @@ -''' Required for tests to see folder and perform tests. ''' diff --git a/tests/modules/__init__.py b/tests/modules/__init__.py deleted file mode 100644 index 224cd58..0000000 --- a/tests/modules/__init__.py +++ /dev/null @@ -1 +0,0 @@ -''' Required for tests to recognize folder and perform tests. ''' diff --git a/tests/modules/test_api.py b/tests/modules/test_api.py deleted file mode 100644 index 1b790e8..0000000 --- a/tests/modules/test_api.py +++ /dev/null @@ -1,107 +0,0 @@ -''' Tests for rec_api.py ''' - -import sys -import unittest - -from io import StringIO -from unittest.mock import patch - -from modules import rec_api - -sys.path.insert(0, "0_1_0/") - - -class TestAPI(unittest.TestCase): - '''Collection of tests for the api module''' - - def test_pull_data_dump(self): - '''Test the pull_data_dump function''' - - system_json = StringIO("""{ - "serial": "536780dfe639468e8e23fc568006950d", - "timezone": "America/New_York", - "CurrentVersion": "0_0_0", - "HUBid": 40, - "Token": "5a12ff36eed2f0647a48af62e635eb8cfd4c5979", - "facility": "3b9fdc97-9649-4c80-8b48-10df647bd032" - }""") - - testdata2 = StringIO("""{ - "serial": "536780dfe639468e8e23fc568006950d", - "timezone": "America/New_York", - "CurrentVersion": "0_0_0", - "HUBid": 40, - "Token": "5a12ff36eed2f0647a48af62e635eb8cfd4c5979", - "facility": "3b9fdc97-9649-4c80-8b48-10df647bd032" - }""") - - testdata3 = StringIO("""{ - "serial": "536780dfe639468e8e23fc568006950d", - "timezone": "America/New_York", - "CurrentVersion": "0_0_0", - "HUBid": 40, - "Token": "5a12ff36eed2f0647a48af62e635eb8cfd4c5979", - "facility": "3b9fdc97-9649-4c80-8b48-10df647bd032" - }""") - - testdata4 = StringIO("""{ - "serial": "536780dfe639468e8e23fc568006950d", - "timezone": "America/New_York", - "CurrentVersion": "0_0_0", - "HUBid": 40, - "Token": "5a12ff36eed2f0647a48af62e635eb8cfd4c5979", - "facility": "3b9fdc97-9649-4c80-8b48-10df647bd032" - }""") - - testdata5 = StringIO("""{ - "serial": "536780dfe639468e8e23fc568006950d", - "timezone": "America/New_York", - "CurrentVersion": "0_0_0", - "HUBid": 40, - "Token": "5a12ff36eed2f0647a48af62e635eb8cfd4c5979", - "facility": "3b9fdc97-9649-4c80-8b48-10df647bd032" - }""") - - testreturn = [{ - "cardNumber": "3132323637373936", - "access_group": 7, - "phone_number": "2403426671", - "address": "123 America Ln", - "city": "USA City", - "state": "PA", - "zip_code": " 1234567", - "username": "GenericMember2", - "first_name": "Gener", - "last_name": "Mem", - "email": "member@email.com", - "restricted_nodes": [] - }, { - "cardNumber": "33", - "access_group": 5, - "phone_number": "2403426671", - "address": "", - "city": "", - "state": "", - "zip_code": "", - "username": "GenericMember3", - "first_name": "Generic", - "last_name": "Member3", - "email": "generic@email.com", - "restricted_nodes": ["123", "shdfhethetbe"] - }] - - with patch('modules.rec_api.op_gpio.open') as mock_system: - mock_system.side_effect = [system_json] - - with patch('modules.rec_api.open') as mock_open: - mock_open.side_effect = [system_json, testdata2, testdata3, testdata4, testdata5] - - with patch("modules.rec_api.requests.get") as mocked_requests: - mocked_requests.return_value.json.return_value = testreturn - self.assertTrue(rec_api.pull_data_dump()) - mock_open.assert_called() - mocked_requests.assert_called() - - -if __name__ == '__main__': - unittest.main() diff --git a/tests/modules/test_lan.py b/tests/modules/test_lan.py deleted file mode 100644 index 495ae19..0000000 --- a/tests/modules/test_lan.py +++ /dev/null @@ -1,107 +0,0 @@ -''' Unit testing for rec_lan.py ''' - -import sys -import unittest - -from unittest.mock import patch - -from modules import rec_lan - -sys.path.insert(0, "openpod/") - - -class TestLan(unittest.TestCase): - ''' Tests for the lan module ''' - - def test_monitor_network(self): - ''' - Confirm that the network monitor is running. - ''' - with patch('modules.rec_lan.test_network') as mocked_test_network: - mocked_test_network.return_value = 0 - - with patch('modules.rec_lan.threading') as mocked_threading: - self.assertTrue(rec_lan.monitor_network()) - mocked_threading.Timer.assert_called() - - def test_test_network(self): - ''' - Confirms all tests are called properly. - ''' - with patch('modules.rec_lan.networked') as mocked_networked: - mocked_networked.return_value = False - self.assertEqual(rec_lan.test_network(), 0) - - mocked_networked.return_value = True - self.assertNotEqual(rec_lan.test_network(), 0) - - with patch('modules.rec_lan.internet_on') as mocked_internet_on: - mocked_internet_on.return_value = False - self.assertEqual(rec_lan.test_network(), 1) - - mocked_internet_on.return_value = True - self.assertNotEqual(rec_lan.test_network(), 1) - - with patch('modules.rec_lan.recursion_connection') as mocked_recursion_connection: - mocked_recursion_connection.return_value = False - self.assertEqual(rec_lan.test_network(), 2) - - mocked_recursion_connection.return_value = True - self.assertEqual(rec_lan.test_network(), 3) - - self.assertTrue(mocked_networked.called) - self.assertTrue(mocked_internet_on.called) - self.assertTrue(mocked_recursion_connection.called) - - def test_networked(self): - ''' - Confirm that the network is active. - ''' - with patch('modules.rec_lan.get_ip') as mocked_get_ip: - mocked_get_ip.return_value = ("127.0.0.1", "127.0.0.1") - self.assertFalse(rec_lan.networked()) - - mocked_get_ip.return_value = ("192.168.1.1", "192.168.1.1") - self.assertTrue(rec_lan.networked()) - - def test_internet_on(self): - ''' - Confirms active internet connection. - ''' - self.assertTrue(rec_lan.internet_on()) - - with patch('modules.rec_lan.requests.get') as mocked_requests: - mocked_requests.return_value.status_code = None - self.assertFalse(rec_lan.internet_on()) - - def test_recursion_connection(self): - ''' - Confirms that the recursion server is reachable. - ''' - self.assertTrue(rec_lan.recursion_connection()) - - with patch('modules.rec_lan.requests.get') as mocked_requests: - mocked_requests.return_value.status_code = None - self.assertFalse(rec_lan.internet_on()) - - def test_get_ip(self): - ''' - Verify that the ip address is obtained. - ''' - with patch('modules.rec_lan.internet_on') as mocked_internet_on: - mocked_internet_on.return_value = True - - with patch('modules.rec_lan.requests.get') as mocked_requests: - mocked_requests.return_value.text = "0.0.0.0" - public_ip, local_ip = rec_lan.get_ip() - - self.assertEqual((rec_lan.get_ip())[0], '0.0.0.0') - self.assertEqual(public_ip, '0.0.0.0') - self.assertNotEqual(local_ip, '127.0.0.1') - - mocked_internet_on.return_value = False - self.assertNotEqual((rec_lan.get_ip())[0], '0.0.0.0') - - -if __name__ == '__main__': - unittest.main() diff --git a/tests/modules/test_log.py b/tests/modules/test_log.py deleted file mode 100644 index b3005e6..0000000 --- a/tests/modules/test_log.py +++ /dev/null @@ -1,31 +0,0 @@ -''' Tests for rec_log.py ''' - -# import sys -# import logging -# import unittest - -# #from modules import rec_lan, rec_api, rec_xbee, rec_log - -# sys.path.insert(0, "openpod/") - -# class Testlog(unittest.TestCase): -# ''' Tests for the log module ''' - -# def __init__(self, *args, **kwargs): -# self.reset() -# logging.Handler.__init__(self, *args, **kwargs) - -# def emit(self, record): -# self.messages[record.levelname.lower()].append(record.getMessage()) - -# def reset(self): -# self.messages = { -# 'debug': [], -# 'info': [], -# 'warning': [], -# 'error': [], -# 'critical': [], -# } - -# if __name__ == '__main__': -# unittest.main() diff --git a/tests/modules/test_lookup.py b/tests/modules/test_lookup.py deleted file mode 100644 index 27932ee..0000000 --- a/tests/modules/test_lookup.py +++ /dev/null @@ -1,383 +0,0 @@ -''' -Tests for the lookup module. -''' - -import sys -import unittest - -from io import StringIO -from unittest.mock import patch - -from modules import rec_lookup - -sys.path.insert(0, "0_1_0/") - - -class TestLookup(unittest.TestCase): - '''Lokup Tests''' - - def setUp(self): - self.nodes_json = StringIO( - '''[ - { - "id": 1, - "name": "Test Node", - "mac": "0011223344556677", - "tool": false, - "door": false, - "qr_toggle": false, - "hub": 1, - "facility": "7659e76b-470c-4d5f-bff4-fcc120f08848", - "qr_code": null - } - ]''' - ) - - self.nodes_alternative_json = StringIO( - '''[ - { - "id": 1, - "name": "Test Node", - "mac": "0000000000000000", - "tool": false, - "door": false, - "qr_toggle": false, - "hub": 1, - "facility": "7659e76b-470c-4d5f-bff4-fcc120f08848", - "qr_code": null - } - ]''' - ) - - self.nodes_empty_json = StringIO('') - - def test_count_matching_mac(self): - ''' - Confirms MAC conflection is counted correctly. - ''' - count_result = rec_lookup.count_matching_mac("12345678") - self.assertEqual(count_result, 0) - - self.assertEqual( - rec_lookup.count_matching_mac("0011223344556677"), - 0 - ) - - with patch('modules.rec_lookup.open') as mock_open: - mock_open.return_value = self.nodes_json - - count_result = rec_lookup.count_matching_mac("0011223344556677") - - mock_open.assert_called() - self.assertEqual(count_result, 1) - - with patch('modules.rec_lookup.open') as mock_open: - mock_open.return_value = self.nodes_alternative_json - - count_result = rec_lookup.count_matching_mac("0011223344556677") - - mock_open.assert_called() - self.assertEqual(count_result, 0) - - with patch('modules.rec_lookup.open') as mock_open: - mock_open.return_value = self.nodes_empty_json - - self.assertEqual( - rec_lookup.count_matching_mac("0011223344556677"), - 0 - ) - - mock_open.assert_called() - -class TestLookUpAccessRequest(unittest.TestCase): # pylint: disable=R0904 - '''Access Request Tests''' - - def setUp(self): - ''' - Collection of JSON used for testing. - ''' - self.system_json = StringIO( - '''{ - "serial": "536780dfe639468e8e23fc568006950d", - "timezone": "America/New_York", - "CurrentVersion": "0_0_0", - "HUBid": 40, - "Token": "5a12ff36eed2f0647a48af62e635eb8cfd4c5979", - "facility": "3b9fdc97-9649-4c80-8b48-10df647bd032" - }''' - ) - - self.nodes_json = StringIO( - '''[ - { - "id": 1, - "name": "Test Node", - "mac": "0011223344556677", - "tool": false, - "door": false, - "qr_toggle": false, - "hub": 1, - "facility": "7659e76b-470c-4d5f-bff4-fcc120f08848", - "qr_code": null - } - ]''' - ) - - self.members_json = StringIO( - '''[ - { - "cardNumber": "313233343536373839", - "access_group": 123, - "phone_number": "1234567890", - "address": "1331 12th ave", - "city": "Altoona", - "state": "PA", - "zip_code": "16601", - "username": "BestName", - "first_name": "John", - "last_name": "Doe", - "email": "email@email.com", - "restricted_nodes": [0,9,8] - } - ]''' - ) - - self.owners_json = StringIO( - '''[ - { - "facility": "3b9fdc97-9649-4c80-8b48-10df647bd032", - "cardNumber": "30393837363534333231", - "phone_number": null, - "address": null, - "city": null, - "state": null, - "zip_code": null, - "username": "OwnerUserName", - "first_name": "Jim", - "last_name": "John", - "email": "email@email.com" - } - ]''' - ) - - self.permissions_json = StringIO( - '''[ - { - "id": 1, - "name": "General Access", - "startTime": "20:20:20", - "endTime": "23:23:23", - "monday": true, - "tuesday": true, - "wednesday": true, - "thursday": true, - "friday": true, - "saturday": true, - "sunday": true, - "twenty_four_seven": false, - "default_fallback": true, - "facility": "3b9fdc97-9649-4c80-8b48-10df647bd032", - "allowedNodes": [1, 4, 6] - } - ]''' - ) - - # ----------------------------------- _alt ----------------------------------- # - - self.system_json = StringIO( - '''{ - "serial": "536780dfe639468e8e23fc568006950d", - "timezone": "America/New_York", - "CurrentVersion": "0_0_0", - "HUBid": 40, - "Token": "5a12ff36eed2f0647a48af62e635eb8cfd4c5979", - "facility": "3b9fdc97-9649-4c80-8b48-10df647bd032" - }''' - ) - - self.nodes_json_alt = StringIO( - '''[ - { - "id": 1, - "name": "Test Node", - "mac": "0011223344556677", - "tool": false, - "door": false, - "qr_toggle": false, - "hub": 1, - "facility": "7659e76b-470c-4d5f-bff4-fcc120f08848", - "qr_code": null - } - ]''' - ) - - self.members_json_alt = StringIO( - '''[ - { - "cardNumber": "313233343536373839", - "access_group": 123, - "phone_number": "1234567890", - "address": "1331 12th ave", - "city": "Altoona", - "state": "PA", - "zip_code": "16601", - "username": "BestName", - "first_name": "John", - "last_name": "Doe", - "email": "email@email.com", - "restricted_nodes": [0,9,8] - } - ]''' - ) - - self.owners_json_alt = StringIO( - '''[ - { - "facility": "3b9fdc97-9649-4c80-8b48-10df647bd032", - "cardNumber": "30393837363534333231", - "phone_number": null, - "address": null, - "city": null, - "state": null, - "zip_code": null, - "username": "OwnerUserName", - "first_name": "Jim", - "last_name": "John", - "email": "email@email.com" - } - ]''' - ) - - self.permissions_json_alt = StringIO( - '''[ - { - "id": 1, - "name": "General Access", - "startTime": "20:20:20", - "endTime": "23:23:23", - "monday": true, - "tuesday": true, - "wednesday": true, - "thursday": true, - "friday": true, - "saturday": true, - "sunday": true, - "twenty_four_seven": false, - "default_fallback": true, - "facility": "3b9fdc97-9649-4c80-8b48-10df647bd032", - "allowedNodes": [1, 4, 6] - } - ]''' - ) - - def test_files_opened(self): - ''' - Confirms that all the files are correctly opened and read. - ''' - with patch('modules.rec_lookup.open') as mock_open: - mock_open.side_effect = [ - self.system_json, - self.nodes_json, # Opened from conversion function. - self.owners_json, # Opened from owner check function. - self.members_json, # Opened from get_details function. - self.permissions_json, # Opened from get_group_details function. - ] - - self.assertAlmostEqual(rec_lookup.access_request(313131, '0011223344556677'), 2) - mock_open.assert_called() - - def test_mac_to_id(self): - ''' - Confirms that the mac address is converted to the node id. - ''' - with patch('modules.rec_lookup.open') as mock_open: - mock_open.return_value = self.nodes_json - - node_id = rec_lookup.mac_to_id('0011223344556677') - mock_open.assert_called() - self.assertEqual(node_id, 1) - - with patch('modules.rec_lookup.open') as mock_open: - mock_open.return_value = self.nodes_json_alt - - node_id = rec_lookup.mac_to_id('9911223344556677') - mock_open.assert_called() - self.assertEqual(node_id, '9911223344556677') - - def test_is_owner(self): - ''' - Confirms that the owner check function returns the correct value. - ''' - with patch('modules.rec_lookup.open') as mock_open: - mock_open.return_value = self.owners_json - - owner = rec_lookup.is_owner('30393837363534333231') - mock_open.assert_called() - self.assertTrue(owner) - - with patch('modules.rec_lookup.open') as mock_open: - mock_open.return_value = self.owners_json_alt - - owner = rec_lookup.is_owner('99393837363534333231') - mock_open.assert_called() - self.assertFalse(owner) - - def test_get_details(self): - ''' - Verifies that the correct details are returned. - ''' - with patch('modules.rec_lookup.open') as mock_open: - mock_open.return_value = self.members_json - - user = rec_lookup.get_details('313233343536373839') - mock_open.assert_called() - self.assertTrue(user['found']) - - with patch('modules.rec_lookup.open') as mock_open: - mock_open.return_value = self.members_json_alt - - user = rec_lookup.get_details('993233343536373839') - mock_open.assert_called() - self.assertFalse(user['found']) - - def test_get_group_details(self): - ''' - Verifies that the correct details are returned. - ''' - with patch('modules.rec_lookup.open') as mock_open: - mock_open.return_value = self.permissions_json - - group = rec_lookup.get_group_details(1) - mock_open.assert_called() - self.assertTrue(group['found']) - - with patch('modules.rec_lookup.open') as mock_open: - mock_open.return_value = self.permissions_json_alt - - group = rec_lookup.get_group_details(69) - mock_open.assert_called() - self.assertFalse(group['found']) - - def test_access_request_combinations(self): - ''' - Checks that the access request function returns the correct values. - ''' - with patch('modules.rec_lookup.open') as mock_open: - mock_open.side_effect = [ - self.system_json, - self.nodes_json, # Opened from conversion function. - self.owners_json, # Opened from owner check function. - self.members_json, # Opened from get_details function. - self.permissions_json, # Opened from get_group_details function. - ] - - self.assertEqual( - rec_lookup.access_request(313131, '0011223344556677'), - 2 - ) - - mock_open.assert_called() - - -if __name__ == '__main__': - unittest.main() diff --git a/tests/modules/test_mqtt.py b/tests/modules/test_mqtt.py deleted file mode 100644 index 813860b..0000000 --- a/tests/modules/test_mqtt.py +++ /dev/null @@ -1,49 +0,0 @@ -''' -Tests MQTT functionality. -''' -import unittest -from unittest.mock import patch - -from modules import rec_mqtt - - -class DummyMessage: # pylint: disable=R0903 - '''Creating a mock response''' - - def __init__(self, command): - self.payload=command - - topic='test_hub' - -class TestOnMessage(unittest.TestCase): - '''unit tests for the MQTT module''' - - def test_function_calls(self): - ''' - Checks that the function calls are correct. - ''' - with patch('modules.rec_mqtt.rec_api.link_hub') as mock_link: - self.assertTrue( - rec_mqtt.on_message('test_client', None, DummyMessage('170')) - ) - mock_link.assert_called() - - with patch('modules.rec_mqtt.rec_api.pull_data_dump') as mock_pull: - rec_mqtt.on_message('test_client', None, DummyMessage('186')) - mock_pull.assert_called() - - with patch('modules.rec_mqtt.mqtt_start_update') as mock_update: - rec_mqtt.on_message('test_client', None, DummyMessage('202')) - mock_update.assert_called() - - with patch('modules.rec_mqtt.rec_api.update_time_zone') as mock_timezone: - rec_mqtt.on_message('test_client', None, DummyMessage('218')) - mock_timezone.assert_called() - - with patch('modules.rec_mqtt.mqtt_restart_system') as mock_restart: - rec_mqtt.on_message('test_client', None, DummyMessage('234')) - mock_restart.assert_called() - - with patch('modules.rec_mqtt.zip_send') as mock_zip: - rec_mqtt.on_message('test_client', None, DummyMessage('250')) - mock_zip.assert_called() diff --git a/tests/modules/test_xbee.py b/tests/modules/test_xbee.py deleted file mode 100644 index 51d8fb5..0000000 --- a/tests/modules/test_xbee.py +++ /dev/null @@ -1,40 +0,0 @@ -# import os -# import pty -# import json -# import logging -# import unittest - -# from io import StringIO -# from unittest.mock import patch, mock_open, Mock - -# import sys -# sys.path.insert(0, "0_1_0/") - -# from modules.rec_xbee import configure_xbee - -# class TestXbee(unittest.TestCase): -# def test_confXBee(self): -# systemJSON = StringIO("""{ -# "serial" : "536780dfe639468e8e23fc568006950d", -# "XBEE_KY" : "11111111111111111111111111111111", -# "timezone": "America/New_York", -# "CurrentVersion": "0_0_0", -# "HUBid": 40, -# "Token": "5a12ff36eed2f0647a48af62e635eb8cfd4c5979", -# "facility": "3b9fdc97-9649-4c80-8b48-10df647bd032" -# }""") - -# master, slave = pty.openpty() -# s_name = os.ttyname(slave) - -# with patch('modules.rec_xbee.open') as mock_open: -# mock_open.side_effect = [systemJSON] - -# with patch('modules.rec_xbee.serial.Serial.write') as mock_write: -# configure_xbee() #Call the function to run. -# mock_open.assert_called() #TEST - systemJSON was called - - - -# if __name__ == '__main__': -# unittest.main() diff --git a/tests/test_installer.py b/tests/test_installer.py deleted file mode 100644 index e69de29..0000000 diff --git a/tests/test_pod.py b/tests/test_pod.py deleted file mode 100644 index aaf5721..0000000 --- a/tests/test_pod.py +++ /dev/null @@ -1,21 +0,0 @@ -''' Unit test for pod.py ''' - -import sys -# import unittest - -sys.path.insert(0, "openpod/") - -# import hub - -# class TestHub(unittest.TestCase): -# ''' -# General tests for the hub.py file -# ''' -# def test_xbee_flag_set_true(self): -# ''' -# Check if the xbee flag is set to true. -# ''' -# global XBEE_FLAG -# XBEE_FLAG = False -# hub.incoming_xbee_data() -# self.assertTrue(XBEE_FLAG) diff --git a/tests/test_updater.py b/tests/test_updater.py deleted file mode 100644 index 145bcce..0000000 --- a/tests/test_updater.py +++ /dev/null @@ -1,42 +0,0 @@ -''' -Tests for update.py -''' - -import sys -import unittest - -from unittest.mock import patch - -import updater # OpenPod Updater - -sys.path.insert(0, "openpod/") - - -@unittest.skip("Refactored updater.py") -class TestUpdater(unittest.TestCase): - '''Collection of tests.''' - - def setUp(self): - - self.system_json = { - "serial": "536780dfe639468e8e23fc568006950d", - "timezone": "America/New_York", - "version": "0_0_0", - "HUBid": 40, - "Token": "5a12ff36eed2f0647a48af62e635eb8cfd4c5979", - "facility": "3b9fdc97-9649-4c80-8b48-10df647bd032" - } - - def test_get_current_versions(self): - ''' - Verify that the current versions are returned. - ''' - - with patch('updater.open') as mock_open: - mock_open.side_effect = [self.system_json, self.system_json] - self.assertEqual(updater.update_pod(), "0_0_0") - mock_open.assert_called() - - -if __name__ == '__main__': - unittest.main() From 925f49206b240528c150bbe8d29c46e5761b693e Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Mon, 23 Jan 2023 23:22:19 -0500 Subject: [PATCH 130/164] Update CD_dev-release.yml --- .github/workflows/CD_dev-release.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/CD_dev-release.yml b/.github/workflows/CD_dev-release.yml index 5fbc138..663d547 100644 --- a/.github/workflows/CD_dev-release.yml +++ b/.github/workflows/CD_dev-release.yml @@ -85,7 +85,7 @@ jobs: git fetch origin git checkout dev-release - git reset --hard origin/master-ci + git merge master-ci --no-edit rm -rf LICENSE rm -rf README.md From e6f70eafe8b7c14cb513a55ecdf1ee0761fcd656 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Mon, 23 Jan 2023 23:24:09 -0500 Subject: [PATCH 131/164] Update CD_dev-release.yml --- .github/workflows/CD_dev-release.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/CD_dev-release.yml b/.github/workflows/CD_dev-release.yml index 663d547..81588cd 100644 --- a/.github/workflows/CD_dev-release.yml +++ b/.github/workflows/CD_dev-release.yml @@ -85,7 +85,7 @@ jobs: git fetch origin git checkout dev-release - git merge master-ci --no-edit + git pull origin master-ci rm -rf LICENSE rm -rf README.md From 0a5e1c0cf92fc83b7df968852f927d18214ed6c1 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Mon, 23 Jan 2023 23:26:28 -0500 Subject: [PATCH 132/164] Update CD_dev-release.yml --- .github/workflows/CD_dev-release.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/CD_dev-release.yml b/.github/workflows/CD_dev-release.yml index 81588cd..0310a8c 100644 --- a/.github/workflows/CD_dev-release.yml +++ b/.github/workflows/CD_dev-release.yml @@ -85,7 +85,7 @@ jobs: git fetch origin git checkout dev-release - git pull origin master-ci + git pull origin master-ci --ff-only rm -rf LICENSE rm -rf README.md From 68dd577b9c3c329acff572a1c4342cfe7093dc02 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Mon, 23 Jan 2023 23:29:35 -0500 Subject: [PATCH 133/164] Update CD_dev-release.yml --- .github/workflows/CD_dev-release.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/CD_dev-release.yml b/.github/workflows/CD_dev-release.yml index 0310a8c..2a4f936 100644 --- a/.github/workflows/CD_dev-release.yml +++ b/.github/workflows/CD_dev-release.yml @@ -85,7 +85,7 @@ jobs: git fetch origin git checkout dev-release - git pull origin master-ci --ff-only + git pull origin master-ci --rebase rm -rf LICENSE rm -rf README.md From 7d51025edf67eb53bbf2dc9216347852cf1d2046 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Mon, 23 Jan 2023 23:36:32 -0500 Subject: [PATCH 134/164] Update CD_dev-release.yml --- .github/workflows/CD_dev-release.yml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/.github/workflows/CD_dev-release.yml b/.github/workflows/CD_dev-release.yml index 2a4f936..8377333 100644 --- a/.github/workflows/CD_dev-release.yml +++ b/.github/workflows/CD_dev-release.yml @@ -85,7 +85,8 @@ jobs: git fetch origin git checkout dev-release - git pull origin master-ci --rebase + git pull origin dev-release --rebase + git merge -X theirs master-ci rm -rf LICENSE rm -rf README.md From 7fdebb4a11b8c165e00fa05aace305cd7e99cdb6 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Mon, 23 Jan 2023 23:38:01 -0500 Subject: [PATCH 135/164] Update CD_dev-release.yml --- .github/workflows/CD_dev-release.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/CD_dev-release.yml b/.github/workflows/CD_dev-release.yml index 8377333..2e85b6e 100644 --- a/.github/workflows/CD_dev-release.yml +++ b/.github/workflows/CD_dev-release.yml @@ -86,7 +86,7 @@ jobs: git checkout dev-release git pull origin dev-release --rebase - git merge -X theirs master-ci + git merge -X theirs origin/master-ci rm -rf LICENSE rm -rf README.md From 048d31a180e8f81d39faddbf4bf94039220a7bc5 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Mon, 23 Jan 2023 23:40:11 -0500 Subject: [PATCH 136/164] Update CD_dev-release.yml --- .github/workflows/CD_dev-release.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/CD_dev-release.yml b/.github/workflows/CD_dev-release.yml index 2e85b6e..1c20eff 100644 --- a/.github/workflows/CD_dev-release.yml +++ b/.github/workflows/CD_dev-release.yml @@ -95,5 +95,5 @@ jobs: git add . - git commit -m "Sanitize dev-release" + git diff-index --quiet HEAD || git commit -m "Sanitize dev-release" git push origin dev-release --force From da91dfac6500aff0e75b4932a8471b28e9d2dfd1 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Mon, 23 Jan 2023 23:43:45 -0500 Subject: [PATCH 137/164] Update CD_dev-release.yml --- .github/workflows/CD_dev-release.yml | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/.github/workflows/CD_dev-release.yml b/.github/workflows/CD_dev-release.yml index 1c20eff..142b65e 100644 --- a/.github/workflows/CD_dev-release.yml +++ b/.github/workflows/CD_dev-release.yml @@ -44,12 +44,7 @@ jobs: target_branch: master-ci github_token: ${{ secrets.GITHUB_TOKEN }} - - name: Webhook - if: steps.wait-for-build.outputs.conclusion == 'success' - uses: distributhor/workflow-webhook@v3 - env: - webhook_url: "https://dev.recursion.space/webhooks/github/" - webhook_secret: "Y0uR5ecr3t" + # master-ci -> dev-release sanitize: @@ -78,6 +73,7 @@ jobs: fetch-depth: 0 - name: Remove Bloat + id: remove-bloat if: steps.wait-for-merge.outputs.conclusion == 'success' run: | git config user.name github-actions @@ -97,3 +93,10 @@ jobs: git diff-index --quiet HEAD || git commit -m "Sanitize dev-release" git push origin dev-release --force + + - name: Webhook + if: steps.remove-bloat.outputs.conclusion == 'success' + uses: distributhor/workflow-webhook@v3 + env: + webhook_url: "https://dev.recursion.space/webhooks/github/" + webhook_secret: "Y0uR5ecr3t" From c413ab89bf04faad7962d37bb614b7c1bd8e547a Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Mon, 23 Jan 2023 23:48:32 -0500 Subject: [PATCH 138/164] Update CD_dev-release.yml --- .github/workflows/CD_dev-release.yml | 1 - 1 file changed, 1 deletion(-) diff --git a/.github/workflows/CD_dev-release.yml b/.github/workflows/CD_dev-release.yml index 142b65e..4579b5e 100644 --- a/.github/workflows/CD_dev-release.yml +++ b/.github/workflows/CD_dev-release.yml @@ -95,7 +95,6 @@ jobs: git push origin dev-release --force - name: Webhook - if: steps.remove-bloat.outputs.conclusion == 'success' uses: distributhor/workflow-webhook@v3 env: webhook_url: "https://dev.recursion.space/webhooks/github/" From 3778814ba7a91996fc669bb12a248ca5a188f17d Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Mon, 23 Jan 2023 23:52:41 -0500 Subject: [PATCH 139/164] Update CD_dev-release.yml --- .github/workflows/CD_dev-release.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/workflows/CD_dev-release.yml b/.github/workflows/CD_dev-release.yml index 4579b5e..365c5d1 100644 --- a/.github/workflows/CD_dev-release.yml +++ b/.github/workflows/CD_dev-release.yml @@ -88,6 +88,7 @@ jobs: rm -rf README.md rm -rf docs rm -rf tests + rm -rf installer.sh git add . From bf9388ff07edc7a0d5f17cd5d16b6480011cf5e6 Mon Sep 17 00:00:00 2001 From: github-actions Date: Tue, 24 Jan 2023 04:53:30 +0000 Subject: [PATCH 140/164] Sanitize dev-release --- installer.sh | 268 --------------------------------------------------- 1 file changed, 268 deletions(-) delete mode 100644 installer.sh diff --git a/installer.sh b/installer.sh deleted file mode 100644 index 278049d..0000000 --- a/installer.sh +++ /dev/null @@ -1,268 +0,0 @@ -#!/bin/bash - -# Installer for OpenPod, for more information see https://github.com/blokbot-io/OpenBlok/blob/master/install.sh - - -# ---------------------------------------------------------------------------- # -# Disable Prompts # -# ---------------------------------------------------------------------------- # -export DEBIAN_FRONTEND=noninteractive -sed -i "/#\$nrconf{restart} = 'i';/s/.*/\$nrconf{restart} = 'a';/" /etc/needrestart/needrestart.conf - - -# ---------------------------------------------------------------------------- # -# Help # -# ---------------------------------------------------------------------------- # -Help() -{ - # Display Help - echo "OpenPod instalation script" - echo - echo "h Display this help" - echo "b Set custom branch for OpenPod" - echo "u Set custom URL for OpenPod" - echo "d Enable debug mode" -} - -# TO FIX: ARGS VS OPTS - -# ---------------------------------------------------------------------------- # -# Defaults # -# ---------------------------------------------------------------------------- # -DEBUG=flase # -d -REPO='https://github.com/RecursionSpace/OpenPod' - -# ---------------------------------------------------------------------------- # -# Options # -# ---------------------------------------------------------------------------- # -while getopts ":hbdu" flags; do - case "${flags}" in - h) # display Help - Help - exit;; - b) # Custom branch - BRANCH="${OPTARG}";; - d) # Enable debug mode - DEBUG=true ;; - u) # Custom URL endpoint - URL="${OPTARG}";; - \?) echo "Invalid option: -${OPTARG}" >&2; - exit 1 ;; - esac -done - -if [ $DEBUG ]; then - BRANCH='dev-release' - URL='dev.recursion.space' - API_URL='dev.api.recursion.space' -elif [ ! $DEBUG ]; then - BRANCH='release' - URL='recursion.space' - API_URL='api.recursion.space' -fi - -echo "Installing OpenPod with the following options:" -echo "Debug | $DEBUG" -echo "Repo | $REPO" -echo "Branch | $BRANCH" -echo "URL | $URL" -echo "API_URL | $API_URL" - -# -------------------------------- Verify Root ------------------------------- # -if [ "$EUID" -ne 0 ] - then echo "Please run as root with sudo." - exit -fi - -# --------------------------------- SSH User --------------------------------- # -echo "Verifying SSH user 'openpod'" - -if ! id -u "openpod" >/dev/null 2>&1; then - echo "Creating user 'openpod'" - useradd -m -s /bin/bash openpod - usermod -aG sudo openpod - mkdir -p ~openpod/.ssh/ && touch ~openpod/.ssh/authorized_keys - echo "openpod ALL=(ALL) NOPASSWD:ALL" | sudo tee -a /etc/sudoers > /dev/null -else - echo "User 'openpod' already exists, skipping..." - mkdir -p ~openpod/.ssh/ && touch ~openpod/.ssh/authorized_keys -fi - -# ---------------------------- Update System Time ---------------------------- # -sudo timedatectl set-timezone UTC - - -# ---------------------------------------------------------------------------- # -# Dependencies # -# ---------------------------------------------------------------------------- # - -# ------------------------------ build-essential ----------------------------- # -# Required to install RPi.GPIO - https://github.com/ynsta/steamcontroller/issues/42 -REQUIRED_PKG="build-essential" -PKG_OK=$(dpkg-query -W --showformat='${Status}\n' $REQUIRED_PKG|grep "install ok installed") -if [ "" = "$PKG_OK" ]; then - echo "No $REQUIRED_PKG. Setting up $REQUIRED_PKG..." - sudo apt-get install build-essential -y -else - echo "build-essential already installed, skipping..." -fi - -# ---------------------------------- chrony ---------------------------------- # -REQUIRED_PKG="chrony" -PKG_OK=$(dpkg-query -W --showformat='${Status}\n' $REQUIRED_PKG|grep "install ok installed") -if [ "" = "$PKG_OK" ]; then - echo "No $REQUIRED_PKG. Setting up $REQUIRED_PKG..." - sudo apt-get install chrony -y - sudo chronyd -q -else - echo "chrony already installed, skipping..." -fi - -# ----------------------------------- unzip ---------------------------------- # -REQUIRED_PKG="unzip" -PKG_OK=$(dpkg-query -W --showformat='${Status}\n' $REQUIRED_PKG|grep "install ok installed") -if [ "" = "$PKG_OK" ]; then - echo "No $REQUIRED_PKG. Setting up $REQUIRED_PKG..." - sudo apt-get install unzip -y -else - echo "unzip already installed, skipping..." -fi - -# ------------------------------------ jq ------------------------------------ # -REQUIRED_PKG="jq" -PKG_OK=$(dpkg-query -W --showformat='${Status}\n' $REQUIRED_PKG|grep "install ok installed") -if [ "" = "$PKG_OK" ]; then - echo "No $REQUIRED_PKG. Setting up $REQUIRED_PKG..." - sudo apt-get install jq -y -else - echo "jq already installed, skipping..." -fi - -# -------------------------------- Python 3.11 ------------------------------- # -pytohn_version=$(python -c 'import sys; print(".".join(map(str, sys.version_info[0:2])))') -if [ "$pytohn_version" != "3.11" ]; then - sudo apt install software-properties-common -y - yes '' | sudo add-apt-repository ppa:deadsnakes/ppa - sudo apt-get install python3.11 -y - sudo update-alternatives --install /usr/bin/python python /usr/bin/python3.11 1 - sudo update-alternatives --install /usr/bin/python python /usr/bin/python3.10 1 -else - echo "Python 3.11 already installed" -fi - -# -------------------------------- Python-Dev -------------------------------- # -REQUIRED_PKG="python3.11-dev" -PKG_OK=$(dpkg-query -W --showformat='${Status}\n' $REQUIRED_PKG|grep "install ok installed") -if [ "" = "$PKG_OK" ]; then - echo "No $REQUIRED_PKG. Setting up $REQUIRED_PKG..." - sudo apt-get install python3.11-dev -y -else - echo "python3.11-dev already installed, skipping..." -fi - -# ------------------------ Python Virtual Environment ------------------------ # -REQUIRED_PKG="python3.11-venv" -PKG_OK=$(dpkg-query -W --showformat='${Status}\n' $REQUIRED_PKG|grep "install ok installed") -if [ "" = "$PKG_OK" ]; then - echo "No $REQUIRED_PKG. Setting up $REQUIRED_PKG..." - sudo apt-get install python3.11-venv -y -else - echo "python3.11-venv already installed, skipping..." -fi - -# ---------------------------------------------------------------------------- # -# OpenPod # -# ---------------------------------------------------------------------------- # - -# -------------------------- Clear Previous Install -------------------------- # -sudo rm -rf /opt/OpenPod -sudo systemctl stop openpod.service - -# ------------------------------- Clone OpenPod ------------------------------ # -set -e # Exit when any command fails. -sudo mkdir -p /opt -cd /opt -sudo git clone --single-branch --branch $BRANCH "${REPO}".git -cd OpenPod - -# ----------------------------- Setup Enviroment ----------------------------- # -sudo python3.11 -m venv /opt/OpenPod/env -source /opt/OpenPod/env/bin/activate -pip install --no-input -U -r /opt/OpenPod/requirements.txt - -# ---------------------------- Create Directories ---------------------------- # -sudo mkdir -p /opt/OpenPod/logs -sudo mkdir -p /opt/OpenPod/data - -# ------------------------------- Create Files ------------------------------- # -# Log Location -sudo touch /opt/OpenPod/logs/RecursionLog.log -sudo touch /opt/OpenPod/logs/System.Snapshot - -# Data Location -sudo touch /opt/OpenPod/data/dump.json -sudo touch /opt/OpenPod/data/nodes.json -sudo touch /opt/OpenPod/data/owners.json -sudo touch /opt/OpenPod/data/permissions.json - -# -------------------------------- system.json ------------------------------- # -sudo touch /opt/OpenPod/system.json -serial_uuid=$(cat /proc/sys/kernel/random/uuid) -serial=${serial_uuid//-} -xbee_uuid=$(cat /proc/sys/kernel/random/uuid) -openpod_version=$(git rev-parse HEAD) -echo '{ - "uuid": "'"$serial_uuid"'", - "debug": '$DEBUG', - "serial": "'"$serial"'", - "timezone": "UTC", - "url": "'"$URL"'", - "api_url": "'"$API_URL"'", - "XBEE": { - "KY": "'"$xbee_uuid"'", - "OP": false - }, - "GPIO": { - "LED_IO": 23, - "LED_STAT": 17 - }, - "version": "'"$openpod_version"'", - "OpenPod": { - "repo": "'"$REPO"'", - "branch": "'"$BRANCH"'", - "commit": "'"$openpod_version"'" - } -}' > /opt/OpenPod/system.json - -# --------------------------- Create Version Folder -------------------------- # -mkdir -p /opt/OpenPod/versions/"$openpod_version" -sudo cp -a /opt/OpenPod/openpod/. /opt/OpenPod/versions/"$openpod_version"/ -sudo rm -rf /opt/OpenPod/openpod - -# --------------------------- Setup OpenPod Service -------------------------- # -cat < /etc/systemd/system/openpod.service -[Unit] -Description=OpenPod | Recursion.Space -After=network.target -StartLimitIntervalSec=0 - -[Service] -Type=simple -User=root -WorkingDirectory=/opt/OpenPod - -ExecStart = /bin/bash -c "exec /opt/OpenPod/env/bin/python3.11 \ - /opt/OpenPod/versions/$(jq '.version' /opt/OpenPod/system.json | xargs)/pod.py" - -Restart=always -RestartSec=10 - -[Install] -WantedBy=multi-user.target -EOF - -sudo systemctl enable --now openpod.service -sudo systemctl daemon-reload - -echo "- OpenPod is now installed -" -exit 0 From 05ddd34e219cb58f8e7ff4b943399dcea7bed41a Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Mon, 23 Jan 2023 23:56:29 -0500 Subject: [PATCH 141/164] Update updater.py --- openpod/updater.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/openpod/updater.py b/openpod/updater.py index c02401b..7986f03 100644 --- a/openpod/updater.py +++ b/openpod/updater.py @@ -47,7 +47,7 @@ def update_pod(): # Copy the files to the root directory. os.makedirs(f"/opt/OpenPod/versions/{latest_version['hash']}/", exist_ok=True) shutil.move( - f"OpenPod-{latest_version['hash']}/OpenPod-{latest_version['hash']}/OpenPod/", + f"OpenPod-{latest_version['hash']}/OpenPod/", f"/opt/OpenPod/versions/{latest_version['hash']}/" ) @@ -64,6 +64,6 @@ def update_pod(): if os.path.exists(f"{latest_version['hash']}.zip"): os.remove(f"{latest_version['hash']}.zip") - shutil.rmtree(f"{latest_version['hash']}/", ignore_errors=True) + shutil.rmtree(f"OpenPod-{latest_version['hash']}/", ignore_errors=True) sys.exit() # Force OpenPod to restart. From 89ee33f7265e7579d98ca8b52dd51a3132fd57f4 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Tue, 24 Jan 2023 00:02:47 -0500 Subject: [PATCH 142/164] Delete boot.sh --- boot.sh | 53 ----------------------------------------------------- 1 file changed, 53 deletions(-) delete mode 100644 boot.sh diff --git a/boot.sh b/boot.sh deleted file mode 100644 index d135a58..0000000 --- a/boot.sh +++ /dev/null @@ -1,53 +0,0 @@ -#!/bin/bash - -# Called by the @reboot conjob, will remain running at all times. -# Used to be located /opt/OpenPod/boot.sh -# Required chmod +x /opt/OpenPod/boot.sh -# Ran with a crontab entry @reboot /opt/OpenPod/boot.sh - - -while : -do - ethChecks=0 #Variable to store the number of delays waiting on an internet connection. - until [[ ( "$ethChecks" -gt 6 ) ]]; #Check to see if the eth adapter has started running yet or 6 attempts have been made. - do - if grep -q 'eth0' ifconfig -s; then - break - fi - (( ethChecks++ )) - sleep 10 - done - - cd /opt/OpenPod/ || exit #Makesure that everthing is being refrenced from the executable folder. - - if [ ! -f system.json ]; then - Token=false - else - Token=$(jq -e '.Token' system.json | xargs) #Read in needed API token, results in null of key is not set. - fi - - if [[ $Token != false && $Token != null ]]; then - CurrentVersion=$(jq '.CurrentVersion' system.json | xargs) #Gets the current version of the program that is available localy. - LatestVersion=$(curl --header "Authorization: Token $Token" https://api.recursion.space/v1/info/hub_version/) #Gets the latest version number available from the server via API. - LatestVersion=$(jq '.[0] | .current_production_version' <<< "$LatestVersion" | xargs) #Isolates just the version number. - if [[ $CurrentVersion != "$LatestVersion" ]]; then #Only pulls new version if it is diffrent from the current version on the system. - curl --header "Authorization: Token $Token" -O --remote-header-name https://recursion.space/updatehub/ #Downloads the latest zip file. - unzip "$LatestVersion".zip -d "$LatestVersion" #Unpacks the zip into a folder of the same name. - rm "$LatestVersion".zip #Clean up and delete the zip file. - python3 "$LatestVersion"/HUB_Launcher.py "$LatestVersion" #Run Launcher from new version. - else - python3 "$LatestVersion"/HUB_Launcher.py "$LatestVersion" - fi - else - echo "Token Not Found" - (cd /opt/OpenPod/ && sudo python3 0_1_0/HUB_Launcher.py) #Updated after last upgrade, the default will be latest version form git. - fi - - status=$? - - if [ $status -ne 0 ]; then - python3 "$CurrentVersion"/HUB_Launcher.py #Run the launcher program from the previous version as a fall back. - fi - - sleep 10 #Delay to prevent constant polling. -done From c0cf5ca2b9b27a2571fd8859c59b39ee449d8eb7 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Tue, 24 Jan 2023 00:14:31 -0500 Subject: [PATCH 143/164] Update updater.py --- openpod/updater.py | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/openpod/updater.py b/openpod/updater.py index 7986f03..e978570 100644 --- a/openpod/updater.py +++ b/openpod/updater.py @@ -47,12 +47,14 @@ def update_pod(): # Copy the files to the root directory. os.makedirs(f"/opt/OpenPod/versions/{latest_version['hash']}/", exist_ok=True) shutil.move( - f"OpenPod-{latest_version['hash']}/OpenPod/", + f"OpenPod-{latest_version['hash']}/openpod/", f"/opt/OpenPod/versions/{latest_version['hash']}/" ) except RuntimeError as err: exception_log.error("Unable to pull update with error: %s", err) + except FileNotFoundError as err: + exception_log.error("Unable to find file with error: %s", err) else: # Update the version number in the config file. @@ -66,4 +68,4 @@ def update_pod(): shutil.rmtree(f"OpenPod-{latest_version['hash']}/", ignore_errors=True) - sys.exit() # Force OpenPod to restart. + os.system("sudo systemctl restart openpod.service") From 474568f974c0c4440140a762300ed4c88f374434 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Tue, 24 Jan 2023 00:44:20 -0500 Subject: [PATCH 144/164] Update updater.py --- openpod/updater.py | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/openpod/updater.py b/openpod/updater.py index e978570..b8d4607 100644 --- a/openpod/updater.py +++ b/openpod/updater.py @@ -8,9 +8,9 @@ # Triggered by the user from the web interface to update the current version. import os -import sys import shutil import zipfile +from distutils.dir_util import copy_tree import urllib.request import requests @@ -46,9 +46,10 @@ def update_pod(): # Copy the files to the root directory. os.makedirs(f"/opt/OpenPod/versions/{latest_version['hash']}/", exist_ok=True) - shutil.move( - f"OpenPod-{latest_version['hash']}/openpod/", - f"/opt/OpenPod/versions/{latest_version['hash']}/" + + copy_tree( + f"OpenPod-{latest_version['hash']}/openpod", + f"/opt/OpenPod/versions/{latest_version['hash']}" ) except RuntimeError as err: @@ -58,8 +59,8 @@ def update_pod(): else: # Update the version number in the config file. - op_config.set('version', latest_version['version']) - op_config.set(['OpenPod', 'commit'], latest_version['hash']) + op_config.set_value('version', latest_version['version']) + op_config.set_nested_value(['OpenPod', 'commit'], latest_version['hash']) finally: # Clean Up From 639f5af511c9c4f51ec5d1f7f222f49b52f2af7e Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Tue, 24 Jan 2023 00:59:31 -0500 Subject: [PATCH 145/164] Update updater.py --- openpod/updater.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/openpod/updater.py b/openpod/updater.py index b8d4607..96ef436 100644 --- a/openpod/updater.py +++ b/openpod/updater.py @@ -10,7 +10,6 @@ import os import shutil import zipfile -from distutils.dir_util import copy_tree import urllib.request import requests @@ -47,9 +46,10 @@ def update_pod(): # Copy the files to the root directory. os.makedirs(f"/opt/OpenPod/versions/{latest_version['hash']}/", exist_ok=True) - copy_tree( + shutil.copytree( f"OpenPod-{latest_version['hash']}/openpod", - f"/opt/OpenPod/versions/{latest_version['hash']}" + f"/opt/OpenPod/versions/{latest_version['hash']}", + dirs_exist_ok=True ) except RuntimeError as err: From d5ce27c3c53609cbb1b0296e565890b10218292c Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Tue, 24 Jan 2023 01:07:53 -0500 Subject: [PATCH 146/164] Update installer.sh --- installer.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/installer.sh b/installer.sh index 278049d..4439e8e 100644 --- a/installer.sh +++ b/installer.sh @@ -251,8 +251,8 @@ Type=simple User=root WorkingDirectory=/opt/OpenPod -ExecStart = /bin/bash -c "exec /opt/OpenPod/env/bin/python3.11 \ - /opt/OpenPod/versions/$(jq '.version' /opt/OpenPod/system.json | xargs)/pod.py" +ExecStart = /bin/bash -c "exec /opt/OpenPod/env/bin/python3.11 \\ + /opt/OpenPod/versions/\$(jq '.version' /opt/OpenPod/system.json | xargs)/pod.py" Restart=always RestartSec=10 From eaa86fdcbd328a079ee3c8addc881fa5fe8f2bc7 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Tue, 24 Jan 2023 01:21:05 -0500 Subject: [PATCH 147/164] Delete launcher.py --- openpod/launcher.py | 105 -------------------------------------------- 1 file changed, 105 deletions(-) delete mode 100644 openpod/launcher.py diff --git a/openpod/launcher.py b/openpod/launcher.py deleted file mode 100644 index 64f90dd..0000000 --- a/openpod/launcher.py +++ /dev/null @@ -1,105 +0,0 @@ -#!/usr/bin/env python3 -''' -- Initiates the LED visuals - -- Installs new requirements as part of an update -(if needed, might want to have a septate process for this) - -- Checks for connection to server, -- If no connection is found then LED indication is set. -- Pulls configuration files from server. -- Launches main program -''' -# pylint: skip-file - -import sys -import json -import threading -import subprocess - - -# ---------------------------------------------------------------------------- # -# Check Requirements # -# ---------------------------------------------------------------------------- # -from .modules import rec_lan, rec_log -from .modules.rec_log import exception_log - -from .settings import LED_STAT -import settings - -if settings.Pi: - try: - from modules import rec_xbee # pylint: disable=C0412 - except ModuleNotFoundError as err: - exception_log.error("%s", err) - - try: - from modules import rec_gpio # pylint: disable=C0412 - except ModuleNotFoundError as err: - exception_log.error("%s", err) - - -# ---------------------------------------------------------------------------- # -# Program Start Visual # -# ---------------------------------------------------------------------------- # -if settings.Pi: - led_stat_thread = threading.Thread(target=rec_gpio.led_stat_thread) - led_stat_thread.start() - - rec_gpio.state(LED_STAT, 1, 0) - - -# ---------------------------------------------------------------------------- # -# Update version in system.json # -# ---------------------------------------------------------------------------- # -# Only updates if called by bash script and environmental variable was passed in. -if len(sys.argv) > 1: - with open("system.json", "r+", encoding="utf-8") as file: - data = json.load(file) - data.update({"CurrentVersion": sys.argv[1]}) - file.seek(0) - json.dump(data, file) - file.truncate() - - -# ---------------------------------------------------------------------------- # -# Logs Settings For Debugging # -# ---------------------------------------------------------------------------- # -try: - public, local = rec_lan.get_ip() - rec_log.snapshot(public, local) - -except RuntimeError as err: - exception_log.error("FATAL - Generate Snapshot - Error: %s", err) - -finally: - exception_log.debug("Launcher - Exiting Settings for Debugging") - -# ---------------------------------------------------------------------------- # -# XBee Configuration # -# ---------------------------------------------------------------------------- # -if settings.Pi: - with open('system.json', 'r', encoding="utf-8") as file: - system_data = json.load(file) - - if system_data.get('XBEE_OP', False) is False: - try: - rec_xbee.xbee_info() - except UnboundLocalError as err: - exception_log.error("Unable to capture XBee info - Error: %s", err) - -# ---------------------------------------------------------------------------- # -# Main HUB Program # -# ---------------------------------------------------------------------------- # -try: - import pod # pylint: disable=W0611 - -except RuntimeError as err: - exception_log.error("Could not start hub.py with error: %s", err) - - # Performs Seppuku to triger bash script and perform update pull (student). - subprocess.call(['pkill', '-f', 'hub.py']) - - # Performs Seppuku to triger bash script and perform update pull - # (can't kill the master before the student). - subprocess.call(['pkill', '-f', 'HUB_Launcher.py']) From 924034c416a6960b9ce98e3148dcdd8696428a53 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Tue, 24 Jan 2023 01:26:45 -0500 Subject: [PATCH 148/164] Update CD_dev-release.yml --- .github/workflows/CD_dev-release.yml | 3 +++ 1 file changed, 3 insertions(+) diff --git a/.github/workflows/CD_dev-release.yml b/.github/workflows/CD_dev-release.yml index 365c5d1..176e28f 100644 --- a/.github/workflows/CD_dev-release.yml +++ b/.github/workflows/CD_dev-release.yml @@ -82,6 +82,9 @@ jobs: git checkout dev-release git pull origin dev-release --rebase + + git checkout master-ci installer.sh + git merge -X theirs origin/master-ci rm -rf LICENSE From 93880009060939b9fd85a1f1249c4afc297acb5e Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Tue, 24 Jan 2023 01:28:58 -0500 Subject: [PATCH 149/164] Update CD_dev-release.yml --- .github/workflows/CD_dev-release.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/CD_dev-release.yml b/.github/workflows/CD_dev-release.yml index 176e28f..ec85ce7 100644 --- a/.github/workflows/CD_dev-release.yml +++ b/.github/workflows/CD_dev-release.yml @@ -83,7 +83,7 @@ jobs: git checkout dev-release git pull origin dev-release --rebase - git checkout master-ci installer.sh + git checkout origin/master-ci installer.sh git merge -X theirs origin/master-ci From 1124d17b9746c8080d2bc550c25f9ae10c9fd526 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Tue, 24 Jan 2023 01:30:14 -0500 Subject: [PATCH 150/164] Update CD_dev-release.yml --- .github/workflows/CD_dev-release.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/CD_dev-release.yml b/.github/workflows/CD_dev-release.yml index ec85ce7..cb7951d 100644 --- a/.github/workflows/CD_dev-release.yml +++ b/.github/workflows/CD_dev-release.yml @@ -83,7 +83,7 @@ jobs: git checkout dev-release git pull origin dev-release --rebase - git checkout origin/master-ci installer.sh + git ls-files -d | xargs git checkout -- git merge -X theirs origin/master-ci From f7af8b3ff8c356ad4beb39c8131718cdb4cab7f3 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Tue, 24 Jan 2023 01:36:38 -0500 Subject: [PATCH 151/164] Update CD_dev-release.yml --- .github/workflows/CD_dev-release.yml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/.github/workflows/CD_dev-release.yml b/.github/workflows/CD_dev-release.yml index cb7951d..11a5e66 100644 --- a/.github/workflows/CD_dev-release.yml +++ b/.github/workflows/CD_dev-release.yml @@ -81,10 +81,11 @@ jobs: git fetch origin git checkout dev-release - git pull origin dev-release --rebase git ls-files -d | xargs git checkout -- + git pull origin dev-release --rebase + git merge -X theirs origin/master-ci rm -rf LICENSE From b80629189618a3d237da5386b46ad774d5a4022e Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Tue, 24 Jan 2023 01:38:06 -0500 Subject: [PATCH 152/164] Update CD_dev-release.yml --- .github/workflows/CD_dev-release.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/CD_dev-release.yml b/.github/workflows/CD_dev-release.yml index 11a5e66..d37e76e 100644 --- a/.github/workflows/CD_dev-release.yml +++ b/.github/workflows/CD_dev-release.yml @@ -84,7 +84,7 @@ jobs: git ls-files -d | xargs git checkout -- - git pull origin dev-release --rebase + git pull origin dev-release --rebase --autostash git merge -X theirs origin/master-ci From 92ce4aaf3368016363806d7df463dc1feb74d591 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Tue, 24 Jan 2023 01:40:16 -0500 Subject: [PATCH 153/164] Update CD_dev-release.yml --- .github/workflows/CD_dev-release.yml | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/.github/workflows/CD_dev-release.yml b/.github/workflows/CD_dev-release.yml index d37e76e..69ccd4e 100644 --- a/.github/workflows/CD_dev-release.yml +++ b/.github/workflows/CD_dev-release.yml @@ -86,7 +86,9 @@ jobs: git pull origin dev-release --rebase --autostash - git merge -X theirs origin/master-ci + git rm installer.sh + + git merge -X theirs origin/master-ci --a rm -rf LICENSE rm -rf README.md From a23e39d9abcf54e8ba55bbbd3c513306e78b56f5 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Tue, 24 Jan 2023 01:44:46 -0500 Subject: [PATCH 154/164] Update CD_dev-release.yml --- .github/workflows/CD_dev-release.yml | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/.github/workflows/CD_dev-release.yml b/.github/workflows/CD_dev-release.yml index 69ccd4e..7f21b14 100644 --- a/.github/workflows/CD_dev-release.yml +++ b/.github/workflows/CD_dev-release.yml @@ -86,9 +86,13 @@ jobs: git pull origin dev-release --rebase --autostash + git checkout origin/master-ci + git rm installer.sh - git merge -X theirs origin/master-ci --a + git checkout dev-release + + git merge -X theirs master-ci --a rm -rf LICENSE rm -rf README.md From f010b6235cddb122d3c76b27b18c98e9e4c738ed Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Tue, 24 Jan 2023 01:46:08 -0500 Subject: [PATCH 155/164] Update CD_dev-release.yml --- .github/workflows/CD_dev-release.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/CD_dev-release.yml b/.github/workflows/CD_dev-release.yml index 7f21b14..ce26e0b 100644 --- a/.github/workflows/CD_dev-release.yml +++ b/.github/workflows/CD_dev-release.yml @@ -92,7 +92,7 @@ jobs: git checkout dev-release - git merge -X theirs master-ci --a + git merge -X theirs master-ci rm -rf LICENSE rm -rf README.md From 58ae925219ae4637900d5c26d5d80033ba6cd039 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Tue, 24 Jan 2023 01:48:21 -0500 Subject: [PATCH 156/164] Update CD_dev-release.yml --- .github/workflows/CD_dev-release.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/CD_dev-release.yml b/.github/workflows/CD_dev-release.yml index ce26e0b..9054c1f 100644 --- a/.github/workflows/CD_dev-release.yml +++ b/.github/workflows/CD_dev-release.yml @@ -86,7 +86,7 @@ jobs: git pull origin dev-release --rebase --autostash - git checkout origin/master-ci + git checkout master-ci git rm installer.sh From b06070f7d8372f59ca09681595b3223c97c5d401 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Tue, 24 Jan 2023 01:50:18 -0500 Subject: [PATCH 157/164] Update CD_dev-release.yml --- .github/workflows/CD_dev-release.yml | 3 +++ 1 file changed, 3 insertions(+) diff --git a/.github/workflows/CD_dev-release.yml b/.github/workflows/CD_dev-release.yml index 9054c1f..15a6e00 100644 --- a/.github/workflows/CD_dev-release.yml +++ b/.github/workflows/CD_dev-release.yml @@ -90,6 +90,9 @@ jobs: git rm installer.sh + git add installer.sh + git commit -m "Remove installer.sh" + git checkout dev-release git merge -X theirs master-ci From c80cd883b219fa668ffc09c7710eaf8e39fd6bab Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Tue, 24 Jan 2023 01:56:05 -0500 Subject: [PATCH 158/164] Update CD_dev-release.yml --- .github/workflows/CD_dev-release.yml | 10 ---------- 1 file changed, 10 deletions(-) diff --git a/.github/workflows/CD_dev-release.yml b/.github/workflows/CD_dev-release.yml index 15a6e00..40d8e2c 100644 --- a/.github/workflows/CD_dev-release.yml +++ b/.github/workflows/CD_dev-release.yml @@ -78,21 +78,11 @@ jobs: run: | git config user.name github-actions git config user.email github-actions@github.com - git fetch origin git checkout dev-release - git ls-files -d | xargs git checkout -- - git pull origin dev-release --rebase --autostash - git checkout master-ci - - git rm installer.sh - - git add installer.sh - git commit -m "Remove installer.sh" - git checkout dev-release git merge -X theirs master-ci From 22e6a28e08fd324238ceace6bc73f12aa1714d9d Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Tue, 24 Jan 2023 01:58:34 -0500 Subject: [PATCH 159/164] Update CD_dev-release.yml --- .github/workflows/CD_dev-release.yml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/CD_dev-release.yml b/.github/workflows/CD_dev-release.yml index 40d8e2c..68453cd 100644 --- a/.github/workflows/CD_dev-release.yml +++ b/.github/workflows/CD_dev-release.yml @@ -81,11 +81,11 @@ jobs: git checkout dev-release - git pull origin dev-release --rebase --autostash + git pull origin dev-release git checkout dev-release - git merge -X theirs master-ci + git merge -X theirs origin/master-ci rm -rf LICENSE rm -rf README.md @@ -96,7 +96,7 @@ jobs: git add . git diff-index --quiet HEAD || git commit -m "Sanitize dev-release" - git push origin dev-release --force + git push origin dev-release - name: Webhook uses: distributhor/workflow-webhook@v3 From b25bea323a62c8a584834a4da472bc882620a1af Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Tue, 24 Jan 2023 02:00:56 -0500 Subject: [PATCH 160/164] Update CD_dev-release.yml --- .github/workflows/CD_dev-release.yml | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/.github/workflows/CD_dev-release.yml b/.github/workflows/CD_dev-release.yml index 68453cd..86656a8 100644 --- a/.github/workflows/CD_dev-release.yml +++ b/.github/workflows/CD_dev-release.yml @@ -83,9 +83,7 @@ jobs: git pull origin dev-release - git checkout dev-release - - git merge -X theirs origin/master-ci + git merge master-ci rm -rf LICENSE rm -rf README.md From 72e907405b82700aec68712f641d68d129d18685 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Tue, 24 Jan 2023 02:03:34 -0500 Subject: [PATCH 161/164] Update CD_dev-release.yml --- .github/workflows/CD_dev-release.yml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/CD_dev-release.yml b/.github/workflows/CD_dev-release.yml index 86656a8..dbca892 100644 --- a/.github/workflows/CD_dev-release.yml +++ b/.github/workflows/CD_dev-release.yml @@ -80,10 +80,9 @@ jobs: git config user.email github-actions@github.com git checkout dev-release - git pull origin dev-release - git merge master-ci + git merge origin/master-ci rm -rf LICENSE rm -rf README.md @@ -93,7 +92,8 @@ jobs: git add . - git diff-index --quiet HEAD || git commit -m "Sanitize dev-release" + git diff-index --quiet HEAD || git commit -m "Merge master-ci into dev-release and remove unnecessary files" + git push origin dev-release - name: Webhook From 178a2a3f9f48cba0b6358bfd562f9e290ef59152 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Tue, 24 Jan 2023 02:08:01 -0500 Subject: [PATCH 162/164] Create installer.sh --- installer.sh | 268 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 268 insertions(+) create mode 100644 installer.sh diff --git a/installer.sh b/installer.sh new file mode 100644 index 0000000..4439e8e --- /dev/null +++ b/installer.sh @@ -0,0 +1,268 @@ +#!/bin/bash + +# Installer for OpenPod, for more information see https://github.com/blokbot-io/OpenBlok/blob/master/install.sh + + +# ---------------------------------------------------------------------------- # +# Disable Prompts # +# ---------------------------------------------------------------------------- # +export DEBIAN_FRONTEND=noninteractive +sed -i "/#\$nrconf{restart} = 'i';/s/.*/\$nrconf{restart} = 'a';/" /etc/needrestart/needrestart.conf + + +# ---------------------------------------------------------------------------- # +# Help # +# ---------------------------------------------------------------------------- # +Help() +{ + # Display Help + echo "OpenPod instalation script" + echo + echo "h Display this help" + echo "b Set custom branch for OpenPod" + echo "u Set custom URL for OpenPod" + echo "d Enable debug mode" +} + +# TO FIX: ARGS VS OPTS + +# ---------------------------------------------------------------------------- # +# Defaults # +# ---------------------------------------------------------------------------- # +DEBUG=flase # -d +REPO='https://github.com/RecursionSpace/OpenPod' + +# ---------------------------------------------------------------------------- # +# Options # +# ---------------------------------------------------------------------------- # +while getopts ":hbdu" flags; do + case "${flags}" in + h) # display Help + Help + exit;; + b) # Custom branch + BRANCH="${OPTARG}";; + d) # Enable debug mode + DEBUG=true ;; + u) # Custom URL endpoint + URL="${OPTARG}";; + \?) echo "Invalid option: -${OPTARG}" >&2; + exit 1 ;; + esac +done + +if [ $DEBUG ]; then + BRANCH='dev-release' + URL='dev.recursion.space' + API_URL='dev.api.recursion.space' +elif [ ! $DEBUG ]; then + BRANCH='release' + URL='recursion.space' + API_URL='api.recursion.space' +fi + +echo "Installing OpenPod with the following options:" +echo "Debug | $DEBUG" +echo "Repo | $REPO" +echo "Branch | $BRANCH" +echo "URL | $URL" +echo "API_URL | $API_URL" + +# -------------------------------- Verify Root ------------------------------- # +if [ "$EUID" -ne 0 ] + then echo "Please run as root with sudo." + exit +fi + +# --------------------------------- SSH User --------------------------------- # +echo "Verifying SSH user 'openpod'" + +if ! id -u "openpod" >/dev/null 2>&1; then + echo "Creating user 'openpod'" + useradd -m -s /bin/bash openpod + usermod -aG sudo openpod + mkdir -p ~openpod/.ssh/ && touch ~openpod/.ssh/authorized_keys + echo "openpod ALL=(ALL) NOPASSWD:ALL" | sudo tee -a /etc/sudoers > /dev/null +else + echo "User 'openpod' already exists, skipping..." + mkdir -p ~openpod/.ssh/ && touch ~openpod/.ssh/authorized_keys +fi + +# ---------------------------- Update System Time ---------------------------- # +sudo timedatectl set-timezone UTC + + +# ---------------------------------------------------------------------------- # +# Dependencies # +# ---------------------------------------------------------------------------- # + +# ------------------------------ build-essential ----------------------------- # +# Required to install RPi.GPIO - https://github.com/ynsta/steamcontroller/issues/42 +REQUIRED_PKG="build-essential" +PKG_OK=$(dpkg-query -W --showformat='${Status}\n' $REQUIRED_PKG|grep "install ok installed") +if [ "" = "$PKG_OK" ]; then + echo "No $REQUIRED_PKG. Setting up $REQUIRED_PKG..." + sudo apt-get install build-essential -y +else + echo "build-essential already installed, skipping..." +fi + +# ---------------------------------- chrony ---------------------------------- # +REQUIRED_PKG="chrony" +PKG_OK=$(dpkg-query -W --showformat='${Status}\n' $REQUIRED_PKG|grep "install ok installed") +if [ "" = "$PKG_OK" ]; then + echo "No $REQUIRED_PKG. Setting up $REQUIRED_PKG..." + sudo apt-get install chrony -y + sudo chronyd -q +else + echo "chrony already installed, skipping..." +fi + +# ----------------------------------- unzip ---------------------------------- # +REQUIRED_PKG="unzip" +PKG_OK=$(dpkg-query -W --showformat='${Status}\n' $REQUIRED_PKG|grep "install ok installed") +if [ "" = "$PKG_OK" ]; then + echo "No $REQUIRED_PKG. Setting up $REQUIRED_PKG..." + sudo apt-get install unzip -y +else + echo "unzip already installed, skipping..." +fi + +# ------------------------------------ jq ------------------------------------ # +REQUIRED_PKG="jq" +PKG_OK=$(dpkg-query -W --showformat='${Status}\n' $REQUIRED_PKG|grep "install ok installed") +if [ "" = "$PKG_OK" ]; then + echo "No $REQUIRED_PKG. Setting up $REQUIRED_PKG..." + sudo apt-get install jq -y +else + echo "jq already installed, skipping..." +fi + +# -------------------------------- Python 3.11 ------------------------------- # +pytohn_version=$(python -c 'import sys; print(".".join(map(str, sys.version_info[0:2])))') +if [ "$pytohn_version" != "3.11" ]; then + sudo apt install software-properties-common -y + yes '' | sudo add-apt-repository ppa:deadsnakes/ppa + sudo apt-get install python3.11 -y + sudo update-alternatives --install /usr/bin/python python /usr/bin/python3.11 1 + sudo update-alternatives --install /usr/bin/python python /usr/bin/python3.10 1 +else + echo "Python 3.11 already installed" +fi + +# -------------------------------- Python-Dev -------------------------------- # +REQUIRED_PKG="python3.11-dev" +PKG_OK=$(dpkg-query -W --showformat='${Status}\n' $REQUIRED_PKG|grep "install ok installed") +if [ "" = "$PKG_OK" ]; then + echo "No $REQUIRED_PKG. Setting up $REQUIRED_PKG..." + sudo apt-get install python3.11-dev -y +else + echo "python3.11-dev already installed, skipping..." +fi + +# ------------------------ Python Virtual Environment ------------------------ # +REQUIRED_PKG="python3.11-venv" +PKG_OK=$(dpkg-query -W --showformat='${Status}\n' $REQUIRED_PKG|grep "install ok installed") +if [ "" = "$PKG_OK" ]; then + echo "No $REQUIRED_PKG. Setting up $REQUIRED_PKG..." + sudo apt-get install python3.11-venv -y +else + echo "python3.11-venv already installed, skipping..." +fi + +# ---------------------------------------------------------------------------- # +# OpenPod # +# ---------------------------------------------------------------------------- # + +# -------------------------- Clear Previous Install -------------------------- # +sudo rm -rf /opt/OpenPod +sudo systemctl stop openpod.service + +# ------------------------------- Clone OpenPod ------------------------------ # +set -e # Exit when any command fails. +sudo mkdir -p /opt +cd /opt +sudo git clone --single-branch --branch $BRANCH "${REPO}".git +cd OpenPod + +# ----------------------------- Setup Enviroment ----------------------------- # +sudo python3.11 -m venv /opt/OpenPod/env +source /opt/OpenPod/env/bin/activate +pip install --no-input -U -r /opt/OpenPod/requirements.txt + +# ---------------------------- Create Directories ---------------------------- # +sudo mkdir -p /opt/OpenPod/logs +sudo mkdir -p /opt/OpenPod/data + +# ------------------------------- Create Files ------------------------------- # +# Log Location +sudo touch /opt/OpenPod/logs/RecursionLog.log +sudo touch /opt/OpenPod/logs/System.Snapshot + +# Data Location +sudo touch /opt/OpenPod/data/dump.json +sudo touch /opt/OpenPod/data/nodes.json +sudo touch /opt/OpenPod/data/owners.json +sudo touch /opt/OpenPod/data/permissions.json + +# -------------------------------- system.json ------------------------------- # +sudo touch /opt/OpenPod/system.json +serial_uuid=$(cat /proc/sys/kernel/random/uuid) +serial=${serial_uuid//-} +xbee_uuid=$(cat /proc/sys/kernel/random/uuid) +openpod_version=$(git rev-parse HEAD) +echo '{ + "uuid": "'"$serial_uuid"'", + "debug": '$DEBUG', + "serial": "'"$serial"'", + "timezone": "UTC", + "url": "'"$URL"'", + "api_url": "'"$API_URL"'", + "XBEE": { + "KY": "'"$xbee_uuid"'", + "OP": false + }, + "GPIO": { + "LED_IO": 23, + "LED_STAT": 17 + }, + "version": "'"$openpod_version"'", + "OpenPod": { + "repo": "'"$REPO"'", + "branch": "'"$BRANCH"'", + "commit": "'"$openpod_version"'" + } +}' > /opt/OpenPod/system.json + +# --------------------------- Create Version Folder -------------------------- # +mkdir -p /opt/OpenPod/versions/"$openpod_version" +sudo cp -a /opt/OpenPod/openpod/. /opt/OpenPod/versions/"$openpod_version"/ +sudo rm -rf /opt/OpenPod/openpod + +# --------------------------- Setup OpenPod Service -------------------------- # +cat < /etc/systemd/system/openpod.service +[Unit] +Description=OpenPod | Recursion.Space +After=network.target +StartLimitIntervalSec=0 + +[Service] +Type=simple +User=root +WorkingDirectory=/opt/OpenPod + +ExecStart = /bin/bash -c "exec /opt/OpenPod/env/bin/python3.11 \\ + /opt/OpenPod/versions/\$(jq '.version' /opt/OpenPod/system.json | xargs)/pod.py" + +Restart=always +RestartSec=10 + +[Install] +WantedBy=multi-user.target +EOF + +sudo systemctl enable --now openpod.service +sudo systemctl daemon-reload + +echo "- OpenPod is now installed -" +exit 0 From 24ea30a01233937a72b6091dacdf12f511dc0ef9 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Tue, 24 Jan 2023 02:08:15 -0500 Subject: [PATCH 163/164] Update CD_dev-release.yml --- .github/workflows/CD_dev-release.yml | 1 - 1 file changed, 1 deletion(-) diff --git a/.github/workflows/CD_dev-release.yml b/.github/workflows/CD_dev-release.yml index dbca892..dba6175 100644 --- a/.github/workflows/CD_dev-release.yml +++ b/.github/workflows/CD_dev-release.yml @@ -88,7 +88,6 @@ jobs: rm -rf README.md rm -rf docs rm -rf tests - rm -rf installer.sh git add . From 2b80e790da7b92291ecf4977b3716b64196c4120 Mon Sep 17 00:00:00 2001 From: Justin Merrell Date: Tue, 24 Jan 2023 23:27:26 -0500 Subject: [PATCH 164/164] fix: release trigger --- .github/workflows/CD_Release.yml | 6 +++--- installer.sh | 1 + 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/.github/workflows/CD_Release.yml b/.github/workflows/CD_Release.yml index 305a587..1c17ddf 100644 --- a/.github/workflows/CD_Release.yml +++ b/.github/workflows/CD_Release.yml @@ -1,9 +1,9 @@ -# Continious Deployment - Production +# Continuous Deployment - Production name: Release - Version & Deploy on: - push: - branches: [ release ] + release: + types: [published] workflow_dispatch: diff --git a/installer.sh b/installer.sh index 4439e8e..ce50343 100644 --- a/installer.sh +++ b/installer.sh @@ -265,4 +265,5 @@ sudo systemctl enable --now openpod.service sudo systemctl daemon-reload echo "- OpenPod is now installed -" +echo "Serial: $serial" exit 0