Set Up and Run SmokeDetector

Setting Up SmokeDetector

Basic setup

To set up SmokeDetector (SD), please execute the following commands:

git clone https://github.com/Charcoal-SE/SmokeDetector.git
cd SmokeDetector
git checkout deploy
sudo pip3 install -r requirements.txt --upgrade
pip3 install --user -r user_requirements.txt --upgrade

Next, copy config.sample to a new file called config, and edit the values required. If you are going to be running this SmokeDetector instance as a normal SD instance with metasmoke, see the "config" file with MS below.

You should now be able to run SmokeDetector.

Running SmokeDetector with metasmoke

You will need a keybase account. You will also need to have the "smoke detector runner" role on metasmoke (MS). Contact an MS admin to request the role. When given the role, you should separately be added to the "charcoal" channel on keybase.

The "config" file with MS

In the files section for the keybase "charcoal" channel, there is a "config" file. You should use that config file as a basis for the "config" file you use in your "SmokeDetector" directory.

Set location to the name you choose for your SD instance. This should be your username, followed by a slash and an identifier that's unique among all your Smokey instances. This is the name that your instance will display in messages in chat and what will be displayed on MS for status.
Set metasmoke_key
- On metasmoke in the dropdown for your username, select "My SmokeDetectors".
- Click on "create a new key" and enter the name you used for location above.
- Once you complete the dialog, your SD instance will be listed and you can copy the "Key" to metasmoke_key.
Save the "config" file, as those are the only changes you need.

When testing: use a "rooms_custom.yml" file

If a file "rooms_custom.yml" exists in the SD directory, then it will be used instead of the "rooms.yml" file. If you are not running a full SD instance, which is reporting into Charcoal HQ and all other rooms, you will need to create a "rooms_custom.yml" which describes the rooms in which you want SD to be reporting and accepting commands. For more information about the format used in "rooms.yml" and "rooms_custom.yml", please see: Rooms configuration.

Run SmokeDetector

At this point, you should be able to run SmokeDetector using the instructions below.

Setting up an AWS EC2 linux instance for SmokeDetector

Using a AWS EC2 instance can be a quick way to get an SD instance up and running. A T2.micro instance, what's available from the AWS Free Tier for 12 months, doesn't have enough CPU credits, at 6/hour, for the instance to be the active SD instance full-time. Makyen ran a t4g.small instance for a couple of years as the standby SD instance. During that time, it was clear that a t4g.small didn't earn enough CPU credits per hour, at 24/hour, to run SD full time. Running two t4g.small instances, for a cumulative 48 CPU credits per hour, and ping-ponging the active instance back-and forth as the available CPU credit ran out, indicated that 48 CPU credits per hour isn't sufficient to run SD full time, as of late 2023.

However, a T2.micro instance does appear to be enough for SD to run as a standby instance to fill in on a very irregular basis without CPU charges, assuming it's not running for more than a couple/few hours. An additional note is that SD on the AWS Linux AMI described below did report some out of memory errors when running on a T3.micro (1GB) without any swap configured. On the other hand, SD functions on a AWS Lightsail instance with 512MB RAM and 2GB swap configured, but that Lightsail instance doesn't have enough CPU credits to operate SD for more than a few hours (and more cannot be purchased). On yet another hand, a t4g.nano, which has the same 512MB of RAM and 2GB of swap, but 2 vCPU compared to 1 vCPU for the low-end Lightsail, can be configured in unlimited mode with respect to CPU Credit usage, so can run SD indefinitely, just with potential substantial CPU credit charges if running full-time. The T4g.nano can also be significantly less expensive (~50%) than the lowest-end Lightsail, if run as a standby SD instance and the T4g.nano has been purchased as a reserved instance.

To set-up a AWS EC2 instance for SD you can do the following:

Note: This installs Python 3.11, which has significant performance improvements over earlier versions. It's expected that you will use the python3.11 and pip3.11 commands anywhere the python3 and pip3 commands are stated in other sections below. You can make that the default for the shell by adding the aliases:

alias python3='python3.11'
alias pip3='pip3.11'

Create the EC2 instance (launch a AWS instance tutorial)
- Create an AWS account (address, verifiable phone number, and credit card required) and Sign-in to the AWS console.
- Click "Launch Instance ->"
- Click "Select" for "Amazon Linux 2023 AMI - ami-06d4b7182ac3480fa (64-bit (x86)) / ami-0090be1905998682a (64-bit (Arm))", which is the first one in the Amazon Machine Image (AMI) list.
  - You can pick any flavor of Linux you're familiar with, just remember to substitute installation commands for the correct package manager. For example, the above version of Linux uses yum. Other versions may use apt-get, rpm, or pacman etc. In addition, other versions of Linux may already have versions of some of the packages installed in the commands below, or may be missing yet other packages. The commands listed below were only tested in the AMI mentioned above.
- Click "Review and Launch"
- Click "Launch"
- Create a public key pair, or use one you've previously created.
  - If creating a new public key pair, download the private key to somewhere on your system from which you can run ssh.
- Click "View Instances"
- Select the new instance
- Click "Connect"
- Use ssh to connect to EC2 instance:
  - Copy-&-paste the example ssh line in the popup into a command line running in the directory where you stored the .pem file with the private key (created or selected earlier).

Execute the following commands. You can copy-&-paste all of them at one time. The commands above the blank line get your EC2 instance to where SmokeDetector can be installed (only tested in the AMI mentioned above). The commands below the blank line clone SmokeDetector from GitHub and install its Python dependencies (the commands below the blank line are a duplicate of those at the top of this page, but are included below to make copy-&-pasting easier):

sudo yum -y update
sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
sudo sh -c 'echo "/swapfile swap swap defaults 0 0" >> /etc/fstab'
sudo yum -y install gcc
sudo yum -y install git
sudo yum -y install python3.11 python3.11-devel python3.11-pip python3.11-wheel
sudo python3.11 -m pip install pip --upgrade

git clone --depth=100 --no-single-branch https://github.com/Charcoal-SE/SmokeDetector.git
cd SmokeDetector
git config user.email "smokey@erwaysoftware.com"
git config user.name "SmokeDetector"
git checkout deploy
sudo /usr/local/bin/pip3.11 install -r requirements.txt --upgrade
pip3.11 install --user -r user_requirements.txt --upgrade

Create the "config" file
- See The "config" file with MS above if you are going to use this SD instance with metasmoke.
- If you are not going to run this SD instance with metasmoke, then copy config.sample to a new file called config, and edit the values required.

Running SmokeDetector

To run, use python3 nocrash.py, or python3 nocrash.py standby (preferably in a daemon-able mode, like a screen session). You can also use python3.11 ws.py, but then SmokeDetector will be shut down after 6 hours; when running from nocrash.py, ws.py will automatically be restarted. (This is to be sure that closed WebSockets, if any, are reopened.)

`nocrash.py`

nocrash.py is the controlling Python code for SmokeDetector. It runs ws.py, which is the SD instance, in a subprocess and restarts the SD instance when it stops. ws.py may stop as the result of various chat !!/ commands, or do to errors. The syntax for nocrash.py is:

python3 nocrash.py [standby] [--loglevel=(debug|info|warning|error)] [no_se_activity_scan] [no_deletion_watcher] [no_edit_watcher] [no_chat_ws_activity_timeout]

standby — Starts the SmokeDetector instance in standby mode.
--loglevel=(debug|info|warning|error) — Set the level of log output to send to the console. By default, there's quite a bit of information logged to the console while scanning.
no_se_activity_scan — Causes the SmokeDetector instance not to watch the SE WebSocket for new posts, even when the instance is active. This gives you an SD instance which will listen for commands in chat, but won't be scanning for new posts, other than ones you !!/scan or !!/report. If you are not testing something which is directly involved with scanning posts from the SE WebSocket (i.e. you don't need to scan a large number of posts, or test/use the SE WebSocket and per-site post fetching queues), then using no_se_activity_scan will probably be quite beneficial to you, as it will allow you to focus on what you are testing without seeing large amounts of output from scanning posts. This can also be used at times when the SE WebSocket is having problems.
no_deletion_watcher — Disables SmokeDetector's DeletionWatcher. The deletion watcher uses the SE WebSocket to watch for the deletion of posts which have been reported by SD. Using this in combination with no_se_activity_scan and no_edit_watcher will result in SD not opening the SE WebSocket, which can be helpful if the SE WebSocket is having problems, testing, etc. Using all those options will result in an SD instance which is monitoring chat for commands, which will allow performing chat commands without SD scanning SE sites based on any WebSocket reports.
no_edit_watcher — Disables SmokeDetector's EditWatcher. The edit watcher uses the SE WebSocket to watch for edits on posts which have been scanned by SD. Using this in combination with no_se_activity_scan and no_deletion_watcher will result in SD not opening the SE WebSocket, which can be helpful if the SE WebSocket is having problems, testing, etc. Using all those options will result in an SD instance which is monitoring chat for commands, which will allow performing chat commands without SD scanning SE sites based on any WebSocket reports.
no_chat_ws_activity_timeout — Disables the periodic check for recent activity on the SE chat WebSockets. Under normal conditions, every 90 seconds SmokeDetector checks for failure of the WebSockets used for each SE chat server by verifying that there's been activity on the WebSocket within the last 60 seconds. If there has been no activity, then the SmokeDetector instance automatically reboots. Using this option disables these checks, and, thus, prevents those reboots. This can be needed in test instances where the instance might be configured to listen to only a single room with little or no activity.