Flagging Website 1.0 with Postgres database 🎊 🎉

README.md

@@ -1,100 +1,19 @@
 # Flagging Website
 
-This is the code base for the [Charles River Watershed Association's](https://crwa.org/) ("CRWA") flagging website. The flagging website hosts an interface for the CRWA's staff to monitor the outputs of a predictive model that determines whether it is reasonably safe to swim or boat in the Charles River.
-
-This code base is built in Python 3.7+ and utilizes the Flask library heavily. The website can be run locally in development mode, and it can be deployed to Heroku using Gunicorn.
-
-## For Developers
-
-Please read the [Flagging Website wiki](https://github.com/codeforboston/flagging/wiki) for on-boarding documents, code style guide, and development requirements.
-
-For strict documentation of the website sans project management stuff, read the docs [here](https://codeforboston.github.io/flagging/).
-
-## Setup (Dev)
-
-These are the steps to set the code up in development mode.
-
-**On Windows:** (in progress)
-First install Postgressql using this link: https://www.postgresql.org/download/
-
-Then activate the Postgressql services in the Windows Services Panel.
-
-Set environment variable `POSTGRES_PASSWORD` to be whatever you want.
-
-Powershell and CMD (Windows):
-
-```commandline
-set POSTGRES_PASSWORD=enter_password_here
-createdb -U postgres flagging
-psql -U postgres -d flagging -c "DROP USER IF EXISTS flagging; CREATE USER flagging SUPERUSER PASSWORD '%POSTGRES_PASSWORD%'"
-```
-
-open up a Command Prompt terminal window (the default in PyCharm on Windows), point the terminal to the project directory if it's not already there, and enter:
-
-```commandline
-run_windows_dev
-```
-
-If you are in PowerShell (default VSC terminal), use `start-process run_windows_dev.bat` instead.
-
-**On OSX or Linux:** We need to setup postgres database first thus developers should install PostgreSQL with Homebrew then start the PostgreSQL database service. 
+![](flagging_site/static/images/og_preview.png)
 
-```
-brew install postgresql
-brew services start postgresql
-```
+**Our website is live at: [https://crwa-flagging.herokuapp.com/](https://crwa-flagging.herokuapp.com/)**
 
-To begin initialize a database which we call `flagging`, enter into the bash terminal: 
+## Overview
 
-```shell script
-export POSTGRES_PASSWORD=enter_password_here
-createdb flagging
-psql -U *enter_username_here* -d flagging -c "DROP USER IF EXISTS flagging; CREATE USER flagging SUPERUSER PASSWORD '${POSTGRES_PASSWORD}'"
-```
-
-Note postgres password can be any password and postgres usernme can be default username  `postgres` or your OS username. 
-
-To run the website, in the project directory `flagging` enter:
-
-```shell script
-sh run_unix_dev.sh
-```
-
-After you run the script for your respective OS, it will ask you if you want to use online mode. If you do not have the "vault password," say yes (`y`)
-
-After that, it will ask if you have the vault password. If you do, enter it here. If not, you can skip this.
-
-Note that the website will _not_ without either the vault password or offline mode turned on; you must do one or the other.
-
-Next it will ask you to for the postgres password that you exported earlier. If you are in online mode, you can skip this as you do in the password section. 
-
-## Deploy
-
-1. Download Heroku.
-
-2. Set the vault password:
-
-```shell script
-heroku config:set VAULT_PASSWORD=replace_me_with_pw
-```
-
-3. Everything else should be set:
-
-```shell script
-heroku create crwa-flagging-staging
-git push heroku master
-```
-
-## Run tests
-
-Tests are written in Pytest. To run tests, run the following on your command line:
+This is the code base for the [Charles River Watershed Association's](https://crwa.org/) ("CRWA") flagging website. The flagging website hosts an interface for the CRWA's staff to monitor the outputs of a predictive model that determines whether it is reasonably safe to swim or boat in the Charles River.
 
-```shell script
-python -m pytest ./tests -s
-```
+This code base is built in Python 3.7+ and utilizes the Flask library heavily. The website can be run locally in development mode, and it can be deployed to Heroku using Gunicorn.
 
-Note: the test may require you to enter the vault password if it is not already in your environment variables.
+## For Developers and Maintainers
 
+**[Read our documentation here.](https://codeforboston.github.io/flagging/)** Our documentation contains information on everything related to the website, including [first time setup](https://codeforboston.github.io/flagging/setup/).
 
-## Start DB
+## Credits 
 
+This website was built by volunteers at [Code for Boston](https://www.codeforboston.org/) in collaboration with the Charles River Watershed Association.

docs/README.md

@@ -2,14 +2,26 @@
 
 The full docs are available at: https://codeforboston.github.io/flagging/
 
+### Info
+
+The docs were made with [Material for MkDocs](https://squidfunk.github.io/mkdocs-material/).
+
+There are a lot of extensions included in this documentation in addition to Material:
+
+- Most of the extensions come from [PyMdown Extensions](https://facelessuser.github.io/pymdown-extensions/).
+- We use a tool called [Mermaid](https://mermaid-js.github.io/mermaid-live-editor/) for our flow charts.
+- We use the [macros plugin](https://squidfunk.github.io/mkdocs-material/reference/variables/) but only to parameterize the flagging website URL (the `flagging_website_url` field inside of `mkdocs.yml`). 
+
+All of these tools are added and configured inside `mkdocs.yml`. Note you need to pip install them for them to work when you deploy; see deployment script below.
+
 ### Deploying / Refreshing the Docs
 
 If you have write permission to the upstream repository (i.e. you are a project manager), point your terminal to this directory and run the following:
 
 ```shell script
 python3 -m venv mkdocs_env
 source mkdocs_env/bin/activate
-pip install mkdocs pymdown-extensions mkdocs-material pygments
+pip install mkdocs pymdown-extensions mkdocs-material mkdocs-macros-plugin pygments
 mkdocs gh-deploy --remote-name upstream
 deactivate
 source ../venv/bin/activate

docs/docs/about.md

@@ -4,7 +4,7 @@
 
 Of the many services that the CRWA provides to the greater Boston community, one of those is monitoring whether it is safe to swim and/or boat in the Charles River. The CRWA Flagging Program uses a system of color-coded flags to indicate whether or not the river's water quality is safe for boating at various boating locations between Watertown and Boston. Flag colors are based on E. coli and cyanobacteria (blue-green algae) levels; blue flags indicate suitable boating conditions and red flags indicate potential health risks.
 
-See the website's [about page](https://crwa-flagging.herokuapp.com/about) for more about the website functionality and how it relates to the flagging program's objectives.
+See the website's [about page]({{ flagging_website_url }}/about) for more about the website functionality and how it relates to the flagging program's objectives.
 
 See the [development resources overview](development/history) for more information on how this project started and how we came to make the design decisions that you see here today.
 

docs/docs/cloud/heroku_deployment.md

@@ -8,32 +8,29 @@ The following tools are required to deploy the website:
 - [Heroku CLI](https://devcenter.heroku.com/articles/heroku-cli)
 - [Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git)
 
+Additionally, you will have to have set up the website locally to deploy to Heroku, notably including setting up the Postgres database.
+
+???+ tip
+    Read the [setup guide](../setup) to learn how to run the website locally.
+
 ## First Time Deployment
 
 ???+ note
-    In this section, the project name is assumed to be `crwa-flagging`. If you are deploying to another URL, such as `crwa-flagging-staging` or another personal domain, then replace each reference to `crwa-flagging` with that.
+    In this section, the project name is assumed to be `crwa-flagging`. If you are deploying to another URL, such as `crwa-flagging-staging` or a personal domain, then replace each reference to `crwa-flagging` with that.
 
 If you've never deployed the app from your computer, follow these instructions.
 
-1. If you have not already done so, pull the repository to your computer, and then change your directory to it:
+1. If you have not already done so, follow the [setup guide](../setup). The following should now be true:
 
-```shell
-git clone https://github.com/codeforboston/flagging.git
-cd ./flagging
-```
+  - Your terminal is pointed to the root directory of the project `/flagging`.
+  - You should have a copy of the `VAULT_PASSWORD`.
+  - The Postgres database should be set up and up-to-date locally.
+  - Your Heroku account needs to be "verified," which means it needs to have a valid credit card registered to it. Registering a credit card does not incur any charges on its own. See [here](https://devcenter.heroku.com/categories/billing) for Heroku's billing page for more information.
 
-  Additionally, make sure the `VAULT_PASSWORD` environment variable is set if it has not already been:
-
-=== "Windows (CMD)"
-    ```shell
-    set VAULT_PASSWORD=replace_me_with_pw
-    ```
-=== "OSX (Bash)"
-    ```shell
-    export VAULT_PASSWORD=replace_me_with_pw
-    ```
+???+ tip
+    If you are especially weary of cloud costs, you can plug a prepaid credit card in with a small nominal amount for the balance. See [here](https://help.heroku.com/U32408KR/does-heroku-support-pre-paid-credit-cards-for-billing-and-account-verification) for more.
 
-2. Login to Heroku, and add Heroku as a remote repo using Heroku's CLI:
+2. Via the command line: login to Heroku, and add Heroku as a remote repo using Heroku's CLI:
 
 ```shell
 heroku login
@@ -42,28 +39,69 @@ heroku git:remote -a crwa-flagging
 
 3. Add the vault password as an environment variable to Heroku.
 
-=== "Windows (CMD)"
-    ```shell
-    heroku config:set VAULT_PASSWORD=%VAULT_PASSWORD%
+```shell
+heroku config:set VAULT_PASSWORD=vault_password_goes_here -a crwa-flagging
+```
+
+4. You need to setup the `FLASK_ENV` environment variable. This is mainly used for the scheduler and other potential add-ons as a way to ensure that the production config is always being used.
+
+```shell
+heroku config:set FLASK_ENV=production -a crwa-flagging
+```
+
+5. Add a `BASIC_AUTH_USERNAME` and `BASIC_AUTH_PASSWORD` for the admin panel. The username can be whatever you want; e.g. `admin` does the trick. The password should be hard to guess.
+
+```shell
+heroku config:set BASIC_AUTH_USERNAME=admin -a crwa-flagging
+heroku config:set BASIC_AUTH_PASSWORD=admin_password_goes_here -a crwa-flagging
+```
+
+???+ danger
+    The password should be _unique_ to this platform, not a universal password you use for everything. The password is _not_ encrypted when stoerd, and it is visible to anyone who has access to the Heroku dashboard.
+
+6. Add some Heroku add-ons.
+
+```shell
+heroku addons:create scheduler -a crwa-flagging
+heroku addons:create heroku-postgresql -a crwa-flagging
+```
+
+???+ success
+    Pay attention to the last line of output: 
+
     ```
-=== "OSX (Bash)"
-    ```shell
-    heroku config:set VAULT_PASSWORD=${VAULT_PASSWORD}
+    Creating heroku-postgresql on ⬢ crwa-flagging... free
+    Database has been created and is available
+     ! This database is empty. If upgrading, you can transfer
+     ! data from another database with pg:copy
+    Created postgresql-ukulele-12345 as DATABASE_URL
     ```
+
+    In the above case, `postgresql-ukulele-12345` is the "name" of the PostgreSQL add-on's dyno. We will be using this name in the next step.
 
-4. Now deploy the app!
+
+7. Push your local copy of the flagging database to the cloud. In the following command, replace `postgresql-ukulele-12345` with whatever the name of your PostgreSQL dyno is, which should have output from the previous step.
+
+```shell
+heroku pg:push flagging postgresql-ukulele-12345 -a crwa-flagging
+```
+
+???+ note
+    The above command assumes that your local Postgres database is up-to-date.
+
+8. Deploy the app.
 
 ```shell
 git push heroku master
 ```
 
-5. Now try the following:
+9. Now try the following:
 
 ```shell
-heroku logs --tail
+heroku logs --tail -a crwa-flagging
 ```
 
-6. If everything worked out, you should see the following at or near the bottom of the log:
+10. If everything worked out, you should see the following at or near the bottom of the log:
 
 ```
 2020-06-13T23:17:54.000000+00:00 app[api]: Build succeeded
@@ -72,9 +110,23 @@ heroku logs --tail
 ???+ note
     If you see instead see something like `[...] State changed from starting to crashed`, then read the rest of the output to see what happened. The most common error when deploying to production will be a `RuntimeError: Unable to load the vault; bad password provided` which is self-explanatory. Update the password, and the website will automatically attempt to redeploy. If you don't see that error, then try to self-diagnose.
 
-7. Go see the website for yourself!
+11. Go see the website for yourself!
+
+12. You are still not done; you need to do one more step, which is to set up the task scheduler. Run the following command:
+
+```shell
+heroku addons:open scheduler -a crwa-flagging
+```
+
+13. The above command should open up a new window in your browser that lets you add a command that runs on a schedule. That command you set should be `python3 -m flask update-website`, and you should run it once a day at 11:00 AM UTC:
+
+![](../img/scheduler_config.png)
 
-8. You are still not done; you need to do one more step, which is to set up the task scheduler.
+???+ tip
+    If you want to have the website update more than once a day, it's probably better to run multiple scheduled jobs that run the same command on 24 hour intervals than it is to run a single scheduled job on hourly intervals.
+
+???+ note
+    The `update-website` command sends out a Tweet as well as re-running the predictive model. You can make the scheduled task only update the website without sending a tweet by replacing `update-website` with `update-db`.
 
 ## Subsequent Deployments
 
@@ -97,6 +149,13 @@ git push heroku master
     git pull heroku master
     ```
 
+3. If you make any changes that affect the database, you should create the database locally, and then push it to the cloud, similar to the step above where we push the database for the first time. Note that updating the database this way requires that you reset it first.
+
+```shell
+heroku pg:reset -a crwa-flagging
+heroku pg:push flagging postgresql-ukulele-12345 -a crwa-flagging
+```
+
 ## Staging and Production Split
 
 It is recommended, though not required, that you have both "staging" and "production" environments for the website (see [here](https://en.wikipedia.org/wiki/Deployment_environment#Staging) for an explanation), and furthermore it is recommended you deploy to staging and play around with the website to see if it looks right before you ever deploy to production.
@@ -130,21 +189,10 @@ git remote -v
     upstream        https://github.com/codeforboston/flagging.git (push)
     ```
 
-3. Now all of your `heroku` commands are going to require specifying the app, but the steps to deploy in staging are otherwise similar to the production deployment:
+3. Now all of your `heroku` commands are going to require specifying this new app instance, but the steps to deploy in staging are otherwise similar to the production deployment, with the exception of `git push heroku master`.
 
+4. Deployment via git requires pushing to the new remote like so:
 
-=== "Windows (CMD)"
-    ```shell
-    heroku config:set --app crwa-flagging-staging VAULT_PASSWORD=%VAULT_PASSWORD%
-    git push staging master
-    heroku logs --app crwa-flagging-staging --tail
-    ```
-
-=== "OSX (Bash)"
-    ```shell
-    heroku config:set --app crwa-flagging-staging VAULT_PASSWORD=${VAULT_PASSWORD}
-    git push staging master
-    heroku logs --app crwa-flagging-staging --tail
-    ```
-
-4. Check out the website in the staging environment and make sure it looks right.
+```
+git push staging master
+```

docs/docs/index.md

@@ -17,7 +17,7 @@ To export the model outputs using an iFrame, use the following HTML:
 
 ```html
 <div style="position: relative;overflow: hidden;width: 100%;padding-top: 75%">
-<iframe src="http://crwa-flagging.herokuapp.com/flags" style="position: absolute;top: 0;left: 0;bottom: 0;right: 0; width: 100%; height: 100%"></iframe>
+<iframe src="{{ flagging_website_url }}/flags" style="position: absolute;top: 0;left: 0;bottom: 0;right: 0; width: 100%; height: 100%"></iframe>
 ```
 
 ## For Developers

docs/docs/setup.md

@@ -138,7 +138,16 @@ After you get everything set up, you should run the website at least once. Te pr
     5. Set up the Postgres database and update it with data.
     6. Run the actual website.
 
-2. For the first prompt, type `y` to run the website in offline mode. For the subsequent two prompts, ++enter++ through them without inputting anything.
+???+ tip
+    If you are receiving any errors related to the Postgres database and you are certain that Postgres is running on your computer, you can modify the `POSTGRES_USERNAME` and `POSTGRES_PASSWORD` environment variables to connect to your local Postgres instance properly.
+
+2. You will be prompted asking if you want to run the website in offline mode. "Offline mode" is a way to run the website with dummy data without accessing the credentials. It is useful for anyone who wants to run a demo of the website regardless of their affiliation with the CRWA or this project. It can also be useful for development purposes.
+
+???+ tip
+    - If you have a working `VAULT_PASSWORD`, type ++n++ -> ++enter++. This runs the website as normal.
+    - If you do _not_ have the `VAULT_PASSWORD`, type ++y++ -> ++enter++ to turn offline mode on.
+
+3. Now just wait for the database to start filling in and for the website to eventually run.
 
 ???+ success
     You should be good if you eventually see something like the following in your terminal:
@@ -151,4 +160,6 @@ After you get everything set up, you should run the website at least once. Te pr
      * Restarting with stat
     ```
 
-3. Point your browser of choice to the URL shown in the terminal output. If everything worked out, the website should be running on your local computer!
+4. Point your browser of choice to the URL shown in the terminal output. If everything worked out, the website should be running on your local computer!
+
+![](img/successful_run.png)

docs/mkdocs.yml

@@ -33,6 +33,8 @@ theme:
 repo_name: codeforboston/flagging
 repo_url: https://github.com/codeforboston/flagging
 edit_uri: ""
+plugins:
+  - macros
 markdown_extensions:
 - admonition
 - pymdownx.tabbed # https://facelessuser.github.io/pymdown-extensions/
@@ -50,6 +52,7 @@ markdown_extensions:
       format: !!python/name:pymdownx.superfences.fence_div_format
 - sane_lists
 extra:
+  flagging_website_url: https://crwa-flagging.herokuapp.com
   social:
   - icon: fontawesome/brands/github
     link: https://github.com/codeforboston/flagging

flagging_site/__init__.py

@@ -1,3 +1,3 @@
-__version__ = '0.3.0'
+__version__ = '1.0.0'
 
 from .app import create_app