diff --git a/.flaskenv b/.flaskenv index 1698ff4c..b419f3a4 100644 --- a/.flaskenv +++ b/.flaskenv @@ -1 +1,2 @@ -FLASK_APP=flagging_site \ No newline at end of file +FLASK_APP="flagging_site:create_app" +FLASK_ENV=development diff --git a/.gitignore b/.gitignore index 6fc28f7f..21770205 100644 --- a/.gitignore +++ b/.gitignore @@ -2,9 +2,13 @@ docs/site/ .vscode/ *.code-workspace .idea/ -keys.json .DS_Store Pipfile +Pipfile.lock +pgadmin4/ +mkdocs_env/ +secrets.json +keys.json # Byte-compiled / optimized / DLL files __pycache__/ @@ -144,5 +148,3 @@ dmypy.json # Cython debug symbols cython_debug/l - -flagging_site/keys.json \ No newline at end of file diff --git a/Procfile b/Procfile index 81305c19..8c440d4e 100644 --- a/Procfile +++ b/Procfile @@ -1 +1,8 @@ -web: gunicorn "flagging_site:create_app()" \ No newline at end of file +# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +# This file is used to deploy the website to Heroku +# +# See here for more: +# https://devcenter.heroku.com/articles/procfile +# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +web: gunicorn "flagging_site:create_app('production')" diff --git a/README.md b/README.md index d043842f..8f20add0 100644 --- a/README.md +++ b/README.md @@ -1,62 +1,19 @@ # Flagging Website -This is the code base for the [Charles River Watershed Association's](https://crwa.org/) ("CRWA") flagging website. The flagging website hosts an interface for the CRWA's staff to monitor the outputs of a predictive model that determines whether it is reasonably safe to swim or boat in the Charles River. - -This code base is built in Python 3.7+ and utilizes the Flask library heavily. The website can be run locally in development mode, and it can be deployed to Heroku using Gunicorn. - -## For Developers - -Please read the [Flagging Website wiki](https://github.com/codeforboston/flagging/wiki) for on-boarding documents, code style guide, and development requirements. - -For strict documentation of the website sans project management stuff, read the docs [here](https://codeforboston.github.io/flagging/). - -## Setup (Dev) - -These are the steps to set the code up in development mode. - -**On Windows:** open up a Command Prompt terminal window (the default in PyCharm on Windows), point the terminal to the project directory if it's not already there, and enter: - -```commandline -run_windows_dev -``` +![](flagging_site/static/images/og_preview.png) -If you are in PowerShell (default VSC terminal), use `start-process run_windows_dev.bat` instead. +**Our website is live at: [https://crwa-flagging.herokuapp.com/](https://crwa-flagging.herokuapp.com/)** -**On OSX or Linux:** open up a Bash terminal, and in the project directory enter: +## Overview -```shell script -sh run_unix_dev.sh -``` - -After you run the script for your respective OS, it will ask you if you want to use online mode. If you do not have the "vault password," say yes (`y`) - -After that, it will ask if you have the vault password. If you do, enter it here. If not, you can skip this. - -Note that the website will _not_ without either the vault password or offline mode turned on; you must do one or the other. - -## Deploy - -1. Download Heroku. - -2. Set the vault password: - -```shell script -heroku config:set VAULT_PASSWORD=replace_me_with_pw -``` - -3. Everything else should be set: +This is the code base for the [Charles River Watershed Association's](https://crwa.org/) ("CRWA") flagging website. The flagging website hosts an interface for the CRWA's staff to monitor the outputs of a predictive model that determines whether it is reasonably safe to swim or boat in the Charles River. -```shell script -heroku create crwa-flagging-staging -git push heroku master -``` +This code base is built in Python 3.7+ and utilizes the Flask library heavily. The website can be run locally in development mode, and it can be deployed to Heroku using Gunicorn. -## Run tests +## For Developers and Maintainers -Tests are written in Pytest. To run tests, run the following on your command line: +**[Read our documentation here.](https://codeforboston.github.io/flagging/)** Our documentation contains information on everything related to the website, including [first time setup](https://codeforboston.github.io/flagging/setup/). -```shell script -python -m pytest ./tests -s -``` +## Credits -Note: the test may require you to enter the vault password if it is not already in your environment variables. +This website was built by volunteers at [Code for Boston](https://www.codeforboston.org/) in collaboration with the Charles River Watershed Association. diff --git a/docs/README.md b/docs/README.md index d56b933d..b9591829 100644 --- a/docs/README.md +++ b/docs/README.md @@ -2,23 +2,38 @@ The full docs are available at: https://codeforboston.github.io/flagging/ +### Info + +The docs were made with [Material for MkDocs](https://squidfunk.github.io/mkdocs-material/). + +There are a lot of extensions included in this documentation in addition to Material: + +- Most of the extensions come from [PyMdown Extensions](https://facelessuser.github.io/pymdown-extensions/). +- We use a tool called [Mermaid](https://mermaid-js.github.io/mermaid-live-editor/) for our flow charts. +- We use the [macros plugin](https://squidfunk.github.io/mkdocs-material/reference/variables/) but only to parameterize the flagging website URL (the `flagging_website_url` field inside of `mkdocs.yml`). + +All of these tools are added and configured inside `mkdocs.yml`. Note you need to pip install them for them to work when you deploy; see deployment script below. + ### Deploying / Refreshing the Docs If you have write permission to the upstream repository (i.e. you are a project manager), point your terminal to this directory and run the following: ```shell script -mkdocs gh-deploy --remote-branch upstream +python3 -m venv mkdocs_env +source mkdocs_env/bin/activate +pip install mkdocs pymdown-extensions mkdocs-material mkdocs-macros-plugin pygments +mkdocs gh-deploy --remote-name upstream +deactivate +source ../venv/bin/activate ``` If you do not have write permission to the upstream repository, you can do one of the following: 1. (Preferred) Ask a project manager to refresh the pages after you've made changes to the docs. - 2. Run `mkdocs gh-deploy` on your own fork, and then do a pull request to `codeforboston:gh-pages` - - If you are a project manager but you're having issues, you can do a more manual git approach to updating the docs: + 2. Run `mkdocs gh-deploy` on your own fork, and then do a pull request to `codeforboston:gh-pages`: ```shell script mkdocs gh-deploy git checkout gh-pages -git push upstream gh-pages -``` \ No newline at end of file +git push origin gh-pages +``` diff --git a/docs/docs/about.md b/docs/docs/about.md new file mode 100644 index 00000000..820a0337 --- /dev/null +++ b/docs/docs/about.md @@ -0,0 +1,32 @@ +# About + +## Flagging Website + +Of the many services that the CRWA provides to the greater Boston community, one of those is monitoring whether it is safe to swim and/or boat in the Charles River. The CRWA Flagging Program uses a system of color-coded flags to indicate whether or not the river's water quality is safe for boating at various boating locations between Watertown and Boston. Flag colors are based on E. coli and cyanobacteria (blue-green algae) levels; blue flags indicate suitable boating conditions and red flags indicate potential health risks. + +See the website's [about page]({{ flagging_website_url }}/about) for more about the website functionality and how it relates to the flagging program's objectives. + +See the [development resources overview](development/history) for more information on how this project started and how we came to make the design decisions that you see here today. + +## Code for Boston + +Code for Boston is the group that built the CRWA's flagging website. You can find a list of individual contributors [here](https://github.com/codeforboston/flagging/graphs/contributors) + +Code for Boston is a volunteer Civic Technology meetup. We are part of the [Code for America Brigade network](http://www.codeforamerica.org/brigade/about), and are made up of developers, designers, data geeks, citizen activists, and many others who use creative technology to solve civic and social problems. + +## Charles River + +![](https://www.epa.gov/sites/production/files/styles/large/public/2015-04/cr-watershed-map.jpg) + +[Via the EPA:](https://www.epa.gov/charlesriver/about-charles-river#HistoricalTimeline) + +> The Charles River flows 80 miles from Hopkinton, Mass. to Boston Harbor. The Charles River is the most prominent urban river in New England. It is a major source of recreation and a readily-available connection to the natural world for residents of the Boston metropolitan area. The entire Charles River drains rain and melted snow from a watershed area of 310 square miles. + +## Charles River Watershed Association (CRWA) + +The Charles River Watershed Association ("CRWA") was formed in 1965, the same year that Dirty Water peaked at #11 on the Billboard singles chart. [Via the CRWA's website:](https://www.crwa.org/about.html) + +> CRWA is one of the country’s oldest watershed organizations and has figured prominently in major cleanup and protection efforts. Since our earliest days of advocacy, we have worked with government officials and citizen groups from 35 Massachusetts watershed towns from Hopkinton to Boston. + +The EPA also relies on sample data collected by the CRWA to construct its report card. + diff --git a/docs/docs/admin.md b/docs/docs/admin.md new file mode 100644 index 00000000..26d7226e --- /dev/null +++ b/docs/docs/admin.md @@ -0,0 +1,20 @@ +# Admin Panel + +???+ Note + This page discusses how to use the admin panel for the website. For how to set up the admin page username and password during deployment, see the [Heroku deployment](cloud/heroku_deployment) documentation. + +The admin panel is used to manually override the model outputs during events and advisories that would adversely effect the river quality. + +You can reach the admin panel by going to `/admin` after the base URL for the flagging website. (You need to it in manually.) + +You will be asked a username and password, which will be provided to you by the person who deployed the website. Enter the correct credentials to enter the admin panel. + +???+ note + In "development" mode, the default username is `admin` and the password is `password`. In production, the environment variables `BASIC_AUTH_USERNAME` and `BASIC_AUTH_PASSWORD` are used to set the credentials. + +## Cyanobacteria Overrides + +There should be a link to this page in the admin navigation bar. +On this page, one can add an override for a reach with a start time and end time, +and if the current time is between those times then the reach will be marked as +unsafe on the main website, regardless of the model data. diff --git a/docs/docs/background.md b/docs/docs/background.md deleted file mode 100644 index ecd71956..00000000 --- a/docs/docs/background.md +++ /dev/null @@ -1,46 +0,0 @@ -# Background -# Charles River - -![](https://www.epa.gov/sites/production/files/styles/large/public/2015-04/cr-watershed-map.jpg) - -[Via the EPA:](https://www.epa.gov/charlesriver/about-charles-river#HistoricalTimeline) - -> The Charles River flows 80 miles from Hopkinton, Mass. to Boston Harbor. The Charles River is the most prominent urban river in New England. It is a major source of recreation and a readily-available connection to the natural world for residents of the Boston metropolitan area. The entire Charles River drains rain and melted snow from a watershed area of 310 square miles. - -Throughout most of the 20th century, the Charles River in Boston was known for its contaminated water. The reputation of the Charles River was popularized out of state by the song [Dirty Water by the Standells](https://en.wikipedia.org/wiki/Dirty_Water), which peaked at #11 on the Billboard singles chart on June 11, 1965. (The song has a chorus containing the lines "Well I love that dirty water / Boston you're my home.") - -Starting in the late 80s, efforts were made to start cleaning up the Charles River. In 1988, as the result of a lawsuit from the Conservation Law Foundation (CLF), the Massachusetts Water Resources Authority (MWRA) created a combined sewer overflow system to address sewage in the Charles River. In 1995, the CRWA, EPA, municipalities, and Massachusetts state agencies launched the Clean Charles Initiative, which included a report card for the Charles River that is issued by EPA scientists annually. The first grade the Charles River received was a D for the year 1995. The Charles River's grade [peaked](https://www.wbur.org/earthwhile/2019/06/12/charles-river-water-quality-report-card-2018) at A- in 2013 and 2018. - -## Charles River Watershed Association - -The Charles River Watershed Association ("CRWA") was formed in 1965, the same year that Dirty Water peaked at #11 on the Billboard singles chart. [Via the CRWA's website:](https://www.crwa.org/about.html) - -> CRWA is one of the country’s oldest watershed organizations and has figured prominently in major cleanup and protection efforts. Since our earliest days of advocacy, we have worked with government officials and citizen groups from 35 Massachusetts watershed towns from Hopkinton to Boston. - -The EPA also relies on sample data collected by the CRWA to construct its report card. - -## Flagging Program - -Of the many services that the CRWA provides to the greater Boston community, one of those is monitoring whether it is safe to swim and/or boat in the Charles River. Traditionally, this was accomplished by running some data through a predictive model hosted on a PHP website and outputting the results through that PHP website. However, that website is currently out of commission. At Code for Boston, we attempted to fix the website, although we have had trouble maintaining a steady stream of PHP expertise inside the "Safe Water Project" (the flagging website's parent project). So we are going to be focusing now on building the website from scratch in Python. See the "Stack Justification" documentation for why we chose this path, and why we chose Python + Flask. - -## Code for Boston - -Code for Boston is a volunteer Civic Technology meetup. We are part of the Code for America Brigade network, and are made up of developers, designers, data geeks, citizen activists, and many others who use creative technology to solve civic and social problems. We aim to find creative means of technology to help better the lives of individuals in our communities. We meet every Tuesday - -## More Information on CRWA - -- [This WBUR article](https://www.wbur.org/news/2017/09/08/charles-river-water-quality-swimming) provides a great overview of the CRWA and its monitoring programs. All volunteers should read it! - -- The CRWA periodically sends a report to the Governor of Massachusetts on the status of the Charles River. [This is the CRWA's latest report.](https://drive.google.com/file/d/1dnDQYMbYvY7U40Fn33Y9xiL7oYcPO6L_/view?usp=sharing) - -## More Information on Code For Boston - -If you are interested more about our organization, go to our [website](https://www.codeforboston.org/about/). - -If you wish to contact the project team - -- Join our [Slack channel](https://cfb-public.slack.com): `#water`. - - - Meet us on Tuesday Night for our general meetings by [Signing up to attend Code for Boston events here](https://www.meetup.com/Code-for-Boston/). - - - More contact information on various projects in this [tab of our website](https://www.codeforboston.org/projects/). Search for our Safe Water Project which will provide link to our github project and google hangout/video chat link. diff --git a/docs/docs/cloud/heroku_deployment.md b/docs/docs/cloud/heroku_deployment.md new file mode 100644 index 00000000..98cfd502 --- /dev/null +++ b/docs/docs/cloud/heroku_deployment.md @@ -0,0 +1,198 @@ +# Remote Deployment + +???+ note + This guide is an instruction manual on how to deploy the flagging website to the internet via Heroku. If you just want to run the website locally, you do not need Heroku. + +The following tools are required to deploy the website: + +- [Heroku CLI](https://devcenter.heroku.com/articles/heroku-cli) +- [Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) + +Additionally, you will have to have set up the website locally to deploy to Heroku, notably including setting up the Postgres database. + +???+ tip + Read the [setup guide](../setup) to learn how to run the website locally. + +## First Time Deployment + +???+ note + In this section, the project name is assumed to be `crwa-flagging`. If you are deploying to another URL, such as `crwa-flagging-staging` or a personal domain, then replace each reference to `crwa-flagging` with that. + +If you've never deployed the app from your computer, follow these instructions. + +1. If you have not already done so, follow the [setup guide](../setup). The following should now be true: + + - Your terminal is pointed to the root directory of the project `/flagging`. + - You should have a copy of the `VAULT_PASSWORD`. + - The Postgres database should be set up and up-to-date locally. + - Your Heroku account needs to be "verified," which means it needs to have a valid credit card registered to it. Registering a credit card does not incur any charges on its own. See [here](https://devcenter.heroku.com/categories/billing) for Heroku's billing page for more information. + +???+ tip + If you are especially weary of cloud costs, you can plug a prepaid credit card in with a small nominal amount for the balance. See [here](https://help.heroku.com/U32408KR/does-heroku-support-pre-paid-credit-cards-for-billing-and-account-verification) for more. + +2. Via the command line: login to Heroku, and add Heroku as a remote repo using Heroku's CLI: + +```shell +heroku login +heroku git:remote -a crwa-flagging +``` + +3. Add the vault password as an environment variable to Heroku. + +```shell +heroku config:set VAULT_PASSWORD=vault_password_goes_here -a crwa-flagging +``` + +4. You need to setup the `FLASK_ENV` environment variable. This is mainly used for the scheduler and other potential add-ons as a way to ensure that the production config is always being used. + +```shell +heroku config:set FLASK_ENV=production -a crwa-flagging +``` + +5. Add a `BASIC_AUTH_USERNAME` and `BASIC_AUTH_PASSWORD` for the admin panel. The username can be whatever you want; e.g. `admin` does the trick. The password should be hard to guess. + +```shell +heroku config:set BASIC_AUTH_USERNAME=admin -a crwa-flagging +heroku config:set BASIC_AUTH_PASSWORD=admin_password_goes_here -a crwa-flagging +``` + +???+ danger + The password should be _unique_ to this platform, not a universal password you use for everything. The password is _not_ encrypted when stoerd, and it is visible to anyone who has access to the Heroku dashboard. + +6. Add some Heroku add-ons. + +```shell +heroku addons:create scheduler -a crwa-flagging +heroku addons:create heroku-postgresql -a crwa-flagging +``` + +???+ success + Pay attention to the last line of output: + + ``` + Creating heroku-postgresql on ⬢ crwa-flagging... free + Database has been created and is available + ! This database is empty. If upgrading, you can transfer + ! data from another database with pg:copy + Created postgresql-ukulele-12345 as DATABASE_URL + ``` + + In the above case, `postgresql-ukulele-12345` is the "name" of the PostgreSQL add-on's dyno. We will be using this name in the next step. + + +7. Push your local copy of the flagging database to the cloud. In the following command, replace `postgresql-ukulele-12345` with whatever the name of your PostgreSQL dyno is, which should have output from the previous step. + +```shell +heroku pg:push flagging postgresql-ukulele-12345 -a crwa-flagging +``` + +???+ note + The above command assumes that your local Postgres database is up-to-date. + +8. Deploy the app. + +```shell +git push heroku master +``` + +9. Now try the following: + +```shell +heroku logs --tail -a crwa-flagging +``` + +10. If everything worked out, you should see the following at or near the bottom of the log: + +``` +2020-06-13T23:17:54.000000+00:00 app[api]: Build succeeded +``` + +???+ note + If you see instead see something like `[...] State changed from starting to crashed`, then read the rest of the output to see what happened. The most common error when deploying to production will be a `RuntimeError: Unable to load the vault; bad password provided` which is self-explanatory. Update the password, and the website will automatically attempt to redeploy. If you don't see that error, then try to self-diagnose. + +11. Go see the website for yourself! + +12. You are still not done; you need to do one more step, which is to set up the task scheduler. Run the following command: + +```shell +heroku addons:open scheduler -a crwa-flagging +``` + +13. The above command should open up a new window in your browser that lets you add a command that runs on a schedule. That command you set should be `python3 -m flask update-website`, and you should run it once a day at 11:00 AM UTC: + +![](../img/scheduler_config.png) + +???+ tip + If you want to have the website update more than once a day, it's probably better to run multiple scheduled jobs that run the same command on 24 hour intervals than it is to run a single scheduled job on hourly intervals. + +???+ note + The `update-website` command sends out a Tweet as well as re-running the predictive model. You can make the scheduled task only update the website without sending a tweet by replacing `update-website` with `update-db`. + +## Subsequent Deployments + +1. Heroku doesn't allow you to redeploy the website unless you create a new commit. Add some updates if you need to with `git add .` then `git commit -m "describe your changes here"`. + +???+ note + In the _very_ rare case you simply need to redeploy without making any changes to the site, in lieu of the above, simply do `git commit --allow-empty -m "redeploy"`. + +2. Once you have done that, Heroku will redeploy the site when you merge your working branch: + +```shell +git push heroku master +``` + +???+ tip + If you are having any issues here related to merge conflicts, instead of deleting everything and starting over, try to pull the data from the `heroku` branch in and merge it into your local branch. + + ```shell + git fetch heroku + git pull heroku master + ``` + +3. If you make any changes that affect the database, you should create the database locally, and then push it to the cloud, similar to the step above where we push the database for the first time. Note that updating the database this way requires that you reset it first. + +```shell +heroku pg:reset -a crwa-flagging +heroku pg:push flagging postgresql-ukulele-12345 -a crwa-flagging +``` + +## Staging and Production Split + +It is recommended, though not required, that you have both "staging" and "production" environments for the website (see [here](https://en.wikipedia.org/wiki/Deployment_environment#Staging) for an explanation), and furthermore it is recommended you deploy to staging and play around with the website to see if it looks right before you ever deploy to production. + +Managing effectively two separate Heroku apps from a single repository requires a bit of knowledge about how git works. Basically what you're doing is connecting to two separate remote git repositories. The default remote repo is called `heroku` and it was created by Heroku's CLI. But since you now have two Heroku remote repositories, the Heroku CLI doesn't know what it's supposed to name the 2nd one. So you have to manually name it using git. + +1. Run the following command to create a staging environment if it does not already exist. + +```shell +heroku create crwa-flagging-staging +``` + +2. Once it exists, add the staging environment as a remote; check to make sure all the remotes look right. The `heroku` remote should correspond with the production environment, and the `staging` remote should correspond with the staging environment you just created. + +```shell +git remote add staging https://git.heroku.com/crwa-flagging-staging.git +git remote -v +``` + +???+ success + The above command should output something like this: + + ``` + heroku https://git.heroku.com/crwa-flagging.git (fetch) + heroku https://git.heroku.com/crwa-flagging.git (push) + origin https://github.com/YOUR_USERNAME_HERE/flagging.git (fetch) + origin https://github.com/YOUR_USERNAME_HERE/flagging.git (push) + staging https://git.heroku.com/crwa-flagging-staging.git (fetch) + staging https://git.heroku.com/crwa-flagging-staging.git (push) + upstream https://github.com/codeforboston/flagging.git (fetch) + upstream https://github.com/codeforboston/flagging.git (push) + ``` + +3. Now all of your `heroku` commands are going to require specifying this new app instance, but the steps to deploy in staging are otherwise similar to the production deployment, with the exception of `git push heroku master`. + +4. Deployment via git requires pushing to the new remote like so: + +``` +git push staging master +``` diff --git a/docs/docs/cloud/index.md b/docs/docs/cloud/index.md new file mode 100644 index 00000000..3d4a6147 --- /dev/null +++ b/docs/docs/cloud/index.md @@ -0,0 +1,5 @@ +# Overview + +The flagging website is designed to be hosted on [Heroku](https://heroku.com/). The guide for how to set up deployment is available [here](heroku_deployment). + +The full cloud deployment depends not only on Heroku but also Twitter's development API. The Twitter bot only needs to be set up once and, notwithstanding exigent circumstances (losing the API key, migrating the bot, or handling a Twitter ban), the Twitter bot does not need any additional maintenance. Nevertheless, there is documentation for how to set up the Twitter bot [here](twitter_bot). diff --git a/docs/docs/cloud/twitter_bot.md b/docs/docs/cloud/twitter_bot.md new file mode 100644 index 00000000..b92beff1 --- /dev/null +++ b/docs/docs/cloud/twitter_bot.md @@ -0,0 +1,61 @@ +# Twitter Bot + +Every time the website updates, it sends out a tweet. In order for it to do that though, you need to set up a Twitter account. + +## First Time Setup + +Follow these steps to set up the Twitter bot for the first time, such as on a new Twitter account. + +1. Create a [Twitter](https://twitter.com/) account that will host the bot, or login to an account you already have that you want to send automated tweets from. + +2. Go to [https://apps.twitter.com/](https://apps.twitter.com/) and sign up for a development account. Note that you will need both a valid phone number and a valid email tied to the developer account in order to use development features. + +???+ note + You will have to wait an hour or two for Twitter.com to get back to you and approve your developer account. + +3. Once you are approved, go to the [Twitter Developer Portal](https://developer.twitter.com/). Click on the app you created, and in the `Settings` tab, ensure that the App permissions are set to *Read and Write* instead of only *Read*. + +???+ tip + If at some point during step 3 Twitter starts throwing API keys at you, ignore it for now. We'll get all the keys we need in next couple steps. + +4. In the code base, use the `VAULT_PASSWORD` to unzip the `vault.7z` manually. You should have a file called `secrets.json`. Open up `secrets.json` in the plaintext editor of your choosing. + +???+ danger + Make sure that you delete the unencrypted, unarchived version of the `secrets.json` file after you are done with it. + +5. Now go back to your browser with the Twitter Developer Portal. At the top of the screen, flip to the `Keys and tokens`. Now it's time to go through the dashboard and get your copy+paste ready. We will be inserting these values into the `secrets.json` (remember to wrap the keys in double quotes `"like this"` when you insert them). + + - The `API Key & Secret` should should go in the corresponding fields for `#!json "api_key": "..."` and `#!json "api_key_secret": "..."`. + - The `Bearer Token` should go in the field `#!json "bearer_token": "..."`. + - The `Access Token & Secret` should go in the corresponding fields for `#!json "access_token": "..."` and `#!json "access_token_secret": "..."`. _But first, you will need to regenerate the `Access Token & Secret` so that it has both read and write permissions._ + +???+ success + The `secrets.json` file should look something like this, with the ellipses replacing the actual values: + + ```json + { + "SECRET_KEY": "...", + "HOBOLINK_AUTH": { + "password": "...", + "user": "...", + "token": "..." + }, + "TWITTER_AUTH": { + "api_key": "...", + "api_key_secret": "...", + "bearer_token": "...", + "access_token": "...", + "access_token_secret": "..." + } + } + ``` + +6. Rezip the file. Enter the `VAULT_PASSWORD` when prompted (happens twice). + +```shell +cd flagging_site +7z a -p vault.7z secrets.json +cd .. +``` + +7. Delete delete the unencrpyted, unarchived version of the `secrets.json` file. diff --git a/docs/docs/deployment.md b/docs/docs/deployment.md deleted file mode 100644 index bfe57dcc..00000000 --- a/docs/docs/deployment.md +++ /dev/null @@ -1,93 +0,0 @@ -# Production Deployment - -The following tools are required to deploy the website: - -- [Heroku CLI](https://devcenter.heroku.com/articles/heroku-cli) -- [Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) - -## Deploying for the first time - -If you've never deployed the app from your computer, follow these instructions. - -Note: the project name here is assumed to be `crwa-flagging`. - -1. If you have not already done so, pull the repository to your computer, and then change your directory to it. - -```shell -git clone https://github.com/codeforboston/flagging.git -cd ./flagging -``` - -2. Login to Heroku, and add Heroku as a remote repo using Heroku's CLI: - -```shell -heroku login -heroku git:remote -a crwa-flagging -``` - -3. Add the vault password as an environment variable to Heroku. - -```shell -heroku config:set VAULT_PASSWORD=replace_me_with_pw -``` - -4. Now deploy the app! - -```shell -git push heroku master -``` - -5. Now try the following: - -```shell -heroku logs --tail -``` - -6. If everything worked out, you should see the following at or near the bottom of the log: - -```shell -2020-06-13T23:17:54.000000+00:00 app[api]: Build succeeded -``` - -If you see insted see something like `2020-06-13T23:17:54.000000+00:00 heroku[web.1]: State changed from starting to crashed`, then read the rest of the output to see what happened (there will likely be a lot of stuff, so dig through it). The most common error when deploying to production will be a `RuntimeError: Unable to load the vault; bad password provided` which is self-explanatory. Update the password, and the website will automatically attempt to redeploy. If you don't see that error, then try to self-diagnose. - -7. Go see the website for yourself! - -## Subsequent deployments - -1. Heroku doesn't allow you to redeploy the website unless you create a new commit. Add some updates if you need to with `git add .` then `git commit -m "describe your changes here"`. - -2. Once you have done that, Heroku will simply redeploy the site when you merge your working branch: - -```shell -git push heroku master -``` - -## Staging and Production Split - -It is recommended, though not required, that you have both "staging" and "production" environments for the website (see [here](https://en.wikipedia.org/wiki/Deployment_environment#Staging) for an explanation), and furthermore it is recommended you deploy to staging and play around with the website to see if it looks right before you deploy to prodution. - -Managing effectively two separate Heroku apps from a single repository requires a bit of knowledge about how git works. Basically what you're doing is connecting to two separate remote git repositories. The default remote repo is called `heroku` and it was created by Heroku's CLI. But since you now have two Heroku remotes, the Heroku CLI doesn't know what it's supposed to name the 2nd one. So you have to manually name it using git. - -1. Run the following command to create a staging environment if it does not already exist. - -```shell -heroku create crwa-flagging-staging -``` - -2. Once it exists, add the staging environment as a remote; check to make sure all the remotes look right. The `heroku` remote should correspond with the production environment, and the `staging` remote should correspond with the staging environment you just created. - -```shell -git remote add staging https://git.heroku.com/crwa-flagging-staging.git -git remote -v -``` - -3. Now all of your `heroku` commands are going to require specifying the app, but the steps to deploy in staging are otherwise similar to the production deployment: - -```shell -heroku config:set --app crwa-flagging-staging VAULT_PASSWORD=replace_me_with_pw -git push staging master -heroku logs --app crwa-flagging-staging --tail -``` - -4. Check out the website and make sure it looks right. diff --git a/docs/docs/development/data.md b/docs/docs/development/data.md new file mode 100644 index 00000000..1d10ad80 --- /dev/null +++ b/docs/docs/development/data.md @@ -0,0 +1,120 @@ +# Data + +# High-Level Overview + +Here is a "TLDR" of the data engineering for this website: + +- To get data, we ping two different APIs, combine the responses from those API requests, do some processing and feature engineering of the data, and then run a predictive model on the processed data. + +- To store the data and then later retrieve it for the front-end of the website, we use PostgreSQL database. + +- To actually run the functionality that gets data, processes it, and stores it. we run a [scheduled job](https://en.wikipedia.org/wiki/Job_scheduler) that runs the command `flask update-db` at a set time intervals. + +Actually setting up the database requires a few additional steps during either remote or local deployment (`flask create-db` and `flask init-db`), however those steps are covered elsewhere in the docs. + +The `update_database()` inside of `database.py` runs four functions elsewhere in the data folder. This flow chart shows how those functions relate to one another (each block is a function; the arrows represent that the function's output is used as an input in the function being pointed at). + +```mermaid +graph TD +A(get_live_hobolink_data) --> C(process_data) +B(get_live_usgs_data) --> C +C --> D(all_models) +``` + +The rest of this document explains in more detail what's happening in these functions individually. + +## Sources + +There are two sources of data for our website: + +1. An API hosted by the USGS National Water Information System API that's hooked up to a Waltham based stream gauge (herein "USGS" data); +2. An API for a HOBOlink RX3000 Remote Monitoring Station device stationed on the Charles River (herein "HOBOlink"). + +### USGS + +The code for retrieving and processing the HOBOlink data is in `flagging_site/data/usgs.py`. + +The USGS API very is straightforward. It's a very typical REST API that takes "GET" requests and return well-formatted json data. Our preprocessing of the USGS API consists of parsing the JSON into a Pandas dataframe. + +The data returned by the USGS API is in 15 minute increments, and it measures the stream flow (cubic feet per second) of the Charles River out in Waltham. + +### HOBOlink + +The code for retrieving and processing the HOBOlink data is in `flagging_site/data/hobolink.py`. + +The HOBOlink device captures various information about the Charles River at the location it's stationed: + +- Air temperature +- Water temperature +- Wind speed +- Photosynthetically active radiation (i.e. sunlight) +- Rainfall + +The HOBOlink data is accessed through a REST API using some credentials stored in the `vault.zip` file. + +The data actually returned by the API is a combination of a yaml file with a CSV below it, and we just use the CSV part. We then do the following to preprocess the CSV: + +- We remove all timestamps ending `:05`, `:15`, `:25`, `:35`, `:45`, and `:55`. These only contain battery information, not weather information. The final dataframe returned is ultimately in 10 minute increments. +- We make the timestamp consistently report eastern standard times. +- We consolidate duplicative columns. The HOBOlink API has a weird issue where sometimes it splits columns of data with the same name, seemingly at random. This issue causes serious data issues if untreated (at one point, it caused our model to fail to update for a couple days), so our function cleans the data. + +As you can see from the above, the HOBOlink API is a bit finicky for whatever reason, but we have a good data processing solution for these problems. + +The HOBOlink data is also notoriously slow to retrieve (regardless of whether you ask for 1 hour of data or multiple weeks of data), which is why we belabored building the database portion of the flagging website out in the first place. The HOBOlink API does not seem to be rate limited or subject to fees that scale with usage. + +???+ tip + You can manually download the latest raw data from this device [here](https://www.hobolink.com/p/0cdac4a6910cef5a8883deb005d73ae1). If you want some preprocessed data that implements the above modifications to the output, there is a better way to get that data explained in the shell guide. + +### Combining the data + +Additional information related to combining the data and how the models work is in the [Predictive Models](../predictive_models) page. + +## Postgres Database + +PostgresSQL is a free, open-source database management system, and it's what our website uses to store data. + +**On OSX or Linux:** + +We need to setup postgres database first thus enter into the bash terminals: + +``` +brew install postgresql +brew services start postgresql +``` +Explanation: We will need to install postgresql in order to create our database. With postgresql installed, we can start up database locally or in our computer. We use `brew` from homebrew to install and start postgresql services. To get homebrew, consult with this link: https://brew.sh/ + +To begin initialize a database, enter into the bash terminal: + +```shell script +export POSTGRES_PASSWORD=*enter_password_here* +createdb -U *enter_username_here* flagging +psql -U *enter_username_here* -d flagging -c "DROP USER IF EXISTS flagging; CREATE USER flagging SUPERUSER PASSWORD '${POSTGRES_PASSWORD}'" +``` +Explanation: Postgres password can be any password you choose. We exported your chosen postgres password into `POSTGRES_PASSWORD`, an environment variable, which is a variable set outside a program and is independent in each session. Next, we created a database called `flagging` using a username/rolename, which needs to be a Superuser or having all accesses of postgres. By default, the Superuser rolename can be `postgres` or the username for you OS. To find out, you can go into psql terminal, which we will explain below, and enter `\du` to see all usernames. Finally, we add the database `flagging` using the env variable in which we save our password. + +You can see the results using the postgresql terminal which you can open by entering: +``` +psql +``` + +Below are a couple of helpful commands you can use in the postgresql: + +``` +\q --to quit +\c *database_name* --to connect to database +\d --show what tables in current database +\du --show database users +\dt --show tables of current database +``` + +To run the website, in the project directory `flagging` enter: + +```shell script +sh run_unix_dev.sh +``` + +Running the bash script `run_unix_dev.sh` found in the `flagging` folder. Inside the scirpt, it defines environment variables `FLASK_APP` and `FLASK_ENV` which we need to find app.py. We also export the user input for offline mode, vault password, and postgres password for validation. Finally we initialize a database with a custom flask command `flask init-db` and finally run the flask application `flask run`. + +Regarding in how flask application connects to postgresql, `database.py` creates an object `db = SQLAlchemy()` which we will refer again in `app.py` to configure the flask application to support postgressql `from .data import db` `db.init_app(app)`. (We can import the `db` object beecause `__init__.py` make the object available as a global variable) + +Flask supports creating custom commands `init-db` for initializing database and `update-db` for updating database. `init-db` command calls `init_db` function from `database.py` and essentially calls `execute_sql()` which executes the sql file `schema.sql` that creates all the tables. Then calls `update_database()` which fills the database with data from usgs, hobolink, etc. `update-db` command primarily just udpates the table thus does not create new tables. Note: currently we are creating and deleting the database everytime the bashscript and program runs. \ No newline at end of file diff --git a/docs/docs/development/index.md b/docs/docs/development/index.md new file mode 100644 index 00000000..ebc22901 --- /dev/null +++ b/docs/docs/development/index.md @@ -0,0 +1,58 @@ +# Development - Overview + +The Development guide is aimed at users who wish to understand the code base and make changes to it if need be. + +This overview page describes at a high-level what the website's infrastructure is, how it all relates, and why those things are in the app. + +!!! tip + Make sure to go through the [setup guide](../setup) before doing anything in the development guide. + +## Dependency Diagram + +```mermaid +classDiagram +Heroku <.. gunicorn +gunicorn <.. Flask : create_app() +gunicorn : /../Procfile +Heroku <.. PostgreSQL +class Flask +Flask : /app.py +Flask : create_app() +Flask : app = Flask(...) +class Config +Config : /config.py +Config : config = get_config_from_env(...) +class vault +vault : /vault.7z +vault : /app.py +Config <.. vault : update_config_from_vault(app) +class Swagger +Swagger : /app.py +Swagger : Swagger(app, ...) +Flask <.. Swagger : init_swagger(app) +Swagger ..> blueprints : wraps RESTful API +Flask <.. Config : app.config.from_object(config) +class SQLAlchemy +SQLAlchemy : /data/database.py +SQLAlchemy : db = SqlAlchemy() +class Jinja2 +Jinja2 : /app.py +Flask <.. Jinja2 : Built-in Flask +SQLAlchemy <.. PostgreSQL: Connected via psycopg2 +Flask <.. SQLAlchemy : db.init_app(app) +class blueprints +blueprints : blueprints/flagging.py +blueprints : blueprints/api.py +blueprints : app.register_blueprint(...) +Flask <.. blueprints +Jinja2 <.. blueprints : Renders HTML +class Admin +Admin : /admin.py +Admin: admin = Admin(...) +SQLAlchemy <.. Admin +Flask <.. Admin : init_admin(app) +class BasicAuth +BasicAuth : /admin.py +BasicAuth : auth = BasicAuth() +BasicAuth ..> Admin +``` diff --git a/docs/docs/development/predictive_models.md b/docs/docs/development/predictive_models.md new file mode 100644 index 00000000..048847be --- /dev/null +++ b/docs/docs/development/predictive_models.md @@ -0,0 +1,137 @@ +# Predictive Models + +The Flagging Website is basically just a deployed predictive model, so in a sense this document covers the real core of the code base. This page explains the models and the data transformations that occur from the original data. At the bottom, there are some notes on how to change the model coefficients and rerun the website with new coefficients. + +The predictive models are stored in the file `/flagging_site/data/models.py`. These models are run as part of the `update-db` command. The input for the models are a combination of the HOBOlink and USGS data with some transformations of the data. The outputs are stored in the SQL table named `model_outputs`. + +???+ tip + There is a fair bit of Pandas in this document and it may be intimidating. However, if you only want to change the model's coefficients and nothing more, you won't need to touch the Pandas directly. + +## Data Transformations + +The model combines data from the HOBOlink device and USGS. These two data sources are run through the function `process_data()`, in which the data is aggregated to hourly intervals (USGS is every 15 minutes and HOBOlink is every 10 minutes). + +Once the data sources are aligned, additional feature transformations are performed such as rolling averages, rolling sums, and a measure of when the last significant rainfall was. + +The feature transformations the CRWA uses depends on the year of the model, so by the time you may be reading this, this information may be outdated. Previous versions included rolling average wind speeds and air/water temperatures over 24 hours. The current version of the model (as of 2020) calculates the following: + +- Rolling 24 hours of the PAR (photosynthetically active radiation) and the stream flow (cubic feet per second). +- Rolling sum of rainfall over the following intervals: 0-24h, 0-48h, and 24-48h. +- The numbers of days since the last "significant rainfall," where significant rain is defined as when the rolling sum of the last 24 hours of rainfall is at least 0.20 inches. + +???+ tip + If you look at the code, you'll see a lot of stuff like `#!python rolling(24)`. The reason `rolling` works is because the dataframe is sorted already by timestamp at that point by `#!python df = df.sort_values('time')`. + +???+ note + We use 28 days of HOBOlink data to process the model. For most features, we only need the last 48 hours worth of data to calculate the most recent value, however the last significant rainfall feature requires a lot of historic data because it is not technically bounded or transformed otherwise. This means that even when calculating 1 row of output data, i.e. the latest hour of data, we still need 28 days. + + In the deployed model, if we do not see any significant rainfall in the last 28 days, we return the difference between the timestamp and the earliest time in the dataframe, `#!python df['time'].min()`. In this scenario, the data will no longer be temporally consistent: a calculation right now will have `28.0` for `'days_since_sig_rain'`, but 12 hours from now it will be 27.5. This is fine though because the model will basically never predict E. coli blooms with 28+ days since significant rain, even when the data is not censored. + + Unfortunately there's no pretty way to implement `days_since_sig_rain`, so the Pandas code that does all of this is one of the more inscrutable parts of the codebase. Note that `'last_sig_rain'` is calculating the timestamp of the last significant rain, and `'days_since_sig_rain'` calculates the time delta and translates into days: + + ```python + df['sig_rain'] = df['rain_0_to_24h_sum'] >= SIGNIFICANT_RAIN + df['last_sig_rain'] = ( + df['time'] + .where(df['sig_rain']) + .ffill() + .fillna(df['time'].min()) + ) + df['days_since_sig_rain'] = ( + (df['time'] - df['last_sig_rain']).dt.seconds / 60 / 60 / 24 + ) + ``` + +## Model Overviews + +Each model function defined in `models.py` is formatted like this: + +1. Take the last few rows of the input dataframe (I discuss what the input dataframe is later on this page). Each row is an hour of data on the condition of the Charles River and its surrounding environment, so for example, taking the last 24 rows is equivalent to taking the last 24 hours of data. +2. Predict the probability of the water being unsafe using a logistic regression fit, with the coefficients in the log odds form (so the dot product of the parameters and the data returns a predicted log odds of the target variable). +3. To get the probability of a log odds, we run it through a logistic function (`sigmoid()`, defined at the top of `models.py`). +4. We check whether the function is above or below the target threshold for safety, defined by `SAFETY_THRESHOLD`. +5. Lastly, we return a dataframe with 5 columns of data: `'reach'`, `'time'`, `'log_odds'` (step 2), `'probability'` (step 3), and `'safe'` (step 4). Each row in output corresponds to a row of input data. + +Here is an example function. It should be pretty easy to track the steps outlined above with the code below. + +```python +def reach_3_model(df: pd.DataFrame, rows: int = 48) -> pd.DataFrame: + """ + a- rainfall sum 0-24 hrs + b- rainfall sum 24-48 hr + d- Days since last rain + 0.267*a + 0.1681*b - 0.02855*d + 0.5157 + + Args: + df: (pd.DataFrame) Input data from `process_data()` + rows: (int) Number of rows to return. + + Returns: + Outputs for model as a dataframe. + """ + df = df.tail(n=rows).copy() + + df['log_odds'] = ( + 0.5157 + + 0.267 * df['rain_0_to_24h_sum'] + + 0.1681 * df['rain_24_to_48h_sum'] + - 0.02855 * df['days_since_sig_rain'] + ) + + df['probability'] = sigmoid(df['log_odds']) + df['safe'] = df['probability'] <= SAFETY_THRESHOLD + df['reach'] = 3 + + return df[['reach', 'time', 'log_odds', 'probability', 'safe']] +``` + +## Editing the Models + +???+ note + + This section covers making changes to the following: + + - The coefficients for the models. + - The safety threshold. + - The model features. + + If you want to do anything more complicated, such as adding a new source of information to the model, that is outside the scope of this document. To accomplish that, you'll need to do more sleuthing into the code to really understand it. + +???+ note + Making any changes covered in this section is relatively easy, but you'll still need to actually deploy the changes to Heroku if you want them to be on the live site. Read the [deployment guide](../../deployment) for more. + +### Model coefficients + +As covered in the last section, each model's coefficients are represented as log odds ratios. Don't be confused by this statement though: this is how logistic regression is represented in all statistical software packages-- `Logit` in Python's Statsmodels, `logit` in Stata, and `glm` in R-- since that's what's being calculated mathematically when a logistic regression is calculated. I only emphasize this to point out that to get a probability, the final log odds needs to be logistically transformed (which is done via the `sigmoid()` function) after the linear terms are summed up. + +The code representing the logistic model prediction was organized for maximum legibility: the first number is the constant term, and the remaining coefficients are aligned next to the column name. Note the final coefficient in this particular example is a negative coefficient and is thus subtracted. + +```python +df['log_odds'] = ( + 0.5157 + + 0.267 * df['rain_0_to_24h_sum'] + + 0.1681 * df['rain_24_to_48h_sum'] + - 0.02855 * df['days_since_sig_rain'] +) +``` + +Changing the coefficients is as simple as just changing one of those numbers next to its respective column name, inside of its respective model for any particular reach. + +### Safety threshold + +The safety threshold is defined near the top of the document: + +```python +SAFETY_THRESHOLD = 0.65 +``` + +This represents a 65% threshold for whether or not we consider the water safe or not. The `SAFETY_THRESHOLD` value is just used as a placeholder/convenience for whatever the default threshold should be. You can always change this value to be lower or higher, and additionally you can replace `SAFETY_THRESHOLD` inside of a model function + +???+ warning + Hopefully this goes without saying, but if you are going to change the threshold, please have a good, scientifically and statistically justifiable reason for doing so! + +### Feature transformations + +Feature transformations occur in the `process_data()` function after the data has been aggregated by hour, merged, and sorted by timestamp. + +If you want to add some feature transformations, my suggestion is you try to learn from existing examples and copy+paste with the necessary replacements. If you have a feature that can't be built from a copy+paste, that's where you'll possibly need to learn a bit of Pandas. diff --git a/docs/docs/development_resources/index.md b/docs/docs/development_resources/index.md new file mode 100644 index 00000000..59578e67 --- /dev/null +++ b/docs/docs/development_resources/index.md @@ -0,0 +1,49 @@ +# Stack + +## Project History + +Traditionally, the CRWA Flagging Program was hosted on a PHP-built website that hosted a predictive model and ran it. However, that website was out of commission due to some bugs and the CRWA's lack of PHP development resources. + +We at Code for Boston attempted to fix the website, although we have had trouble maintaining a steady stream of PHP expertise, so we rebuilt the website from scratch in Python. The project's source code is available [on GitHub](https://github.com/codeforboston/flagging/wiki), and the docs we used for project management and some dev stuff are available in [the repo's wiki](https://github.com/codeforboston/flagging/wiki). + +## Why Python? + +Python proves to be an excellent choice for the development of this website. Due to how the CRWA tends to staff its team (academics and scientists), Python is the most viable language that a website can be built in while still being maintainable by the CRWA. The two most popular coding languages in academia are R and Python. You can't really build a website in R (you technically can, but really really shouldn't for a lot of reasons). So the next best option is Python. + +Even if the CRWA does not staff people based on their Python knowledge (we do not expect that they will do this), they are very likely have connections to various people who know Python. It is unlikely that the CRWA will have as many direct ties to people who have Javascript or PHP knowledge. Because long-term maintainability is such a high priority, Python is the sensible technical solution. + +Not only is Python way more popular than PHP in academia, it's [the most popular programming language](http://pypl.github.io/PYPL.html) _in general_. This means that Python is a natural fit for any organization's coding projects that do not have specialized needs for a particular coding language. + +## Why Flask? + +Once we have decided on Python for web development, we need to make a determination on whether to use Django or Flask, the two leading frameworks for building websites in Python. + +Django is designed for much more complicated websites than what we would be building. Django has its own idiom that takes a lot of time to learn and get used to. On the other hand, Flask is a very simple and lightweight framework built mainly around the use of its "`app.route()`" decorator. + +## Why Heroku? + +Heroku's main advantage is that we can run it for free; the CRWA does not want to spend money if they can avoid doing so. + +One alternative was [Google Cloud](https://cloud.google.com/free/docs/gcp-free-tier#always-free), specifically the [Google App Engine](https://cloud.google.com/appengine/docs/standard/python3/building-app). + +We did not do this mainly as it is more work to set up for developers and controlling costs requires extra care. E.g. the always free tier of Google Cloud still requires users to plug in a payment method. Developers who want to test Google Cloud functionality would also run into some of those limitations too, depending on their past history with Google Cloud. + +With that said, Heroku does provide some excellent benefits focused around how lightweight it is. Google Cloud is not shy about the fact that it can host massive enterprise websites with extremely complicated infrastructural needs. Don't get me wrong: Heroku can host large websites too. But Heroku supports small to medium sites extremely well, and it is really nice for open source websites in particular. + +- Heroku is less opinionated about how you manage your website, whereas Google Cloud products tend to push you toward Google's various Python integrations and APIs. +- Google Cloud is a behemoth of various services that can overwhelm users, whereas Heroku is conceptually easier to understand. +- Heroku integrates much more nicely into Flask's extensive use of CLIs. For example, Heroku's task scheduler tool (which is very easy to set up) can simply run a command line script built in Flask. Google App Engine lets you do a simple cron job setup that [sends GET requests to your app](https://cloud.google.com/appengine/docs/flexible/python/scheduling-jobs-with-cron-yaml), but doing something that doesn't publicly expose the interface requires use of [three additional services](https://cloud.google.com/python/getting-started/background-processing): Pub/Sub, Firestore, and Cloud Scheduler. +- We want to publicly host this website, but we don't want to expose the keys we use for various things. This is a bit easier to do with Heroku, as it has the concept of an environment that lives on the instance's memory and can be set through the CLI. Google App Engine lets you configure the environment [only](https://cloud.google.com/appengine/docs/flexible/python/reference/app-yaml) through `app.yaml`, which is an issue because it means we'd need to gitignore the `app.yaml`. (We want to just gitignore the keys, not the whole cloud deployment config!) + +???+ warning + If you ever want to run this website on Google App Engine, you'll have to make some changes to the repository (such as adding an `app.yaml`), and it may also involve making changes to the code-- mainly the data backend and the task scheduler interface. + +## Why Pandas? + +We made the decision to use Pandas to manipulate data in the backend of the website because it has an interface that should feel familiar to users of Stata, R, or other statistical software packages commonly used by scientists and academics. This ultimately helps with the maintainability of the website for its intended audience. Data manipulation in SQL can sometimes be unintuitive and require advanced trickery (CTEs, window functions) compared to Pandas code that achieves the same results. Additionally, SQL code tends to be formatted in a non-chronological way, e.g. subqueries run before the query that references them, but occur somewhere in the middle of a query. This isn't hard if you use SQL a bit, but it's not intuitive until you've done a bit of SQL. + +It's true that Pandas is not as efficient as SQL, but we're not processing millions of rows of data, we're only processing a few hundred rows at a time and at infrequent intervals. (And even if efficiency was a concern, we'd sooner use something like [Dask](https://dask.org/) than SQL.) + +One possible downside of Pandas compared to SQL is that SQL has been around for a very long time, and is more of a "standardized" thing than Pandas is or perhaps ever will be. We went with the choice for Pandas after discussing it with some academic friends, but we are aware that in the non-academic world, there are more people who know SQL than Pandas. + +We do use SQL in this website, but primarily to store and retrieve data and to access some features that integrate nicely with the SQLAlchemy ORM (notably the Flask-Admin extension). diff --git a/docs/docs/development_resources/learning_resources.md b/docs/docs/development_resources/learning_resources.md new file mode 100644 index 00000000..ffd7ba06 --- /dev/null +++ b/docs/docs/development_resources/learning_resources.md @@ -0,0 +1,39 @@ +# Learning Resources + +???+ tip + Unless you want to overhaul the website or do some bugfixing, you _probably_ don't need to learn any of the frameworks here. + + The Flagging Website documentation is detailed, self-contained, and should cover the vast majority of use cases for the Flagging website from an administrative perspective, such as updating the predictive model and deploying the website to Heroku. + +The code base is mainly built with the following frameworks; all of these but the last one are Python frameworks: + +- Pandas (data manipulation framework, built on top of another framework called `numpy`.) +- Flask (web framework that handles routing of the website) +- Jinja2 (text markup framework that is used for adding programmatic logic to statically rendered HTML pages.) +- Click (CLI building framework) +- Postgresql + +These frameworks may be intimidating if this is your first time seeing them and you want to make changes to the site. This page has some learning resources that can help you learn these frameworks. + +## Flask & Jinja2 + +The [official Flask tutorial](https://flask.palletsprojects.com/en/1.1.x/tutorial/) is excellent and worth following if you want to learn both Flask and Jinja2. + +???+ tip + Our website's code base is organized somewhat similar to the code base built in the official Flask tutorial. If you are confused by how the code base is organized, going through the tutorial may help clarify some of our design choices. For more examples of larger Flask websites, check out the [flask-bones](https://github.com/cburmeister/flask-bones) template; we did not explicitly reference it in constructing our website but it nevertheless follows a lot of the same ideas we use. + +## Pandas + +The Pandas documentation has excellent resources for users who are coming from R, Stata, or SAS: [https://pandas.pydata.org/docs/getting_started/comparison/comparison_with_r.html](https://pandas.pydata.org/docs/getting_started/comparison/comparison_with_r.html) + +## Click + +Click is pretty easy to understand: it lets you wrap your Python functions with decorators to make the code run on the command line. We use Click to do a lot of our database management. + +The [homepage for Click's documentation](https://click.palletsprojects.com/en/7.x/) should give you a good idea of what Click is all about. Additionally, Flask's documentation has a page [here](https://flask.palletsprojects.com/en/1.1.x/cli/) that discusses Flask's integration with Click. + +## Postgresql + +We do not do anything crazy with Postgresql. We made a deliberate decision to only use SQL for retrieving and storing data, and to avoid some of the more intermediate to advanced aspects of Postgres such as CTEs, views, and so on. Actual data manipulation is done in Pandas. + +A simple [intro SQL tutorial](https://www.khanacademy.org/computing/computer-programming/sql) should be more than sufficient for understanding the SQL we use in this code base. diff --git a/docs/docs/development_resources/shell.md b/docs/docs/development_resources/shell.md new file mode 100644 index 00000000..7c01f8c9 --- /dev/null +++ b/docs/docs/development_resources/shell.md @@ -0,0 +1,92 @@ +# Flask Shell Documentation + +The shell is used to access app functions and data, such as Hobolink and USGS +data and access to the database. + +The reason why the shell is useful is because there may be cases where you want to play around with the app's functions. For example, maybe you see something that seems fishy in the data, so you want to have direct access to the function the website is running. You may also want to + +The way Flask works makes it impossible to run the website's functions outside the Flask app context, which means importing the functions into a naked shell doesn't work as intended. The `flask shell` provides all the tools needed to let coders access the functions the exact same way the website does, except in a shell environment. + +## Run the Shell + +1. Open up a terminal at the `flagging` folder. + +2. Activate a Python virtual environment: + +```shell +python3 -m venv venv +source venv/bin/activate +python3 -m pip install -r requirements.txt +``` + +3. Set up the `FLASK_ENV` environment variable: + +```shell +export FLASK_ENV=development +``` + +4. Run the shell: + +```shell +flask shell +``` + +And you should be good to go! The functions listed below should be available for use, and the section below contains some example use cases for the shell. + +???+ tip + To exit from the shell, type `exit()` then ++enter++. + +## Available Shell Functions and Variables + +- **`app`** (*flask.Flask*): + The actual Flask app instance. +- **`db`** (*flask_sqlalchemy.SQLAlchemy*): + The object used to interact with the Postgres database. +- **`get_live_hobolink_data`** (*(Optional[str]) -> pd.DataFrame*): + Gets the HOBOlink data table based on the given "export" name. +- **`get_live_usgs_data`** (*() -> pd.DataFrame*): + Gets the USGS data table. +- **`get_data`** (*() -> pd.DataFrame*): + Gets the Hobolink and USGS data tables and returns a combined table. +- **`process_data`** (*(pd.DataFrame, pd.DataFrame) -> pd.DataFrame*): + Combines the Hobolink and USGS tables. +- **`compose_tweet`** (*() -> str*): + Generates a message for Twitter that represents the current status of the flagging program (note: this function does not actually send the Tweet to Twitter.com). + +Additionally, Pandas and Numpy are already pre-imported via `#!python import pandas as pd` and `#!python import numpy as np`. + +???+ tip + To add more functions and variables that pre-load in the Flask shell, simply add another entry to the dictionary returned by the function `#!python make_shell_context()` in `flagging_site/app.py:creat_app()`. + +???+ tip + All of the website's functions can be run in the Flask shell, even those that are not pre-loaded in the shell's global context. All you need to do is import it. For example, let's say you want to get the un-parsed request object from USGS.gov. You can import the function we use and run it like this: + + ```python + # (in Flask shell) + from flagging_site.data.usgs import request_to_usgs + res = request_to_usgs() + print(res.json()) + ``` + +## Example 1: Export Hobolink Data to CSV + +Here we assume you have already started the Flask shell. +This example shows how to download the Hobolink data and +save it as a CSV file. + +```python +# (in Flask shell) +hobolink_data = get_live_hobolink_data() +hobolink_data.to_csv('path/where/to/save/my-CSV-file.csv') +``` + +Downloading the data may be useful if you want to see + +## Example 2: Preview Tweet + +Let's say you want to preview a Tweet that would be sent out without actually sending it. The `compose_tweet()` function returns a string of this message: + +```python +# (in Flask shell) +print(compose_tweet()) +``` diff --git a/docs/docs/export.md b/docs/docs/export.md new file mode 100644 index 00000000..0ddb1e20 --- /dev/null +++ b/docs/docs/export.md @@ -0,0 +1,4 @@ +# Export Data + +## HTML iFrame + diff --git a/docs/docs/img/github_fork.png b/docs/docs/img/github_fork.png new file mode 100644 index 00000000..c0b9e32a Binary files /dev/null and b/docs/docs/img/github_fork.png differ diff --git a/docs/docs/img/scheduler_config.png b/docs/docs/img/scheduler_config.png new file mode 100644 index 00000000..26436487 Binary files /dev/null and b/docs/docs/img/scheduler_config.png differ diff --git a/docs/docs/img/successful_run.png b/docs/docs/img/successful_run.png new file mode 100644 index 00000000..f6a16853 Binary files /dev/null and b/docs/docs/img/successful_run.png differ diff --git a/docs/docs/index.md b/docs/docs/index.md index cc8e042b..8f8db631 100644 --- a/docs/docs/index.md +++ b/docs/docs/index.md @@ -1,9 +1,30 @@ -# Safe Water Project Guide +!!! note + This project is still currently under development. If you are interested in joining our team and contributing, [read our project wiki](https://github.com/codeforboston/flagging/wiki) for more info. -Welcome to the Safe Water MKDocs! +Welcome to the CRWA Flagging Website Documentation! -This site provides first-time users background about this project and guide to deploying and using the website. +This site provides developers and maintainers information about the CRWA's flagging website, including information on: deploying the website, developing the website, using the website's code locally, and using the website's admin panel. -## Quick Blurb of this Project +## For Website Administrators -Charles River Watershed Association monitors Charles river for diseases and originally had out-of-comissioned PHP website that keep tracks of its temperature and volume flow. This project aims to update the website wih a new website using flask framework. More information about deployment, background, and how it works in other web pages. +If the website is already deployed and you want to implement a manual override, you do not need to follow the setup guide. All you need to do is read the [admin](admin) guide to manage the website while it's deployed. + +## Connecting to Weebly + +The outputs of the model can be exported as an iFrame, which allows the flagging website's live data to be viewed on a statically rendered web page (such as those hosted by Weebly). + +To export the model outputs using an iFrame, use the following HTML: + +```html +
+ +``` + +## For Developers + +Start by following the [setup guide](setup). Once you have the website setup locally, you now have access to the following: + +- Deploy the website to Heroku (guide [here](deployment)) +- Manually run commands and download data through the [shell](shell). +- Make changes to the predictive model, including revising its coefficients. (Guide is currently WIP) +- (Advanced) Make other changes to the website. diff --git a/docs/docs/setup.md b/docs/docs/setup.md new file mode 100644 index 00000000..08b54886 --- /dev/null +++ b/docs/docs/setup.md @@ -0,0 +1,165 @@ +# Setup + +This is a guide on how to do your first-time setup for running the website locally and getting ready to make changes to the code base. If you are a developer, you should follow this guide before doing anything else! + +This guide assumes you're a complete beginner at Python, Git, Postgres, etc. so don't be intimidated if you don't know what all of these things are. The main thing this guide assumes is that you know how to open up a terminal in your respective operating system (command prompt or "CMD" in Windows, and bash in OSX). + +## Dependencies + +Install all of the following programs onto your computer: + +**Required:** + +- [Python 3](https://www.python.org/downloads/) - specifically 3.7 or higher +- [Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) (first time setup guide [here](https://git-scm.com/book/en/v2/Getting-Started-First-Time-Git-Setup)) +- [Postgres](https://www.postgresql.org/) _(see installation instructions below)_ +- [7zip](https://www.7-zip.org/) (If on OSX, install via Homebrew: `brew install p7zip`) +- _(OSX only)_ [Homebrew](https://brew.sh/) + +**Recommended:** + +- A good text editor or IDE, such as [Atom.io](https://atom.io/) (which is lightweight and beginner friendly) or [PyCharm](https://www.jetbrains.com/pycharm/) (which is powerful but bulky and geared toward advanced users). +- [Heroku CLI](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) _(required for remote deployment to Heroku.)_ + +**Other:** + +- It is strongly recommend that you create a [GitHub account](https://github.com/) if you haven't done so already. The GitHub account should have the same email as the one registered to your `git config --global user.email` that you set in the first time git setup. + +???+ warning + _(Windows users only)_ At least two Windows users have had problems getting Python working in Windows for the first time. Check out some people troubleshooting various Python installation related issues [on StackOverflow](https://stackoverflow.com/questions/13596505/python-not-working-in-command-prompt). Also note that the command to run Python in Windows may be `python`, `python3`, `py`, or `py3`. Figure out which one works for you. + +### Postgres installation + +=== "Windows (CMD)" + 1. Download [here](https://www.postgresql.org/download/windows/) and install via the executable. + + 2. (If you had any terminals open, close out and reopen after Postgres installation.) + + 3. Open command prompt and try the following (case-sensitive): `psql -V` If it returns the version number then you're set. + + 4. If you get an error about the command not being recognized, then it might mean you need to manually add Postgres's bin to your PATH ([see here](https://stackoverflow.com/a/11567231)). + +=== "OSX (Bash)" + + 1. If you do not have Homebrew installed, install it from [here](https://brew.sh/). + + 2. Via a bash terminal: `brew install postgres` + + 3. Test that it works by running (case-sensitive): `psql -V`. If it returns the version number then you're set. + +???+ tip + Chances are you are not going to need Postgres to run in the background constantly, so you should learn how to turn it off and back on. + + === "Windows (CMD)" + + **Turn Postgres on/off:** + + 1. Go to the Start menu and open up "Run..." + + 2. `services.msc` -> ++enter++. This opens the Services panel. + + 3. Look for the name _postgresql_ and start/stop Postgres. + + **Keep Postgres from running at startup:** + + (Via the Services panel) As long as the service is "manual" and not automatic, it will not load at startup. + + === "OSX (Bash)" + **Turn Postgres on:** + + ```shell + pg_ctl -D /usr/local/var/postgres start + ``` + + **Turn Postgres off:** + + ```shell + pg_ctl -D /usr/local/var/postgres stop + ``` + + **Keep Postgres from running at startup:** + + Some solutions [here](https://superuser.com/questions/244589/prevent-postgresql-from-running-at-startup). + +## Download and Setup the Code Base + +The flagging website is open source; the whole website's source code is available on GitHub. This section of the setup guide shows you the preferred way to install it and set up the code on a local computer. + +1. Fork the [main GitHub repo](https://github.com/codeforboston/flagging/) to your personal GitHub account. You can do that by going to the Code For Boston flagging repo and clicking on this button: + +![](img/github_fork.png) + +2. Point your terminal (Bash on OSX, command prompt on Windows) to the directory you want to put the `/flagging` project folder inside of. E.g. if you want the project folder to be located at `/Documents/flagging`, then point your directory to `/Documents`. You can change directories using the `cd` command: `cd "path/goes/here"` + +3. Run the following to download your fork and setup the connection to the upstream remote. Replace `YOUR_USERNAME_HERE` (in the first line) with your actual GitHub username. + +```shell +git clone https://github.com/YOUR_USERNAME_HERE/flagging/ +cd flagging +git remote add upstream https://github.com/codeforboston/flagging.git +git fetch upstream +``` + +4. In your newly created `flagging` folder, create a file called `.env` and add the vault password to it. (Replace `vault_password_here` with the actual vault password if you have it; otherwise just copy+paste and run the command as-is): + +```shell +echo "VAULT_PASSWORD=vault_password_here" > .env +``` + +???+ danger + If you do any commits to the repo, _please make sure `.env` is properly gitignored!_ (`.flaskenv` does not need to be gitignored, only `.env`.) The vault password is sensitive information; it should not be shared with others and it should not be posted online publicly. + +## Run the Website Locally + +???+ note + From here on in the documentation, each terminal command assumes your terminal's working directory is pointed toward the `flagging` directory. + +After you get everything set up, you should run the website at least once. Te process of running the website installs the remaining dependencies, and sets up a virtual environment to work in. + +1. Run the following: + +=== "Windows (CMD)" + ```shell + run_windows_dev + ``` + +=== "OSX (Bash)" + ```shell + sh run_unix_dev.sh + ``` + +???+ note + The script being run is doing the following, in order: + + 1. Set up a "virtual environment" (basically an isolated folder inside your project directory that we install the Python packages into), + 2. install the packages inside of `requirements.txt`; this can take a while during your first time. + 3. Set up some environment variables that Flask needs. + 4. Prompts the user to set some options for the deployment. (See step 2 below.) + 5. Set up the Postgres database and update it with data. + 6. Run the actual website. + +???+ tip + If you are receiving any errors related to the Postgres database and you are certain that Postgres is running on your computer, you can modify the `POSTGRES_USERNAME` and `POSTGRES_PASSWORD` environment variables to connect to your local Postgres instance properly. + +2. You will be prompted asking if you want to run the website in offline mode. "Offline mode" is a way to run the website with dummy data without accessing the credentials. It is useful for anyone who wants to run a demo of the website regardless of their affiliation with the CRWA or this project. It can also be useful for development purposes. + +???+ tip + - If you have a working `VAULT_PASSWORD`, type ++n++ -> ++enter++. This runs the website as normal. + - If you do _not_ have the `VAULT_PASSWORD`, type ++y++ -> ++enter++ to turn offline mode on. + +3. Now just wait for the database to start filling in and for the website to eventually run. + +???+ success + You should be good if you eventually see something like the following in your terminal: + + ``` + * Serving Flask app "flagging_site:create_app" (lazy loading) + * Environment: development + * Debug mode: on + * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit) + * Restarting with stat + ``` + +4. Point your browser of choice to the URL shown in the terminal output. If everything worked out, the website should be running on your local computer! + +![](img/successful_run.png) diff --git a/docs/docs/system.md b/docs/docs/system.md deleted file mode 100644 index dad6054f..00000000 --- a/docs/docs/system.md +++ /dev/null @@ -1,8 +0,0 @@ -# Website Explained - -## Diagram -![Image of Yaktocat](Flaggin_System.png) - -## Explanation - -Here is a tentative explanantion of how the website works. Currently it is a flask web application that creates a main web application using `create_app()` function and retrieve configuration options from `config.py` and keys from the `vault.zip`. Then joins mini web apps by registering blueprints found inside the `blueprints` directory. Particularly the main web app will be joining web app `flagging.py` to retrieve data from USGS and Hobolink api. With this information, we generate predictive data based on multiple logistic models to determine if river is safe or not. The website displays that data calling `render_template()` which renders `output_model.html` with the Jinja template engine. Moreover, we save that data inside a SQL database hosted in heroku, which will also where we deploy the flask web application. diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index 24e52fba..dceeba7c 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -1,8 +1,64 @@ -site_name: Water MKDocs +site_name: CRWA Flagging Website Documentation +site_description: Guide on developing and deploying the flagging website nav: - - Home: index.md - - Background: background.md - - Deployment: deployment.md - - System: system.md - -theme: readthedocs +- Home: index.md +- About: about.md +- Setup: setup.md +- Admin: admin.md +- Cloud: + - Overview: cloud/index.md + - Heroku Deployment: cloud/heroku_deployment.md + - Twitter Bot: cloud/twitter_bot.md +- Development: + - Overview: development/index.md + # - Config: development/config.md + - Data: development/data.md + - Predictive Models: development/predictive_models.md + # - Front-End: development/front-end.md +- Development Resources: + - Overview: development_resources/index.md + - Learning Resources: development_resources/learning_resources.md + - Shell: development_resources/shell.md +theme: + name: material + palette: + scheme: default + primary: teal + accent: cyan + icon: + logo: fontawesome/regular/flag + font: + text: Opens Sans + code: Roboto Mono +repo_name: codeforboston/flagging +repo_url: https://github.com/codeforboston/flagging +edit_uri: "" +plugins: + - macros +markdown_extensions: +- admonition +- pymdownx.tabbed # https://facelessuser.github.io/pymdown-extensions/ +- pymdownx.keys +- pymdownx.details +- pymdownx.inlinehilite +- pymdownx.magiclink: + repo_url_shorthand: true + user: squidfunk + repo: mkdocs-material +- pymdownx.superfences: + custom_fences: + - name: mermaid + class: mermaid + format: !!python/name:pymdownx.superfences.fence_div_format +- sane_lists +extra: + flagging_website_url: https://crwa-flagging.herokuapp.com + social: + - icon: fontawesome/brands/github + link: https://github.com/codeforboston/flagging + - icon: fontawesome/brands/meetup + link: https://www.meetup.com/Code-for-Boston/ + - icon: fontawesome/brands/twitter + link: https://twitter.com/codeforboston +extra_javascript: +- https://unpkg.com/mermaid@8.4.6/dist/mermaid.min.js diff --git a/flagging_site/__init__.py b/flagging_site/__init__.py index 11ea373b..21bfb84b 100644 --- a/flagging_site/__init__.py +++ b/flagging_site/__init__.py @@ -1,4 +1,3 @@ - -__version__ = '0.3.0' +__version__ = '1.0.0' from .app import create_app diff --git a/flagging_site/admin.py b/flagging_site/admin.py new file mode 100644 index 00000000..b58e6b73 --- /dev/null +++ b/flagging_site/admin.py @@ -0,0 +1,118 @@ +import os +from flask import Flask +from flask import redirect +from flask import request +from flask import Response +from flask_admin import Admin +from flask_admin import BaseView +from flask_admin import expose +from flask_admin.contrib import sqla +from werkzeug.exceptions import HTTPException + +from flask_basicauth import BasicAuth +from werkzeug.exceptions import HTTPException + +from .data import db + +admin = Admin(template_mode='bootstrap3') + +basic_auth = BasicAuth() + + +# Taken from https://computableverse.com/blog/flask-admin-using-basicauth +class AuthException(HTTPException): + def __init__(self, message): + """HTTP Forbidden error that prompts for login""" + super().__init__(message, Response( + 'You could not be authenticated. Please refresh the page.', + status=401, + headers={'WWW-Authenticate': 'Basic realm="Login Required"'} + )) + + +def init_admin(app: Flask): + """Registers the Flask-Admin extensions to the app, and attaches the + model views to the admin panel. + + Args: + app: A Flask application instance. + """ + basic_auth.init_app(app) + admin.init_app(app) + + with app.app_context(): + # Register /admin sub-views + from .data.cyano_overrides import CyanoOverridesModelView + admin.add_view(CyanoOverridesModelView(db.session)) + admin.add_view(LogoutView(name="Logout")) + + +# Adapted from https://computableverse.com/blog/flask-admin-using-basicauth +class AdminModelView(sqla.ModelView): + """ + Extension of SQLAlchemy ModelView that requires BasicAuth authentication, + and shows all columns in the form (including primary keys). + """ + + def __init__(self, model, *args, **kwargs): + # Show all columns in form + self.column_list = [c.key for c in model.__table__.columns] + self.form_columns = self.column_list + + super().__init__(model, *args, **kwargs) + + def is_accessible(self): + """ + Protect admin pages with basic_auth. + If logged out and current page is /admin/, then ask for credentials. + Otherwise, raises HTTP 401 error and redirects user to /admin/ on the + frontend (redirecting with HTTP redirect causes user to always be + redirected to /admin/ even after logging in). + + We redirect to /admin/ because our logout method only works if the path to + /logout is the same as the path to where we put in our credentials. So if + we put in credentials at /admin/cyanooverride, then we would need to logout + at /admin/cyanooverride/logout, which would be difficult to arrange. Instead, + we always redirect to /admin/ to put in credentials, and then logout at + /admin/logout. + """ + if not basic_auth.authenticate(): + if '/admin/' == request.path: + raise AuthException('Not authenticated. Refresh the page.') + else: + raise HTTPException( + 'Attempted to visit admin page but not authenticated.', + Response( + ''' + Not authenticated. Navigate to /admin/ to login. + + ''', + status=401 # 'Forbidden' status + ) + ) + else: + return True + + def inaccessible_callback(self, name, **kwargs): + """Ask for credentials when access fails""" + return redirect(basic_auth.challenge()) + + +class LogoutView(BaseView): + @expose('/') + def index(self): + """ + To log out of basic auth for admin pages, + we raise an HTTP 401 error (there isn't really a cleaner way) + and then redirect on the frontend to home. + """ + raise HTTPException( + 'Logged out.', + Response( + ''' + Successfully logged out. + + ''', + status=401 + ) + ) diff --git a/flagging_site/app.py b/flagging_site/app.py index a723042a..2e4c923f 100644 --- a/flagging_site/app.py +++ b/flagging_site/app.py @@ -2,15 +2,27 @@ This file handles the construction of the Flask application object. """ import os +import click +import time +import json +import decimal +import datetime from typing import Optional +from typing import Dict +from typing import Union + from flask import Flask +from flask.json import JSONEncoder + +import py7zr +from lzma import LZMAError +from py7zr.exceptions import PasswordRequired -from .data.keys import get_keys from .config import Config from .config import get_config_from_env -def create_app(config: Optional[Config] = None) -> Flask: +def create_app(config: Optional[Union[Config, str]] = None) -> Flask: """Create and configure an instance of the Flask application. We use the `create_app` scheme over defining the `app` directly at the module level so the app isn't loaded immediately by importing the module. @@ -21,13 +33,16 @@ def create_app(config: Optional[Config] = None) -> Flask: Returns: The fully configured Flask app instance. """ - app = Flask(__name__, instance_relative_config=True) + app = Flask(__name__) # Get a config for the website. If one was not passed in the function, then # a config will be used depending on the `FLASK_ENV`. - if not config: + if config is None: # Determine the config based on the `FLASK_ENV`. config = get_config_from_env(app.env) + elif isinstance(config, str): + # If config is string, parse it as if it's an env. + config = get_config_from_env(config) app.config.from_object(config) @@ -39,52 +54,119 @@ def create_app(config: Optional[Config] = None) -> Flask: # blueprints are imported is: If BLUEPRINTS is in the config, then import # only from that list. Otherwise, import everything that's inside of # `blueprints/__init__.py`. - from . import blueprints - register_blueprints_from_module(app, blueprints) + from .blueprints.api import bp as api_bp + app.register_blueprint(api_bp) + + from .blueprints.flagging import bp as flagging_bp + app.register_blueprint(flagging_bp) # Add Swagger to the app. Swagger automates the API documentation and # provides an interface for users to query the API on the website. - add_swagger_plugin_to_app(app) + init_swagger(app) # Register the database commands - # from .data import db - # db.init_app(app) + from .data import db + db.init_app(app) - # And we're all set! We can hand the app over to flask at this point. - return app + # Register admin + from .admin import init_admin + init_admin(app) + # Register Twitter bot + from .twitter import init_tweepy + init_tweepy(app) -def update_config_from_vault(app: Flask) -> None: - """ - This updates the state of the `app` to have the keys from the vault. The - vault also stores the "SECRET_KEY", which is a Flask builtin configuration - variable (i.e. Flask treats the "SECRET_KEY" as special). So we also - populate the "SECRET_KEY" in this step. + class CustomJSONEncoder(JSONEncoder): + """Add support for Decimal types""" - If we fail to load the vault in development mode, then the user is warned - that the vault was not loaded successfully. In production mode, failing to - load the vault raises a RuntimeError. + def default(self, o): + if isinstance(o, decimal.Decimal): + return float(o) + elif isinstance(o, datetime.date): + return o.isoformat() + else: + return super().default(o) - Args: - app: A Flask application instance. - """ - try: - app.config['KEYS'] = get_keys() - except (RuntimeError, KeyError): - msg = 'Unable to load the vault; bad password provided.' - if app.env == 'production': - raise RuntimeError(msg) + app.json_encoder = CustomJSONEncoder + + @app.before_request + def before_request(): + from flask import g + g.request_start_time = time.time() + g.request_time = lambda: '%.3fs' % (time.time() - g.request_start_time) + + @app.cli.command('create-db') + def create_db_command(): + """Create database (after verifying that it isn't already there).""" + from .data.database import create_db + if create_db(): + click.echo('The database was created.') else: - print(f'Warning: {msg}') - app.config['KEYS'] = None - app.config['SECRET_KEY'] = None - else: - app.config['SECRET_KEY'] = app.config['KEYS']['flask']['secret_key'] + click.echo('The database was already there.') + + @app.cli.command('init-db') + def init_db_command(): + """Clear existing data and create new tables.""" + from .data.database import init_db + init_db() + click.echo('Initialized the database.') + + @app.cli.command('update-db') + def update_db_command(): + """Update the database with the latest live data.""" + from .data.database import update_database + updated = update_database() + click.echo('Updated the database.') + if not updated: + click.echo('Note: while updating database, the predictive model ' + 'did not run.') + return updated + + @app.cli.command('update-website') + @click.pass_context + def update_website_command(ctx): + """Updates the database, then Tweets a message.""" + updated = ctx.invoke(update_db_command) + if updated: + from .twitter import tweet_current_status + msg = tweet_current_status() + click.echo(f'Sent out tweet: {msg!r}') + + # Make a few useful functions available in Flask shell without imports + @app.shell_context_processor + def make_shell_context(): + import pandas as pd + import numpy as np + from flask import current_app + from .blueprints.flagging import get_data + from .data import db + from .data.hobolink import get_live_hobolink_data + from .data.predictive_models import process_data + from .data.usgs import get_live_usgs_data + from .twitter import compose_tweet + + return { + 'pd': pd, + 'np': np, + 'app': current_app, + 'db': db, + 'get_data': get_data, + 'get_live_hobolink_data': get_live_hobolink_data, + 'get_live_usgs_data': get_live_usgs_data, + 'process_data': process_data, + 'compose_tweet': compose_tweet + } + + # And we're all set! We can hand the app over to flask at this point. + return app -def add_swagger_plugin_to_app(app: Flask): - """This function hnadles all the logic for adding Swagger automated +def init_swagger(app: Flask): + """This function handles all the logic for adding Swagger automated documentation to the application instance. + + Args: + app: A Flask application instance. """ from flasgger import Swagger from flasgger import LazyString @@ -94,11 +176,11 @@ def add_swagger_plugin_to_app(app: Flask): 'headers': [], 'specs': [ { - 'endpoint': 'reach_api', - 'route': '/api/reach_api.json', + 'endpoint': 'flagging_api', + 'route': '/api/flagging_api.json', 'rule_filter': lambda rule: True, # all in 'model_filter': lambda tag: True, # all in - } + }, ], 'static_url_path': '/flasgger_static', # 'static_folder': '/static/flasgger', @@ -119,26 +201,75 @@ def add_swagger_plugin_to_app(app: Flask): } app.config['SWAGGER'] = { 'uiversion': 3, - 'favicon': LazyString(lambda: url_for('static', filename='favicon/favicon.ico')) + 'favicon': LazyString( + lambda: url_for('static', filename='favicon/favicon.ico')) } Swagger(app, config=swagger_config, template=template) -def register_blueprints_from_module(app: Flask, module: object) -> None: +def _load_secrets_from_vault( + password: str, + vault_file: str +) -> Dict[str, Union[str, Dict[str, str]]]: + """This code loads the keys directly from the vault zip file. + + The schema of the vault's `secrets.json` file looks like this: + + >>> { + >>> "SECRET_KEY": str, + >>> "HOBOLINK_AUTH": { + >>> "password": str, + >>> "user": str, + >>> "token": str + >>> }, + >>> "TWITTER_AUTH": { + >>> "api_key": str, + >>> "api_key_secret": str, + >>> "access_token": str, + >>> "access_token_secret": str, + >>> "bearer_token": str + >>> } + >>> } + + Args: + vault_password: (str) Password for opening up the `vault_file`. + vault_file: (str) File path of the zip file containing `keys.json`. + + Returns: + Dict of credentials. """ - This function looks within the submodules of a module for objects - specifically named `bp`. It then assumes those objects are blueprints, and - registers them to the app. + with py7zr.SevenZipFile(vault_file, mode='r', password=password) as f: + archive = f.readall() + d = json.load(archive['secrets.json']) + return d + + +def update_config_from_vault(app: Flask) -> None: + """ + This updates the state of the `app` to have the secrets from the vault. The + vault also stores the "SECRET_KEY", which is a Flask builtin configuration + variable (i.e. Flask treats the "SECRET_KEY" as special). So we also + populate the "SECRET_KEY" in this step. + + If we fail to load the vault in development mode, then the user is warned + that the vault was not loaded successfully. In production mode, failing to + load the vault raises a RuntimeError. Args: - app: (Flask) Flask instance to which we will register blueprints. - module: (object) A module that contains submodules which themselves - contain `bp` objects. + app: A Flask application instance. """ - if app.config.get('BLUEPRINTS'): - blueprint_list = app.config['BLUEPRINTS'] - else: - blueprint_list = filter(lambda x: not x.startswith('_'), dir(module)) - for submodule in blueprint_list: - app.register_blueprint(getattr(module, submodule).bp) + try: + secrets = _load_secrets_from_vault( + password=app.config['VAULT_PASSWORD'], + vault_file=app.config['VAULT_FILE'] + ) + # Add 'SECRET_KEY', 'HOBOLINK_AUTH', AND 'TWITTER_AUTH' to the config. + app.config.update(secrets) + except (LZMAError, PasswordRequired, KeyError): + msg = 'Unable to load the vault; bad password provided.' + if app.config.get('VAULT_OPTIONAL'): + print(f'Warning: {msg}') + app.config['SECRET_KEY'] = os.urandom(16) + else: + raise RuntimeError(msg) diff --git a/flagging_site/blueprints/__init__.py b/flagging_site/blueprints/__init__.py index 8600ca44..417875cd 100644 --- a/flagging_site/blueprints/__init__.py +++ b/flagging_site/blueprints/__init__.py @@ -1,3 +1,2 @@ -from . import cyanobacteria from . import flagging from . import api diff --git a/flagging_site/blueprints/api.py b/flagging_site/blueprints/api.py index 1367279a..800d7648 100644 --- a/flagging_site/blueprints/api.py +++ b/flagging_site/blueprints/api.py @@ -1,41 +1,27 @@ -from typing import Optional from typing import List import pandas as pd from flask import Blueprint from flask import render_template from flask import request -from flask_restful import Api -from flask_restful import Resource from flask import current_app - -from ..data.hobolink import get_live_hobolink_data -from ..data.usgs import get_live_usgs_data -from ..data.model import process_data -from ..data.model import reach_2_model -from ..data.model import reach_3_model -from ..data.model import reach_4_model -from ..data.model import reach_5_model +from flask import jsonify +from ..data.predictive_models import latest_model_outputs +from ..data.predictive_models import MODEL_VERSION +from ..data.database import get_boathouse_metadata_dict +from ..data.database import execute_sql from flasgger import swag_from bp = Blueprint('api', __name__, url_prefix='/api') -api = Api(bp) @bp.route('/', methods=['GET']) def index() -> str: + """Landing page for the API.""" return render_template('api/index.html') -def get_data() -> pd.DataFrame: - """Retrieves the data that gets plugged into the the model.""" - df_hobolink = get_live_hobolink_data('code_for_boston_export_21d') - df_usgs = get_live_usgs_data() - df = process_data(df_hobolink, df_usgs) - return df - - def add_to_dict(models, df, reach) -> None: """ Iterates through dataframe from model output, adds to model dict where @@ -53,58 +39,74 @@ def add_to_dict(models, df, reach) -> None: models[f'reach_{reach}'] = df.to_dict(orient='list') -def model_api(reaches: Optional[List[int]], hours: Optional[int]) -> dict: +def model_api(reaches: List[int], hours: int) -> dict: """ Class method that retrieves data from hobolink and usgs and processes data, then creates json-like dictionary structure for model output. returns: json-like dictionary """ - # Set defaults - if reaches is None: - reaches = [2, 3, 4, 5] - if hours is None: - hours = 24 + # First step is to validate inputs - # `hours` must be between 1 and `API_MAX_HOURS` + # `hours` must be an integer between 1 and `API_MAX_HOURS`. Default is 24 if hours > current_app.config['API_MAX_HOURS']: hours = current_app.config['API_MAX_HOURS'] elif hours < 1: hours = 1 - - df = get_data() - - dfs = { - 2: reach_2_model, - 3: reach_3_model, - 4: reach_4_model, - 5: reach_5_model + # `reaches` must be a list of integers. Default is all the reaches. + + # get model output data from database + df = latest_model_outputs(hours) + return { + 'model_version': MODEL_VERSION, + 'time_returned': pd.to_datetime('today'), + 'model_outputs': [ + { + 'predictions': df.loc[ + df['reach'] == int(reach), : + ].drop(columns=['reach']).to_dict(orient='records'), + 'reach': reach + } + for reach in reaches + ] } - main = {} - models = {} - # adds metadata - main['version'] = '2020' - main['time_returned'] = str(pd.to_datetime('today')) +# ======================================== +# The REST API endpoints are defined below +# ======================================== - for reach, model_func in dfs.items(): - if reach in reaches: - _df = model_func(df, hours) - add_to_dict(models, _df, reach) - # adds models dict to main dict - main['models'] = models +@bp.route('/v1/model') +@swag_from('predictive_model_api.yml') +def predictive_model_api(): + """Returns JSON of the predictive model outputs.""" + reaches = request.args.getlist('reach', type=int) or [2, 3, 4, 5] + hours = request.args.get('hours', type=int) or 24 + return jsonify(model_api(reaches, hours)) - return main +@bp.route('/v1/boathouses') +@swag_from('boathouses_api.yml') +def boathouses_api(): + """Returns JSON of the boathouses.""" + boathouse_metadata_dict = get_boathouse_metadata_dict() + return jsonify(boathouse_metadata_dict) -class ReachesApi(Resource): - @swag_from('reach_api.yml') - def get(self): - reaches = request.args.getlist('reach', type=int) - hours = request.args.get('hours', type=int) - return model_api(reaches, hours) +@bp.route('/v1/model_input_data') +@swag_from('model_input_data_api.yml') +def model_input_data_api(): + """Returns records of the data used for the model.""" + df = execute_sql('''SELECT * FROM processed_data ORDER BY time''') + + # Parse the hours + hours = request.args.get('hours', type=int) or 24 + if hours > current_app.config['API_MAX_HOURS']: + hours = current_app.config['API_MAX_HOURS'] + elif hours < 1: + hours = 1 -api.add_resource(ReachesApi, '/v1/model') + return jsonify({ + 'model_input_data': df.tail(n=hours).to_dict(orient='records') + }) diff --git a/flagging_site/blueprints/boathouses_api.yml b/flagging_site/blueprints/boathouses_api.yml new file mode 100644 index 00000000..94b572f7 --- /dev/null +++ b/flagging_site/blueprints/boathouses_api.yml @@ -0,0 +1,30 @@ +JSON of boathouse metadata +--- +tags: + - Boathouse API +responses: + 200: + description: JSON records of the boathouses covered by the predictive model + schema: + id: boathouses + type: object + properties: + boathouses: + description: Records of the boathouses + type: array + items: + type: object + properties: + boathouse: + description: Name of the boathouse + type: string + latitude: + description: Latitude of boathouse's location + type: number + longitude: + description: Longitude of boathouse's location + type: number + reach: + description: Reach that the boathouse is associated with + type: integer + enum: [2, 3, 4, 5] diff --git a/flagging_site/blueprints/cyanobacteria.py b/flagging_site/blueprints/cyanobacteria.py deleted file mode 100644 index 39750c04..00000000 --- a/flagging_site/blueprints/cyanobacteria.py +++ /dev/null @@ -1,7 +0,0 @@ -from flask import Blueprint - -bp = Blueprint('cyanobacteria', __name__, url_prefix='/cyanobacteria') - -# @bp.route('/') -# def index() -> str: -# return 'Hello, world!' diff --git a/flagging_site/blueprints/flagging.py b/flagging_site/blueprints/flagging.py index 62819822..c5739574 100644 --- a/flagging_site/blueprints/flagging.py +++ b/flagging_site/blueprints/flagging.py @@ -1,28 +1,17 @@ import pandas as pd + from flask import Blueprint from flask import render_template from flask import request from flask import current_app -from ..data.hobolink import get_live_hobolink_data -from ..data.usgs import get_live_usgs_data -from ..data.model import process_data -from ..data.model import reach_2_model -from ..data.model import reach_3_model -from ..data.model import reach_4_model -from ..data.model import reach_5_model +from ..data.cyano_overrides import get_currently_overridden_reaches +from ..data.predictive_models import latest_model_outputs +from ..data.database import get_boathouse_by_reach_dict bp = Blueprint('flagging', __name__) -def get_data() -> pd.DataFrame: - """Retrieves the processed data that gets plugged into the the model.""" - df_hobolink = get_live_hobolink_data('code_for_boston_export_21d') - df_usgs = get_live_usgs_data() - df = process_data(df_hobolink, df_usgs) - return df - - def stylize_model_output(df: pd.DataFrame) -> str: """ This function function stylizes the dataframe that we will output for our @@ -35,6 +24,8 @@ def stylize_model_output(df: pd.DataFrame) -> str: Returns: HTML table. """ + df = df.copy() + def _apply_flag(x: bool) -> str: flag_class = 'blue-flag' if x else 'red-flag' return f'{x}' @@ -42,59 +33,48 @@ def _apply_flag(x: bool) -> str: df['safe'] = df['safe'].apply(_apply_flag) df.columns = [i.title().replace('_', ' ') for i in df.columns] + # remove reach number + df = df.drop(columns=['Reach']) + return df.to_html(index=False, escape=False) +def parse_model_outputs(df: pd.DataFrame) -> dict: + df = df.set_index('reach') + + overridden_reaches = get_currently_overridden_reaches() + + flags = { + reach: val['safe'] and reach not in overridden_reaches + for reach, val + in df.to_dict(orient='index').items() + } + + boathouse_statuses = get_boathouse_by_reach_dict() + + # verify that the same reaches are in boathouse list and model outputs + if flags.keys() != boathouse_statuses.keys(): + print('ERROR! the reaches are\'t identical between boathouse list and model outputs!') + + for flag_reach, flag_safe in flags.items(): + boathouse_statuses[flag_reach]['flag'] = flag_safe + + return boathouse_statuses + + @bp.route('/') def index() -> str: """ The home page of the website. This page contains a brief description of the purpose of the website, and the latest outputs for the flagging model. - - Returns: - The website's home page with the latest flag updates. """ - df = get_data() - - homepage = { - 2: { - 'flag': reach_2_model(df, rows=1)['safe'].iloc[0], - 'boathouses': [ - 'Newton Yacht Club', - 'Watertown Yacht Club', - 'Community Rowing, Inc.', - 'Northeastern\s Henderson Boathouse', - 'Paddle Boston at Herter Park' - ] - }, - 3: { - 'flag': reach_3_model(df, rows=1)['safe'].iloc[0], - 'boathouses': [ - 'Harvard\'s Weld Boathouse' - ] - }, - 4: { - 'flag': reach_4_model(df, rows=1)['safe'].iloc[0], - 'boathouses': [ - 'Riverside Boat Club' - ] - }, - 5: { - 'flag': reach_5_model(df, rows=1)['safe'].iloc[0], - 'boathouses': [ - 'Charles River Yacht Club', - 'Union Boat Club', - 'Community Boating', - 'Paddle Boston at Kendall Square' - ] - } - } - - model_last_updated_time = reach_5_model(df, rows=1)['time'].iloc[0] + df = latest_model_outputs() + homepage = parse_model_outputs(df) + model_last_updated_time = df['time'].iloc[0] - return render_template('index.html', homepage=homepage, model_last_updated_time=model_last_updated_time) - # return render_template('index.html', flags=flags) - + return render_template('index.html', + homepage=homepage, + model_last_updated_time=model_last_updated_time) @bp.route('/about') @@ -121,25 +101,33 @@ def output_model() -> str: # Look at no more than x_MAX_HOURS hours = min(max(hours, 1), current_app.config['API_MAX_HOURS']) - df = get_data() + df = latest_model_outputs(hours) - reach_model_mapping = { - 2: reach_2_model, - 3: reach_3_model, - 4: reach_4_model, - 5: reach_5_model - } - - if reach in reach_model_mapping: - reach_func = reach_model_mapping[int(reach)] - reach_html_tables = { - reach: stylize_model_output(reach_func(df, rows=hours)) - } - else: - reach_html_tables = { - reach: stylize_model_output(reach_func(df, rows=hours)) - for reach, reach_func - in reach_model_mapping.items() - } + # reach_html_tables is a dict where the index is the reach number + # and the values are HTML code for the table of data to display for + # that particular reach + reach_html_tables = {} + + # loop through each reach in df + # compare with reach to determine whether to display that reach + # extract the subset from the df for that reach + # then convert that df subset to HTML code + # and then add that HTML subset to reach_html_tables + for i in df['reach'].unique(): + if (reach == -1 or reach == i): + reach_html_tables[i] = stylize_model_output(df.loc[df['reach'] == i]) return render_template('output_model.html', tables=reach_html_tables) + + +@bp.route('/flags') +def flags() -> str: + # TODO: Update to use combination of Boathouses and the predictive model + # outputs + df = latest_model_outputs() + boathouse_statuses = parse_model_outputs(df) + model_last_updated_time = df['time'].iloc[0] + + return render_template('flags.html', + boathouse_statuses=boathouse_statuses, + model_last_updated_time=model_last_updated_time) diff --git a/flagging_site/blueprints/model_input_data_api.yml b/flagging_site/blueprints/model_input_data_api.yml new file mode 100644 index 00000000..a5634898 --- /dev/null +++ b/flagging_site/blueprints/model_input_data_api.yml @@ -0,0 +1,17 @@ +JSON of model input data +--- +tags: + - Model Input Data API +responses: + 200: + description: JSON records of the processed input data used to make predictions + schema: + id: model_input_data + type: object + properties: + model_input_data: + type: array + description: All fields used to predict the state of the Charles River. The schema for the input data is not + maintained in these docs as the specific fields may be subject to occasional change. + items: + type: object diff --git a/flagging_site/blueprints/predictive_model_api.yml b/flagging_site/blueprints/predictive_model_api.yml new file mode 100644 index 00000000..fc2d8159 --- /dev/null +++ b/flagging_site/blueprints/predictive_model_api.yml @@ -0,0 +1,71 @@ +JSON of the predictive model outputs +--- +tags: + - Predictive Model API +parameters: + - name: reach + description: The reach (or reaches) to return model results for. + in: query + type: array + collectionFormat: multi + required: false + default: [2, 3, 4, 5] + items: + type: string + enum: [2, 3, 4, 5] + - name: hours + description: Number of hours of data to return. + in: query + type: integer + enum: [1, 2, 3, 6, 12, 24, 36, 48] + required: false + default: 24 +responses: + 200: + description: Dictionary-like json of the output model + schema: + id: predictive_models + type: object + properties: + version: + description: The model version. Each model version corresponds with its own unique set of coefficients and/or + features. Typically this will be the year of the model. + type: string + time_returned: + description: The timestamp of when the model was run. + type: string + model_outputs: + description: The most recent predictive model outputs + type: array + items: + type: object + properties: + reach: + description: The reach that these model outputs correspond to. + type: integer + predictions: + description: Records of the predictive model outputs. + type: array + items: + type: object + properties: + time: + description: Timestamp for the model results. + type: array + items: + type: string + log-odds: + description: Log odds output of the model. + type: array + items: + type: number + probability: + description: Probability output of the model. + type: array + items: + type: number + safe: + description: Indication of whether or not the water is safe according to the model. + type: array + items: + type: boolean diff --git a/flagging_site/blueprints/reach_api.yml b/flagging_site/blueprints/reach_api.yml deleted file mode 100644 index 3549de16..00000000 --- a/flagging_site/blueprints/reach_api.yml +++ /dev/null @@ -1,64 +0,0 @@ -Endpoint returning json of the output model ---- -tags: - - Reach Model API -parameters: - - name: reach - description: The reach (or reaches) to return model results for. - in: query - type: array - collectionFormat: multi - required: false - default: [2, 3, 4, 5] - items: - type: string - enum: [2, 3, 4, 5] - - name: hours - description: Number of hours of data to return. - in: query - type: int - enum: [1, 2, 3, 6, 12, 24, 36, 48] - required: false - default: 24 -responses: - 200: - description: Dictionary-like json of the output model - schema: - id: model_api - type: object - properties: - version: - description: The model version. Each model version corresponds with its own unique set of coefficients and/or - features. Typically this will be the year of the model. - type: string - time_returned: - description: The timestamp of when the model was run. - type: string - models: - description: - type: object - properties: - model_num: - description: The outputs of the model for the given reach number. - type: object - properties: - time: - description: Timestamp for the model results. - type: array - items: - type: string - log-odds: - description: Log odds output of the model. - type: array - items: - type: number - probability: - description: Probability output of the model. - type: array - items: - type: number - safe: - description: Indication of whether or not the water is safe according to the model. - type: array - items: - type: boolean diff --git a/flagging_site/config.py b/flagging_site/config.py index b443bb12..188c2811 100644 --- a/flagging_site/config.py +++ b/flagging_site/config.py @@ -6,26 +6,26 @@ this module, they won't refresh. """ import os -from typing import Dict, Any, Optional, List +import re from flask.cli import load_dotenv +from distutils.util import strtobool # Constants # ~~~~~~~~~ ROOT_DIR = os.path.abspath(os.path.dirname(__file__)) +QUERIES_DIR = os.path.join(ROOT_DIR, 'data', 'queries') DATA_STORE = os.path.join(ROOT_DIR, 'data', '_store') -VAULT_FILE = os.path.join(ROOT_DIR, 'vault.zip') +VAULT_FILE = os.path.join(ROOT_DIR, 'vault.7z') -# Dotenv -# ~~~~~~ -# If you are using a .env file, please double check that it is gitignored. -# The `.flaskenv` file should not be gitignored, only `.env`. -# See this for more: -# https://flask.palletsprojects.com/en/1.1.x/cli/ -load_dotenv(os.path.join(ROOT_DIR, '..', '.flaskenv')) -load_dotenv(os.path.join(ROOT_DIR, '..', '.env')) +# Load dotenv +# ~~~~~~~~~~~ + +if os.getenv('FLASK_ENV') == 'development': + load_dotenv(os.path.join(ROOT_DIR, '..', '.flaskenv')) + load_dotenv(os.path.join(ROOT_DIR, '..', '.env')) # Configs @@ -51,11 +51,40 @@ def __repr__(self): # ========================================================================== # DATABASE CONFIG OPTIONS - # - # Not currently used, but soon we'll want to start using the config to set - # up references to the database, data storage, and data retrieval. # ========================================================================== - DATABASE: str = None + POSTGRES_USER: str = os.getenv('POSTGRES_USER', os.getenv('USER', 'postgres')) + POSTGRES_PASSWORD: str = os.getenv('POSTGRES_PASSWORD') + POSTGRES_HOST: str = 'localhost' + POSTGRES_PORT: str = '5432' + POSTGRES_DBNAME: str = 'flagging' + + @property + def SQLALCHEMY_DATABASE_URI(self) -> str: + """ + Returns the URI for the Postgres database. + + Example: + >>> Config().SQLALCHEMY_DATABASE_URI + 'postgres://postgres:password_here@localhost:5432/flagging' + """ + if self.OFFLINE_MODE == False: + user = self.POSTGRES_USER + password = self.POSTGRES_PASSWORD + host = self.POSTGRES_HOST + port = self.POSTGRES_PORT + db = self.POSTGRES_DBNAME + return f'postgres://{user}:{password}@{host}:{port}/{db}' + else: + return os.getenv('DATABASE_URL') + + SQLALCHEMY_ECHO: bool = True + SQLALCHEMY_RECORD_QUERIES: bool = True + SQLALCHEMY_TRACK_MODIFICATIONS: bool = False + + QUERIES_DIR: str = QUERIES_DIR + """Directory that contains various queries that are accessible throughout + the rest of the code base. + """ # ========================================================================== # MISC. CUSTOM CONFIG OPTIONS @@ -70,13 +99,26 @@ def __repr__(self): wasn't opened. Usually set alongside DEBUG mode. """ - KEYS: Dict[str, Dict[str, Any]] = None - """These are where the keys from the vault are stored. It should be a dict - of dicts. Each key in the first level dict corresponds to a different - service that needs keys / secured credentials stored. - - Currently, HOBOlink and Flask's `SECRET_KEY` are the two services that pass - through the vault. + VAULT_PASSWORD: str = os.getenv('VAULT_PASSWORD') + + HOBOLINK_AUTH: dict = { + 'password': None, + 'user': None, + 'token': None + } + """Note: Do not fill these out manually; the HOBOlink auth gets populated + from the vault. + """ + + TWITTER_AUTH: dict = { + 'api_key': None, + 'api_key_secret': None, + 'access_token': None, + 'access_token_secret': None, + 'bearer_token': None + } + """Note: Do not fill these out manually; the Twitter auth gets populated + from the vault. """ VAULT_FILE: str = VAULT_FILE @@ -100,26 +142,79 @@ def __repr__(self): when doing requests. """ - BLUEPRINTS: Optional[List[str]] = None - """Names of the blueprints available to the app. We can use this to turn - parts of the website off or on depending on if they're fully developed - or not. If BLUEPRINTS is `None`, then it imports all the blueprints it can - find in the `blueprints` module. + API_MAX_HOURS: int = 48 + """The maximum number of hours of data that the API will return. We are not + trying to be stingy about our data, we just want this in order to avoid any + odd behaviors if the user requests more data than exists. """ - API_MAX_HOURS: int = 48 - """The maximum number of hours of data that the API will return. We are not trying - to be stingy about our data, we just want this in order to avoid any odd behaviors - if the user requests more data than exists. + SEND_TWEETS: bool = strtobool(os.getenv('SEND_TWEETS') or 'false') + """If True, the website behaves normally. If False, any time the app would + send a Tweet, it does not do so. It is useful to turn this off when + developing to test Twitter messages. """ + BASIC_AUTH_USERNAME: str = os.getenv('BASIC_AUTH_USERNAME', 'admin') + BASIC_AUTH_PASSWORD: str = os.getenv('BASIC_AUTH_PASSWORD', 'password') + class ProductionConfig(Config): """The Production Config is used for deployment of the website to the internet. Currently the only part of the website that's pretty fleshed out is the `flagging` part, so that's the only blueprint we import. """ - BLUEPRINTS: Optional[List[str]] = ['flagging', 'api'] + SEND_TWEETS: str = True + + def __init__(self): + """Initializing the production config allows us to ensure the existence + of these variables in the environment.""" + try: + self.VAULT_PASSWORD: str = os.environ['VAULT_PASSWORD'] + self.BASIC_AUTH_USERNAME: str = os.environ['BASIC_AUTH_USERNAME'] + self.BASIC_AUTH_PASSWORD: str = os.environ['BASIC_AUTH_PASSWORD'] + except KeyError: + msg = ( + 'You did not set all of the environment variables required to ' + 'initiate the app in production mode. If you are deploying ' + 'the website to Heroku, read the Deployment docs page to ' + 'learn how to set env variables in Heroku.' + ) + raise KeyError(msg) + + # Production does things in reverse: Instead of defining the database + # using the POSTGRES_* environment variables, the database is set with + # the `DATABASE_URL` (provided automatically by Postgres), and the + # POSTGRES_* variables are not used. + # + # In the rare event that they are needed in production, they are set + # to be consistent with the `DATABASE_URL` below: + postgres_url_schema = re.compile(''' + ^ + postgres:// + ([^\s:@/]+) # Username + :([^\s:@/]+) # Password + @([^\s:@/]+) # Host + :([^\s:@/]+) # Port + /([^\s:@/]*?) # Database + $ + ''', re.VERBOSE) + matches = re.search(postgres_url_schema, self.SQLALCHEMY_DATABASE_URI) + self.POSTGRES_USER = matches.group(1) + self.POSTGRES_PASSWORD = matches.group(2) + self.POSTGRES_HOST = matches.group(3) + self.POSTGRES_PORT = matches.group(4) + self.POSTGRES_DBNAME = matches.group(5) + + @property + def SQLALCHEMY_DATABASE_URI(self) -> str: + return os.getenv('DATABASE_URL') + + +class StagingConfig(ProductionConfig): + """The ProductionConfig is the as the ProductionConfig, except it does not + send Tweets unless told to with a `SEND_TWEETS` environment variable. + """ + SEND_TWEETS = strtobool(os.getenv('SEND_TWEETS') or 'false') class DevelopmentConfig(Config): @@ -138,13 +233,14 @@ class DevelopmentConfig(Config): VAULT_OPTIONAL: bool = True DEBUG: bool = True TESTING: bool = True - OFFLINE_MODE = bool(os.getenv('OFFLINE_MODE', 'false')) + OFFLINE_MODE = strtobool(os.getenv('OFFLINE_MODE') or 'false') class TestingConfig(Config): """The Testing Config is used for unit-testing and integration-testing the website. """ + SEND_TWEETS: bool = False TESTING: bool = True @@ -166,12 +262,13 @@ def get_config_from_env(env: str) -> Config: """ config_mapping = { 'production': ProductionConfig, + 'staging': StagingConfig, 'development': DevelopmentConfig, - 'testing': TestingConfig + 'testing': TestingConfig, } try: config = config_mapping[env] except KeyError: raise KeyError('Bad config passed; the config must be production, ' 'development, or testing.') - return config() + return config() \ No newline at end of file diff --git a/flagging_site/data/README.md b/flagging_site/data/README.md index 2921d1a8..b8ddad45 100644 --- a/flagging_site/data/README.md +++ b/flagging_site/data/README.md @@ -8,3 +8,4 @@ - `model.py`: outputs table model by processing usgs and hobolink data - `task_queue`: set up task queue - `usgs.py`: retrieve hobolink by requesting a response and parsing data from usgs + diff --git a/flagging_site/data/__init__.py b/flagging_site/data/__init__.py index cbeb04b6..a5b67987 100644 --- a/flagging_site/data/__init__.py +++ b/flagging_site/data/__init__.py @@ -2,3 +2,4 @@ The data module contains exactly what you'd expect: everything related to data processing, collection, and storage. """ +from .database import db diff --git a/flagging_site/data/_store/refresh.py b/flagging_site/data/_store/refresh.py index fb38dfb5..21db83f6 100644 --- a/flagging_site/data/_store/refresh.py +++ b/flagging_site/data/_store/refresh.py @@ -5,6 +5,8 @@ This file is a CLI to refresh the data store. You can run it with: `python flagging_site/data/_store/refresh.py` + + """ import os import sys @@ -12,6 +14,8 @@ import click +DATA_STORE_PATH = os.path.dirname(__file__) + @click.command() @click.option('--vault_password', prompt=True, @@ -31,21 +35,17 @@ def refresh_data_store(vault_password: Optional[str] = None) -> None: raise Exception('The app should not be running when the data store is ' 'being refreshed.') - from flagging_site.data.keys import get_data_store_file_path - from flagging_site.data.hobolink import get_live_hobolink_data - from flagging_site.data.hobolink import STATIC_FILE_NAME as hobolink_file + from flagging_site.data.hobolink import HOBOLINK_STATIC_FILE_NAME get_live_hobolink_data('code_for_boston_export_21d')\ - .to_pickle(get_data_store_file_path(hobolink_file)) + .to_pickle(os.path.join(DATA_STORE_PATH, HOBOLINK_STATIC_FILE_NAME)) from flagging_site.data.usgs import get_live_usgs_data - from flagging_site.data.usgs import STATIC_FILE_NAME as usgs_file - get_live_usgs_data().to_pickle(get_data_store_file_path(usgs_file)) + from flagging_site.data.usgs import USGS_STATIC_FILE_NAME + get_live_usgs_data()\ + .to_pickle(os.path.join(DATA_STORE_PATH, USGS_STATIC_FILE_NAME)) if __name__ == '__main__': - try: - sys.path.append('.') - refresh_data_store() - finally: - sys.path.remove('.') + sys.path.append('.') + refresh_data_store() diff --git a/flagging_site/data/cyano_overrides.py b/flagging_site/data/cyano_overrides.py new file mode 100644 index 00000000..03b5c02b --- /dev/null +++ b/flagging_site/data/cyano_overrides.py @@ -0,0 +1,39 @@ +from typing import Set + +from sqlalchemy import Column +from sqlalchemy import Integer +from sqlalchemy import TIMESTAMP +from sqlalchemy import VARCHAR + +from ..admin import AdminModelView +from .database import Base +from .database import execute_sql_from_file + + +class CyanoOverrides(Base): + __tablename__ = 'cyano_overrides' + reach = Column(Integer, primary_key=True) + start_time = Column(TIMESTAMP, primary_key=True) + end_time = Column(TIMESTAMP, primary_key=True) + reason = Column(VARCHAR(255)) + + +class CyanoOverridesModelView(AdminModelView): + form_choices = { + 'reason': [ + ('cyanobacteria', 'Cyanobacteria'), + ('sewage', 'Sewage'), + ('other', 'Other'), + ] + } + + def __init__(self, session): + super().__init__(CyanoOverrides, session) + + +def get_currently_overridden_reaches() -> Set[int]: + return set( + execute_sql_from_file( + 'currently_overridden_reaches.sql' + )["reach"].unique() + ) diff --git a/flagging_site/data/database.py b/flagging_site/data/database.py index fe2bdea2..44c1fdf9 100644 --- a/flagging_site/data/database.py +++ b/flagging_site/data/database.py @@ -1,4 +1,208 @@ +"""This file handles all database stuff, i.e. writing and retrieving data to +the Postgres database. Note that of the functionality in this file is available +directly in the command line. + +While the app is running, the database connection is managed by SQLAlchemy. The +`db` object defined near the top of the file is that connector, and is used +throughout both this file and other files in the code base. The `db` object is +connected to the actual database in the `create_app` function: the app instance +is passed in via `db.init_app(app)`, and the `db` object looks for the config +variable `SQLALCHEMY_DATABASE_URI`. """ -This file should handle all database connection stuff, namely: writing and -retrieving data. -""" \ No newline at end of file +import os +import pandas as pd +from typing import Optional +from flask import current_app +from flask_sqlalchemy import SQLAlchemy +from flask_sqlalchemy import declarative_base +from sqlalchemy.exc import ResourceClosedError +from psycopg2 import connect +from dataclasses import dataclass + +db = SQLAlchemy() +Base = declarative_base() + + +def execute_sql(query: str) -> Optional[pd.DataFrame]: + """Execute arbitrary SQL in the database. This works for both read and + write operations. If it is a write operation, it will return None; + otherwise it returns a Pandas dataframe. + + Args: + query: (str) A string that contains the contents of a SQL query. + + Returns: + Either a Pandas Dataframe the selected data for read queries, or None + for write queries. + """ + with db.engine.connect() as conn: + res = conn.execute(query) + try: + df = pd.DataFrame( + res.fetchall(), + columns=res.keys() + ) + return df + except ResourceClosedError: + return None + + +def execute_sql_from_file(file_name: str) -> Optional[pd.DataFrame]: + """Execute SQL from a file in the `QUERIES_DIR` directory, which should be + located at `flagging_site/data/queries`. + + Args: + file_name: (str) A file name inside the `QUERIES_DIR` directory. It + should be only the file name alone and not the full path. + + Returns: + Either a Pandas Dataframe the selected data for read queries, or None + for write queries. + """ + path = os.path.join(current_app.config['QUERIES_DIR'], file_name) + with current_app.open_resource(path) as f: + return execute_sql(f.read().decode('utf8')) + + +def create_db() -> bool: + """If the database defined by `POSTGRES_DBNAME` doesn't exist, create it + and return True, otherwise do nothing and return False. By default, the + config variable `POSTGRES_DBNAME` is set to "flagging". + + Returns: + bool for whether the database needed to be created. + """ + + # connect to postgres database, get cursor + conn = connect( + user=current_app.config['POSTGRES_USER'], + password=current_app.config['POSTGRES_PASSWORD'], + host=current_app.config['POSTGRES_HOST'], + port=current_app.config['POSTGRES_PORT'], + dbname='postgres' + ) + cursor = conn.cursor() + + # get a list of all databases: + cursor.execute('SELECT datname FROM pg_database;') + + # create a list of all available database names: + db_list = cursor.fetchall() + db_list = [d[0] for d in db_list] + + # if that database is already there, exit out of this function + if current_app.config['POSTGRES_DBNAME'] in db_list: + return False + # if the database isn't already there, proceed ... + + # create the database + cursor.execute('COMMIT;') + cursor.execute('CREATE DATABASE ' + current_app.config['POSTGRES_DBNAME']) + cursor.execute('COMMIT;') + + return True + + +def init_db(): + """This data clears and then populates the database from scratch. You only + need to run this function once per instance of the database. + """ + with current_app.app_context(): + # This file drops the tables if they already exist, and then defines + # the tables. This is the only query that CREATES tables. + execute_sql_from_file('schema.sql') + + # The boathouses table is populated. This table doesn't change, so it + # only needs to be populated once. + execute_sql_from_file('define_boathouse.sql') + + # The function that updates the database periodically is run for the + # first time. + update_database() + + # The models available in Base are given corresponding tables if they + # do not already exist. + Base.metadata.create_all(db.engine) + + +def database_is_empty(): + execute_sql('SELECT * FROM asdf') + + +def update_database(): + """This function basically controls all of our data refreshes. The + following tables are updated in order: + + - usgs + - hobolink + - processed_data + - model_outputs + + The functions run to calculate the data are imported from other files + within the data folder. + """ + options = { + 'con': db.engine, + 'index': False, + 'if_exists': 'replace' + } + + # Populate the `usgs` table. + from .usgs import get_live_usgs_data + df_usgs = get_live_usgs_data() + df_usgs.to_sql('usgs', **options) + + # Populate the `hobolink` table. + from .hobolink import get_live_hobolink_data + df_hobolink = get_live_hobolink_data() + df_hobolink.to_sql('hobolink', **options) + + # Populate the `processed_data` table. + from .predictive_models import process_data + df = process_data(df_hobolink=df_hobolink, df_usgs=df_usgs) + df.to_sql('processed_data', **options) + + # Populate the `model_outputs` table. + from .predictive_models import all_models + model_outs = all_models(df) + model_outs.to_sql('model_outputs', **options) + + return True + + +@dataclass +class Boathouses(db.Model): + reach: int = db.Column(db.Integer, unique=False) + boathouse: str = db.Column(db.String(255), primary_key=True) + latitude: float = db.Column(db.Numeric, unique=False) + longitude: float = db.Column(db.Numeric, unique=False) + + +def get_boathouse_by_reach_dict(): + """ + Return a dict of boathouses, indexed by reach + """ + # return value is an outer dictionary with the reach number as the keys + # and the a sub-dict as the values each sub-dict has the string 'boathouses' + # as the key, and an array of boathouse names as the value + boathouse_dict = {} + + # outer boathouse loop: take one reach at a time + for bh_out in Boathouses.query.distinct(Boathouses.reach): + bh_list = [] + # inner boathouse loop: get all boathouse names within + # the reach (the reach that was selected by outer loop) + for bh_in in Boathouses.query.filter(Boathouses.reach == bh_out.reach).all(): + bh_list.append(bh_in.boathouse) + + boathouse_dict[bh_out.reach] = {'boathouses': bh_list} + + return boathouse_dict + + +def get_boathouse_metadata_dict(): + """ + Return a dictionary of boathouses' metadata + """ + boathouse_query = (Boathouses.query.all()) + return {'boathouses': boathouse_query} diff --git a/flagging_site/data/hobolink.py b/flagging_site/data/hobolink.py index e792e6f9..838d66e9 100644 --- a/flagging_site/data/hobolink.py +++ b/flagging_site/data/hobolink.py @@ -2,22 +2,17 @@ This file handles connections to the HOBOlink API, including cleaning and formatting of the data that we receive from it. """ -# TODO: -# Pandas is inefficient. It should go to SQL, not to Pandas. I am currently -# using pandas because we do not have any cron jobs or any caching or SQL, but -# I think in future versions we should not be using Pandas at all. +import os import io import requests import pandas as pd from flask import abort -from .keys import get_keys -from .keys import offline_mode -from .keys import get_data_store_file_path +from flask import current_app # Constants HOBOLINK_URL = 'http://webservice.hobolink.com/restv2/data/custom/file' -EXPORT_NAME = 'code_for_boston_export' +DEFAULT_HOBOLINK_EXPORT_NAME = 'code_for_boston_export_21d' # Each key is the original column name; the value is the renamed column. HOBOLINK_COLUMNS = { 'Time, GMT-04:00': 'time', @@ -33,11 +28,13 @@ 'Temp': 'air_temp', # 'Batt, V, Charles River Weather Station': 'battery' } -STATIC_FILE_NAME = 'hobolink.pickle' +HOBOLINK_STATIC_FILE_NAME = 'hobolink.pickle' # ~ ~ ~ ~ -def get_live_hobolink_data(export_name: str = EXPORT_NAME) -> pd.DataFrame: +def get_live_hobolink_data( + export_name: str = DEFAULT_HOBOLINK_EXPORT_NAME +) -> pd.DataFrame: """This function runs through the whole process for retrieving data from HOBOlink: first we perform the request, and then we clean the data. @@ -48,8 +45,11 @@ def get_live_hobolink_data(export_name: str = EXPORT_NAME) -> pd.DataFrame: Returns: Pandas Dataframe containing the cleaned-up Hobolink data. """ - if offline_mode(): - df = pd.read_pickle(get_data_store_file_path(STATIC_FILE_NAME)) + if current_app.config['OFFLINE_MODE']: + fpath = os.path.join( + current_app.config['DATA_STORE'], HOBOLINK_STATIC_FILE_NAME + ) + df = pd.read_pickle(fpath) else: res = request_to_hobolink(export_name=export_name) df = parse_hobolink_data(res.text) @@ -57,7 +57,7 @@ def get_live_hobolink_data(export_name: str = EXPORT_NAME) -> pd.DataFrame: def request_to_hobolink( - export_name: str = EXPORT_NAME, + export_name: str = DEFAULT_HOBOLINK_EXPORT_NAME, ) -> requests.models.Response: """ Get a request from the Hobolink server. @@ -71,7 +71,7 @@ def request_to_hobolink( """ data = { 'query': export_name, - 'authentication': get_keys()['hobolink'] + 'authentication': current_app.config['HOBOLINK_AUTH'] } res = requests.post(HOBOLINK_URL, json=data) @@ -93,15 +93,16 @@ def parse_hobolink_data(res: str) -> pd.DataFrame: Returns: Pandas DataFrame containing the HOBOlink data. """ - # TODO: - # The first half of the output is a yaml-formatted text stream. Is there - # anything useful in it? Can we store it and make use of it somehow? if isinstance(res, requests.models.Response): res = res.text - # Turn the text from the API response into a Pandas DataFrame. + # The first half of the text from the response is a yaml. The part below + # the yaml is the actual data. The following lines split the text and grab + # the csv: split_by = '------------' str_table = res[res.find(split_by) + len(split_by):] + + # Turn the text from the API response into a Pandas DataFrame. df = pd.read_csv(io.StringIO(str_table), sep=',') # There is a weird issue in the HOBOlink data where it sometimes returns @@ -133,6 +134,6 @@ def parse_hobolink_data(res: str) -> pd.DataFrame: df = df.loc[df['water_temp'].notna()] # Convert time column to Pandas datetime - df['time'] = pd.to_datetime(df['time']) + df['time'] = pd.to_datetime(df['time'], format='%m/%d/%y %H:%M:%S') return df diff --git a/flagging_site/data/keys.py b/flagging_site/data/keys.py deleted file mode 100644 index d4492524..00000000 --- a/flagging_site/data/keys.py +++ /dev/null @@ -1,84 +0,0 @@ -""" -This file handles their access credentials and tokens for various APIs required -to retrieve data for the website. This file also handles retrieving config -variables, which are either stored in the application config or (if there is no -active Flask app context) the system environment. - -The file that contains the credentials is called "vault.zip", and is referenced -by a constant, `VAULT_FILE`. This file is accessed using a password stored in -the config called `VAULT_PASSWORD`. - -Inside the "vault.zip" file, there is a file named "keys.yml." And this is the -file with all the credentials (plus a Flask secret key). It looks like this: - - flask: - secret_key: "" - hobolink: - password: "" - user: "crwa" - token: "" -""" -import os -import zipfile -import json -from distutils.util import strtobool -from flask import current_app - -from flagging_site.config import VAULT_FILE - - -def get_keys() -> dict: - """Retrieves the keys from the `current_app` if it exists. If not, then this - function tries to load directly from the vault. The reason this function - exists is so that you can use the API wrappers regardless of whether or not - the Flask app is running. - - Note that this function requires that you assign the vault password to the - environmental variable named `VAULT_PASSWORD`. - - Returns: - The full keys dict. - """ - if current_app: - d = current_app.config['KEYS'] - else: - vault_file = os.getenv('VAULT_FILE') or VAULT_FILE - d = load_keys_from_vault(vault_password=os.environ['VAULT_PASSWORD'], - vault_file=vault_file) - return d.copy() - - -def load_keys_from_vault( - vault_password: str, - vault_file: str = VAULT_FILE -) -> dict: - """This code loads the keys directly from the vault zip file. Users should - preference using the `get_keys()` function over this function. - - Args: - vault_password: (str) Password for opening up the `vault_file`. - vault_file: (str) File path of the zip file containing `keys.json`. - - Returns: - Dict of credentials. - """ - pwd = bytes(vault_password, 'utf-8') - with zipfile.ZipFile(vault_file) as f: - with f.open('keys.json', pwd=pwd, mode='r') as keys_file: - d = json.load(keys_file) - return d - - -def offline_mode() -> bool: - if current_app: - return current_app.config['OFFLINE_MODE'] - else: - return bool(strtobool(os.getenv('OFFLINE_MODE', 'false'))) - - -def get_data_store_file_path(file_name: str) -> str: - if current_app: - return os.path.join(current_app.config['DATA_STORE'], file_name) - else: - from ..config import DATA_STORE - return os.path.join(DATA_STORE, file_name) diff --git a/flagging_site/data/model.py b/flagging_site/data/predictive_models.py similarity index 72% rename from flagging_site/data/model.py rename to flagging_site/data/predictive_models.py index a0b46e23..5917d881 100644 --- a/flagging_site/data/model.py +++ b/flagging_site/data/predictive_models.py @@ -36,6 +36,8 @@ import numpy as np import pandas as pd +MODEL_VERSION = '2020' + SIGNIFICANT_RAIN = 0.2 SAFETY_THRESHOLD = 0.65 @@ -79,7 +81,7 @@ def process_data( .agg({ 'pressure': np.mean, 'par': np.mean, - 'rain': sum, + 'rain': np.sum, 'rh': np.mean, 'dew_point': np.mean, 'wind_speed': np.mean, @@ -105,28 +107,16 @@ def process_data( if df.iloc[-1, :][['stream_flow', 'rain']].isna().any(): df = df.drop(df.index[-1]) - # Next, do the following: - # - # 1 day avg of: - # - wind_speed - # - water_temp - # - air_temp - # - stream_flow (via USGS) - # 2 day avg of: - # - par - # - stream_flow (via USGS) - # sum of rain at following increments: - # - 1 day - # - 2 day - # - 7 day - for col in ['par', 'stream_flow']: - df[f'{col}_1d_mean'] = df[col].rolling(24).mean() - - for incr in [24, 48]: - df[f'rain_0_to_{str(incr)}h_sum'] = df['rain'].rolling(incr).sum() - df[f'rain_24_to_48h_sum'] = ( - df[f'rain_0_to_48h_sum'] - df[f'rain_0_to_24h_sum'] - ) + # The code from here on consists of feature transformations. + + # Calculate rolling means + df['par_1d_mean'] = df['par'].rolling(24).mean() + df['stream_flow_1d_mean'] = df['stream_flow'].rolling(24).mean() + + # Calculate rolling sums + df[f'rain_0_to_24h_sum'] = df['rain'].rolling(24).sum() + df[f'rain_0_to_48h_sum'] = df['rain'].rolling(48).sum() + df[f'rain_24_to_48h_sum'] = df[f'rain_0_to_48h_sum'] - df[f'rain_0_to_24h_sum'] # Lastly, they measure the "time since last significant rain." Significant # rain is defined as a cumulative sum of 0.2 in over a 24 hour time period. @@ -134,7 +124,6 @@ def process_data( df['last_sig_rain'] = ( df['time'] .where(df['sig_rain']) - .shift() .ffill() .fillna(df['time'].min()) ) @@ -145,7 +134,7 @@ def process_data( return df -def reach_2_model(df: pd.DataFrame, rows: int = 24) -> pd.DataFrame: +def reach_2_model(df: pd.DataFrame, rows: int = 48) -> pd.DataFrame: """Model params: a- rainfall sum 0-24 hrs d- Days since last rain @@ -154,6 +143,7 @@ def reach_2_model(df: pd.DataFrame, rows: int = 24) -> pd.DataFrame: Args: df: Input data from `process_data()` + rows: (int) Number of rows to return. Returns: Outputs for model as a dataframe. @@ -166,12 +156,15 @@ def reach_2_model(df: pd.DataFrame, rows: int = 24) -> pd.DataFrame: - 0.0362 * df['days_since_sig_rain'] - 0.000312 * df['par_1d_mean'] ) + df['probability'] = sigmoid(df['log_odds']) df['safe'] = df['probability'] <= SAFETY_THRESHOLD - return df[['time', 'log_odds', 'probability', 'safe']] + df['reach'] = 2 + + return df[['reach', 'time', 'log_odds', 'probability', 'safe']] -def reach_3_model(df: pd.DataFrame, rows: int = 24) -> pd.DataFrame: +def reach_3_model(df: pd.DataFrame, rows: int = 48) -> pd.DataFrame: """ a- rainfall sum 0-24 hrs b- rainfall sum 24-48 hr @@ -185,7 +178,6 @@ def reach_3_model(df: pd.DataFrame, rows: int = 24) -> pd.DataFrame: Returns: Outputs for model as a dataframe. """ - df = df.tail(n=rows).copy() df['log_odds'] = ( @@ -194,12 +186,15 @@ def reach_3_model(df: pd.DataFrame, rows: int = 24) -> pd.DataFrame: + 0.1681 * df['rain_24_to_48h_sum'] - 0.02855 * df['days_since_sig_rain'] ) + df['probability'] = sigmoid(df['log_odds']) df['safe'] = df['probability'] <= SAFETY_THRESHOLD - return df[['time', 'log_odds', 'probability', 'safe']] + df['reach'] = 3 + return df[['reach', 'time', 'log_odds', 'probability', 'safe']] -def reach_4_model(df: pd.DataFrame, rows: int = 24) -> pd.DataFrame: + +def reach_4_model(df: pd.DataFrame, rows: int = 48) -> pd.DataFrame: """ a- rainfall sum 0-24 hrs b- rainfall sum 24-48 hr @@ -215,6 +210,7 @@ def reach_4_model(df: pd.DataFrame, rows: int = 24) -> pd.DataFrame: Outputs for model as a dataframe. """ df = df.tail(n=rows).copy() + df['log_odds'] = ( 0.5791 + 0.30276 * df['rain_0_to_24h_sum'] @@ -222,12 +218,15 @@ def reach_4_model(df: pd.DataFrame, rows: int = 24) -> pd.DataFrame: - 0.02267 * df['days_since_sig_rain'] - 0.000427 * df['par_1d_mean'] ) + df['probability'] = sigmoid(df['log_odds']) df['safe'] = df['probability'] <= SAFETY_THRESHOLD - return df[['time', 'log_odds', 'probability', 'safe']] + df['reach'] = 4 + + return df[['reach', 'time', 'log_odds', 'probability', 'safe']] -def reach_5_model(df: pd.DataFrame, rows: int = 24) -> pd.DataFrame: +def reach_5_model(df: pd.DataFrame, rows: int = 48) -> pd.DataFrame: """ c- rainfall sum 0-48 hr d- Days since last rain @@ -242,12 +241,53 @@ def reach_5_model(df: pd.DataFrame, rows: int = 24) -> pd.DataFrame: Outputs for model as a dataframe. """ df = df.tail(n=rows).copy() + df['log_odds'] = ( 0.3333 + 0.1091 * df['rain_0_to_48h_sum'] - 0.01355 * df['days_since_sig_rain'] + 0.000342 * df['stream_flow_1d_mean'] ) + df['probability'] = sigmoid(df['log_odds']) df['safe'] = df['probability'] <= SAFETY_THRESHOLD - return df[['time', 'log_odds', 'probability', 'safe']] + df['reach'] = 5 + + return df[['reach', 'time', 'log_odds', 'probability', 'safe']] + + +def all_models(df: pd.DataFrame, *args, **kwargs): + out = pd.concat([ + reach_2_model(df, *args, **kwargs), + reach_3_model(df, *args, **kwargs), + reach_4_model(df, *args, **kwargs), + reach_5_model(df, *args, **kwargs), + ], axis=0) + out = out.sort_values(['reach', 'time']) + return out + + +def latest_model_outputs(hours: int = 1) -> pd.DataFrame: + from .database import execute_sql_from_file + + if hours == 1: + df = execute_sql_from_file('return_1_hour_of_model_outputs.sql') + + elif hours > 1: + # pull out 48 hours of model outputs + df = execute_sql_from_file('return_48_hours_of_model_outputs.sql') + + # find most recent timestamp + latest_time = df['time'].max() + + # create pandas Timedelta, based on input parameter hours + time_interval = pd.Timedelta(str(hours) + ' hours') + + # reset df to exclude anything from before time_interval ago + df = df[latest_time - df['time'] < time_interval] + + else: + raise ValueError('Hours of data to pull must be a number and it ' + 'cannot be less than one') + + return df diff --git a/flagging_site/data/queries/currently_overridden_reaches.sql b/flagging_site/data/queries/currently_overridden_reaches.sql new file mode 100644 index 00000000..f8a660a4 --- /dev/null +++ b/flagging_site/data/queries/currently_overridden_reaches.sql @@ -0,0 +1,3 @@ +SELECT reach +FROM cyano_overrides +WHERE current_timestamp BETWEEN start_time AND end_time diff --git a/flagging_site/data/queries/define_boathouse.sql b/flagging_site/data/queries/define_boathouse.sql new file mode 100644 index 00000000..f850a5a8 --- /dev/null +++ b/flagging_site/data/queries/define_boathouse.sql @@ -0,0 +1,14 @@ +INSERT INTO boathouses + (reach, boathouse, latitude, longitude) +VALUES + (2, 'Newton Yacht Club', 42.358698, 71.172850), + (2, 'Watertown Yacht Club', 42.361952, 71.167791), + (2, 'Community Rowing, Inc.', 42.358633, 71.165467), + (2, 'Northeastern''s Henderson Boathouse', 42.364135, 71.141571), + (2, 'Paddle Boston at Herter Park', 42.369182, 71.131301), + (3, 'Harvard''s Weld Boathouse', 42.369566, 71.122083), + (4, 'Riverside Boat Club', 42.358272, 71.115763), + (5, 'Charles River Yacht Club', 42.360526, 71.084760), + (5, 'Union Boat Club', 42.357816, 71.073319), + (5, 'Community Boating', 42.359935, 71.073035), + (5, 'Paddle Boston at Kendall Square', 42.362964, 71.082112); diff --git a/flagging_site/data/queries/return_1_hour_of_model_outputs.sql b/flagging_site/data/queries/return_1_hour_of_model_outputs.sql new file mode 100644 index 00000000..3d7a773c --- /dev/null +++ b/flagging_site/data/queries/return_1_hour_of_model_outputs.sql @@ -0,0 +1,5 @@ +-- This query returns the latest values for the model. + +SELECT * +FROM model_outputs +WHERE time = (SELECT MAX(time) FROM model_outputs); diff --git a/flagging_site/data/queries/return_48_hours_of_model_outputs.sql b/flagging_site/data/queries/return_48_hours_of_model_outputs.sql new file mode 100644 index 00000000..9f1d364c --- /dev/null +++ b/flagging_site/data/queries/return_48_hours_of_model_outputs.sql @@ -0,0 +1,8 @@ +-- This query returns up to 48 hours of the latest data + +SELECT * +FROM model_outputs +WHERE time BETWEEN + (SELECT MAX(time) - interval '47 hours' FROM model_outputs) + AND + (SELECT MAX(time) FROM model_outputs) diff --git a/flagging_site/data/queries/schema.sql b/flagging_site/data/queries/schema.sql new file mode 100644 index 00000000..76c2fa53 --- /dev/null +++ b/flagging_site/data/queries/schema.sql @@ -0,0 +1,48 @@ +DROP TABLE IF EXISTS usgs; +CREATE TABLE IF NOT EXISTS usgs ( + time timestamp, + stream_flow decimal, + gage_height decimal +); + +DROP TABLE IF EXISTS hobolink; +CREATE TABLE IF NOT EXISTS hobolink ( + time timestamp, + pressure decimal, + par decimal, -- photosynthetically active radiation + rain decimal, + rh decimal, -- relative humidity + dew_point decimal, + wind_speed decimal, + gust_speed decimal, + wind_dir decimal, + water_temp decimal, + air_temp decimal +); + +DROP TABLE IF EXISTS boathouses; +CREATE TABLE IF NOT EXISTS boathouses ( + reach int, + boathouse varchar(255), + latitude decimal, + longitude decimal +); + +DROP TABLE IF EXISTS model_outputs; +CREATE TABLE IF NOT EXISTS model_outputs ( + reach int, + time timestamp, + log_odds decimal, + probability decimal, + safe boolean +); + +DROP TABLE IF EXISTS cyano_overrides; +CREATE TABLE IF NOT EXISTS cyano_overrides ( + reach int, + start_time timestamp, + end_time timestamp, + reason varchar(255) +); + +COMMIT; diff --git a/flagging_site/data/task_queue.py b/flagging_site/data/task_queue.py deleted file mode 100644 index df5a61b3..00000000 --- a/flagging_site/data/task_queue.py +++ /dev/null @@ -1,4 +0,0 @@ -""" -This file should define and set up all the tasks that need to be pushed through -the task queue, and should also handle the logic of setting up the task queue. -""" \ No newline at end of file diff --git a/flagging_site/data/usgs.py b/flagging_site/data/usgs.py index 936d2bf5..5da92650 100644 --- a/flagging_site/data/usgs.py +++ b/flagging_site/data/usgs.py @@ -5,28 +5,31 @@ Link to the web interface (not the api) https://waterdata.usgs.gov/nwis/uv?site_no=01104500 """ +import os import pandas as pd import requests from flask import abort -from .keys import offline_mode -from .keys import get_data_store_file_path +from flask import current_app # Constants USGS_URL = 'https://waterservices.usgs.gov/nwis/iv/' -STATIC_FILE_NAME = 'usgs.pickle' +USGS_STATIC_FILE_NAME = 'usgs.pickle' # ~ ~ ~ ~ def get_live_usgs_data() -> pd.DataFrame: - """This function runs through the whole process for retrieving data from - usgs: first we perform the request, and then we clean the data. + """This function runs through the whole process for retrieving data from + usgs: first we perform the request, and then we parse the data. Returns: Pandas Dataframe containing the usgs data. """ - if offline_mode(): - df = pd.read_pickle(get_data_store_file_path(STATIC_FILE_NAME)) + if current_app.config['OFFLINE_MODE']: + fpath = os.path.join( + current_app.config['DATA_STORE'], USGS_STATIC_FILE_NAME + ) + df = pd.read_pickle(fpath) else: res = request_to_usgs() df = parse_usgs_data(res) @@ -34,13 +37,12 @@ def get_live_usgs_data() -> pd.DataFrame: def request_to_usgs() -> requests.models.Response: - """ - Get a request from the USGS. + """Get a request from the USGS. Returns: Request Response containing the data from the request. """ - + payload = { 'format': 'json', 'sites': '01104500', @@ -48,7 +50,7 @@ def request_to_usgs() -> requests.models.Response: 'parameterCd': '00060,00065', 'siteStatus': 'all' } - + res = requests.get(USGS_URL, params=payload) if res.status_code // 100 in [4, 5]: error_msg = 'API request to the USGS endpoint failed with status code '\ @@ -89,7 +91,7 @@ def parse_usgs_data(res) -> pd.DataFrame: df['time'] = ( pd.to_datetime(df['time']) # Convert to Pandas datetime format .dt.tz_localize('UTC') # This is UTC; define it as such - .dt.tz_convert('US/Eastern') # Take the UTC time and convert to EST + .dt.tz_convert('US/Eastern') # Take UTC time and convert to EST .dt.tz_localize(None) # Remove the timezone from the datetime ) except TypeError: diff --git a/flagging_site/static/images/github.svg b/flagging_site/static/images/github.svg new file mode 100644 index 00000000..a235d2f4 --- /dev/null +++ b/flagging_site/static/images/github.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/flagging_site/static/images/og_preview.png b/flagging_site/static/images/og_preview.png new file mode 100644 index 00000000..87f31fda Binary files /dev/null and b/flagging_site/static/images/og_preview.png differ diff --git a/flagging_site/static/images/twitter.svg b/flagging_site/static/images/twitter.svg new file mode 100644 index 00000000..a754fd07 --- /dev/null +++ b/flagging_site/static/images/twitter.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/flagging_site/static/style.css b/flagging_site/static/style.css index f0cf052a..2ac42378 100644 --- a/flagging_site/static/style.css +++ b/flagging_site/static/style.css @@ -1,149 +1,176 @@ +/***** Start Element Styling *****/ html { font-family: sans-serif; - background: #dce9f2; - padding: 1rem; } + body { max-width: 1260px; margin: 0 auto; - background: white; + background: #dce9f2; } + h1, h2, h3, h4 { color: #27475e; - margin: 1rem 0; } + a { color: #377ba8; + text-decoration: none; } + hr { border: none; - border-top: 1px solid lightgray; + border-top: thin solid #dce9f2; } + p { line-height: 25px; } -nav h1 { - flex: auto; - margin: 0; + +ul { + line-height: 25px; + list-style-type: square; } -nav h1 a { - text-decoration: none; - padding: 0.25rem 0.5rem; + +nav { + margin: 0.5em; } -nav ul { +/***** End Element Styling *****/ + +/***** Start Common Classes *****/ +.above-fold { + /* Everything above the bottom of the screen */ + min-height: 100vh; display: flex; - list-style: none; - margin: 0; - padding: 0; -} -nav ul li a, -nav ul li span, -header .action { - display: block; - padding: 0.5rem; + flex-direction: column; } -ul { - line-height: 25px; + +.centered { + text-align: center; } -ul.home-boat-house { - list-style-type: square; - padding-left: 15px; - padding-right: 15px; - text-align: left; + +.non-breaking { + display: inline-block; } -section.page-header { +/***** End Common Classes *****/ + +/***** Start Header Section *****/ +.page-header { text-align: center; - border-bottom: 1px solid lightgray; - padding-bottom: 0.7em; + border-bottom: thin solid #dce9f2; + padding: 15px; + background: white; } + .page-header > header { padding: 0.4em 0 0.4em 0; text-align: center; font-size: 1.5em; } + +.header-logo { + max-width: 100%; +} +/***** End Header Section *****/ + +/***** Start Content Section *****/ .content { - padding: 0 1rem 1rem; + background: white; + padding: 24px; + flex-grow: 1; } + +@media(min-width: 1024px) { + .content { + padding: 48px 10%; + } +} + .content > header h1 { flex: auto; margin: 1rem 0 0.25rem 0; } -.flash { - margin: 1em 0; - padding: 1em; - background: #cae6f6; - border: 1px solid #377ba8; -} -.flag { - width: 2rem; - height: auto; - padding-top: 1rem; - padding-left: 1rem; - padding-right: 1rem; -} -.post > header { - display: flex; - align-items: flex-end; - font-size: 0.85em; -} -.post > header > div:first-of-type { - flex: auto; -} -.post > header h1 { - font-size: 1.5em; - margin-bottom: 0; -} -.post .about { - color: slategray; - font-style: italic; -} -.post .body { - white-space: pre-line; -} + .content:last-child { margin-bottom: 0; } + .content form { margin: 1em 0; display: flex; flex-direction: column; } + .content label { font-weight: bold; margin-bottom: 0.5em; } + .content input, .content textarea { margin-bottom: 1em; } + +.content textarea { + min-height: 12em; + resize: vertical; +} +/***** End Content Section *****/ + +/***** Start Footer Section *****/ .footer { + text-align: center; padding: 3em 1em 3em 1em; font-size: 0.85em; + background: white; + border-top: thin solid #dce9f2; +} + +.footer p { + line-height: normal; +} +.footer .footer-section:not(:first-child) { + padding-top: 30px; } -.footer > .footer-content { - width: 30em; + +.footer .social { + display: inline-block; + width: 1.6rem; + height: 1.6rem; text-align: center; - margin: 0 auto; } -.content textarea { - min-height: 12em; - resize: vertical; + +.footer .social img:hover { + -webkit-transition: all 0.25s; + transition: all 0.25s; + opacity: .5; } +/***** End Footer *****/ -.home-centered{ - text-align: center; +/***** Start Home Page *****/ +.home-boat-house { + margin: 5px 0; + text-align: left; } -.home-reach{ + +.home-reach { border: 1px solid black; border-collapse: collapse; padding: 15px; margin: 0 auto; } -input.danger { - color: #cc2f2e; + +.flag { + width: 2rem; + height: auto; + padding-top: 1rem; + padding-left: 1rem; + padding-right: 1rem; } -input[type="submit"] { - align-self: start; - min-width: 10em; + +.twitter-tweet { + margin: 0 auto; } + +/***** End Home Page *****/ diff --git a/flagging_site/templates/about.html b/flagging_site/templates/about.html index 512997c4..400ead70 100644 --- a/flagging_site/templates/about.html +++ b/flagging_site/templates/about.html @@ -10,7 +10,7 @@

Flagging Program

Water Contamination

- CRWA conducts two water quality monitoring events per week during the peak recreational season in the Lower Basin from late June to late October. + CRWA conducts two water quality monitoring events per week during the peak recreational season in the Lower Basin from late June to late October. During monitoring events, CRWA measures water temperature and depth and collects water quality samples at four sampling locations in the Lower Basin. G & L Laboratory in Quincy analyzes the samples for E. coli bacteria. Water temperature and depth are measured in situ with a digital field thermometer and a digital depth finder, respectively. Sampling locations are center channel sites, upstream of the following bridges: North Beacon Street, @@ -81,6 +81,14 @@

Credits

This website is open source. You can view the source code and a list of contributors - here. + here. Documentation for developing and operating this + website is available here.

+
+
+ +
+ {% endblock %} \ No newline at end of file diff --git a/flagging_site/templates/api/index.html b/flagging_site/templates/api/index.html index d9ac11a7..9974ceb8 100644 --- a/flagging_site/templates/api/index.html +++ b/flagging_site/templates/api/index.html @@ -23,6 +23,8 @@

Public Flagging API

Available Endpoints

-{% endblock %} \ No newline at end of file +{% endblock %} diff --git a/flagging_site/templates/base.html b/flagging_site/templates/base.html index 1bb0bdd1..c435b835 100644 --- a/flagging_site/templates/base.html +++ b/flagging_site/templates/base.html @@ -14,40 +14,68 @@ {% block head %} - - {% block title %}{% endblock %} - CRWA Flagging Program - + + {% block title %}{% endblock %} - CRWA Flagging Program + + + + + + + + + + + + + + {% endblock %} -
- {% block header %} - -
{{ self.title() }}
-
- Home -   |   About -   |   Detailed Outputs -   |   API +
+
+ {% block header %} + +
{{ self.title() }}
+ + {% endblock %}
- {% endblock %} -
-
{% block content %}{% endblock %}
+
{% block content %}{% endblock %}
+
{% block footer %} -
-
- Charles River Watershed Association
- 190 Park Rd, Weston, MA 02493
- t (781) 788-0007     - e charles@crwa.org
-
- This website was built by Code for Boston in - collaboration with Charles River Watershed Association.
+
+

Charles River Watershed Association

+

190 Park Rd, Weston, MA 02493

+

+ Phone: (781) 788-0007     + Email: charles@crwa.org +

+
+
+

This website was built by Code for Boston in collaboration with Charles River Watershed Association.

+
+
+   +
{% endblock %}
- \ No newline at end of file + diff --git a/flagging_site/templates/flags.html b/flagging_site/templates/flags.html new file mode 100644 index 00000000..8aae1cd5 --- /dev/null +++ b/flagging_site/templates/flags.html @@ -0,0 +1,45 @@ + +{# + This template is for the flagging iFrame, which is used to embed the flagging model in other websites. +#} + + + + + Flags - CRWA Flagging Program + + + + + + {% for reach, data in boathouse_statuses.items() %} + {% for boathouse in data['boathouses'] %} +
+ {% if data['flag'] %} + + {% else %} + + {% endif %} +
+ {{ boathouse }} +
+ {% endfor %} + {% endfor %} + +
+ ( Last updated: {{ model_last_updated_time }} ) +
+
+ Click here for more information. +
+ + \ No newline at end of file diff --git a/flagging_site/templates/index.html b/flagging_site/templates/index.html index 7260e8c4..4d28a32e 100644 --- a/flagging_site/templates/index.html +++ b/flagging_site/templates/index.html @@ -7,9 +7,9 @@

Overview

safe for boating at eleven boating locations between Watertown and Boston. Flag colors are based on E. coli and cyanobacteria (blue-green algae) levels:

-