Migrate deploy-a-site docs to this repo

README.md

@@ -18,12 +18,17 @@ _In alphabetical order and including links to external repository-based document
 - [Docker](docker/)
     - [Docker for local development](docker/local-development.md)
     - [Templates for containerizing your application](docker/templates/)
-- [GatsbyJS](gatsby/)
-- [Mapping](mapping/)
-    - [Google APIs (Maps, Geocoding)](mapping/google-apis.md)
 - [Environment setup](environment-setup.md)
+- [GatsbyJS](gatsby/)
 - [Heroku](/heroku/)
     - [Deploy a Django app to Heroku](/heroku/deploy-a-django-app.md)
+- [Logging](logging/)
+    - [Sentry](logging/sentry.md)
+    - [Slack](logging/slack.md)
+- [Mapping](mapping/)
+    - [Google APIs (Maps, Geocoding)](mapping/google-apis.md)
+- [Netlify](netlify/)
+    - [Deploy a static site to Netlify](netlify/README.md#deploy-a-static-site-to-netlify)
 - [PostgreSQL](postgres/)
     - [A quick and dirty introduction to `sqlalchemy`](postgres/quick-n-dirty-sqlalchemy.md)
     - [Interacting with a remote database](postgres/Interacting-with-a-remote-database.md)
@@ -36,7 +41,9 @@ _In alphabetical order and including links to external repository-based document
 - [Scraping](scraping/)
     - [`lxml` for web scraping](scraping/lxml-for-web-scraping.md)
 - [Searching data](search/)
-- [Security](https://bit.ly/cryptochecklist)
+- [Security](security/)
+    - [GPG and Blackbox](security/gpg/blackbox.md)
+    - [Crypto checklist](https://bit.ly/cryptochecklist)
 - [Software testing](https://github.com/datamade/testing-guidelines)
 - [The shell and Ubuntu](shell/)
     - [tmux, best practices](shell/tmux-best-practices.md)

django/README.md

@@ -12,3 +12,4 @@ In some cases, it also provides extended documentation for setup and use.
 | Autocomplete | [`django-autocomplete-light`](https://github.com/yourlabs/django-autocomplete-light) | |
 | Cross-browser ES6 support | [`django-compressor`](https://github.com/django-compressor/django-compressor) + [`django-compressor-toolkit`](https://github.com/kottenator/django-compressor-toolkit) | [Link](django-compressor.md) |
 | API | [`django-rest-framework`](https://github.com/encode/django-rest-framework) + [`django-cors-headers`](https://github.com/ottoyiu/django-cors-headers) | [Link](django-rest-framework.md) |
+| File uploads | [`django-storages`](https://django-storages.readthedocs.io/en/latest/) | [Link](file-uploads.md) | 

django/file-uploads.md

@@ -0,0 +1,142 @@
+# File uploads in Django
+
+## Contents
+
+- [Background](#background)
+- [Setting up `django-storages`](#setting-up-django-storages)
+    - [AWS configuration](#aws-configuration)
+        - [Create an S3 bucket](#create-an-s3-bucket)
+        - [Create an IAM user](#create-an-iam-user)
+    - [Django configuration](#django-configuration)
+    - [Docker configuration](#docker-configuration)
+    - [Heroku configuration](#heroku-configuration)
+
+## Background
+
+File uploads can be tricky to handle in Django because uploaded files are not stored under
+version control, and if your application is deployed on Heroku, your application
+filesystem is ephemeral.
+
+We recommend configuring Django to store file uploads in a persistent
+bucket on AWS S3 by using the [`django-storages` package](https://django-storages.readthedocs.io/en/latest/).
+This guide will walk you through the process of setting up this kind of pattern for your project.
+
+For an example of a project that uses this configuration, DataMade developers
+can see [`mn-election-archive`](https://github.com/datamade/mn-election-archive).
+
+## Setting up `django-storages`
+
+### AWS configuration
+
+Configuring AWS requires creating two sets of resources: An S3 bucket to store your
+files in, and a programmatic IAM user that is authorized to read and write files from
+your bucket.
+
+#### Create an S3 bucket
+
+In order to configure your app to store file uploads in S3, you need to have an S3
+bucket to store files in.
+
+Sign into the DataMade AWS tenant and follow the AWS docs for [creating an S3
+bucket](https://docs.aws.amazon.com/AmazonS3/latest/user-guide/create-bucket.html).
+Use the following guidelines:
+
+- Give your S3 bucket the same name as the GitHub repo for your project.
+- Create your bucket in the `us-east-1` region whenever possible.
+- Block all public access to the bucket.
+
+If you've never created an S3 bucket before, ask a Lead Developer to double-check
+your configuration before you start storing files there.
+
+#### Create an IAM user
+
+In order to read and write files to your S3 bucket, your application needs AWS credentials
+for an IAM user that has access to the bucket.
+
+Follow the AWS docs for [creating an IAM
+user](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html#id_users_create_console).
+Use the following guidelines:
+
+- Name your user after the GitHub repo for your project plus the slug `s3-user`,
+  e.g. `bga-payroll-s3-user`.
+- Select `Programmatic access` and **do not** select `AWS Management Console access`.
+- Give your user full read/write permissions for AWS S3, but restrict the resource
+  to only the S3 bucket you created and its contents. If you've never done this before,
+  ask a Lead Developer for assistance.
+
+Once you've created the user, copy the access key ID and secret key ID that AWS displays to
+you. We don't need to store these long-term, since we can always recreate them if we lose them,
+but make sure to keep them on hand to use in the next few steps.
+
+### Django Configuration
+
+Before configuring Django, update your application's `requirements.txt` file to
+add `django-storages` as a dependency. Remember to rebuild your application container
+to ensure that your dependencies are up to date.
+
+`django-storages` can be configured to store files on S3 using variables in your
+`settings.py` file. While there are [a wide array of variables you can set for the
+package](https://django-storages.readthedocs.io/en/latest/backends/amazon-S3.html#settings),
+we recommend the following settings as a baseline:
+
+```python
+DEFAULT_FILE_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
+
+# Replace 'my-s3-bucket' with the name of the bucket you created
+AWS_STORAGE_BUCKET_NAME = os.getenv('AWS_STORAGE_BUCKET_NAME', 'my-s3-bucket')
+
+AWS_LOCATION = os.getenv('AWS_LOCATION', 'dev')  # S3 prefix for uploaded objects
+
+AWS_DEFAULT_ACL = None
+
+if os.getenv('AWS_ACCESS_KEY_ID'):
+    AWS_ACCESS_KEY_ID = os.environ['AWS_ACCESS_KEY_ID']
+
+if os.getenv('AWS_SECRET_ACCESS_KEY'):
+    AWS_SECRET_ACCESS_KEY = os.environ['AWS_SECRET_ACCESS_KEY']
+```
+
+### Docker configuration
+
+In order for file uploads to work properly in development, you'll need to
+update your environment to pass your AWS credentials to your application container.
+
+Update your `docker-compose.yml` file to mount your local AWS config directory
+into your application container so you can upload files as your own user during
+development. Add the following value to the `volumes` attribute
+of your `app` service:
+
+```diff
+services:
+  app:
+    volumes:
+      # Mount the development directory as a volume into the container, so
+      # Docker automatically recognizes your changes.
+      - .:/app
++     # Mount the AWS credentials folder so you can save uploads to S3 in dev.
++     - $HOME/.aws:/root/.aws:ro
+```
+
+In addition, you'll need to adjust the `tests/docker-compose.yml` file to disable
+remote file uploads during testing:
+
+```diff
+services:
+  app:
+    enviroment:
++     DJANGO_STATICFILES_STORAGE: django.contrib.staticfiles.storage.StaticFilesStorage
+```
+
+### Heroku configuration
+
+Finally, update your Heroku configuration to set the right variables for your
+application environments. In each of your staging, production, and review app
+enviroments, set the following config variables:
+
+- `AWS_LOCATION`: `django-storages` will use this value to set the prefix (basically,
+  the containing folder) for uploaded objects in S3. This is useful for keeping
+  development, staging, and production file uploads separate. We recommend
+  setting this to the name of the environment, e.g. it should be `production`
+  in prod and `staging` for staging or review apps.
+- `AWS_ACCESS_KEY_ID`: Set the access key ID you saved during S3 setup above.
+- `AWS_SECRET_ACCESS_KEY`: Set the secret access key you saved during S3 setup above.

logging/README.md

@@ -0,0 +1,12 @@
+# Logging
+
+This directory records best practices for logging exceptions in web applications.
+
+## Guides
+
+- [Sentry: Catch exceptions in running web applications](./sentry.md)
+    - [Logging errors in Django applications](./sentry.md#logging-errors-in-django-applications)
+- [Slack: Get notified of your app errors](./slack.md)
+    - [Push Sentry notifications to Slack](./slack.md#push-sentry-notifications-to-slack)
+    - [Push Netlify notifications to Slack](./slack.md#push-netlify-notifications-to-slack)
+    - [Push Heroku notifications to Slack](./slack.md#push-heroku-notifications-to-slack)

logging/sentry.md

@@ -0,0 +1,153 @@
+# Sentry
+
+This document describes how to configure your applications to log errors to
+[Sentry](https://sentry.io/), DataMade's preferred error logging service.
+
+## Contents
+
+- [Background](#background)
+    - [A note on naming](#a-note-on-naming)
+- [Logging errors in Django applications](#logging-errors-in-django-applications)
+    - [Option 1: Use the Django app template](#option-1-use-the-django-app-template)
+    - [Option 2: Enable Sentry manually](#option-2-enable-sentry-manually)
+        - [Install the Sentry SDK](#install-the-sentry-sdk)
+        - [Thread the DSN into the app environment](#thread-the-dsn-into-the-app-environment)
+        - [Initialize `sentry_sdk` in `settings.py`](#initialize-sentrysdk-in-settingspy)
+        - [Group 400 errors](#group-400-errors)
+
+## Background
+
+We use [Sentry](https://sentry.io) as a way to log errors that occur in our applications.
+Once you're added to the DataMade organization on Sentry, you should be able to create a
+project for your application.
+
+Depending on what kind of application you're building, the code setup for Sentry
+will be different. See the docs below for setting up specific application types, but
+when in doubt, [refer to the Sentry documentation](https://docs.sentry.io/)
+to figure out what steps you'll need to take to configure your application.
+
+### A note on naming
+
+We prefer that Sentry applications share the same name as the GitHub
+repo for the project in question. For example, the project with the GitHub repo
+[`la-metro-dashboard`](https://github.com/datamade/la-metro-dashboard) should also
+have the Sentry application name `la-metro-dashboard`.
+
+## Logging errors in Django applications
+
+Logging errors from Django to Sentry is straightforward using the Sentry Django integration.
+When you create an application in Sentry it will provide you with a "Data Source Name,"
+or "DSN", a secret string that the Sentry SDK can use to push errors to your application.
+You'll then need to update your code to initialize the Sentry SDK and make use of
+this DSN.
+
+### Option 1: Use the Django app template
+
+If you used the [Django app template](/docker/templates/) to create your app, your
+application will already be configured to push errors to Sentry with the Sentry SDK.
+Update your Heroku applications to [add a config
+var](https://devcenter.heroku.com/articles/config-vars#managing-config-vars)
+called `SENTRY_DSN` representing the DSN for your application, and your app will
+be ready to push errors to Sentry.
+
+### Option 2: Enable Sentry manually
+
+If you didn't use the Django app template to create your app, you'll have to take
+a few extra steps to prep your app to log errors to Sentry. These steps are adapted
+from the [official Sentry docs for Django](https://docs.sentry.io/platforms/python/django/).
+
+#### Install the Sentry SDK
+
+Update your app's `requirements.txt` file to install [`sentry-sdk`](https://pypi.org/project/sentry-sdk/).
+Remember to run `docker-compose build` after you update `requirements.txt` to make
+sure that your container image requirements are up to date.
+
+#### Thread the DSN into the app environment
+
+Your Django app will need access to the Sentry DSN string in order to push errors to the
+correct Sentry application. If you're deploying on Heroku, the easiest way to do this
+is with environment variables. Update your [Heroku config
+vars](https://devcenter.heroku.com/articles/config-vars#managing-config-vars) to
+add a new var, `SENTRY_DSN`, representing the DSN string from Sentry.
+
+If you're deploying with the legacy `settings_local.py` deployment method, you
+can also set `SENTRY_DSN` as a global variable in this file and then import it
+in `settings.py`.
+
+#### Initialize `sentry_sdk` in `settings.py`
+
+Edit your app's `settings.py` config file and add a block to initialize
+Sentry when the app starts up. Import the following modules with the rest of your
+`settings.py` imports:
+
+```python
+import sentry_sdk
+from sentry_sdk.integrations.django import DjangoIntegration
+```
+
+If your app threads the Sentry DSN into its environment using an environment variable,
+check that variable to see whether to configure Sentry:
+
+```python
+if os.getenv('SENTRY_DSN'):
+    sentry_sdk.init(
+        dsn=os.environ['SENTRY_DSN'],
+        integrations=[DjangoIntegration()],
+    )
+```
+
+If your app uses the legacy `settings_local.py` deployment method, you can try to
+import `SENTRY_DSN` from the local settings module:
+
+```python
+try:
+    from .settings_local import SENTRY_DSN
+except ImportError:
+    pass
+else:
+    sentry_sdk.init(
+        dsn=SENTRY_DSN,
+        integrations=[DjangoIntegration()],
+    )
+```
+
+#### Group 400 errors
+
+By default, Django will raise an exception any time a user sends a request to your
+app with an HTTP `Host` header that does not match a value in `ALLOWED_HOSTS`.
+This should never happen if a human user is requesting your app via a valid
+web URL, but it can happen frequently if bot scanners are sending requests to your app.
+The can cause lots of annoying Sentry notifications because the `Host` value is
+often different from bot to bot, and Sentry will create a separate issue for each
+`Host`, making it impossible to tell Sentry to ignore the error.
+
+To ignore these errors, write a `before_send` function in a module like `your_app/logging.py`
+to check for this particular error and assign it a custom Sentry fingerprint:
+
+```python
+def before_send(event, hint):
+    """
+    Log 400 Bad Request errors with the same custom fingerprint so that we can
+    group them and ignore them all together. See:
+    https://github.com/getsentry/sentry-python/issues/149#issuecomment-434448781
+    """
+    log_record = hint.get('log_record')
+    if log_record and hasattr(log_record, 'name'):
+        if log_record.name == 'django.security.DisallowedHost':
+            event['fingerprint'] = ['disallowed-host']
+    return event
+```
+
+Then, Update your `sentry_sdk` initialization function to add the function as
+a Sentry hook:
+
+```diff
+    sentry_sdk.init(
+        dsn=SENTRY_DSN,
++       before_send=before_send,
+        integrations=[DjangoIntegration()]
+    )
+```
+
+Now, all 400 errors should be grouped under the same issue in Sentry. Proceed to
+the Sentry dashboard and ignore these errors as needed.