Merge branch 'master' into flowetl-config

.circleci/config.yml

@@ -623,7 +623,7 @@ jobs:
       image: circleci/classic:201808-01
     working_directory: /home/circleci/project
     environment:
-      BRANCH: $CIRCLE_BRANCH
+      GIT_REVISION: $CIRCLE_SHA1
     steps:
       - checkout:
           path: /home/circleci/project/

CHANGELOG.md

@@ -7,32 +7,56 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 ## [Unreleased]
 
 ### Added
-- Buttons to copy token to clipboard and download token as file added to token list page.[#704](https://github.com/Flowminder/FlowKit/issues/704)
--  Two new worked examples: "Cell Towers Per Region" and "Unique Subscriber Counts". [#633](https://github.com/Flowminder/FlowKit/issues/633), [#634](https://github.com/Flowminder/FlowKit/issues/634)
+
+- The dev provisioning Ansible playbook now automatically generates an SSH key pair for the `flowkit` user. [#892](https://github.com/Flowminder/FlowKit/issues/892)
+
+### Changed
+
+- The quick-start script now only pulls the docker images for the services that are actually started up. [#898](https://github.com/Flowminder/FlowKit/issues/898)
+- The quick-start script now uses the environment variable `GIT_REVISION` to control the version to be deployed.
+
+### Fixed
+
+- When creating a new token in FlowAuth, the expiry now always shows the year, seconds till expiry, and timezone. [#260](https://github.com/Flowminder/FlowKit/issues/260)
+- Distances in `Displacement` are now calculated with longitude and latitude the corrcet way around. [#913](https://github.com/Flowminder/FlowKit/issues/913)
+- The quick-start script now works correctly with branches. [#902](https://github.com/Flowminder/FlowKit/issues/902)
+
+### Removed
+
+## [0.6.4]
+
+### Added
+
+- Buttons to copy token to clipboard and download token as file added to token list page. [#704](https://github.com/Flowminder/FlowKit/issues/704)
+- Two new worked examples: "Cell Towers Per Region" and "Unique Subscriber Counts". [#633](https://github.com/Flowminder/FlowKit/issues/633), [#634](https://github.com/Flowminder/FlowKit/issues/634)
 
 ### Changed
+
 - The `FLOWDB_DEBUG` environment variable has been renamed to `FLOWDB_ENABLE_POSTGRES_DEBUG_MODE`.
 - FlowAuth will now automatically set up the database when started without needing to trigger via the cli.
 - FlowAuth now requires that at least one administrator account is created by providing env vars or secrets for:
-    - `FLOWAUTH_ADMIN_PASSWORD`
-    - `FLOWAUTH_ADMIN_USERNAME`
+  - `FLOWAUTH_ADMIN_PASSWORD`
+  - `FLOWAUTH_ADMIN_USERNAME`
 
 ### Fixed
+
 - The `FLOWDB_DEBUG` environment variable used to have no effect. This has been fixed. [#811](https://github.com/Flowminder/FlowKit/issues/811)
 - Previously, queries could be stuck in an executing state if writing their cache metadata failed, they will now correctly show as having errored. [#833](https://github.com/Flowminder/FlowKit/issues/833)
-- Fixed an issue where `Table` objects could be in an inconsistent cache state after resetting cache [#832](https://github.com/Flowminder/FlowKit/issues/832) 
-- FlowAuth's docker container can now be used with a Postgres backing database. [#825](https://github.com/Flowminder/FlowKit/issues/825) 
+- Fixed an issue where `Table` objects could be in an inconsistent cache state after resetting cache [#832](https://github.com/Flowminder/FlowKit/issues/832)
+- FlowAuth's docker container can now be used with a Postgres backing database. [#825](https://github.com/Flowminder/FlowKit/issues/825)
 - FlowAPI now starts up successfully when following the "Secrets Quickstart" instructions in the docs. [#836](https://github.com/Flowminder/FlowKit/issues/836)
 - The command to generate an SSL certificate in the "Secrets Quickstart" section in the docs has been fixed and made more robust [#837](https://github.com/Flowminder/FlowKit/issues/837)
 - FlowAuth will no longer try to initialise the database or create demo data multiple times when running under uwsgi with multiple workers [#844](https://github.com/Flowminder/FlowKit/issues/844)
 - Fixed issue of Multiple tokens don't line up on FlowAuth "Tokens" page [#849](https://github.com/Flowminder/FlowKit/issues/849)
 
 ### Removed
+
 - The `FLOWDB_SERVICES` environment variable has been removed from the toplevel Makefile, so that now `DOCKER_SERVICES` is the only environment variable that controls which services are spun up when running `make up`. [#827](https://github.com/Flowminder/FlowKit/issues/827)
 
 ## [0.6.3]
 
 ### Added
+
 - FlowKit's worked examples are now Dockerized, and available as part of the quick setup script [#614](https://github.com/Flowminder/FlowKit/issues/614)
 - Skeleton for Airflow based ETL system added with basic ETL DAG specification and tests.
 - The docs now contain information about required versions of installation prerequisites [#703](https://github.com/Flowminder/FlowKit/issues/703)
@@ -43,15 +67,15 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - The function `print_dependency_tree()` now takes an optional argument `show_stored` to display information whether dependent queries have been stored or not [#804](https://github.com/Flowminder/FlowKit/issues/804)
 - A new function `plot_dependency_graph()` has been added which allows to conveniently plot and visualise a dependency graph for use in Jupyter notebooks (this requires IPython and pygraphviz to be installed) [#786](https://github.com/Flowminder/FlowKit/issues/786)
 
-
 ### Changed
+
 - Parameter names in `flowmachine.connect()` have been renamed as follows to be consistent with the associated environment variables [#728](https://github.com/Flowminder/FlowKit/issues/728):
-    - `db_port -> flowdb_port`
-    - `db_user -> flowdb_user`
-    - `db_pass -> flowdb_password`
-    - `db_host -> flowdb_host`
-    - `db_connection_pool_size -> flowdb_connection_pool_size`
-    - `db_connection_pool_overflow -> flowdb_connection_pool_overflow`
+  - `db_port -> flowdb_port`
+  - `db_user -> flowdb_user`
+  - `db_pass -> flowdb_password`
+  - `db_host -> flowdb_host`
+  - `db_connection_pool_size -> flowdb_connection_pool_size`
+  - `db_connection_pool_overflow -> flowdb_connection_pool_overflow`
 - FlowAPI and FlowAuth now expect an audience key to be present in tokens [#727](https://github.com/Flowminder/FlowKit/issues/727)
 - Dependent queries are now only included once in the md5 calculation of a given query (in particular, it changes the query ids compared to previous FlowKit versions).
 - Error is displayed in the add user form of Flowauth if username is alredy exists. [#690](https://github.com/Flowminder/FlowKit/issues/690)
@@ -60,6 +84,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - The class `SubscriberSubsetterBase` in FlowMachine no longer inherits from `Query` [#740](https://github.com/Flowminder/FlowKit/issues/740) (this changes the query ids compared to previous FlowKit versions).
 
 ### Fixed
+
 - FlowClient docs rendered to website now show the options available for arguments that require a string from some set of possibilities [#695](https://github.com/Flowminder/FlowKit/issues/695).
 - The Flowmachine loggers are now initialised only once when flowmachine is imported, with a call to `connect()` only changing the log level [#691](https://github.com/Flowminder/FlowKit/issues/691)
 - The FERNET_KEY environment variable for FlowAuth is now named FLOWAUTH_FERNET_KEY
@@ -71,6 +96,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 ## [0.6.2]
 
 ### Added
+
 - Added a new module, `flowkit-jwt-generator`, which generates test JWT tokens for use with FlowAPI [#564](https://github.com/Flowminder/FlowKit/issues/564)
 - A new Ansible playbook was added in `deployment/provision-dev.yml`. In addition to the standard provisioning
   this installs pyenv, Python 3.7, pipenv and clones the FlowKit repository, which is useful for development purposes.
@@ -382,8 +408,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
 - Added Python 3.6 support for FlowClient
 
-
-[Unreleased]: https://github.com/Flowminder/FlowKit/compare/0.6.3...master
+[unreleased]: https://github.com/Flowminder/FlowKit/compare/0.6.4...master
+[0.6.4]: https://github.com/Flowminder/FlowKit/compare/0.6.3...0.6.4
 [0.6.3]: https://github.com/Flowminder/FlowKit/compare/0.6.2...0.6.3
 [0.6.2]: https://github.com/Flowminder/FlowKit/compare/0.6.1...0.6.2
 [0.6.1]: https://github.com/Flowminder/FlowKit/compare/0.6.0...0.6.1

deployment/README.md

@@ -1,4 +1,30 @@
-To provision the machine:
+## Quick start
+
+Make sure you have completed the initial setup steps below (this is only needed once).
+
+Then run the following commands to provision the machine.
+```bash
+export HOST=<host_name_or_ip_address>
+export SSH_PROVISIONING_USER=root
+export PROVISIONING_PLAYBOOK=provision.yml  # alternatively, use "provision-dev.yml"
+
+# Use the values below to keep the default username/password (`flowkit:flowkit`), or
+# change them to use different values. See below how to determine the hashed password.
+export FLOWKIT_USER_NAME=flowkit
+export FLOWKIT_USER_PASSWORD='$6$YaOatFoRa91eOA06$cLJCvJCdd0sLKBEM01eQ2wJ7ZKkTZJz.YWGFK5r0bs4yqiwAz1Lw9pmExiS.PPBBJv13cuBpiHYU88ThX4TeG/'
+
+# Run the provisioning playbook
+pipenv run ansible-playbook -i ${HOST}, --user=${SSH_PROVISIONING_USER} \
+    --extra-vars="username=${FLOWKIT_USER_NAME} password=${FLOWKIT_USER_PASSWORD_SHA512}" \
+    ${PROVISIONING_PLAYBOOK}
+```
+
+See below for an explanation of the different environment variables and how to generate
+a hashed password for the `flowkit` user account.
+
+## First-time setup steps
+
+The following steps only need to be done once.
 
 - Set up the pipenv environment and install auxiliary Ansible roles.
   ```bash
@@ -10,47 +36,38 @@ To provision the machine:
   SSH keys of the users who should be able to log into the `flowkit` account
   once the machine is provisioned.
 
-- Set the environment variables `HOST` and `SSH_PROVISIONING_USER` to the host
-  of the machine to be provisioned and the username of the provisioning user.
-  Note that this user needs admin permissions because it needs to be able to
+## Meaning of the environment variables
+
+- `HOST`: the hostname (or IP address) of the machine to be provisioned.
+
+- `SSH_PROVISIONING_USER`: the user as which the provisioning steps in the Ansible playbook are run.
+
+  Note that this provisioning user needs admin permissions because it needs to be able to
   install packages on the system. For a cloud VM the provisioning user will
   typically be the root user.
 
-  You should also set `PROVISIONING_PLAYBOOK` to either `provision.yml` or
-  `provision-dev.yml`. In addition to the standard provisioning tasks (the
-  same as in `provision.yml`), the latter will also install Python 3 via
-  `pyenv` and clone the FlowKit repository at `/home/flowkit/code/FlowKit`.
-
-  You also need to set the environment variables `FLOWKIT_USER_NAME`
-  and `FLOWKIT_USER_PASSWORD_SHA512`. These specify the username and (hashed)
-  password of the user account that will install FlowKit. The default values
-  are given below (simply use these if you want to keep the default username
-  and password `flowkit:flowkit`). Note that the password env var must contain
-  the password in _hashed_ form. You can determine this using the following
-  command (which presents you with an interactive prompt to enter the password):
-  ```bash
-  pipenv run python -c "from passlib.hash import sha512_crypt; import getpass; print(sha512_crypt.using(rounds=5000).hash(getpass.getpass()))"
-  ```
-  (See the Ansible
-  [FAQ](https://docs.ansible.com/ansible/latest/reference_appendices/faq.html#how-do-i-generate-crypted-passwords-for-the-user-module)
-  for alternative methods to determine a SHA512 hash of the password.)
+- `PROVISIONING_PLAYBOOK`: the Ansible playbook used to provision the machine. There are two options:
 
-  Here is an example how to set the relevant environment variables:
-  ```bash
-  export HOST=<some_host_name>
-  export SSH_PROVISIONING_USER=root
-  export PROVISIONING_PLAYBOOK=provision.yml  # alternatively, use "provision-dev.yml"
-
-  # Use the values below to keep the default username/password (`flowkit:flowkit`), or
-  # change them to use different values. See above how to determine the hashed password.
-  export FLOWKIT_USER_NAME=flowkit
-  export FLOWKIT_USER_PASSWORD='$6$YaOatFoRa91eOA06$cLJCvJCdd0sLKBEM01eQ2wJ7ZKkTZJz.YWGFK5r0bs4yqiwAz1Lw9pmExiS.PPBBJv13cuBpiHYU88ThX4TeG/'
-  ```
+   - `provision.yml`: this performs the standard provisioning tasks needed to get the machine into a state
+     so that FlowKit can be installed and used on it. This mainly installs `docker` and `docker-compose`
+     and sets up  a `flowkit` user account (also see the `FLOWKIT_USER_NAME` and `FLOWKIT_USER_PASSWORD_SHA512`
+     environment variable below).
 
-- Finally, run the following command (make sure you don't forget the comma
-  after `${HOST}` if you type it manually):
-  ```bash
-  pipenv run ansible-playbook -i ${HOST}, --user=${SSH_PROVISIONING_USER} --extra-vars="username=${FLOWKIT_USER_NAME} password=${FLOWKIT_USER_PASSWORD_SHA512}" ${PROVISIONING_PLAYBOOK}
-  ```
+   - `provision-dev.yml`: in addition to the standard tasks performed by `provision.yml`, this performs
+     additional steps which are useful to do development on FlowKit: it installs Python 3 via `pyenv`,
+     clones the FlowKit repository at `/home/flowkit/code/FlowKit` and generates a pair of SSH keys for
+     the `flowkit` user.
+
+- `FLOWKIT_USER_NAME` and `FLOWKIT_USER_PASSWORD_SHA512`: these specify the username and (hashed) password
+   of the user account that will install and run FlowKit. The default values given in the "Quick Start"
+   section above represent the username/password `flowkit:flowkit`, but this can be changed by setting these
+   variables to different values. Note that the password env var must contain the password in _hashed_ form.
+   You can determine this using the following command (which presents you with an interactive prompt to enter the password):
+   ```bash
+   pipenv run python -c "from passlib.hash import sha512_crypt; import getpass; print(sha512_crypt.using(rounds=5000).hash(getpass.getpass()))"
+   ```
+   See the Ansible
+   [FAQ](https://docs.ansible.com/ansible/latest/reference_appendices/faq.html#how-do-i-generate-crypted-passwords-for-the-user-module)
+   for alternative methods to determine a SHA512 hash of the password.
 
 This has been tested with CentOS Linux release 7.5.1804.

deployment/provision-dev.yml

@@ -39,3 +39,11 @@
         dest: "~{{ username }}/code/FlowKit"
         version: master
         update: no
+    - name: "Generate ssh keys for user '{{ username }}'"
+      become: yes
+      become_user: "{{ username }}"
+      openssh_keypair:
+        path: "~{{ username }}/.ssh/id_rsa"
+        type: rsa
+        size: 4096
+        comment: "{{ username }}@{{ ansible_hostname }}"