Skip to content
This repository has been archived by the owner on Jan 25, 2023. It is now read-only.

Feature/ddb backend #77

Closed
wants to merge 26 commits into from
Closed

Feature/ddb backend #77

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
d6f5711
added checking for supervisor existence before installing
pgr-josh-wells Jul 5, 2018
6544406
added dynamo flags and logic to use either s3 or dynamo
pgr-josh-wells Jul 5, 2018
a8e6b72
correct useradd with --system flag
pgr-josh-wells Jul 5, 2018
6cd9bfa
updated dynamo checking for config generation
pgr-josh-wells Jul 5, 2018
fdc8fac
add dynamo documentation
pgr-josh-wells Jul 5, 2018
6d308d2
tf fmt only
pgr-josh-wells Jul 5, 2018
966df0a
adding standard dynamo table and policy to iam role
pgr-josh-wells Jul 5, 2018
36a74d7
adding dynamo arn output
pgr-josh-wells Jul 5, 2018
0761984
adding vault dynamo variables (read/write and name)
pgr-josh-wells Jul 5, 2018
36e1b94
added tf example for vault with DDB backend. lacking infrastructure …
pgr-josh-wells Jul 5, 2018
8d539cd
updated main comment to specify ddb
Jul 6, 2018
14c4e94
updated README to specify using a vault ami only
Jul 6, 2018
6a55724
updated userdata 'data' block to remove consul variables and add dyna…
Jul 6, 2018
93fb7b6
set consul_storage and vault_storage to and removed redundant logic
Jul 6, 2018
f3020a8
removed logic changing dynamo/s3 values and made just list. Added lo…
Jul 6, 2018
2df7148
added Name tag to DDB
Jul 6, 2018
ffabbb8
updated dyanmo output as concat element
Jul 6, 2018
10b5cb1
updated ddb read/write string variables without quotes
Jul 6, 2018
011d71a
update userdata comment
Jul 6, 2018
0b74d0a
updated enable-dynamo and enable_dynamo to include 'backend' with con…
Jul 6, 2018
7449675
updated enable dynamo flag in userdata
Jul 6, 2018
45f7658
updated readme to include enable-dynamo-backend references
Jul 6, 2018
c5f0603
update generate vault config line to include all as list
Jul 6, 2018
481c675
remove checking supervisor pip
Jul 6, 2018
773d47a
updated to allow s3 with ddb backend
pgr-josh-wells Jul 31, 2018
c8d9fab
tf fmt
pgr-josh-wells Jul 31, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions modules/install-vault/install-vault
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,8 @@ function two_way_symlink() {

# Install steps are based on: http://stackoverflow.com/a/31576473/483528
function install_supervisord_amazon_linux {
if [[ ! $(pip list |grep supervisor) ]]; then
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couldn't supervisor be installed via tools other than pip? And won't this return false positives for anything that has the word "supervisor" in the name?

Seems like a better check might be for the presence of supervisorctl in the PATH.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, to avoid wrapping this whole method in an if, consider inverting the check and doing an early return right at the top.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed this check entirely. This was something I did on my local instance because it simply timed out with pip (no internet connectivity). So I had installed the package manually and just ran a check to bypass.

Reverted to the the prior code.


sudo pip install supervisor

# On Amazon Linux, /usr/local/bin is not in PATH for the root user, so we add symlinks to /usr/bin, which is in PATH
Expand All @@ -115,6 +117,7 @@ function install_supervisord_amazon_linux {
create_supervisor_config
sudo chkconfig --add supervisor
sudo chkconfig supervisor on
fi
}

function create_supervisor_config {
Expand Down
20 changes: 18 additions & 2 deletions modules/run-vault/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,22 +63,30 @@ The `run-vault` script accepts the following arguments:
* `user` (optional): The user to run Vault as. Default is to use the owner of `config-dir`.
* `skip-vault-config` (optional): If this flag is set, don't generate a Vault configuration file. This is useful if you
have a custom configuration file and don't want to use any of of the default settings from `run-vault`.
* `--enable-s3-backend` (optional): If this flag is set, an S3 backend will be enabled in addition to the HA Consul backend.
* `--enable-s3-backend` (optional): Cannot be set with `--enable-dynamo`. If this flag is set, an S3 backend will be enabled in addition to the HA Consul backend.
* `--s3-bucket` (optional): Specifies the S3 bucket to use to store Vault data. Only used if `--enable-s3-backend` is set.
* `--s3-bucket-region` (optional): Specifies the AWS region where `--s3-bucket` lives. Only used if `--enable-s3-backend` is set.
* `--enable-dynamo` (optional): Cannot be set with `--enable-s3-backend`. If this flag is set, a DynamoDB backend will be enabled. Consul will __NOT__ be enabled as a backend.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably --enable-dynamo-backend for consistency with --enable-s3-backend?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

find-replace in readme, userdata in example, and run-vault script.

* `--dynamo-table` (optional): Specifies the DynamoDB table to use to store Vault data. Only used if `--enable-dynamo` is set.
* `--dynamo-region` (optional): Specifies the AWS region where `--dynamo-table` lives. Only used if `--enable-dynamo` is set.

Example:

```
/opt/vault/bin/run-vault --tls-cert-file /opt/vault/tls/vault.crt.pem --tls-key-file /opt/vault/tls/vault.key.pem
```

Or if you want to enable an S3 backend:
If you want to enable an S3 backend:

```
/opt/vault/bin/run-vault --tls-cert-file /opt/vault/tls/vault.crt.pem --tls-key-file /opt/vault/tls/vault.key.pem --enable-s3-backend --s3-bucket my-vault-bucket --s3-bucket-region us-east-1
```

OR if you want to enable DynamoDB backend:

```
/opt/vault/bin/run-vault --tls-cert-file /opt/vault/tls/vault.crt.pem --tls-key-file /opt/vault/tls/vault.key.pem --enable-dynamo --dynamo-table my-dynamo-table --dynamo-region us-east-1
```


## Vault configuration
Expand Down Expand Up @@ -134,6 +142,14 @@ available.
* [region](https://www.vaultproject.io/docs/configuration/storage/s3.html#region): Set to the `--s3-bucket-region`
parameter.

* [storage](https://www.vaultproject.io/docs/configuration/index.html#storage): Set the `--enable-dynamo` flag to
configure DynamoDB as the main (HA) storage backend for Vault:

* [table](https://www.vaultproject.io/docs/configuration/storage/dynamodb.html#table): Set to the `--dynamo-table`
parameter.
* [region](https://www.vaultproject.io/docs/configuration/storage/dynamodb.html#region): Set to the `--dynamo-region`
parameter.

### Overriding the configuration

To override the default configuration, simply put your own configuration file in the Vault config folder (default:
Expand Down
73 changes: 68 additions & 5 deletions modules/run-vault/run-vault
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,10 @@ function print_usage {
echo -e " --skip-vault-config\tIf this flag is set, don't generate a Vault configuration file. Optional. Default is false."
echo -e " --enable-s3-backend\tIf this flag is set, an S3 backend will be enabled in addition to the HA Consul backend. Default is false."
echo -e " --s3-bucket\tSpecifies the S3 bucket to use to store Vault data. Only used if '--enable-s3-backend' is set."
echo -e " --s3-bucket-region\tSpecifies the AWS region where `--s3-bucket` lives. Only used if `--enable-s3-backend` is set."
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

backticks were previously used causing this function to behave unexpectedly. Replaced with single quotes per all other items in the list.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whoops! Good fix, thanks!

echo -e " --s3-bucket-region\tSpecifies the AWS region where '--s3-bucket' lives. Only used if '--enable-s3-backend' is set."
echo -e " --enable-dynamo\tIf this flag is set, DynamoDB will be enabled as the backend storage (HA)"
echo -e " --dynamo-region\tSpecifies the AWS region where --dynamo-table lives. Only used if '--enable-dynamo is on'"
echo -e " --dynamo--table\tSpecifies the DynamoDB table to use for HA Storage. Only used if '--enable-dynamo is on'"
echo
echo "Examples:"
echo
Expand Down Expand Up @@ -112,16 +115,34 @@ function generate_vault_config {
local readonly api_addr="$5"
local readonly config_dir="$6"
local readonly user="$7"
local readonly enable_s3_backend="$8"
local readonly s3_bucket="$9"
local readonly s3_bucket_region="${10}"

if [[ "$enable_s3_backend" == "true" ]]; then
local readonly enable_s3_backend="$8"
local readonly s3_bucket="$9"
local readonly s3_bucket_region="${10}"
local readonly enable_dynamo=""
local readonly dynamo_region=""
local readonly dynamo_table=""
fi

if [[ "$enable_dynamo" == "true" ]]; then
local readonly enable_dynamo="$8"
local readonly dynamo_region="$9"
local readonly dynamo_table="${10}"
local readonly enable_s3_backend=""
local readonly s3_bucket=""
local readonly s3_bucket_region=""
fi
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I find changing the list of input params a bit confusing. I'd recommend making this a single list:

local readonly enable_s3_backend="$8"
local readonly s3_bucket="$9"
local readonly s3_bucket_region="${10}"
local readonly enable_dynamo="${11}"
local readonly dynamo_region="${12}"
local readonly dynamo_table="${13}"

The run method should pass all these params, setting them to true, false, empty strings, or proper values, depending on what the user passed in.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree, however, by default the vault config generation just adds the S3 params at the end (no harm if they are empty). I can add extra vault config generation logic (at the bottom) if you'd rather see this as a list.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed it back to a list and updated logic at the bottom (vault generation script) to match


local readonly config_path="$config_dir/$VAULT_CONFIG_FILE"

local instance_ip_address
instance_ip_address=$(get_instance_ip_address)

log_info "Creating default Vault config file in $config_path"
local readonly listener_config=$(cat <<EOF
ui = true
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Enabled vault UI by default

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thx! We should probably make this configurable though with a --disable-ui flag for those that want to leave it off.


listener "tcp" {
address = "0.0.0.0:$port"
cluster_address = "0.0.0.0:$cluster_port"
Expand All @@ -133,6 +154,7 @@ EOF

local consul_storage_type="storage"
local s3_config=""

if [[ "$enable_s3_backend" == "true" ]]; then
s3_config=$(cat <<EOF
storage "s3" {
Expand All @@ -144,7 +166,24 @@ EOF
consul_storage_type="ha_storage"
fi

local readonly consul_storage=$(cat <<EOF

if [[ "$enable_dynamo" == "true" ]]; then
consul_storage=""
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please declare local consul_storage and local vault_storage up above this if/else block. That way, both default to empty strings, and only one gets set to a proper value.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good call. Adjusting.

vault_storage=$(cat <<EOF
storage "dynamodb" {
Copy link

@BMonsalvatge BMonsalvatge Jul 20, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In order to have both dynamodb and s3 backend we would need to be able to make that block similar to how its handled for consul. Currently what I have adjusted on my end, and is working so far with manual testing:

  local consul_storage_type="storage"
  local dynamodb_storage_type="storage"
  local s3_config=""
  local consul_storage=""
  local vault_storage=""

if [[ "$enable_s3_backend" == "true" ]]; then
    s3_config=$(cat <<EOF
storage "s3" {
  bucket = "$s3_bucket"
  region = "$s3_bucket_region"
}\n
EOF
)
    consul_storage_type="ha_storage"
    dynamodb_storage_type="ha_storage"
  fi


  if [[ "$enable_dynamo_backend" == "true" ]]; then
    vault_storage=$(cat <<EOF
$dynamodb_storage_type "dynamodb" {
  ha_enabled = "true"
  region = "$dynamo_region"
  table  = "$dynamo_table"
}

# HA settings
cluster_addr  = "https://$instance_ip_address:$cluster_port"
api_addr      = "$api_addr"

EOF
)

ha_enabled = "true"
region = "$dynamo_region"
table = "$dynamo_table"
}

# HA settings
cluster_addr = "https://$instance_ip_address:$cluster_port"
api_addr = "$api_addr"

EOF
)
else
local readonly consul_storage=$(cat <<EOF
$consul_storage_type "consul" {
address = "127.0.0.1:8500"
path = "vault/"
Expand All @@ -159,9 +198,12 @@ api_addr = "$api_addr"
EOF
)

fi

echo -e "$listener_config" >> "$config_path"
echo -e "$s3_config" >> "$config_path"
echo -e "$consul_storage" >> "$config_path"
echo -e "$vault_storage" >> "$config_path"

chown "$user:$user" "$config_path"
}
Expand Down Expand Up @@ -215,6 +257,9 @@ function run {
local enable_s3_backend="false"
local s3_bucket=""
local s3_bucket_region=""
local enable_dynamo="false"
local dynamo_region=""
local dynamo_table=""
local all_args=()

while [[ $# > 0 ]]; do
Expand Down Expand Up @@ -283,6 +328,17 @@ function run {
s3_bucket_region="$2"
shift
;;
--enable-dynamo)
enable_dynamo="true"
;;
--dynamo-region)
dynamo_region="$2"
shift
;;
--dynamo-table)
dynamo_table="$2"
shift
;;
--help)
print_usage
exit
Expand All @@ -305,6 +361,11 @@ function run {
assert_not_empty "--s3-bucket-region" "$s3_bucket_region"
fi

if [[ "$enable_dynamo" == "true" ]]; then
assert_not_empty "--dynamo-table" "$dynamo_table"
assert_not_empty "--dynamo-region" "$dynamo_region"
fi

assert_is_installed "supervisorctl"
assert_is_installed "aws"
assert_is_installed "curl"
Expand Down Expand Up @@ -336,6 +397,8 @@ function run {

if [[ "$skip_vault_config" == "true" ]]; then
log_info "The --skip-vault-config flag is set, so will not generate a default Vault config file."
elif [[ "$enable_dynamo" == "true" ]]; then
generate_vault_config "$tls_cert_file" "$tls_key_file" "$port" "$cluster_port" "$api_addr" "$config_dir" "$user" "$enable_dynamo" "$dynamo_region" "$dynamo_table"
else
generate_vault_config "$tls_cert_file" "$tls_key_file" "$port" "$cluster_port" "$api_addr" "$config_dir" "$user" "$enable_s3_backend" "$s3_bucket" "$s3_bucket_region"
fi
Expand Down