Vault Production Readiness Checklist

This checklist will prepare you launch production-ready vault clusters into any major Cloud provider or on Premise

1. Infrastructure Architecture
2. Consul Storage Backend
3. Load Balancing
4. Monitoring and Alerting
5. Configuration Management
6. Vault Configuration
7. Identity & Access Management
8. Security Hardening
9. Operational Readiness
10. Observability
11. Governance and compliance

Infrastructure Architecture

☐	Infrastructure Workloads spread accross Availability Zones Nodes in Vault clusters (and Consul clusters if being used as a storage backend) should be spread accross two or more failure domains known as Availability zones. The loss of a single Availability zone should not result result in a loss of service.
☐	Machine Images Created (Virtual Machines only) If you are deploying your Vault nodes on virtual machines, It is reccomended to build re-usable VM images that can be used to create cluster nodes in an immutable way. Tools like Hashicorp Packer are designed to help build repeatable machine images for most virtualised and cloud platform. Machine images should be versioned and should follow a release cycle as new images are produced.
☐	Replication Enabled (Enterprise only) If you are using the Enterprise version of Vault, you can enable replication between two or more Vault clusters in different geographical regions for added protection is Disaster Recovery scenarios. Replication can be configured in Disaster Recovery Mode or Performance Replication mode. If you are planing on using Replication, you need to provision infrastructure in an alternative region, with nodes spread accross multiple Availability Zones. For more information about the Enterprise Replication feature, see the official documentation.
☐	Firewall rules configured to control access to Vault Vault will likely contain business critical secrets which makes it a prime target for malicious actors. Access to vault to should be restricted to your private networks and not be accessible on the internet. The Use of Virtual Private Networks is a commonly used approach to allow access to Vault from unknown networks
☐	Compute Resouces satisfy the minimum requirements Ensure Hardware servers and Virtual Machines have been appropriately resources in accordance with the Deployment System Requirements
☐	Secondary Disk Ensure that vault servers have a secondary disk attached to them. This will help with Audit Device Fault tolerance

Consul Storage Backend

☐	Storage Backend Architecture It is a recommended pattern to use HashiCorp Consul's Key/Value store as the storage backend. The reccomended cluster size for consul is 5 nodes. This cluster size allows for fault tolerance whilst performing maintenence on a a node
☐	Consul Gossip Encryption configured Members of the Consul clusters use a gossip protocol to communicate with eachother and hold leadership elections. This network traffic should be encrypted to minimise security risks. You can read more about consul encryption here.
☐	ACLs The path that Vault uses in Consul's key/value storage to store it's encrypted data should be protected using Consul's ACL system. Once configured, the Management ACL token should be revoked. You can read more about configuring Consul's ACL system here.
☐	Consul Backups scheduled As Consul is being used as a data store that Vault uses, it should be considered a stateful service, and as such, should have a backup strategy. Consul snapshot, in addition to disk backups should be implemented on a regular schedule. For more information about consul snapshot, click here
☐	Consul Clients installed on Vault Nodes Vault should not talk directly to Consul backend as this introduces an increased attack vector. Instead, Consul should be installed on the Vault servers and configured in client mode. The clients will facilitate the communication between Vault and Consul.
☐	Consul UI enabled (Optional) If using a 5 node consul cluster, you can choose to enable the UI; however, it is recommended that the UI is enabled on two nodes only.

Load Balancing

☐	TLS Encryption is configured on Load Balancer Vault’s communications should be encrypted end-to-end with TLS and this should not be terminated at the Load balancer layer. The load balancer should also use the same encryption to communicate with Vault
☐	HTTP Redirects configured With TLS configured, all traffic going via HTTPS will be encrypted; however, we need to ensure that there are no connections to vault via HTTP. The Load balancer should be configured to redirect all HTTP traffic to HTTPS.
☐	Health checks enabled and configured Load balancer health probes can be used to ensure that traffic is only routed to a healthy leader node. Configure routing rules according to these response codes

Monitoring & Alerting

☐	Vault Telemetry Enabled. Vault telemetry should be configured in the telemetry stanza within the Vault config file. This will enable monitoring and alerting with a wide range of open source tools (Telegraf and prometheus)
☐	Consul Telemetry Enabled. Consul telemetry should be configured in the telemetry stanza within the consul config file. This will enable monitoring and alerting with a wide range of open source tools (Telegraf and prometheus)
☐	Vault platform monitoring configured Monitoring system of your choice is configured to monitor and alert on vault application metric thresholds as per the best practice guidance of Hashicorp.
☐	Consul platform Monitoring Configured Monitoring system of your choice is configured to monitor and alert on infrastructure metric thresholds as per the best practice guidance of Hashicorp.
☐	Infrastructure Monitoring Configured Monitoring system of your choice is configured to monitor and alert on consul application metric thresholds as per the best practice guidance of Hashicorp.
☐	Monitoring Dashbaord Created Using a Dashboard tool a of your choice, create a monitoring dashboard for operations staff to easily identify any issues that may be occurring.

Configuration Management

☐	Infrastructure as code written (Virtual Machines only) Code written to deploy the infrastructure for Consul and Vault. Terrafrom is an appropriate tool for this task. Virtual Machine images created from code for Consul and Vault. Packer is a good choice of tool for this. All Virtual infrastructure should be deployed and managed using an Infrastructure as code tool
☐	Vault Platform Configuration Code Written Vault Platform configuration should be described in code using a tool like Terrafrom. Configuration such as Auth Methods, Secrets Engines, Audit Devices and Policies should all be configured using code
☐	Code under Version Control in Source Code Repositories All Infrastructure code and application code should be stored separate source control repositories and be placed under version control. An appropriate branching strategy should be implemented and documented in the README file.
☐	Code Owners in repositories Repository files should have code owners assigned to them to control who can approve Pull Requests that will be merged into the Master branch.
☐	Repository rules implemented Configure the minimum number of Pull Request approvers, restrictions on Pull Request Authors approving their own requests and any other rules that your organisation’s security standards require for Integrity.
☐	Deployment pipelines implemented Code deployments should be automated using deployment pipelines. Where possible, the pipeline should be written as code and stored under version control with the code
☐	Dev/Staging environments created Create development and staging environments for Vault. Staging Environment should be identical to production, with the only divergence being, when pre-production changes are implemented for final testing prior to being deployed to production.
☐	Naming and coding standards established Implement and document naming and coding standards. Naming standards for Namespaces, Policies, Vault Roles, secrets keys and AppRoles. Coding standards where applicable for variable names and function names.
☐	Integration tests written A suite of automated integration tests written to be run either during the deployment pipeline or as a pre-check on your chosen VCS required to pass before a Pull Request can be merged.

Vault Configuration

☐	Enable audit device on 2 files on different disks Vault logs all requests and responses to requests. If Vault is unable to log requests and responses to these requests, it will immediately seize operations. To provide redundancy, each vault node should have 2 file audit devices enabled on separate volumes on separate disks.
☐	Log rotation configured on log files Enable and configure log rotation on the audit files to ensure the disks do not fill up and cause a vault outage.
☐	Configure at least one IDP as an auth method Where appropriate, configure an existing identity provider (or multiple if required) as an authentication method in Vault
☐	Configure the required secrets engines Identify and enable the required secrets engines for your business and technical use cases
☐	Ensure KV v2 is enabled Ensure that Version 2 of the KV secrets engine is used to enable secrets versioning

Identity & Access Management

☐	Create default policies Create default policies that all user entities will inherit according to your business security model. This could be list permissions on a particular KV path for example.
☐	Create Policy mappings for default policies Create a mapping for default policies to ensure all user entities inherit these policies.
☐	Configure aliases for entities when more than one auth method is in use Using the Identity Secrets engine, create aliases to attach vault logins via different auth methods to a single entity to ensure the correct policies are inherited and to make the logging data easier to mine
☐	Design a path structure for KV v2 that matches the way your org works (team based or product/service based) Map you KV path design to the way your organisation works or product groupings.
☐	Meta AppRole process defined Meta Approles are a mechanism that allow an application or service to read the secret id of an app role without exposing this to application developers.

Security Hardening

☐	Ensure TLS is configured on Vault Cluster Enable end-to-end encryption using TLS certificates. Vault agents should also use TLS certificates
☐	Ensure TLS is configured on Consul Clusters Enable end-to-end encryption on consul cluster and agent. More information can be found here.
☐	Enable and configure SELinux / App Armour Enable and config SELinux / app amour depending on your operating system to create sandboxed contexts to reduce blast radius if even the system is compromised.
☐	Randomise the ports used to differ from standard ports for Vault By default, Vault uses port 8200 and 8201. Change the port to a non-standard port to provide extra hardening
☐	Randomise the ports used to differ from standard ports for Consul By default, Consul uses port 8500 and 8501. Change the port to a non-standard port to provide extra hardening
☐	Revoke root token Once initial set-up of Vault cluster has been completed, the root token should be revoked.
☐	Revoke Consul Management ACL token Once the initial set-up of the Consul ACL system has been completed, the management token should be revoked.
☐	Configure server firewalls to only allow access to required ports. Using firewalld or IP Tables, configure these firewalls to limit port access to the vault and consul servers.
☐	Disable SSH Interaction with Vault is done via the API, even when using the CLI. As such, there is no reason to have to SSH on to a vault server (if it’s a virtual machine) so SSH should be disabled to mitigate the risk of unauthorised access to the server.

Operational Readiness

☐	Configure auto-unseal Add a seal stanza to the Vault config file to reduce operational burden on operators. For more information check the auto-unseal documentation here
☐	PGP encryption of unseal/recovery keys Use PGP or Keybase to add an extra layer of security to the distribution of unseal/recovery keys. For more details, see the official documentation here
☐	Backup / restore practice run Practice restoring your Vault platform from a Consul snapshot. Your backup strategy isn’t complete until you have tested this.
☐	Node rebuild practice run Practice building and replacing a node in the vault and consul clusters with zero downtime.
☐	Vault upgrade practice run Practice upgrading Vault binaries to newer versions with zero downtime.
☐	Consul upgrade practice run Practice upgrading Consul binaries to newer versions with zero downtime.
☐	Load testing Consuct load testing to ensure your infrastructure compute resources are sufficient for the load you are expecting. There are projects like wrk That can assist with generating traffic.
☐	Document key holders and contact details Ensure unseal/recovery key holders are documented on a Wiki and this document is kept up-to-date
☐	Trusted Broker/Platform pattern Choose a platform or broker that your business trusts and use this for secure injection of initial secrets. Examples are using Azure as a trusted platform or using Jenkins as a trusted broker. Each organisation will differ with regards to what they trust so this should be a business driven decision.
☐	OS Patching strategy Document an implement an OS patching strategy, whether it’s updating VM images and replacing VMs with up-to-date images or whether its a controlled direct access update by an operator.

Observability

☐	Logs shipping to central logs data warehouse Logs should be streamed to a central data warehouse as log rotation on the servers should be enabled and logs will be lost locally. A platform like splunk is ideal for this use case. There are other viable options available.
☐	Logs data mining scripts written. Decide the value that the log data should provide and write some scripts to extract this value from the data. Scripts can be written in python. Models can also be produced to predict future loads based on existing data sets. This kind of insight can be useful for planning.
☐	Logs alerting configured Some events should generate some kind of alert, for example, a root token being generated should be flagged and alerted on. Ensure these events have alerts configured for them.

Governance and compliance

☐	Threat Model Exercise Conduct a threat modelling exercise using a framework of your organisations choosing and ensure you have documented and mitigated against all identified threats.