This Quick Start deploys Darktrace vSensor virtual threat detection on Google Cloud Platform (GCP). For deploying on Amazon Web Services (AWS), see terraform-aws-vsensor.
Darktrace vSensors analyze raw packet data from mirrored virtual private cloud (VPC) traffic to learn to identify threats. This guide covers the steps necessary to deploy this Quick Start.
Darktrace vSensors accept traffic from mirroring subnets (GCP Virtual Private Cloud Packet Mirroring) and from Darktrace osSensor agents.
- GCP Virtual Private Cloud (VPC) Packet Mirroring copies traffic from VPC subnets you want to monitor.
- Darktrace osSensors can be configured on virtual machines and containerized applications. Darktrace osSensors are available for Linux, Windows, and as a container for any system that can run the Docker Engine.
Darktrace vSensors can also be configured to accept syslog feeds of third-party security information and event management tools.
Darktrace vSensors perform deep packet inspection and extract meta-data from the traffic sources. This meta-data is then submitted to a connected Darktrace cloud service or physical Darktrace deployment over port 443 (TLS). PCAP (packet capture) data is also produced by the vSensor for forensic analysis. This is stored in a Google Cloud Storage bucket.
This guide covers the steps necessary to deploy this Quick Start.
This module sets up the following:
- A VPC (optional), or provide an existing VPC to deploy into
- A Subnet for containing the internal vSensors
- A Cloud Router to provide vSensors with appliance / update connectivity
- A Managed Instance Group containing vSensors with predictive autoscaling
- (Optional) A GCP Storage bucket for storing PCAPS for later analysis
- GCP IAP (SSH-in-Browser) ssh access to the vSensors
- IAM role assignments configuring GCP Ops Agent logging and (optionally) Storage bucket
- (Optional) A Bastion subnet, firewall rules and host for allowing external access to the vSensors
- (Optional) User and public ssh key for ssh public key authentication
This Quick Start terraform module provides the following deployment options:
- Deploy Darktrace vSensor into an existing VPC. This option provisions Darktrace vSensor in your existing GCP infrastructure in a new subnet.
- Deploy Darktrace vSensor into a new VPC. This option builds a new GCP environment that consists of the VPC, subnet, Cloud NAT (Public), and other infrastructure components. It then deploys Darktrace vSensor into this new VPC.
Register a new push token to enable connection between vSensor probes and an existing Darktrace on-premise or cloud instance. All of the vSensor instances in one deployment should share the same push token.
-
Log into the Darktrace console.
-
From the main menu, choose Admin > System Config, then access the "Settings" page.
-
Locate the "Push Probe Tokens" section. At the bottom of the list of probes is a field to create a new token. Enter a label for the vSensor deployment.
-
Choose Add. You will need to record two variables from the resulting window. The vSensor Update Key (also found on the Darktrace customer portal) and the Push Token. The Push Token is only displayed once.
-
In GCP Secret Manager create a new secret with a unique name eg.
darktrace_vsensor_update_keyand the value of the vSensor Update Key obtained previously. -
In GCP Secret Manager create a new secret with a unique name eg.
darktrace_push_tokenand the value of the Push Token obtained previously. (Any additional Darktrace master deployments will require their own push token)
If you have a physical Darktrace deployment behind a firewall, you must grant access to the instance from the IP addresses of your NAT Gateways after deployment.
Note: Darktrace cloud offerings are already configured to allow Push Token access, no firewall changes are necessary.
The shared HMAC secret key between the osSensor and vSensor is optional for the installation. If the HMAC is not provided the module will not set the osSensor shared HMAC secret key on the vSensors, such that osSensors can not connect.
- In GCP Secret Manager create a new secret with a unique name eg.
os_sensor_hmac_tokenand the value of the osSensor shared HMAC secret key obtained previously.
The following GCP APIs are required for deploying this template:
- Compute Engine
- Deployment Manager v2
- Identity and Access Management (IAM)
- IAM Service Account Credentials
- Cloud Resource Manager
- Cloud Logging
- Cloud Monitoring
- Secret Manager
When the module is used to create a new VPC for deploying the vSensors, there are still no subnets in that VPC for packet mirroring, hence the module cannot configure the packet mirroring. Once the terraform deployment has completed you can create the subnets for the instances that traffic will be mirrored. Then, you can either:
- configure the packet mirroring manually,
- provide the names of these subnets to the module (input variable
mirrored_subnets) and run againterraform applywhich will configure the packet mirroring; this way, the packet mirroring resource will be managed by the terraform module and you can use it to change the packet mirroring filter parameter (mirrored_protocols,mirrored_cidr_ranges,mirrored_direction).
The CIDR range for the subnet that the vSensors will be deployed in, should have enough usable IP addresses for the maximum vSensor instances (see mig_max_size) plus one for the Internal passthrough Network Load Balancer. Learn more about how GCP uses the IP addresses in a subnet.
module "vsensors" {
source = "git::https://github.com/darktrace/terraform-gcp-vsensor?ref=<version>"
project_id = "XXXXXXX"
region = "europe-west1"
deployment_prefix = "euw1a"
existing_vpc_name = "XXXXXXX"
mirrored_subnets = ["XXXX-sub1", "XXXX-sub2"]
mig_subnet_cidr = "10.127.2.128/27"
sm_update_key = "sm_test_update_key"
sm_push_token = "sm_test_push_token"
sm_ossensor_hmac = "sm_test_hmac_token"
dt_instance_hostname = "dt-master.company.com"
}
module "vsensors" {
source = "git::https://github.com/darktrace/terraform-gcp-vsensor?ref=<version>"
project_id = "XXXXXXX"
region = "europe-west1"
deployment_prefix = "euw1a"
new_vpc_enable = true
mig_subnet_cidr = "10.127.2.128/27"
sm_update_key = "sm_test_update_key"
sm_push_token = "sm_test_push_token"
sm_ossensor_hmac = "sm_test_hmac_token"
dt_instance_hostname = "dt-master.company.com"
}
| Name | Version |
|---|---|
| terraform | >= 1.4 |
| >= 4.79 | |
| google-beta | >= 4.79 |
| random | ~>3.5 |
| Name | Version |
|---|---|
| >= 4.79 | |
| google-beta | >= 4.79 |
| random | ~>3.5 |
No modules.
| Name | Description | Type | Default | Required |
|---|---|---|---|---|
| bastion_enable | (Optional; applicable only if new_vpc_enable = true) If true a standalone/single bastion host will be installed to provide ssh remote access to the vSensors.A new subnet will be created for the bastion. |
bool |
false |
no |
| bastion_ssh_cidr | (Optional) List of Subnet CIDR ranges that the Bastion will accept SSH from. Add 35.235.240.0/20 to the list to enable GCP IAP (SSH-in-Browser) ssh access to the Bastion. Do not provide any value if bastion is not enabled. | list(string) |
[] |
no |
| bastion_ssh_user_key | (Optional) Bastion username and public ssh key for ssh public key authentication in format 'username:ssh_public_key'. This will be added to the instance metadata: https://cloud.google.com/compute/docs/connect/add-ssh-keys#add_ssh_keys_to_instance_metadata. Example: bastion_ssh_user_key = "test:ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILg6UtHDNyMNAh0GjaytsJdrUxjtLy3APXqZfNZhvCeT test" |
string |
null |
no |
| bastion_subnet_cidr | (Optional) Subnet CIDR range that the Bastion will be deployed in (must not overlap with vSensor or other subnets in the VPC). Example: 10.127.2.0/27. Do not provide any value if bastion is not enabled. | string |
null |
no |
| deployment_prefix | The name of the deployment and VM instance. A string that will be used to create deployment ID and resource names. It can contain between two and five lowercase alphabet characters and digits. It must start with lowercase alphabet character." |
string |
n/a | yes |
| dt_instance_hostname | Host name of the Darktrace Master instance. | string |
n/a | yes |
| dt_instance_port | Connection port between vSensor and the Darktrace Master instance. | number |
443 |
no |
| existing_vpc_name | The existing VPC network name where the vSensors will be deployed. If new_vpc_enable = true this will be ignored. |
string |
"" |
no |
| mig_instance_type | The machine type to use to create the vSensor. Sizing requirements can be found at https://customerportal.darktrace.com/product-guides/main/vsensor-requirements. | string |
"e2-standard-2" |
no |
| mig_max_size | Maximum number of vSensor instances in the Autoscaling group. It is recommended to be set larger than the mig_min_size to allow Autoscaling and instance replacement actions to work correctly. |
number |
3 |
no |
| mig_min_size | Minimum number of vSensor instances in the Autoscaling group. | number |
2 |
no |
| mig_ssh_user_key | vSensor username and public ssh key for ssh public key authentication in format 'username:ssh_public_key'. This will be added to the instance metadata: https://cloud.google.com/compute/docs/connect/add-ssh-keys#add_ssh_keys_to_instance_metadata. Example: mig_ssh_user_key = "test:ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILg6UtHDNyMNAh0GjaytsJdrUxjtLy3APXqZfNZhvCeT test" |
string |
null |
no |
| mig_subnet_cidr | Subnet range that the vSensors will be deployed in (must not overlap with bastion or other subnets in VPC). | string |
n/a | yes |
| mig_zone | The distribution policy for the vSensors managed instance group. You can specify one or more values, for example ["europe-west1-b", "europe-west1-c"]. The default (empty) means all zones in the region. |
list(string) |
[] |
no |
| mirrored_cidr_ranges | IP CIDR ranges that apply as a filter on the source (ingress) or destination (egress) IP in the IP header. Only IPv4 is supported. Use 0.0.0.0/0 to allow all IP ranges. Example: ["10.0.0.0/24", "10.0.1.0/24"]. The default value is "0.0.0.0/0". |
list(string) |
[ |
no |
| mirrored_direction | Direction of traffic to mirror. Possible values are: INGRESS, EGRESS, BOTH. NOTE: any setting other than BOTH may lead to unidirectional traffic alerts on the Darktrace master instance. The default value is BOTH. |
string |
"BOTH" |
no |
| mirrored_protocols | (Optional). IP protocols to mirror. Possible IP protocols including ah, esp, icmp, ipip, sctp, tcp, udp, or an IP protocol number between 0 and 255. Example: ["tcp", "udp"] to mirror only tcp and udp protocols. Do not provide any value to mirror all IP protocols. The default is to mirror all IP protocols. Changing the value will force resource replacement (destroy and then create replacement). |
list(string) |
[] |
no |
| mirrored_subnets | (Optional) Names of the subnets to be mirrored. For example ["mirror-subnet-1", "mirror-subnet-2"]. If left empty, no packet mirroring policy will be created. In such cases, packet mirroring policy can be manually created later, or once the module has completed, mirrored_subnets can be added and another terraform apply will create the policy. |
list(string) |
[] |
no |
| new_vpc_enable | If true will create a new VPC network. The name of the new network will be of the form deployment_prefix-<random-string>-. |
bool |
false |
no |
| project_id | Project ID. | string |
n/a | yes |
| region | The region that the resources will be deployed into. | string |
n/a | yes |
| retention_time_days | Number of days to retain captured packets in the bucket. Longer retention will increase storage costs. Set to 0 to disable PCAPs and Storage bucket. | number |
7 |
no |
| sm_ossensor_hmac | (Optional)Secret Manager Secret Name of the hash-based message authentication code (HMAC) token to authenticate osSensors with vSensor. | string |
"" |
no |
| sm_push_token | Secret Manager Secret Name of the push token. The push token is used to authenticate the vSensor with the Darktrace Master instance. | string |
n/a | yes |
| sm_update_key | Secret Manager Secret Name of the vSensor update key. If you don't have one, contact your Darktrace representative. | string |
n/a | yes |
| ssh_iap | (Forces re-creating the vSensor) Enable or disable GCP IAP (SSH-in-Browser) ssh access to the vSensors. Accepted values are enable or disable. The default is enable. After the vSensors have been deployed, changing the value will force re-creating the vSensors in the create before destroy fashion. | string |
"enable" |
no |
| Name | Description |
|---|---|
| bastion_external_ip | Public IP address used for the Bastion VM. |
| lb_ip | LB fronend IP. |
| mig_distribution_zones | Zones configured for vSensors managed instance group (MIG). |
| mig_subnets_id | Subnet id where the vSensors are deployed. |
| mirrored_protocols | List of the packet mirroring protocols. |
| nat_external_ip | Public IP address used for NAT router to allow vSensors access to Darktrace cloud service / physical Darktrace deployment and software updates. |
| pcaps_bucket_name | The name of the bucket that stores the PCAPs. |
| pcaps_bucket_self_link | The URI of the bucket that stores the PCAPs. |
| pcaps_bucket_url | The base URL of the bucket that stores the PCAPs. |
| project_id | The project id. |
| project_number | The project number. |
| service_account_vsensor_email | The service account email. |
| vpc_id | The ID of the new VPC. |
| vpc_name | The name of the new VPC. |
| vpc_self_link | The self_link of the new VPC. |
If you have a physical Darktrace deployment behind a firewall, you must grant access to the instance to the NAT router public IP addresses (output_nat_external_ip).
If mirrored_subnets was left empty, no Packet Mirroring Policy will be created. The module allows this to be configured post-deployment:
- Provide the names of the subnets to be mirrored (input variable
mirrored_subnets); - Run
terraform applyagain;
Alternatively, this can be configured manually from the GCP console.
The shared HMAC secret key between the osSensor and vSensor is optional for the installation. If the HMAC is not provided the module will not set the osSensor shared HMAC secret key on the vSensors.
Later, the deployment might require setting the HMAC on the vSensors. In that case:
- Configure the
sm_ossensor_hmacwith the HMAC secret key; - Run
terraform applyagain;
Be aware, that the above will force vSensors replacement and some downtime is expected. It is advised that such changes are planned for a Maintenance Window.
In scenarios of invalid configuration parameters, upstream internet access blocked and other cases of the VM instances failing to install and configure Darktrace vSensor, the Managed Instance Group will fail to deploy.
In most situations, logging will be available from the vSensor start up in "Logs Explorer". In "LOG NAME" look for vsensor-userdata.
If no userdata logging is provided in "Logs Explorer", you may need to connect directly to the vSensor via GCP IAP (SSH-in-Browser) or ssh from a Bastion and retrieve /var/log/user-data.log directly.
For situations where the error is not clear, contact Darktrace Customer Support via the Darktrace Customer Portal, provide the terraform error and one of the above userdata logs if available.
If Packet mirroring was configured via Terraform, changes in the packet mirroring filter, for example the mirrored_protocols, will replace the resource google_compute_packet_mirroring in a 'destroy and then create replacement' manner. You will encounter a short downtime and it is advised that such changes are planned for a Maintenance Window.
Google Ops Agent is configured on the vSensors and bastion automatically to provide more detailed statistics including CPU, memory and bandwidth.
It also provides Log Explorer logs in four 'Log Name' groups to support debugging:
vsensor-syslogFor general syslog logging.vsensor-updatesFor logging fromapt/dpkgpackage updating of the vSensor and OS packages.vsensor-servicesFor logging from the main vSensor product components.vsensor-userdataFor logging from the initial vSensor installation.
The module uses regional resources.
Each new deployment into a new region or GCP project should be given a new push token generated from the master instance UI config page. Each push token refers to a scalable set of vSensors, which share the same Cloud Storage bucket and config. When querying for pcap data, the master instance only queries one vSensor in the probe group (to avoid querying the same Cloud Storage bucket multiple times). If the same push token is reused in multiple environments, querying for PCAP data will only search one of the environments and miss the rest.