Skip to content
Branch: master
Find file History
xingao267 and Copybara-Service Fix build/test of the oss version.
PiperOrigin-RevId: 259420089
Latest commit e308095 Jul 22, 2019
Permalink
Type Name Latest commit message Commit time
..
Failed to load latest commit information.
apply allow default resource to handle name being set outside properties block Jul 12, 2019
cmd add --generated_fields_path Jul 19, 2019
config
deploymentmanager Deploy CHC type provider in DPT Jun 27, 2019
rule_generator Support python3 only May 27, 2019
rulegen migrate forseti installation logic to Terraform Jul 8, 2019
samples Remove sandbox demo from samples. Jul 22, 2019
starlark Enforce at least 1 file match if imports is specified. Jul 22, 2019
templates Enable CHC dataset template to support notificationConfig Jul 4, 2019
testconf add --generated_fields_path Jul 19, 2019
utils make output_yaml_path optional Jul 17, 2019
BUILD use go config loader in python code Jul 11, 2019
README.md make output_yaml_path optional Jul 17, 2019
create_project.py add --generated_fields_path Jul 19, 2019
create_project_test.py rm datathon sample Jul 18, 2019
generate_rules.py Fix role name in rule generator: from roles/editors to roles/editor. Jan 16, 2019
grant_forseti_access.py Fix duplicate flags in grant_forseti_access.py. Dec 4, 2018
project_config.yaml.schema allow default resource to handle name being set outside properties block Jul 12, 2019
requirements.txt Support python3 only May 27, 2019

README.md

Google Cloud Healthcare Data Protection Toolkit

GoDoc

Tools to configure GCP environments aimed at healthcare use cases.

Setup Instructions

To use the templates and script in this folder, first decide if you want your audit logs saved in the same project as the hosted data (local audit logs), or in a separate project (remote audit logs). Remote audit logs can be especially beneficial if you have several data projects.

  1. Complete Script Prerequisites the first time using these scripts.
  2. Create Groups for the dataset and audit logs (if using remote audit logs) project(s).
  3. Create a YAML config for the project(s) you want to deploy.
  4. Create New Projects using the create_project.py script. This will create the audit logs project (if required) and all data hosting projects.

Script Prerequisites

NOTE: If running through Cloud Shell, all of the following dependencies are already available.

Python Dependencies

Install Python dependencies with the following command:

$ pip3 install -r requirements.txt

Create Groups

Before using the setup scripts, you will need to create the following groups for your dataset projects. We also provide recommended names based on best practices.

  • Owners Group: {PROJECT_ID}-owners@{DOMAIN}. Provides owners role to the project, which allows members to do anything permitted by organization policies within the project. Users should only be added to the owners group short term.
  • Auditors Group: {PROJECT_ID}-auditors@{DOMAIN}. The auditors group has permission to list resources, view IAM configurations and view contents of audit logs, but not to view any hosted data. If you have multiple data projects, you may want a single auditors group across all projects.
  • Data Read-only Group: {PROJECT_ID}-readonly@{DOMAIN}. Members of this group have read-only access to the hosted data in the project.
  • Data Read/Write Group: {PROJECT_ID}-readwrite@{DOMAIN}. Members of this group have permission to read and write hosted data in the project.

If you are using a separate Audit Logs project, then the audit logs project will also need its own Owners Group and an Auditors group, but no data groups.

Create a YAML Config

Edit a copy of the file samples/project_with_remote_audit_logs.yaml or samples/project_with_local_audit_logs.yaml, depending on whether you are using remote or local audit logs. The schema for these YAML files is in project_config.yaml.schema.

  • The overall section contains organization and billing details applied to all projects. Omit the organziation_id if the projects are not being deployed under a GCP organization.
  • If using remote audit logs, include the audit_logs_project section, which describes the project that will host audit logs.
  • Under projects, list each data hosting project to create.

WARNING: these samples use newer configuration that is incompatible with the previous and require --enable_new_style_resources to be passed to the create_project script. This flag will soon become default and the old style configs will be deprecated. Old style config examples can be found here.

Create New Projects

Use the create_project.py script to create an audit logs project (if using remote audit logs) and one or more data hosting projects.

  1. Make sure the user running the script is in the owners group(s) of all projects that will be created, including the audit logs project (if used). You should remove the user from these groups after successful deployment.

  2. If not already logged in, run gcloud init to log in as a user with permission to create projects under the specified organization and billing account

  3. If you provided a stackdriver_alert_email in any project, then when prompted during the script, follow the instructions to create new Stackdriver Accounts for these projects.

  4. If you provided a forseti config and the project hasn't been deployed you may be prompted for additional steps during the Forseti instance installation.

  5. Optional: pass a --projects flag listing the projects you wish to deploy, or * (default) to deploy all projects.

    WARNING: deploying a project that was previously deployed will trigger an update.

  6. If the projects were deployed successfully, the script will write a YAML add a generated_fields block in --project_yaml. These fields are used to generate monitoring rules.

$ git clone https://github.com/GoogleCloudPlatform/healthcare
$ cd healthcare
# Note: --incompatible_use_python_toolchains is not needed after Bazel 0.27
# See https://github.com/bazelbuild/bazel/issues/7899.
$ bazel run --incompatible_use_python_toolchains deploy:create_project -- --project_yaml=${PROJECT_CONFIG?} --projects=${PROJECTS?} --output_yaml_path=${PROJECT_CONFIG?} --nodry_run

If the script fails at any point, try to correct the error, sync the output yaml file with the input and try again.

Disabled Unneeded APIs

NOTE: This will be moved to create_project.py.

List the APIs that are enabled for your project, and remove any that you no longer require:

gcloud services list --project ${PROJECT_ID?}

...

gcloud services --project ${PROJECT_ID?} disable ${SERVICE_NAME}

Updates

You may wish to modify a project to add additional resources or change an existing setting. The create_project.py script can be used to also update a project. Listing a previously deployed project in the --projects flag (or setting the flag to be "*" for all projects), will trigger an update.

WARNING: Deployment Manager will run with deletion policy "ABANDON". Thus, if a resource is removed from a project, it will become unmonitored rather than deleted. It is the responsibility of the user to manually remove stale references including IAM bindings and enabled APIs.

Steps

The script create_project.py takes in YAML config files and creates one or more projects. It creates an audit logs project if audit_logs_project is provided and a forseti project if forseti is provided. It then creates a data hosting project for each project listed under projects. For each new project, the script performs the following steps:

  1. Create a new GCP project.

  2. Enable billing on the project.

  3. Enable required services:

    • deploymentmanager.googleapis.com (for Deployment Manager)
    • cloudresourcemanager.googleapis.com (for project level IAM policy changes)
    • iam.googleapis.com (if custom IAM roles were defined)
  4. Grant the Deployment Manager service account the required roles:

    • roles/owner
    • roles/storage.admin
  5. Create custom boot image(s), if specified on a gce_instance resource

  6. Deploy all project resources including:

    • Default resources:
      • IAM Policies to grant owner and auditor groups project level access
      • Log sink export to the logs_bq_dataset
      • Log metrics for alerting on bigquery ACL, IAM and bucket ALC accesses.
    • User defined resources (possible resources can be found in the project_config.yaml.schema file under the gcp_project.resources definition).

    TIP: To view deployed resources, open the project in the GCP console and navigate to the Deployment Manager page. Click the expanded config under the data-protect-toolkit-resources deployment to view the list of resources deployed.

  7. Deploys audit resources to hold exported logs

    • The audit resources will be deployed in the remote audit logs project if one was specified.
  8. Removes the user from the project's owners.

  9. Prompts the user to create a Stackdriver account (currently this must be done using the Stackdriver UI).

  10. Creates Stackdriver Alerts for IAM changes and unexpected GCS bucket access.

  11. If a forseti block is defined:

    • Runs the Forseti installer to deploy a Forseti instance (user may be prompted during installation)
    • Grants permissions for each project to the Forseti service account so they may be monitored.
    • Generates Forseti rules and writes them to the Forseti server bucket.

Resources

Resources can be added in the config in a uniform way, but may use different tools underneath to do the actual deployment. Since each resource may have a very large and complex schema, we cannot cover all of it in our tooling layers. Thus, we only implement a subset and allow the users to set additional allowed fields as they wish. This is done through the properties block available for most resources. See documentation below for each resource.

See the samples/ directory for sample resource definitions.

NOTE: project_config.yaml.schema provides documentation on our subset. Please reference it before adding your resource to your config.

NOTE: Dependencies can be set implicitly in deployment manager resource properties. See https://cloud.google.com/deployment-manager/docs/step-by-step-guide/using-references. Dependencies are only supported between deployment manager resources.

Resource Deployment Tool
bq_dataset Deployment Manager (CFT)
chc_datasets (ALPHA) Deployment Manager
enabled_api Gcloud
forseti Terraform (CFT)
gce_firewall Deployment Manager (CFT)
gce_instance Deployment Manager (CFT)
gcs_bucket Deployment Manager (CFT)
gke_cluster Deployment Manager (CFT)
gke_workload Kubectl
iam_custom_role Deployment Manager (CFT)
iam_policy Deployment Manager (CFT)
pubsub Deployment Manager (CFT)
vpc_network Deployment Manager (CFT)
service_account (ALPHA) Deployment Manager (Direct)

Debug

These are solutions to non-typical problems encountered:

Bucket Permission Denied

Typically the error message will contain the following during a failed deployment manager deployment:

"ResourceType":"storage.v1.bucket", "ResourceErrorCode":"403"

This is due to the Deployment Manager Service account not having storage admin permissions. There can be a delay of up to 7 minutes for permission changes to propagate. After recent changes, deployment manager permissions will no longer be revoked so just retry deployment of all projects that failed after at least 7 minutes.

NOTE: if remote audit logs failed to deploy due to this error then you will need to re-deploy the audit logs project first.

You can’t perform that action at this time.