Merge forsetisecurity.org-dev into forsetisecurity.org (#2078)

* Added 2.0 upgrade instruction (#1769) * Added 2.0 upgrade instruction * Addressed PR comment * Updated broken urls * Added missing gsuite scope (#1772) * Update upgrade.md (#1774) * Added compute engine disk resource (#1779) * Update CSCC doc (#1778) * versioned 2.1 * revert branch to stable * Updated generated configuration files * update versioning (#1791) * Added space between comment and function * Updated app.yaml file * updates * updates * fixed version redirect issues * Update forsetisecurity.org-dev faqs for VM instances security updates… (#1799) * Update forsetisecurity.org-dev faqs for VM instances security updates issue#830 * Changes to PR comments Update forsetisecurity.org-dev faqs for VM instances security updates issue#830 * Adding information about building the website server locally for development. * Adding information about building the website server locally for development. * Making a set of small changes pr comments in PR/1804 * Fixing title that got munged in merge * Making more changes to website section * Responding to an additional comment. * Responding to comments about dep location and ruby virtual environment. * Fixing issue 1801 where links to GitHub for latest are broken. * Update CSCC notifier documentation to list the additional requirements for CSCC (#1808) * Update doc to list the additional requirements for CSCC * Update how to enable API * Fixing issue 1801 where links to GitHub for latest are broken. (#1807) * Responding to comments in PR/1804 * Making an additional round of changes to PR/1804 * Making an additional round of changes to PR/1804 * Noting the updated G Suite Domain-wide Delegation screen. * Fixing typo in documentation * Fixing typo in org policy permission configuration * Updating documentation as `service-management enable` is now `services enable` $ gcloud beta service-management enable admin.googleapis.com ERROR: (gcloud.beta.service-management.enable) The `service-management enable` command has been replaced by `services enable`. * Documenting solution to virtualenv issues on clean Linux install This took literally hours of searching to figure out, so saving others time. * Adding some additional clarification to setup process, specifically g… (#1818) * Adding some additional clarification to setup process, specifically granting ACLs. * Fixing small typo * Responding to comment. * Adding clarifying details for CloudSQL setup (#1819) * Adding clarifying details for CloudSQL setup For example, a new screen UI was introduced in the CloudSQL creation process. * Making requested changes. * Adjusting indent to be four spaces * Adjusting overflow indent * Fix for "mysql_config not found" Showing how to fix "EnvironmentError: mysql_config not found" when MySQL isn't available due to version/license issues. * Updated installation instructions to include steps to install server instance only (#1824) * Updated installation instructions to include steps to install server instance only * Addressed PR comments * Executing specific subset of tests (#1852) Adding some examples for running specific subset of tests. * Adding solution for error repeatedly encountered in forseti setup (#1851) * Adding solution for error repeatedly encountered in forseti setup "bash: workon: command not found" and its solution on Linux machine. * Update setup.md * Add FAQ for adding documentation (#1863) * Add FAQ for adding documentation * tweak * tweak * tweak * tweak * Add the -r flag for requirements.txt (#1877) * Add the -r flag for requirements.txt (#1876) * Add the -r flag for requirements.txt (#1875) * Noting process for running unit tests in Docker after making changes (#1855) * Noting process for running unit tests in Docker after making changes * Updated website with instructions to modify cron job (#1906) * updated website with instructions to modify cron job * addressed comment on cron job update * adjusted line * adjusted line break * Forsetisecurity.org Add snapshots to Resource Coverage on website (#1912) * Add compute snapshots to covered resources * Fix link for compute engine disks * Updated documentation that G Suite is optional * Updated documentation * recent changes updated (#1933) * Add entry in FAQ for Terraform deployment (#1939) * Add entry in FAQ for Terraform deployment * Fixed typo * Update deploy-using-terraform.md * v2.2 website created (#1948) * Updated website with Slack information and office hours (#1950) * Updated website with Slack and office hours info * Updated website with slack info and office hours * Update upgrade instructions (#1966) * updates * git command updated * updates * pin sphinx version to 177 (#1987) * Update rules.md with new BigQuery syntax (#1984) * Update rules.md with new BigQuery syntax Partially fix #1937. * update deprecated notice * typos corrected (#1988) * created version v2.3.0 * added sphinx docs * Fix testing instructions (#1993) * Fix testing instructions * Adjust docs to match @blueandgold's suggestion from #1998 * Document the new sequential step-upgrade process. (#2016) * Document new upgrade process. * updates * formated md * Addressed PR comments * updates * updates * Added compute instance reset step * addressed PR comments * fixed broken urls * Add Documentation to Install and Configure Non-Org Root (#2039) * add doc for non-org root * tweak * tweak * tweak * tweak * tweak * reorganize doc for clarity * address comments * website publish v2.4 * Cloud Asset Inventory documentation (#2054) * CAI documentation * updated cai doc page * addressed comments * added bucket level role * documented steps to enable cai while upgrading * added gcloud command * fixed quotes around command * added steps to upgrade from v2.4 to v2.5 * reorganized steps by placing both the gcloud command steps together * need to add a service account role * updates * updates * updates * Added lien scanner config update instructions * travis updates * updates * Created v2.5.0.
forseti-security · Oct 5, 2018 · 9a6ad72 · 9a6ad72
1 parent 6a66735
commit 9a6ad72
Show file tree

Hide file tree

Showing 567 changed files with 116,543 additions and 26 deletions.
diff --git a/_config.yml b/_config.yml
@@ -293,4 +293,33 @@ defaults:
   - scope:
       path: _docs/v2.4/faq
     values:
-      category: FAQ (v2.4)
+      category: FAQ (v2.4)
+  - scope:
+      path: _docs/v2.5
+    values:
+      category: Overview (v2.5)
+      layout: docs
+  - scope:
+      path: _docs/v2.5/concepts
+    values:
+      category: Concepts (v2.5)
+  - scope:
+      path: _docs/v2.5/setup
+    values:
+      category: Setup (v2.5)
+  - scope:
+      path: _docs/v2.5/use
+    values:
+      category: Use (v2.5)
+  - scope:
+      path: _docs/v2.5/configure
+    values:
+      category: Configure (v2.5)
+  - scope:
+      path: _docs/v2.5/develop
+    values:
+      category: Develop (v2.5)
+  - scope:
+      path: _docs/v2.5/faq
+    values:
+      category: FAQ (v2.5)
diff --git a/_data/doc_categories.yml b/_data/doc_categories.yml
@@ -85,4 +85,16 @@ v2.4:
   href: docs/$$VERSION$$/develop/
 - title: FAQ
   href: docs/$$VERSION$$/faq/
-
+v2.5:
+- title: Concepts
+  href: docs/$$VERSION$$/concepts/
+- title: Setup
+  href: docs/$$VERSION$$/setup/
+- title: Configure
+  href: docs/$$VERSION$$/configure/
+- title: Use
+  href: docs/$$VERSION$$/use/
+- title: Develop
+  href: docs/$$VERSION$$/develop/
+- title: FAQ
+  href: docs/$$VERSION$$/faq/
diff --git a/_docs/_latest/setup/upgrade.md b/_docs/_latest/setup/upgrade.md
@@ -171,12 +171,12 @@ Learn more about [Updating a Deployment](https://cloud.google.com/deployment-man
 1. Download the latest copy of your Forseti server deployment template file from the Forseti server GCS 
 bucket to your cloud shell (located under `forseti-server-xxxxxx/deployment_templates`) by running command  
 `gsutil cp gs://YOUR_FORSETI_GCS_BUCKET/deployment_templates/deploy-forseti-server-<LATEST_TEMPLATE>.yaml 
-deployment_templates/deploy-forseti-server-xxxxx-2-1-0.yaml`.
-1. Open up the deployment template `deployment_templates/deploy-forseti-server-xxxxx-2-1-0.yaml` for edit.
+deployment-templates/deploy-forseti-server-xxxxx-2-1-0.yaml`.
+1. Open up the deployment template `deployment-templates/deploy-forseti-server-xxxxx-2-1-0.yaml` for edit.
     1. Update the `forseti-version` inside the deployment template to `tags/v2.1.0`.
-1. Upload file `deployment_templates/deploy-forseti-server-xxxxx-2-1-0.yaml` back to the GCS bucket 
+1. Upload file `deployment-templates/deploy-forseti-server-xxxxx-2-1-0.yaml` back to the GCS bucket 
 (`forseti-server-xxxxxx/deployment_templates`) by running command  
-`gsutil cp deployment_templates/deploy-forseti-server-xxxxx-2-1-0.yaml gs://YOUR_FORSETI_GCS_BUCKET/
+`gsutil cp deployment-templates/deploy-forseti-server-xxxxx-2-1-0.yaml gs://YOUR_FORSETI_GCS_BUCKET/
 deployment_templates/deploy-forseti-server-xxxxx-2-1-0.yaml`.
 1. Navigate to [Deployment Manager](https://console.cloud.google.com/dm/deployments) and 
 copy the deployment name for Forseti server.
@@ -235,8 +235,8 @@ update the configuration file.
 1. Download the latest copy of your Forseti server deployment template file from the Forseti server GCS 
 bucket to your cloud shell (located under `forseti-server-xxxxxx/deployment_templates`) by running command  
 `gsutil cp gs://YOUR_FORSETI_GCS_BUCKET/deployment_templates/deploy-forseti-server-<LATEST_TEMPLATE>.yaml 
-deployment_templates/deploy-forseti-server-xxxxx-2-2-0.yaml`.
-1. Open up the deployment template `deployment_templates/deploy-forseti-server-xxxxx-2-2-0.yaml` for edit.
+deployment-templates/deploy-forseti-server-xxxxx-2-2-0.yaml`.
+1. Open up the deployment template `deployment-templates/deploy-forseti-server-xxxxx-2-2-0.yaml` for edit.
     1. Update the `forseti-version` inside the deployment template to `tags/v2.2.0`.
     1. Add the following fields to the compute engine section inside your deployment template.  
     `region` - The region of your VM, e.g. us-central1,  
@@ -267,10 +267,10 @@ deployment_templates/deploy-forseti-server-xxxxx-2-2-0.yaml`.
         run-frequency: ...
         
         ```
-1. Upload file `deployment_templates/deploy-forseti-server-xxxxx-2-2-0.yaml` back to the GCS bucket 
+1. Upload file `deployment-templates/deploy-forseti-server-xxxxx-2-2-0.yaml` back to the GCS bucket 
 (`forseti-server-xxxxxx/deployment_templates`) by running command  
-`gsutil cp deployment_templates/deploy-forseti-server-xxxxx-2-1-0.yaml gs://YOUR_FORSETI_GCS_BUCKET/
-deployment_templates/deploy-forseti-server-xxxxx-2-2-0.yaml`.
+`gsutil cp deployment-templates/deploy-forseti-server-xxxxx-2-1-0.yaml gs://YOUR_FORSETI_GCS_BUCKET/
+deployment-templates/deploy-forseti-server-xxxxx-2-2-0.yaml`.
 1. Navigate to [Deployment Manager](https://console.cloud.google.com/dm/deployments) and 
 copy the deployment name for Forseti server.
 1. Run command `gcloud deployment-manager deployments update DEPLOYMENT_NAME --config deploy-forseti-server-xxxxx-2-2-0.yaml`  
@@ -299,14 +299,14 @@ Example command: `gcloud compute instances reset forseti-server-vm-70ce82f --zon
     1. Run command `cd forseti-security` to navigate to the forseti-security directory.
     1. Run command `git checkout tags/v2.3.0` to checkout version `v2.3.0` of Forseti Security.
 1. Download the latest copy of your Forseti server deployment template file from the Forseti server GCS 
-bucket to your cloud shell (located under `forseti-server-xxxxxx/deployment_templates`) by running command  
+bucket to your cloud shell (located under `forseti-server-xxxxxx/deployment-templates`) by running command  
 `gsutil cp gs://YOUR_FORSETI_GCS_BUCKET/deployment_templates/deploy-forseti-server-<LATEST_TEMPLATE>.yaml 
-deployment_templates/deploy-forseti-server-xxxxx-2-3-0.yaml`.
+deployment-templates/deploy-forseti-server-xxxxx-2-3-0.yaml`.
 1. Open up the deployment template `deployment_templates/deploy-forseti-server-xxxxx-2-3-0.yaml` for edit.
     1. Update the `forseti-version` inside the deployment template to `tags/v2.3.0`.
-1. Upload file `deployment_templates/deploy-forseti-server-xxxxx-2-3-0.yaml` back to the GCS bucket 
+1. Upload file `deployment-templates/deploy-forseti-server-xxxxx-2-3-0.yaml` back to the GCS bucket 
 (`forseti-server-xxxxxx/deployment_templates`) by running command  
-`gsutil cp deployment_templates/deploy-forseti-server-xxxxx-2-3-0.yaml gs://YOUR_FORSETI_GCS_BUCKET/
+`gsutil cp deployment-templates/deploy-forseti-server-xxxxx-2-3-0.yaml gs://YOUR_FORSETI_GCS_BUCKET/
 deployment_templates/deploy-forseti-server-xxxxx-2-3-0.yaml`.
 1. Navigate to [Deployment Manager](https://console.cloud.google.com/dm/deployments) and 
 copy the deployment name for Forseti server.
@@ -334,7 +334,7 @@ update the configuration file.
     ```
 
     **Notifier**
-    - Update the `resources` section to include `mode` and `organization_id`.
+    - Update the `resources` section to include `log_sink_violations`.
     ```
     notifier:
         ...
@@ -379,12 +379,12 @@ update the configuration file.
 1. Download the latest copy of your Forseti server deployment template file from the Forseti server GCS 
 bucket to your cloud shell (located under `forseti-server-xxxxxx/deployment_templates`) by running command  
 `gsutil cp gs://YOUR_FORSETI_GCS_BUCKET/deployment_templates/deploy-forseti-server-<LATEST_TEMPLATE>.yaml 
-deployment_templates/deploy-forseti-server-xxxxx-2-4-0.yaml`.
-1. Open up the deployment template `deployment_templates/deploy-forseti-server-xxxxx-2-4-0.yaml` for edit.
+deployment-templates/deploy-forseti-server-xxxxx-2-4-0.yaml`.
+1. Open up the deployment template `deployment-templates/deploy-forseti-server-xxxxx-2-4-0.yaml` for edit.
     1. Update the `forseti-version` inside the deployment template to `tags/v2.4.0`.
-1. Upload file `deployment_templates/deploy-forseti-server-xxxxx-2-4-0.yaml` back to the GCS bucket 
+1. Upload file `deployment-templates/deploy-forseti-server-xxxxx-2-4-0.yaml` back to the GCS bucket 
 (`forseti-server-xxxxxx/deployment_templates`) by running command  
-`gsutil cp deployment_templates/deploy-forseti-server-xxxxx-2-4-0.yaml gs://YOUR_FORSETI_GCS_BUCKET/
+`gsutil cp deployment-templates/deploy-forseti-server-xxxxx-2-4-0.yaml gs://YOUR_FORSETI_GCS_BUCKET/
 deployment_templates/deploy-forseti-server-xxxxx-2-4-0.yaml`.
 1. Navigate to [Deployment Manager](https://console.cloud.google.com/dm/deployments) and 
 copy the deployment name for Forseti server.
@@ -401,6 +401,164 @@ Forseti is updated to be usable on a non organization resource.
 {% endcapture %}
 {% include site/zippy/item.html title="Upgrading 2.3.0 to 2.4.0" content=upgrading_2_3_0_to_2_4_0 uid=5 %}
 
+{% capture upgrading_2_4_0_to_2_5_0 %}
+
+Starting v2.5, Forseti Inventory will be integrated with the Cloud 
+Asset Inventory (CAI) service when Forseti is first deployed. CAI
+integration is supported only if the root resource is `organization`.
+
+Below are the steps to upgrade from v2.4.0 to v2.5.0
+
+1. Open cloud shell when you are in the Forseti project on GCP.
+1. Checkout forseti with tag v2.5.0 by running the following commands:
+    1. If you already have the forseti-security folder under your cloud shell directory, 
+    run command `rm -rf forseti-security` to delete the folder.
+    1. Run command `git clone https://github.com/GoogleCloudPlatform/forseti-security.git` to 
+    clone the forseti-security directory to cloud shell.
+    1. Run command `cd forseti-security` to navigate to the forseti-security directory.
+    1. Run command `git checkout tags/v2.5.0` to checkout version `v2.5.0` of Forseti Security.
+1. Download the latest copy of your Forseti server deployment template file from the Forseti server GCS 
+bucket to your cloud shell (located under `forseti-server-xxxxxx/deployment_templates`) by running command  
+`gsutil cp gs://YOUR_FORSETI_GCS_BUCKET/deployment_templates/deploy-forseti-server-<LATEST_TEMPLATE>.yaml 
+deployment-templates/deploy-forseti-server-xxxxx-2-5-0.yaml`.
+1. Open up the deployment template `deployment-templates/deploy-forseti-server-xxxxx-2-5-0.yaml` for edit.
+    1. Update the `forseti-version` inside the deployment template to `tags/v2.5.0`.
+    1. (Server only changes) Add the following lines under sections `imports` and `resources` to allow 
+    deployment template to create a new GCS bucket to store the CAI data dump. Please update `{BUCKET_LOCATION}` 
+    to point to the location of your bucket, e.g. `us-central1`.   
+    ```
+    imports:
+    ...
+    - path: storage/bucket_cai.py
+      name: bucket_cai.py
+    ...
+    
+    resources:
+    ...
+    # Cloud Storage
+    ...
+    - name: forseti-cai-export
+      type: bucket_cai.py
+      properties:
+        location: {BUCKET_LOCATION}
+        retention_days: 14
+    ...
+    ```
+1. Upload file `deployment-templates/deploy-forseti-server-xxxxx-2-5-0.yaml` back to the GCS bucket 
+(`forseti-server-xxxxxx/deployment_templates`) by running command  
+`gsutil cp deployment-templates/deploy-forseti-server-xxxxx-2-5-0.yaml gs://YOUR_FORSETI_GCS_BUCKET/
+deployment_templates/deploy-forseti-server-xxxxx-2-5-0.yaml`.
+1. Navigate to [Deployment Manager](https://console.cloud.google.com/dm/deployments) and 
+copy the deployment name for Forseti server.
+1. Run command `gcloud deployment-manager deployments update DEPLOYMENT_NAME --config deploy-forseti-server-xxxxx-2-5-0.yaml`
+If you see errors while running the deployment manager update command, please refer to below section 
+`Error while running deployment manager` for details on how to workaround the error.
+1. Reset the Forseti server VM instance for changes in startup script to take effect.  
+You can reset the VM by running command `gcloud compute instances reset MY_FORSETI_SERVER_INSTANCE --zone MY_FORSETI_SERVER_ZONE`  
+Example command: `gcloud compute instances reset forseti-server-vm-70ce82f --zone us-central1-c`
+1. Repeat step `3-8` for Forseti client.
+1. Configuration file `forseti_conf_server.yaml` updates:  
+    **Inventory**
+    - Add `cai` section. 
+       ```
+       inventory:
+       ...
+            api_quota:
+            ...
+           
+            cai:
+                enabled: true    
+                gcs_path: gs://forseti-cai-export
+       ...
+       ```
+    - Add the cloudasset api quota.
+        ```
+        inventory:
+        ...
+            api_quota:
+            ...
+                cloudasset:
+                    max_calls: 1
+                    period: 1.0
+            ...
+        ```
+    - Update the IAM and logging api quota.
+        ```
+        inventory:
+        ...
+            api_quota:
+            ...
+                iam:
+                    max_calls: 90
+                    period: 1.0
+                logging:
+                    max_calls: 9
+                    period: 1.0
+            ...
+        ```
+    **Scanner**
+    - Update the `scanners` section to include `lien`.
+    ```
+    scanner:
+    ...
+        scanners:
+            ...
+            - name: lien
+              enabled: true
+            ...
+    ```
+
+    **Notifier**
+    - Update the `resources` section to include `lien_violations`.
+    ```
+    notifier:
+        ...
+        resources:
+            ...
+            - resource: lien_violations
+              should_notify: true
+              notifiers:
+                # Email violations
+                - name: email_violations
+                  configuration:
+                    sendgrid_api_key: {SENDGRID_API_KEY}
+                    sender: {EMAIL_SENDER}
+                    recipient: {EMAIL_RECIPIENT}
+                # Upload violations to GCS.
+                - name: gcs_violations
+                  configuration:
+                    data_format: csv
+                    # gcs_path should begin with "gs://"
+                    gcs_path: gs://{FORSETI_BUCKET}/scanner_violations
+            ...
+        ...
+    ```
+1. Forseti server service account roles updates:
+   1. Assign role `roles/storage.objectAdmin` to the service account on the CAI bucket.
+    ```
+    gsutil iam ch serviceAccount:SERVICE_ACCOUNT_NAME@PROJECT_ID.iam.gserviceaccount.com:objectAdmin YOUR_CAI_BUCKET
+    ```
+    Example:  
+    ```
+    gsutil iam ch serviceAccount:forseti-server-gcp-637723d@joeupdate210.iam.gserviceaccount.com:objectAdmin gs://forseti-cai-export
+    ```
+
+    1. Assign role `roles/cloudasset.viewer` to the service account on the organization level.   
+    ```
+    gcloud organizations add-iam-policy-binding {ORGANIZATION_ID} --member=serviceAccount:{SERVICE_ACCOUNT_NAME}@{PROJECT_ID}.iam.gserviceaccount.com --role=roles/cloudasset.viewer
+    ```
+    Example:  
+    ```
+    gcloud organizations add-iam-policy-binding 1234567890 --member=serviceAccount:forseti-server-gcp-ea370bd@my_gcp_project.iam.gserviceaccount.com --role=roles/cloudasset.viewer
+    ```
+1. Forseti project API updates:
+   1. Enable `Cloud Asset API` under APIs & Services from the GUI or by running the following command on cloud shell:  
+    `gcloud beta services enable cloudasset.googleapis.com`
+1. Create a copy and upload [lien_rules.yaml](https://github.com/GoogleCloudPlatform/forseti-security/blob/dev/rules/lien_rules.yaml) to `rules` directory under your Forseti server GCS bucket.
+
+{% endcapture %}
+{% include site/zippy/item.html title="Upgrading 2.4.0 to 2.5.0" content=upgrading_2_4_0_to_2_5_0 uid=6 %}
+
 {% capture deployment_manager_error %}
 
 If you get the following error while running the deployment manager:
@@ -449,4 +607,4 @@ This will recreate the VM with updated fields.
 * Configure Forseti to send
 [email notifications]({% link _docs/latest/configure/notifier/index.md %}#email-notifications-with-sendgrid).
 * Enable [G Suite data collection]({% link _docs/latest/configure/inventory/gsuite.md %})
-for processing by Forseti.
+for processing by Forseti.
diff --git a/_docs/v2.5/concepts/architecture.md b/_docs/v2.5/concepts/architecture.md
@@ -0,0 +1,50 @@
+---
+title: Architecture
+order: 001
+---
+
+# {{ page.title }}
+
+This page describes how Forseti is built, and how the components of Forseti fit
+together.
+
+---
+
+## Overview
+
+Forseti gives you tools to understand all the resources you have in Google Cloud
+Platform (GCP). The core Forseti modules work together to provide complete
+information so you can take action action to secure resources and minimize
+security risks.
+
+ * Inventory regularly collects data from your GCP resources and makes it
+   available to other modules.
+ * Scanner periodically compares your rules about GCP resource policies against
+   the policies collected by Inventory, and saves the output for your review.
+ * Enforcer uses Google Cloud APIs to change resource policy state to match the
+   state you define.
+ * Notifier keeps you up to date about Forseti findings and actions.
+ * Explain helps you understand, test, and develop Cloud Identity and Access
+   Management (Cloud IAM) policies.
+
+---
+
+**The image below shows how data flows through Forseti**
+
+{% responsive_image path: images/docs/concepts/forseti-architecture.png alt: "forseti architecture" %}
+
+{:style="list-style-type: upper-alpha;"}
+
+ 1. Inventory collects information about your GCP resources and G Suite.
+ 1. Inventory stores information in Cloud SQL for your review and use by other
+    Forseti modules.
+ 1. Scanner compares the data collected by Inventory to the policy rules you
+    set.
+ 1. Notifier sends Scanner & Inventory results to one or more of the following channels you
+    configure: Cloud Storage, SendGrid, and Slack.
+ 1. You use Explain to query and understand your Cloud IAM policies.
+ 1. Enforcer uses Google Cloud APIs to make sure policies match your desired
+    state.
+ 1. You use the command-line interface to query Forseti data using gRPC.
+ 1. You use Data Studio or MySQL Workbench to visualize the Forseti data stored
+    in Cloud SQL.