Dec 10, 2018
Disable pulling bigquery data from CAI (#2377)
* Disabled pulling bigquery data from CAI

* Addressed PR comments

* updates

* pylint updates

* pylint updates
Assets 2

Summary

Inventory

  • Added new resources from the Cloud Asset API
    • Cloud IAM Grantable Roles
    • Cloud IAM Organization Roles
    • Cloud IAM Project Roles
    • Cloud Pub/Sub
    • Cloud Storage IAM Policies

Notifier

  • Added G Suite DwD status in Inventory Summary email

All changes

9ab73c2 Ensure CSCC alpha mode is the default. (#2318)
78889be Cherry picked #2261 into release-2.8.0. (#2262)
ecd741d Increased pool_size to 50 when creating mysql engine.
4ae557e Increment version to 2.8.0
d98b327 Updated reference to outdated field. (#2252)
ea976e8 (dev) Updated db migrate logic to remove cai_temporary_store (#2248)
eb538a6 Add Support for CSCC Beta API (#2130)
e90364b Merge pull request #2202 from GoogleCloudPlatform/easRevision
8ef6336 Merge branch 'dev' into easRevision
9b0b919 fix test
8e07d8a Update CloudAsset name max length to 512 chars in temporary table. (#2242)
a36509f styling
f58ac90 Add G Suite DwD enabled/disabled field in Inventory Summary email (#2201)
82130c3 New Cloud Asset API resources (#2235)
037b557 Update apt_packages to include new mysql name (#2173)
493a15d Remove apt-get upgrade from startup scripts in client & server vm. (#2195)
daa835a addressing some of the suggestions from last PR
e0107ae Initial commit of project access scanner (#2004)
f1400f8 Define variable before try block. (#2191)
33e6ede Throw A Clear Exception Error When A Scanner Doesn't Exist (#2160)
3323ec5 Globally Rename find_policy_violations() to find_violations() (#2172)
83545d6 Prevent output to the console (#2185)
fa21c02 Add VS Code to gitignore (#2186)
a071f95 Servicemanagement client update (#2175)
75af1e2 Integrate Release 2.7.0 Back into Dev (#2174)

@joecheuk joecheuk released this Nov 7, 2018

Assets 2

Summary

Inventory

  • Added bigquery datasets and service accounts from Cloud Asset Inventory.
  • Improved the inventory email summary, with a new detail section that breaks out resources in different states(e.g. active vs pending delete projects).

Scanner

  • Added Kubernetes Engine Scanner that uses JMESPath language and query.

Infrastructure

  • Reverted the workaround that restarted the server on every cron run.

Thanks to our contributors!

All changes

172e0d1 (HEAD -> release-2.7.0, origin/release-2.7.0) Prevent output to the console (#2185)
b877df8 Merge branch 'release-2.7.0' of github.com:GoogleCloudPlatform/forseti-security into release-2.7.0
github.com:GoogleCloudPlatform/forseti-security into release-2.7.0
2e86035 Merge branch 'dev' into release-2.7.0
2ccf8b4 (origin/dev, dev) Revert "Add OpenCensus instrumentation for tracing" (#2171)
37d047d Don't warn for known handlers. (#2167)
4bac8b0 Increment Forseti version.
ca65e41 Fix applies_to in location scanner rule (#2161)
3ec2263 Add OpenCensus instrumentation for tracing (#1926)
fd0945e Add support for IAM ServiceAccounts from Asset Inventory data. (#2152)
e0a2b92 Update docstrings for project_number args. (#2151)
64a1408 Support Bigquery Datasets and Access Policies from Cloud Assets data. (#2150)
337a7ca Remove private API flag for cloud asset API. (#2148)
34e00c5 Support specific ids in location rules applies_to (#2142)
918b4e0 Add jmespath scanner for KE (#1991)
faab5eb Revert "Restart the server at the beginning of the cronjob, temp fix to #1832" (#2146)
7522fc8 Separate active/pending delete project status in inventory summary email (#2132)
bf63266 Fix Flake8 styling issues in dev causing Travis build to fail (#2137)
f0f40d4 (origin/coontr) Release 2 6 0 merge to dev (#2141)
099b7c0 Cai resources importer (#2114)

@red2k18 red2k18 released this Oct 18, 2018 · 2 commits to release-2-6-0 since this release

Assets 2

Summary

Inventory

  • Added organization policies to inventory crawler and model.
  • Added all supported resource types from CAI to Forseti Inventory.

Scanner

  • Added location Scanner.
  • Updated the violations generated by the log sink scanner to contain proper full name.

Infrastructure

  • Added alter column functionality and increased the size of asset_data column.
  • Added role roles/orgpolicy.policyViewer to the forseti server service account.

Thanks to our contributors!

All changes

7275da1 Add organization policies to inventory crawler and model. (#2105)
97d6a74 Merge 2.5 to dev (#2107)
b5849fc Add better logs and details for CAI json parse errors. (#2104)
f14d4ce Added alter column functionality and increased the size of asset_data column (#2099)
8087f5f Update full_name in log sink scanner. (#2103)
45f92ff Updated Config files with location details to create bucket for CAI (#2101)
1adfe13 Cast lien violations to list (#2100)
96dc0e7 Add location scanner (#2075)
8a0ca24 Fix issue #2090. (#2091)
8839f16 Add all supported resource types from CAI to Forseti Inventory. (#2081)
9019130 Fix a bad code refactoring (#2072)
00bfc23 fix lt, gt for source and target tags (#2050)
395f7c3 fix firewall test (#2071)
e143101 fix cloudasset_test.py (#2066)

@joecheuk joecheuk released this Oct 18, 2018

Assets 2

Summary

Inventory

  • Integration with Cloud Asset Inventory (CAI). CAI is a new GCP service that provides data across different resources. We will keep on updating the integration as CAI onboards new resources. This integration also significantly reduces the overall time to build the inventory.
  • Updated quota configuration to improve performance when getting data from the GCP APIs.
    • increased iam API calls from 18 to 90 requests per second.
    • increased logging API calls from 1 to 9 requests per second.

Scanner

  • Added Project Lien scanner.
  • Fixed sqlalchemy error in bigquery scanner.

Infrastructure

  • New GCS bucket created to store the data dump from CAI.
  • Forseti server service account permission updates:
    • Added role roles/cloudasset.viewer on the organization level.
    • Added role roles/storage.objectAdmin to the newly created CAI bucket on the bucket level.
  • Enabled new cloudasset API.

Upgrade

  • Please refer to the upgrade instructions on our official website.

Thanks to our contributors!

All changes

d035a3f Merge branch 'dev' of github.com:GoogleCloudPlatform/forseti-security into release-2.5.0
a80818e Change to new CAI package name (#2074)
316c239 Workaround on the different namings in firewall resource data from CAI.
095f7bb Updated contributors.md and incremented version to 2.5.0
bc9598a Merge 2.4.0 to dev (#2068)
e53b8ac (dev) Model Delete returns result (#2044)
8dbc837 Finish one query before executing the others. (#2067)
eb01373 fix client config path (#2065)
3b01742 Add Lien scanner (#2041)
1693306 CAI Integration - Installer (#2061)
aca0132 Typo in help text of gcloud command (#2045)
7381a93 Update quota configuration. (#2043)
fa492c8 Make Firewall Scanner code maintainable (#2015)
a48f501 Added check to see if gsuite_superadmin_email is set. (#2040)
9661165 Adding version number into logs (#2037)
b63dcf2 CloudAsset data integration (#2034)
017d8a6 Add support for downloading files to the storage client. (#2028)
a811249 Update Forseti inventory storage to store the temporary CAI asset data. (#2019)

@red2k18 red2k18 released this Sep 26, 2018

Assets 2

RELEASE NOTE v2.4.0

Summary

Inventory

  • Added project liens as a new resource to Inventory.

Scanner

  • Added support for Forseti to run from a folder, instead of an organization.
  • Added resource_name column to all scanners and violations which contain human readable names.

Infrastructure

  • Improved firewall rules by removing default rules and creating Forseti specific firewall rules for the client, and with more organized priorities for both server and client firewall rules.
  • Updated Docker files to speed up build during development.
  • Updated Docker wrapper scripts to correctly return error code when any bash script test runners fail.
  • Improved naming quality of the Docker setup and install scripts, based on their functionalities.

Thanks to our contributors!

All changes

754ac74 (HEAD -> rele-2.4.0, tag: v2.4.0, origin/rele-2.4.0) version updated
dcaec81 (origin/dev, origin/HEAD, dev) Merge release-2.3.0 to dev (#2020)
2592fbd (rel-2.4.0) Added resource_name column to all the scanners, added schema update handles in scanner dao (#1864)
a6a45cc Add project Liens to inventory (#2011)
4cae7e2 Clean up and improve firewall rules (#2005)
d48e131 Create a new API client for the new Cloud Asset API. (#2008)
e985fe8 Allow data model to be built without an organization root. (#1770)
f5d61c0 Properly trap and return errors correctly in travis scripts (#2006)
6ed8234 Rename docker run script to clarify what it does (#1998)
38a7fe0 Adjust Dockerfiles to speed up builds during development (#1992)

@red2k18 red2k18 released this Sep 6, 2018 · 9 commits to release-2-3-0 since this release

Assets 2

Summary

Installer

  • More robust installation process by handling ssh failure gracefully, and by enabling additional Google APIs in case they are not enabled by default.

Scanner

  • KE Scanner: Kubernetes rule updated to scan for the below vulnerabilities.
    • CVE-2018-5390 describes a kernel-level networking vulnerability that increases the effectiveness of denial of service (DoS) attacks against vulnerable systems over TCP connections.
    • CVE-2018-5391 describes a kernel-level networking vulnerability that increases the effectiveness of denial of service (DoS) attacks against vulnerable systems over IP connections.
  • BigQuery Scanner: Enhanced BigQuery rules syntax to support bindings in BigQuery rules
    with backward compatibility.
  • IAM Scanner:
    • Added a new rule to scan for bucket with allUsers permission set in IAM policy.
    • allAuthenticatedUsers can now be audited and added the associated rule.
    • Billing account can now be audited.

Upgrade instructions

  • If you deployed version v2.0.0 or v2.1.0, due to a deployment script change in v2.2.0, you will need to be on the older git tag of your Forseti to run the deployment manager update command.
  • Complete instructions
    • Download the Forseti server deployment template from the Forseti server GCS bucket, the template is located under folder deployment_templates.
    • Update the fields in the deployment template according to the Forseti official website, look at section Change deployment properties under Upgrading 2.X installations.
      • Specifically, change these default values for the newly added fields in the deploy-forseti-server-{HASH}.yaml to:
        • vpc-host-project-id: {YOUR_PROJECT_ID}
        • vpc-host-network: default
        • vpc-host-subnetwork: default
    • Put the deployment template file under forseti-security/deployment-templates/ in your cloud shell.
    • Make sure the git tag of forseti-security is on the same Forseti version you ran the deployment on (e.g. if you deployed version v2.0.0, you will need to make sure forseti-security is on git tag v2.0.0).
    • You can verify the git tag by running command git status under the forseti-security folder.
    • Run command gcloud deployment-manager deployments update {DEPLOYMENT_NAME} --config path/to/deploy-forseti-server-{HASH}.yaml to do the update.

Thanks to our contributors!

All changes

a4df217 (HEAD -> release-2-3-0, tag: v2.3.0, origin/release-2-3-0) added contributors and modified version
5c928f2 (dev) Merge branch 'dev' of github.com:GoogleCloudPlatform/forseti-security into dev
a8322ea (origin/dev, origin/HEAD) kubernetes rule updated to address the latest vulnerabilities (#1990)
70d45ea Merge branch 'dev' of github.com:GoogleCloudPlatform/forseti-security into dev
df48b43 Stop matching unset member fields in BigQuery ACLs (#1989)
6c6de3f Support multiple dataset ids in BigQuery rules (#1986)
092c460 (firewallrule) Merge branch 'dev' of github.com:GoogleCloudPlatform/forseti-security into dev
bf9b667 Fixed IAM Scanner so it audits allAuthenticatedUsers correctly (#1983)
b105396 Support bindings in BigQuery rules (#1977)
42b767f Merge branch 'dev' of github.com:GoogleCloudPlatform/forseti-security into dev
af66fd5 Add support for billing_account to IAM scanner (#1975)
6b05607 adding Michael Capicotto as a contributor (#1970)
9221815 Handle SSH failure gracefully during the installation process (#1969)
d3d0bfc more APIs require enabling (#1967)
1e3a337 (fixiamscanner) Merge branch 'dev' of github.com:GoogleCloudPlatform/forseti-security into dev
87477eb Removed upgrade option (#1965)
02323bd (authusers) Merge branch 'dev' of github.com:GoogleCloudPlatform/forseti-security into dev
e47b2aa Added a new rule to scan for bucket with allUsers in IAM policy (#1964)
d80d53e (allauthusers) Merge branch 'dev' of github.com:GoogleCloudPlatform/forseti-security into dev
9a66da6 Default non-existent fields in bigquery acl to glob (#1958)
a742de4 (1909authusers) Merge branch 'dev' of github.com:GoogleCloudPlatform/forseti-security into dev
1c99b00 Use the file_loader util method for safe yaml loading. (#1959)
df01f4e Remove the AE for now while it's reworked. (#1961)
99c7105 [Issue 1848] Fixing parameterized test to mock out logger to fix log pollution (#1929)
3fda0e0 Merge stable to dev (#1956)
035d553 [Fixes #783] Support whitelist mode in Bigquery scanner (#1925)
037039c Add billing accounts to log sink scanner (#1922)
94c2171 Removed unused constant MESSAGE_GSUITE_DATA_COLLECTION (#1953)

@joecheuk joecheuk released this Aug 22, 2018 · 7 commits to release-2.2.0 since this release

Assets 2

Summary

Installer

  • Shared VPC Supports: Installer will now be able to handle deployment with shared VPC, by specifying the following flags at the start of the deployment:
    • vpc-host-project-id
    • vpc-host-network
    • vpc-host-subnetwork
  • G Suite updates: G Suite integration is now optional. Forseti will not inventory any G Suite groups/users if the G Suite super admin email is not provided. You can learn more about the details here.
  • Templatize Forseti server region and Zone.

Inventory

  • Compute Engine Disk Snapshots: Your Compute Engine Disk Snapshots information is now inventoried.
  • Container: masterAuth attribute for container clusters is now retained, but the actual data value is redacted.

Notifier

  • CSCC API Mode: Improved usability of Forseti findings in Cloud Security Command Center by including more useful information (e.g. rule name and db source) to the display item. You can find the instructions on how to setup CSCC integration here.

Scanner

  • Group Scanner: Updated to avoid scanning members with no rule.
  • BigQuery Scanner: Updated to respect resources.
  • IAM Scanner: Updated to audit allUsers correctly.

Enforcer

  • Enforcer is now updated to use the common gcp_api compute client.

Fixes/Updates

  • API client: Add mixins for Insert, Update, Delete actions
  • Logger: Updated to use exception() instead of error() when it's logging inside an except block, so the stack trace will also be produced.
  • Group scanner test is re-enabled.
  • Suppressed noisy app errors in unit tests.

Upgrade instructions

  • If you deployed version v2.0.0 or v2.1.0, due to a deployment script change in v2.2.0, you will need to be on the older git tag of your Forseti to run the deployment manager update command.
  • Complete instructions
    • Download the Forseti server deployment template from the Forseti server GCS bucket, the template is located under folder deployment_templates.
    • Update the fields in the deployment template according to the Forseti official website, look at section Change deployment properties under Upgrading 2.X installations.
      • Specifically, change these default values for the newly added fields in the deploy-forseti-server-{HASH}.yaml to:
        • vpc-host-project-id: {YOUR_PROJECT_ID}
        • vpc-host-network: default
        • vpc-host-subnetwork: default
    • Put the deployment template file under forseti-security/deployment-templates/ in your cloud shell.
    • Make sure the git tag of forseti-security is on the same Forseti version you ran the deployment on (e.g. if you deployed version v2.0.0, you will need to make sure forseti-security is on git tag v2.0.0).
    • You can verify the git tag by running command git status under the forseti-security folder.
    • Run command gcloud deployment-manager deployments update {DEPLOYMENT_NAME} --config path/to/deploy-forseti-server-{HASH}.yaml to do the update.

Thanks to our contributors!

All changes

ab2346b (tag: v2.2.0, origin/release-2.2.0, release-2.2.0) Merge branch 'dev' of github.com:GoogleCloudPlatform/forseti-security into release-2.2.0
50db302 (origin/dev, origin/HEAD) Improve CSCC usability (#1907)
074769c Added space
b8418c3 Increment version to 2.2.0
30aff07 (dev) Merge stable to dev (#1940)
e08356c Updated Installer prompt that G Suite is optional (#1936)
cee4193 Updated Installer with G Suite optional (#1934)
402a22b [Issue 1848] Fix for more log pollution of tests. (#1921)
479bf7d Updated the group scanner to avoid scanning members with no rule (#1905)
94f60fc Removed unused variable in required section when generating the deployment template. (#1924)
5069400 [Fixes #1865] Fix bigquery scanner to respect resources (#1884)
6abf60a [Issue 1848] Mock out server errors for invalid arguments to eliminate log pollution in tests. (#1919)
088f46d Templatize Forseti server region and Zone (#1887)
639bd6d Removed sample from actual rule names (#1916)
5cf529d Fix dev installer (#1917)
e5ab500 [Fixes #1859] Remove dependency on the _metadata server module from google.auth (#1860)
f511cd2 Inventory and model compute snapshots (#1893)
33101ea Fixes #1871, Update Enforcer to use the common gcp_api compute client. (#1904)
f2a959a [Issue 1848] Mock out logger to fix almost all remaining instances of pollution of test logs. Remaining issus involve server and will likely require some production refactoring. (#1903)
03c81c6 [Issue 1848] Mock out logger to fix pollution of test logs. (#1899)
6249e00 Updated logger to use exception() instead of error() when it's logging inside an except block. (#1897)
6d82da4 Add a flake8 test (pycodestyle) to check for pep8 related stye (#1896)
b509dbc Added try catch before uploading files to gcs bucket. (#1895)
c1aae0f update stacktrace in broad excepts issue#1797 (#1836)
ad10bf6 Remove cluster auth data, but keep keys (#1888)
8af6b33 [Issue #1848] Fixing logging to use Forsetting logging infrastructure. (#1890)
181a559 Fixing CrawlerTest to use Forseti Logging infrastructure (#1889)
c0b783e Update docker_unittest_forseti.sh (#1886)
d4b907a Restore VPC Support (#1874)
705dc31 Collapse apt layers in base dockerfile (#1883)
e3e1116 Fix iam scanner so that it audits allUsers correctly (#1878)
26fdd51 Re-enable groups scanner test (#1873)
9fddb9b (break_down_query, alpha_role_handling) Add Compute client methods to insert, update and delete firewall rules. (#1872)
363ecb2 Add Billing Account log sinks to Inventory (#1839)
f1da48b Added requriemodel decorator to the scanner run method, pin the version of rumael.yaml library (#1870)
4fc6568 Fixing copy and paste error in test description (#1867)
0298934 [Issue 1848] Fix a test that is emitting errors and polluting the logs. (#1857)
3b1a974 Clean up test dependencies (#1858)
3a5f894 Refactor server.py to move config classes into base/config.py. (#1854)
6f942a0 Fixes to gcloud.py and Service Account Support (#1815)

@joecheuk joecheuk released this Aug 16, 2018 · 1 commit to stable since this release

Assets 2

RELEASE NOTE v2.1.0

Summary

Installer

  • Force the Forseti server to restart at the beginning of the cron run as a temporary fix to the auth issue #1832.
  • Forseti installation process can now be automated by passing in flags for all the prompted values.

Inventory

  • Compute Engine Disks: Your Compute Engine Disks information are now inventoried.
  • Log sinks (Exports): Your Log sink information are now inventoried.

Notifier

  • CSCC API Mode: Forseti notifier is now integrated using CSCC API, once you have the Forseti server service account setup with CSCC, Forseti will directly send violations to CSCC.

To setup the Forseti server service account with CSCC, you will need to whitelist your Forseti project with CSCC and assign role securityCenter.editor to the Forseti server service account on the organization level. You learn more about the configuration here.

Fixes

  • Data model: Increased column size for some of the columns.
  • Service account key scanner: Updated to not use execute multiple queries in the same session during the use of yield_per().

Upgrade notes

To enable the CSCC API mode, add the following to the notifier section in your forseti_conf_server.yaml file.

Note: mode can be either 'api' or 'bucket' and if mode is 'bucket', you will need to specify a gcs_path.

notifier:
    violation:
        cscc:
            enabled: true
            mode: api
            organization_id: organizations/<your_organization_id>
            # gcs_path should begin with "gs://"
            gcs_path:

Thanks to our contributors!

All changes

9d3a465 (tag: v2.0.1) Added init file for discovery documents
7fd0d63 Increased column size for data model
aae7c41 Use absolute path to detech discovery_documents folder
e504c70 Updated version to 2.0.1
9cbbe49 Updated variable in the configs yaml file (#1781)
af627a3 (dev) Merge stable to dev (#1780)
1e765e3 Alpha sort the scanner maps issue#1654 (#1777)
f8f1a95 Updated broken urls (#1773)
84bf37a Add Compute Engine disks to crawler and data model. (#1766)
fcdb3d8 Merge stable to dev (#1764)
a897a3b Fix broken links on README.md (#1751)
6c2d82c update iter_foo and fetch_foo issue#1702 (#1760)
6384eb4 Updated to not query again in the same session during yield_per() (#1763)
431e1ef Tidy-Up CSCC API (#1757)
b20482a Updated the dataset type_name to use dataset_policy/{dataset_id} format. (#1759)
cbf25f6 Fixing exemption typo issue#1643 (#1749)
d4edf80 Fix CSCC Notifier Test (#1750)
23122bc Adding working changes for CSCC API integration (#1746)
ea0d66c Add Log Sinks (Exports) to the Forseti Inventory (#1681)
c8d1485 Added Service Account support for installing Forseti (#1706)
0a1c691 Merge RC3 into Dev (#1726)
0f38839 Handle the deprecated port field in backend service. (#1717)
e2f3112 Update build and code coverage status for 2.0 branches. (#1723)
08867da Fix blacklist scanner to handle network intefaces that do not have external internet access. (#1721)
ce2f60e Updated output (#1714)
4ddd003 update cloudsql naming (#1695)
da6daa1 fix service account key scanner name in sample config (#1691)
4036ed4 updated installation instructions (#1689)
2c0c575 Updated hardcoded resource types in violation to use the resource types defined in the ResourceType class (#1665)

Assets 2

Summary

All Changes

6687c0d Removed the execution of run_forseti.sh from the startup script
c7ffe8a Pin idna to version 2.6 and pip to version 9.0.3
415f40b Added location check for KE pipeline
70aa0b1 Update ke scanner rules