Templatize Forseti server region and Zone #1887

johnrevans6 · 2018-08-15T04:27:17Z

Currently, The deployment template for the Forseti server explicitly references ref.cloudsql-instance.region as the server's subnet region, and likewise, references ref.cloudsql-instance.region-c as the server's GCE zone. Abstracted this away into FORSETI_SERVER_REGION, and FORSETI_SERVER_ZONE template variables to better match the Forseti client template.

blueandgold

Thanks for this! Just a couple of comments.

blueandgold · 2018-08-17T04:53:33Z

rules/bigquery_rules.yaml

@@ -24,7 +24,7 @@ rules:
    resource:
      - type: organization
        resource_ids:
-          - {ORGANIZATION_ID}
+          - 826592752744


We should not need this. :)

Are you saying these files can be removed?

We should not have the specific value here. The placeholder will be filled in by the Installer during deployment.

blueandgold · 2018-08-17T05:10:25Z

install/gcp/installer/forseti_server_installer.py

@@ -272,6 +272,8 @@ def get_deployment_values(self):
            'FORSETI_BUCKET': bucket_name[len('gs://'):],
            'BUCKET_LOCATION': self.config.bucket_location,
            'GCP_SERVER_SERVICE_ACCOUNT': self.gcp_service_acct_email,
+            'FORSETI_SERVER_REGION': self.config.cloudsql_region,


This is an improvement, and will be accepted. But the inherent oddity is still present... that is we are re-using a value for the cloudsql for the VM. Either that flag should be renamed --cloudsql-region to just --deployment-region, or a new flag for the VM should be added --vm_region. I realized this is a bigger change, so thoughts?

Been thinking about this. I guess my question is, is the intent to have the Cloud SQL instance in the same region as the Forseti Server and Client? If so, then we should definitely rename the flag to --deployment-region to capture that. If not, then you want a separate flag for the VM region.

Correct, the intent is to have everything in the same region. We don't have any use case for them to be in separate region.

blueandgold · 2018-08-17T05:12:38Z

deployment-templates/compute-engine/server/forseti-instance-server.py.schema

@@ -25,7 +25,7 @@ required:
 - zone
 - service-account
 - service-account-scopes
- service-account-gsuite
+# - service-account-gsuite


Why is this commented out?

At the time of this PR, this requirement isn't used in the template, and thus breaks the build. I commented it out because I assumed this was an introduction by the Forseti team, and since it is in dev, I figured I would hold off on further action until I had a chance to speak with you more about it.

We used to have a dedicated gsuite service account for integrating with gsuite, but has since been removed. Perhaps this was missed in the clean-up. Are you sure that this needs to be commented out? I am somewhat surprised since we did the 2.0 & 2.1 releases with this in place, without anything breaking. Can you please confirm?

blueandgold

Thanks for this PR!

blueandgold · 2018-08-20T14:42:36Z

rules/bigquery_rules.yaml

@@ -24,7 +24,7 @@ rules:
    resource:
      - type: organization
        resource_ids:
-          - {ORGANIZATION_ID}
+          - 826592752744


We should not have the specific value here. The placeholder will be filled in by the Installer during deployment.

blueandgold · 2018-08-20T14:44:00Z

install/gcp/installer/forseti_server_installer.py

@@ -272,6 +272,8 @@ def get_deployment_values(self):
            'FORSETI_BUCKET': bucket_name[len('gs://'):],
            'BUCKET_LOCATION': self.config.bucket_location,
            'GCP_SERVER_SERVICE_ACCOUNT': self.gcp_service_acct_email,
+            'FORSETI_SERVER_REGION': self.config.cloudsql_region,


Correct, the intent is to have everything in the same region. We don't have any use case for them to be in separate region.

blueandgold · 2018-08-20T14:50:27Z

deployment-templates/compute-engine/server/forseti-instance-server.py.schema

@@ -25,7 +25,7 @@ required:
 - zone
 - service-account
 - service-account-scopes
- service-account-gsuite
+# - service-account-gsuite


We used to have a dedicated gsuite service account for integrating with gsuite, but has since been removed. Perhaps this was missed in the clean-up. Are you sure that this needs to be commented out? I am somewhat surprised since we did the 2.0 & 2.1 releases with this in place, without anything breaking. Can you please confirm?

* Updated hardcoded resource types in violation to use the resource types defined in the ResourceType class (#1665) * Updated hardcoded resource types in violation to use the resource type defined in the ResourceType class. * removed unused comments * Updated unit tests * Addressed PR comments * Updated unit test * Removed warning messages * updated installation instructions (#1689) * fix service account key scanner name in sample config (#1691) * update cloudsql naming (#1695) * Updated output (#1714) * Fix blacklist scanner to handle network intefaces that do not have external internet access. (#1721) * updates * updates * Update build and code coverage status for 2.0 branches. (#1723) * Update travis status in README * tweak table * tweak * address comment * Handle the deprecated port field in backend service. (#1717) * Added error handling in scanners * Addressed PR comment * updates * updates * updates * updates * Addressed PR comments * Updates * updates * updates * updates * updates * updates * Merge RC3 into Dev (#1726) * clean up the code to change inventory_index_id arg to type int64 everywhere except at the parser level (#1561) * fixes * Updated inventory_index_id in notifier * updates * updated docstring * updated unit tests * Updated notifier unit tests * pylint fixes * Changed notifier inventory_index_id default value to 0 * cli better grpc error handling (#1562) * Improved grpc error handling. * added unit test * Fix Missing Scopes in Local Environment (#1564) * Fix missing scope * fix test * CLI stacktrace clean up (#1584) * CLI stacktrace clean up * Addressed PR comment * pylint fixes * Updated firewall rule to not call the parse function when the time is empty (#1588) * Updated firewall rule to not call the parse function when the time is not passed in * Added comment * handle error from saving cscc (#1589) * Clarify required flags (#1592) * Clarify required flags * Clarify required flags * Clarify required flags * fix test * Update resource name to use hash of timestamp & org id (#1591) * Updated the resource identifier * Add identifier generation * pylint fixes * Update cloudsql instance after identifier is generated * Lower cased the identifier * Pass identifer to client if both client and server are installed at the same time * Addressed PR comments * Updated kubernetes violation to use full name instead of parent full name (#1594) * Added cluster name to kubernetes full name * Updated to use resource full name instead of parent full name * skip child resource when getting ancestors * Updated unit tests * Removed google.apputils usage in stubs (#1596) * Removed apputils usage * Added default args back for testing purpose * removed unused import * Updated lock file location (#1597) * updated lock file location * added sudo * Lock on README instead so we don't need to create any new file * added lock file * update_flag (#1602) * Disable scanners that have rules all commented out (#1603) * Update ke scanner rules (#1604) * Updated logs (#1606) * Added cloud function script to export inventory summary to bigquery (#1607) * Added cloud function script to export inventory summary to bigquery * Updates * Port Inventory email to 2.0 (#1598) * Port Inventory email to 2.0 * tweak * tweak * add timestamp * add timestamp * fix inventory summary tests * fix test * fix tests * update the config files * tweak * Remove default plain text output style (#1608) * Updated the inventory to output a viewmodel instead * Removed plain text output format option * updated show config method * Updated unit tests * Increased column size for inventory errors and message (#1615) * Increased column size for inventory errors * increased column size for message column in model * Fix scanner_iter() db session out of sync when used by multiple simultaneous queries. (#1718) * Fix db session out of sync * tweak * tweak * Remove additional db query which conflicts with existing db session (#1720) * Fix blacklist scanner to handle network intefaces that do not have external internet access. (#1721) * updates * updates * add blank line * Added Service Account support for installing Forseti (#1706) * Added Support for running installer as Service Account * Added service account file credential path lookup * Fixed typo * Ignore domain check if using Service Account * Minor cleanup * Bug fix * Fix trailing whitespace * Fix linter errors * Fix linter errors * Fix lynter errors * Fix linter errors * Added --service-account-key-path support * Removed credentials lookup and added key_path support to activate_service_account * Fix key path * Fix key path * Fix linter errors * Fix linter errors * Update gcp_installer.py * Update config.py * Update gcloud.py * Update gcloud.py * Update gcloud.py * Update gcloud.py * Update gcp_installer.py * Update gcloud.py * Update gcloud.py * Update gcp_installer.py * Update gcp_installer.py * Update config.py * Update gcp_installer.py * Update forseti_installer.py * Update forseti_installer.py * Add Log Sinks (Exports) to the Forseti Inventory (#1681) * Add Stackdriver Logging GCP API for sinks * Add Stackdriver Logging methods to GCP ApiClient * Add logging sinks to the crawler * Add logging sinks to Modeller resources tables * Remove unnecessary split on name in FolderSinkIterator * Adding working changes for CSCC API integration (#1746) * Fix CSCC Notifier Test (#1750) Get Travis back to Green after merging the initial CSCC API PR * Fixing exemption typo issue#1643 (#1749) * Fixing exemption typo issue#1643 * Fixing exemption typo issue#1643 * Updated the dataset type_name to use dataset_policy/{dataset_id} format. (#1759) * Updated the dataset type_name to use dataset_policy/{dataset_id} format. * Removed extra post action call * Tidy-Up CSCC API (#1757) * Tidy up CSCC API * tweak * tweak * tweak * tweak * tweak * tweak * fix tests * add test * fix tests * add log message * tweak * fix test * Updated to not query again in the same session during yield_per() (#1763) * Updated to not query again in the same session during yield_per() * format updates * Added error checks * Get type_name from full_name * update iter_foo and fetch_foo issue#1702 (#1760) * Fix broken links on README.md (#1751) * Merge stable to dev (#1764) * Merge 2.0 release branch into stable branch. (#1732) * Updated hardcoded resource types in violation to use the resource types defined in the ResourceType class (#1665) * Updated hardcoded resource types in violation to use the resource type defined in the ResourceType class. * removed unused comments * Updated unit tests * Addressed PR comments * Updated unit test * Removed warning messages * updated installation instructions (#1689) * fix service account key scanner name in sample config (#1691) * update cloudsql naming (#1695) * Merge Dev into 2.0.0 release branch (#1727) * clean up the code to change inventory_index_id arg to type int64 everywhere except at the parser level (#1561) * fixes * Updated inventory_index_id in notifier * updates * updated docstring * updated unit tests * Updated notifier unit tests * pylint fixes * Changed notifier inventory_index_id default value to 0 * cli better grpc error handling (#1562) * Improved grpc error handling. * added unit test * Fix Missing Scopes in Local Environment (#1564) * Fix missing scope * fix test * CLI stacktrace clean up (#1584) * CLI stacktrace clean up * Addressed PR comment * pylint fixes * Updated firewall rule to not call the parse function when the time is empty (#1588) * Updated firewall rule to not call the parse function when the time is not passed in * Added comment * handle error from saving cscc (#1589) * Clarify required flags (#1592) * Clarify required flags * Clarify required flags * Clarify required flags * fix test * Update resource name to use hash of timestamp & org id (#1591) * Updated the resource identifier * Add identifier generation * pylint fixes * Update cloudsql instance after identifier is generated * Lower cased the identifier * Pass identifer to client if both client and server are installed at the same time * Addressed PR comments * Updated kubernetes violation to use full name instead of parent full name (#1594) * Added cluster name to kubernetes full name * Updated to use resource full name instead of parent full name * skip child resource when getting ancestors * Updated unit tests * Removed google.apputils usage in stubs (#1596) * Removed apputils usage * Added default args back for testing purpose * removed unused import * Updated lock file location (#1597) * updated lock file location * added sudo * Lock on README instead so we don't need to create any new file * added lock file * update_flag (#1602) * Disable scanners that have rules all commented out (#1603) * Update ke scanner rules (#1604) * Updated logs (#1606) * Added cloud function script to export inventory summary to bigquery (#1607) * Added cloud function script to export inventory summary to bigquery * Updates * Port Inventory email to 2.0 (#1598) * Port Inventory email to 2.0 * tweak * tweak * add timestamp * add timestamp * fix inventory summary tests * fix test * fix tests * update the config files * tweak * Remove default plain text output style (#1608) * Updated the inventory to output a viewmodel instead * Removed plain text output format option * updated show config method * Updated unit tests * Increased column size for inventory errors and message (#1615) * Increased column size for inventory errors * increased column size for message column in model * Updated output (#1714) * Fix scanner_iter() db session out of sync when used by multiple simultaneous queries. (#1718) * Fix db session out of sync * tweak * tweak * Remove additional db query which conflicts with existing db session (#1720) * Fix blacklist scanner to handle network intefaces that do not have external internet access. (#1721) * updates * updates * Fix blacklist scanner to handle network intefaces that do not have external internet access. (#1721) * updates * updates * Update build and code coverage status for 2.0 branches. (#1723) * Update travis status in README * tweak table * tweak * address comment * Handle the deprecated port field in backend service. (#1717) * Added error handling in scanners * Addressed PR comment * updates * updates * updates * updates * Addressed PR comments * Updates * updates * updates * updates * updates * updates * add blank line * Merge RC3 into Dev (#1726) * clean up the code to change inventory_index_id arg to type int64 everywhere except at the parser level (#1561) * fixes * Updated inventory_index_id in notifier * updates * updated docstring * updated unit tests * Updated notifier unit tests * pylint fixes * Changed notifier inventory_index_id default value to 0 * cli better grpc error handling (#1562) * Improved grpc error handling. * added unit test * Fix Missing Scopes in Local Environment (#1564) * Fix missing scope * fix test * CLI stacktrace clean up (#1584) * CLI stacktrace clean up * Addressed PR comment * pylint fixes * Updated firewall rule to not call the parse function when the time is empty (#1588) * Updated firewall rule to not call the parse function when the time is not passed in * Added comment * handle error from saving cscc (#1589) * Clarify required flags (#1592) * Clarify required flags * Clarify required flags * Clarify required flags * fix test * Update resource name to use hash of timestamp & org id (#1591) * Updated the resource identifier * Add identifier generation * pylint fixes * Update cloudsql instance after identifier is generated * Lower cased the identifier * Pass identifer to client if both client and server are installed at the same time * Addressed PR comments * Updated kubernetes violation to use full name instead of parent full name (#1594) * Added cluster name to kubernetes full name * Updated to use resource full name instead of parent full name * skip child resource when getting ancestors * Updated unit tests * Removed google.apputils usage in stubs (#1596) * Removed apputils usage * Added default args back for testing purpose * removed unused import * Updated lock file location (#1597) * updated lock file location * added sudo * Lock on README instead so we don't need to create any new file * added lock file * update_flag (#1602) * Disable scanners that have rules all commented out (#1603) * Update ke scanner rules (#1604) * Updated logs (#1606) * Added cloud function script to export inventory summary to bigquery (#1607) * Added cloud function script to export inventory summary to bigquery * Updates * Port Inventory email to 2.0 (#1598) * Port Inventory email to 2.0 * tweak * tweak * add timestamp * add timestamp * fix inventory summary tests * fix test * fix tests * update the config files * tweak * Remove default plain text output style (#1608) * Updated the inventory to output a viewmodel instead * Removed plain text output format option * updated show config method * Updated unit tests * Increased column size for inventory errors and message (#1615) * Increased column size for inventory errors * increased column size for message column in model * Fix scanner_iter() db session out of sync when used by multiple simultaneous queries. (#1718) * Fix db session out of sync * tweak * tweak * Remove additional db query which conflicts with existing db session (#1720) * Fix blacklist scanner to handle network intefaces that do not have external internet access. (#1721) * updates * updates * add blank line * Run cloudsqlproxy service as root (#1762) * Removed User= from cloudsqlproxy service * Addressed PR comment * Add Compute Engine disks to crawler and data model. (#1766) * Add Compute Engine disks to crawler and data model. * Updated broken urls (#1773) * Alpha sort the scanner maps issue#1654 (#1777) * Merge stable to dev (#1780) * Merge 2.0 release branch into stable branch. (#1732) * Updated hardcoded resource types in violation to use the resource types defined in the ResourceType class (#1665) * Updated hardcoded resource types in violation to use the resource type defined in the ResourceType class. * removed unused comments * Updated unit tests * Addressed PR comments * Updated unit test * Removed warning messages * updated installation instructions (#1689) * fix service account key scanner name in sample config (#1691) * update cloudsql naming (#1695) * Merge Dev into 2.0.0 release branch (#1727) * clean up the code to change inventory_index_id arg to type int64 everywhere except at the parser level (#1561) * fixes * Updated inventory_index_id in notifier * updates * updated docstring * updated unit tests * Updated notifier unit tests * pylint fixes * Changed notifier inventory_index_id default value to 0 * cli better grpc error handling (#1562) * Improved grpc error handling. * added unit test * Fix Missing Scopes in Local Environment (#1564) * Fix missing scope * fix test * CLI stacktrace clean up (#1584) * CLI stacktrace clean up * Addressed PR comment * pylint fixes * Updated firewall rule to not call the parse function when the time is empty (#1588) * Updated firewall rule to not call the parse function when the time is not passed in * Added comment * handle error from saving cscc (#1589) * Clarify required flags (#1592) * Clarify required flags * Clarify required flags * Clarify required flags * fix test * Update resource name to use hash of timestamp & org id (#1591) * Updated the resource identifier * Add identifier generation * pylint fixes * Update cloudsql instance after identifier is generated * Lower cased the identifier * Pass identifer to client if both client and server are installed at the same time * Addressed PR comments * Updated kubernetes violation to use full name instead of parent full name (#1594) * Added cluster name to kubernetes full name * Updated to use resource full name instead of parent full name * skip child resource when getting ancestors * Updated unit tests * Removed google.apputils usage in stubs (#1596) * Removed apputils usage * Added default args back for testing purpose * removed unused import * Updated lock file location (#1597) * updated lock file location * added sudo * Lock on README instead so we don't need to create any new file * added lock file * update_flag (#1602) * Disable scanners that have rules all commented out (#1603) * Update ke scanner rules (#1604) * Updated logs (#1606) * Added cloud function script to export inventory summary to bigquery (#1607) * Added cloud function script to export inventory summary to bigquery * Updates * Port Inventory email to 2.0 (#1598) * Port Inventory email to 2.0 * tweak * tweak * add timestamp * add timestamp * fix inventory summary tests * fix test * fix tests * update the config files * tweak * Remove default plain text output style (#1608) * Updated the inventory to output a viewmodel instead * Removed plain text output format option * updated show config method * Updated unit tests * Increased column size for inventory errors and message (#1615) * Increased column size for inventory errors * increased column size for message column in model * Updated output (#1714) * Fix scanner_iter() db session out of sync when used by multiple simultaneous queries. (#1718) * Fix db session out of sync * tweak * tweak * Remove additional db query which conflicts with existing db session (#1720) * Fix blacklist scanner to handle network intefaces that do not have external internet access. (#1721) * updates * updates * Fix blacklist scanner to handle network intefaces that do not have external internet access. (#1721) * updates * updates * Update build and code coverage status for 2.0 branches. (#1723) * Update travis status in README * tweak table * tweak * address comment * Handle the deprecated port field in backend service. (#1717) * Added error handling in scanners * Addressed PR comment * updates * updates * updates * updates * Addressed PR comments * Updates * updates * updates * updates * updates * updates * add blank line * Merge RC3 into Dev (#1726) * clean up the code to change inventory_index_id arg to type int64 everywhere except at the parser level (#1561) * fixes * Updated inventory_index_id in notifier * updates * updated docstring * updated unit tests * Updated notifier unit tests * pylint fixes * Changed notifier inventory_index_id default value to 0 * cli better grpc error handling (#1562) * Improved grpc error handling. * added unit test * Fix Missing Scopes in Local Environment (#1564) * Fix missing scope * fix test * CLI stacktrace clean up (#1584) * CLI stacktrace clean up * Addressed PR comment * pylint fixes * Updated firewall rule to not call the parse function when the time is empty (#1588) * Updated firewall rule to not call the parse function when the time is not passed in * Added comment * handle error from saving cscc (#1589) * Clarify required flags (#1592) * Clarify required flags * Clarify required flags * Clarify required flags * fix test * Update resource name to use hash of timestamp & org id (#1591) * Updated the resource identifier * Add identifier generation * pylint fixes * Update cloudsql instance after identifier is generated * Lower cased the identifier * Pass identifer to client if both client and server are installed at the same time * Addressed PR comments * Updated kubernetes violation to use full name instead of parent full name (#1594) * Added cluster name to kubernetes full name * Updated to use resource full name instead of parent full name * skip child resource when getting ancestors * Updated unit tests * Removed google.apputils usage in stubs (#1596) * Removed apputils usage * Added default args back for testing purpose * removed unused import * Updated lock file location (#1597) * updated lock file location * added sudo * Lock on README instead so we don't need to create any new file * added lock file * update_flag (#1602) * Disable scanners that have rules all commented out (#1603) * Update ke scanner rules (#1604) * Updated logs (#1606) * Added cloud function script to export inventory summary to bigquery (#1607) * Added cloud function script to export inventory summary to bigquery * Updates * Port Inventory email to 2.0 (#1598) * Port Inventory email to 2.0 * tweak * tweak * add timestamp * add timestamp * fix inventory summary tests * fix test * fix tests * update the config files * tweak * Remove default plain text output style (#1608) * Updated the inventory to output a viewmodel instead * Removed plain text output format option * updated show config method * Updated unit tests * Increased column size for inventory errors and message (#1615) * Increased column size for inventory errors * increased column size for message column in model * Fix scanner_iter() db session out of sync when used by multiple simultaneous queries. (#1718) * Fix db session out of sync * tweak * tweak * Remove additional db query which conflicts with existing db session (#1720) * Fix blacklist scanner to handle network intefaces that do not have external internet access. (#1721) * updates * updates * add blank line * Run cloudsqlproxy service as root (#1762) * Removed User= from cloudsqlproxy service * Addressed PR comment * Updated variable in the configs yaml file (#1781) * Updated variable in the configs yaml file * Updated org id in sample file * Updates * Pre release patches- Increased column size, updated discovery_documents path and incremented release version (#1786) * Updated version to 2.0.1 * Use absolute path to detech discovery_documents folder * Increased column size for data model * Added init file for discovery documents * Fix full_res_name value when the root_resource is a project or a folder (#1788) * Improved the keys of Memory.mem to avoid key conflicts. Some GCP resources share the same key (an Instance Group and its Instance Group Manager, for example). (#1796) * Add a scanner for Log Sinks (#1747) * Add Log Sink Rules Engine. * Improve test coverage, fix bugs * Add log sink engine * Add Log Sink scanner to scanne map and configs. * Use empty string instead of none for sink rule violation * Update examples in log_sink_rules.yaml * Shorten the violation id hash. (#1803) * shorten the hash * shorten the hash when we send to CSCC, so that we can preserve the original full hash for other clients that rely on it * fix test * Short the violation hash id to 32 characters for upload to CSCC (#1841) * increment version to 2.1.0 (#1842) * Fixes to gcloud.py and Service Account Support (#1815) * + Fixed silent gcloud errors that are generating errors in the logs + Removed service account name requirement from activate_service_account * Update gcloud.py * Update gcloud.py * Refactor server.py to move config classes into base/config.py. (#1854) * Clean up test dependencies (#1858) * [Issue 1848] Fix a test that is emitting errors and polluting the logs. (#1857) * Fixing copy and paste error in test description (#1867) * Added requriemodel decorator to the scanner run method, pin the version of rumael.yaml library (#1870) * Added requriemodel decorator to scanner run method * pin rumael.yaml version * Add Billing Account log sinks to Inventory (#1839) * Add list and get_iam_policy to Cloud Billing client * Fix lint errors * Add billing account methods to GCP ApiClient * Add billing accounts to crawler * Add billing account sinks to the crawler and model. * Code tidyup. * Add Compute client methods to insert, update and delete firewall rules. (#1872) * Add mixins for Insert, Update, Delete actions * Add support for Dry Run / read only mode to client * Add support to make async calls blocking until call completes * Switch API version for compute.firewalls to v1 instead of beta * Re-enable groups scanner test (#1873) * Re-enable groups scanner test * remove the direct member count * Fix iam scanner so that it audits allUsers correctly (#1878) * Fix iam scanner so that it audits allUsers correctly * update tests * fix line too long * address lint * fix spacing * Collapse apt layers in base dockerfile (#1883) * Restore VPC Support (#1874) * + Fixed silent gcloud errors that are generating errors in the logs + Removed service account name requirement from activate_service_account * Update gcloud.py * Update gcloud.py * add arguments for VPC support * Added VPC flags and debug lines * Added VPC support * Update gcp_installer.py * Update gcloud.py * Rename VPC network flags * Rename VPC variables in config.py * Rename VPC variables * Update VPC client schema variables * Rename VPC server variables * Rename VPC variable names * Rename VPC server schema variables * Rename VPC template variables * Rename VPC template variables * Update deploy-forseti-client.yaml.in * Update VPC variable names * Update forseti_client_installer.py * Rename VPC variables * Rename VPC variable names * Update config.py * Update gcloud.py * Update gcloud.py * Update config.py * Delete forseti-instance-server.py * Update bigquery_rules.yaml * Update group_rules.yaml * Update bucket_rules.yaml * Update iam_rules.yaml * Update log_sink_rules.yaml * Update iap_rules.yaml * Update iam_rules.yaml * Update firewall_rules.yaml * Update cloudsql_rules.yaml * Templatize server region and zone * Renamed check_network_host_project_id to set_network_host_projecT_id * Minor wording change * Revert template Reverting the template back to using refs.cloudsql.region. This is going to require a separate PR. * Update gcloud.py * Update gcloud.py * Update deploy-forseti-server.yaml.in * Update deploy-forseti-server.yaml.in * Update forseti_server_installer.py * Update docker_unittest_forseti.sh (#1886) * Fixing CrawlerTest to use Forseti Logging infrastructure (#1889) * Fixing CrawlerTest to use Forseti Logging infrastructure All logging should be done through the central Forseti logging infrastructure so that we can selectively control which logs go to the console or not in an effort to fix test log pollution. Help fix #1848 * [Issue #1848] Fixing logging to use Forsetting logging infrastructure. (#1890) * [Issue #1848] Fixing logging to use Forsetting logging infrastructure. * Remove cluster auth data, but keep keys (#1888) * update stacktrace in broad excepts issue#1797 (#1836) * update stacktrace in broad excepts issue#1797 * Adding more files to update stacktrace in broad excepts issue#1797 * update dao.py stacktrace in broad excepts issue#1797 * update dao.py logger stacktrace in broad excepts issue#1797 * update threadpool.py logger stacktrace in broad excepts issue#1797 * update threadpool & dao.py logger stacktrace in broad excepts issue#1797 * update and fix dao.py logger stacktrace in broad excepts issue#1797 * update and fix crawler.py logger stacktrace in broad excepts issue#1797 * update import order crawler.py logger stacktrace in broad excepts issue#1797 * fix import order crawler.py logger stacktrace in broad excepts issue#1797 * Added try catch before uploading files to gcs bucket. (#1895) * Added try catch before uploading files to gcs bucket. * Addressed PR comments * Add a flake8 test (pycodestyle) to check for pep8 related stye (#1896) * flake8 support and changes. * name change. * suggested change. * ignore *pb2 files. * stop flake8 and pylint fighting. * exclusion cleanup. * Updated logger to use exception() instead of error() when it's logging inside an except block. (#1897) * Updated LOGGER.error() to LOGGER.except() when it's logging inside an except block. * Fixed unit tests * pylint updates * Addressed PR comment * [Issue 1848] Mock out logger to fix pollution of test logs. (#1899) * [Issue 1848] Mock out logger to fix almost all remaining instances of pollution of test logs. Remaining issus involve server and will likely require some production refactoring. (#1903) * Fixes #1871, Update Enforcer to use the common gcp_api compute client. (#1904) * Update Enforcer to use the common gcp_api compute client. * Remove one off compute API implementation from Enforcer * Update the Compute Insert/Update/Delete firewall rules to take an optional retry argument * Update the common date_time library and remove the requirement for the google python datelib module. * Switch enforcement from running multiple simultaneous operations to running a single operation in blocking mode. This will reduce load on the back end and should improve reliability of operations. * Fix some merge conflicts * Test cleanup. * Fix retry test. * Add operation timeout test. * Inventory and model compute snapshots (#1893) * Implement compute#snapshot inventory * Add compute#snapshots to crawler * Create tests for compute#snapshot inventory * Implement importer for compute#snapshots for modelling * Update forseti-test.db * [Fixes #1859] Remove dependency on the _metadata server module from google.auth (#1860) * Remove dependency on the _metadata server module from google.auth * Fix getheader. * Fix dev installer (#1917) * Removed sample from actual rule names (#1916) * Templatize Forseti server region and Zone (#1887) * Templatized Forseti Server Region and Zone * Templatized Forseti Server Region and Zone * Removed rules directory from PR tracking * Update forseti-instance-server.py.schema * Readd rules now that branch is clean * Untrack rules directory * Readd rules * Revert rules files * [Issue 1848] Mock out server errors for invalid arguments to eliminate log pollution in tests. (#1919) * [Fixes #1865] Fix bigquery scanner to respect resources (#1884) * Pipe resources through scanner and resources and actually use them * Add test to ensure inapplicable resource rules are not matched * Fix lint * Fix lint #2... * Fix bigquery scanner hierarchy and resource struct * Create project resource from dao Resource * Document ValueError * Use type instead of type_name * Don't conver violations to set * rename resource to parent_project, fix full name in test, and rename gen to be clearer * Fix lint * Removed unused variable in required section when generating the deployment template. (#1924) * Updated the group scanner to avoid scanning members with no rule (#1905) * Updated group scanner logic to avoid scanning members with no rules and scanning the same member multiple times. * docstring updates * Updates * updates * Updated unit tests * updates * [Issue 1848] Fix for more log pollution of tests. (#1921) * Updated Installer with G Suite optional (#1934) * Removed sample from actual rule names * updated installer with G Suite optional * Updated Installer prompt that G Suite is optional (#1936) * Removed sample from actual rule names * Updated Installer code to not block if the field is empty * Updated Installer prompt that G Suite is Optional * Merge stable to dev (#1940) * Merge 2.0 release branch into stable branch. (#1732) * Updated hardcoded resource types in violation to use the resource types defined in the ResourceType class (#1665) * Updated hardcoded resource types in violation to use the resource type defined in the ResourceType class. * removed unused comments * Updated unit tests * Addressed PR comments * Updated unit test * Removed warning messages * updated installation instructions (#1689) * fix service account key scanner name in sample config (#1691) * update cloudsql naming (#1695) * Merge Dev into 2.0.0 release branch (#1727) * clean up the code to change inventory_index_id arg to type int64 everywhere except at the parser level (#1561) * fixes * Updated inventory_index_id in notifier * updates * updated docstring * updated unit tests * Updated notifier unit tests * pylint fixes * Changed notifier inventory_index_id default value to 0 * cli better grpc error handling (#1562) * Improved grpc error handling. * added unit test * Fix Missing Scopes in Local Environment (#1564) * Fix missing scope * fix test * CLI stacktrace clean up (#1584) * CLI stacktrace clean up * Addressed PR comment * pylint fixes * Updated firewall rule to not call the parse function when the time is empty (#1588) * Updated firewall rule to not call the parse function when the time is not passed in * Added comment * handle error from saving cscc (#1589) * Clarify required flags (#1592) * Clarify required flags * Clarify required flags * Clarify required flags * fix test * Update resource name to use hash of timestamp & org id (#1591) * Updated the resource identifier * Add identifier generation * pylint fixes * Update cloudsql instance after identifier is generated * Lower cased the identifier * Pass identifer to client if both client and server are installed at the same time * Addressed PR comments * Updated kubernetes violation to use full name instead of parent full name (#1594) * Added cluster name to kubernetes full name * Updated to use resource full name instead of parent full name * skip child resource when getting ancestors * Updated unit tests * Removed google.apputils usage in stubs (#1596) * Removed apputils usage * Added default args back for testing purpose * removed unused import * Updated lock file location (#1597) * updated lock file location * added sudo * Lock on README instead so we don't need to create any new file * added lock file * update_flag (#1602) * Disable scanners that have rules all commented out (#1603) * Update ke scanner rules (#1604) * Updated logs (#1606) * Added cloud function script to export inventory summary to bigquery (#1607) * Added cloud function script to export inventory summary to bigquery * Updates * Port Inventory email to 2.0 (#1598) * Port Inventory email to 2.0 * tweak * tweak * add timestamp * add timestamp * fix inventory summary tests * fix test * fix tests * update the config files * tweak * Remove default plain text output style (#1608) * Updated the inventory to output a viewmodel instead * Removed plain text output format option * updated show config method * Updated unit tests * Increased column size for inventory errors and message (#1615) * Increased column size for inventory errors * increased column size for message column in model * Updated output (#1714) * Fix scanner_iter() db session out of sync when used by multiple simultaneous queries. (#1718) * Fix db session out of sync * tweak * tweak * Remove additional db query which conflicts with existing db session (#1720) * Fix blacklist scanner to handle network intefaces that do not have external internet access. (#1721) * updates * updates * Fix blacklist scanner to handle network intefaces that do not have external internet access. (#1721) * updates * updates * Update build and code coverage status for 2.0 branches. (#1723) * Update travis status in README * tweak table * tweak * address comment * Handle the deprecated port field in backend service. (#1717) * Added error handling in scanners * Addressed PR comment * updates * updates * updates * updates * Addressed PR comments * Updates * updates * updates * updates * updates * updates * add blank line * Merge RC3 into Dev (#1726) * clean up the code to change inventory_index_id arg to type int64 everywhere except at the parser level (#1561) * fixes * Updated inventory_index_id in notifier * updates * updated docstring * updated unit tests * Updated notifier unit tests * pylint fixes * Changed notifier inventory_index_id default value to 0 * cli better grpc error handling (#1562) * Improved grpc error handling. * added unit test * Fix Missing Scopes in Local Environment (#1564) * Fix missing scope * fix test * CLI stacktrace clean up (#1584) * CLI stacktrace clean up * Addressed PR comment * pylint fixes * Updated firewall rule to not call the parse function when the time is empty (#1588) * Updated firewall rule to not call the parse function when the time is not passed in * Added comment * handle error from saving cscc (#1589) * Clarify required flags (#1592) * Clarify required flags * Clarify required flags * Clarify required flags * fix test * Update resource name to use hash of timestamp & org id (#1591) * Updated the resource identifier * Add identifier generation * pylint fixes * Update cloudsql instance after identifier is generated * Lower cased the identifier * Pass identifer to client if both client and server are installed at the same time * Addressed PR comments * Updated kubernetes violation to use full name instead of parent full name (#1594) * Added cluster name to kubernetes full name * Updated to use resource full name instead of parent full name * skip child resource when getting ancestors * Updated unit tests * Removed google.apputils usage in stubs (#1596) * Removed apputils usage * Added default args back for testing purpose * removed unused import * Updated lock file location (#1597) * updated lock file location * added sudo * Lock on README instead so we don't need to create any new file * added lock file * update_flag (#1602) * Disable scanners that have rules all commented out (#1603) * Update ke scanner rules (#1604) * Updated logs (#1606) * Added cloud function script to export inventory summary to bigquery (#1607) * Added cloud function script to export inventory summary to bigquery * Updates * Port Inventory email to 2.0 (#1598) * Port Inventory email to 2.0 * tweak * tweak * add timestamp * add timestamp * fix inventory summary tests * fix test * fix tests * update the config files * tweak * Remove default plain text output style (#1608) * Updated the inventory to output a viewmodel instead * Removed plain text output format option * updated show config method * Updated unit tests * Increased column size for inventory errors and message (#1615) * Increased column size for inventory errors * increased column size for message column in model * Fix scanner_iter() db session out of sync when used by multiple simultaneous queries. (#1718) * Fix db session out of sync * tweak * tweak * Remove additional db query which conflicts with existing db session (#1720) * Fix blacklist scanner to handle network intefaces that do not have external internet access. (#1721) * updates * updates * add blank line * Run cloudsqlproxy service as root (#1762) * Removed User= from cloudsqlproxy service * Addressed PR comment * Merge release-2.1 into stable (#1787) * Updated hardcoded resource types in violation to use the resource types defined in the ResourceType class (#1665) * Updated hardcoded resource types in violation to use the resource type defined in the ResourceType class. * removed unused comments * Updated unit tests * Addressed PR comments * Updated unit test * Removed warning messages * updated installation instructions (#1689) * fix service account key scanner name in sample config (#1691) * update cloudsql naming (#1695) * Updated output (#1714) * Fix blacklist scanner to handle network intefaces that do not have external internet access. (#1721) * updates * updates * Update build and code coverage status for 2.0 branches. (#1723) * Update travis status in README * tweak table * tweak * address comment * Handle the deprecated port field in backend service. (#1717) * Added error handling in scanners * Addressed PR comment * updates * updates * updates * updates * Addressed PR comments * Updates * updates * updates * updates * updates * updates * Merge RC3 into Dev (#1726) * clean up the code to change inventory_index_id arg to type int64 everywhere except at the parser level (#1561) * fixes * Updated inventory_index_id in notifier * updates * updated docstring * updated unit tests * Updated notifier unit tests * pylint fixes * Changed notifier inventory_index_id default value to 0 * cli better grpc error handling (#1562) * Improved grpc error handling. * added unit test * Fix Missing Scopes in Local Environment (#1564) * Fix missing scope * fix test * CLI stacktrace clean up (#1584) * CLI stacktrace clean up * Addressed PR comment * pylint fixes * Updated firewall rule to not call the parse function when the time is empty (#1588) * Updated firewall rule to not call the parse function when the time is not passed in * Added comment * handle error from saving cscc (#1589) * Clarify required flags (#1592) * Clarify required flags * Clarify required flags * Clarify required flags * fix test * Update resource name to use hash of timestamp & org id (#1591) * Updated the resource identifier * Add identifier generation * pylint fixes * Update cloudsql instance after identifier is generated * Lower cased the identifier * Pass identifer to client if both client and server are installed at the same time * Addressed PR comments * Updated kubernetes violation to use full name instead of parent full name (#1594) * Added cluster name to kubernetes full name * Updated to use resource full name instead of parent full name * skip child resource when getting ancestors * Updated unit tests * Removed google.apputils usage in stubs (#1596) * Removed apputils usage * Added default args back for testing purpose * removed unused import * Updated lock file location (#1597) * updated lock file location * added sudo * Lock on README instead so we don't need to create any new file * added lock file * update_flag (#1602) * Disable scanners that have rules all commented out (#1603) * Update ke scanner rules (#1604) * Updated logs (#1606) * Added cloud function script to export inventory summary to bigquery (#1607) * Added cloud function script to export inventory summary to bigquery * Updates * Port Inventory email to 2.0 (#1598) * Port Inventory email to 2.0 * tweak * tweak * add timestamp * add timestamp * fix inventory summary tests * fix test * fix tests * update the config files * tweak * Remove default plain text output style (#1608) * Updated the inventory to output a viewmodel instead * Removed plain text output format option * updated show config method * Updated unit tests * Increased column size for inventory errors and message (#1615) * Increased column size for inventory errors * increased column size for message column in model * Fix scanner_iter() db session out of sync when used by multiple simultaneous queries. (#1718) * Fix db session out of sync * tweak * tweak * Remove additional db query which conflicts with existing db session (#1720) * Fix blacklist scanner to handle network intefaces that do not have external internet access. (#1721) * updates * updates * add blank line * Added Service Account support for installing Forseti (#1706) * Added Support for running installer as Service Account * Added service account file credential path lookup * Fixed typo * Ignore domain check if using Service Account * Minor cleanup * Bug fix * Fix trailing whitespace * Fix linter errors * Fix linter errors * Fix lynter errors * Fix linter errors * Added --service-account-key-path support * Removed credentials lookup and added key_path support to activate_service_account * Fix key path * Fix key path * Fix linter errors * Fix linter errors * Update gcp_installer.py * Update config.py * Update gcloud.py * Update gcloud.py * Update gcloud.py * Update gcloud.py * Update gcp_installer.py * Update gcloud.py * Update gcloud.py * Update gcp_installer.py * Update gcp_installer.py * Update config.py * Update gcp_installer.py * Update forseti_installer.py * Update forseti_installer.py * Add Log Sinks (Exports) to the Forseti Inventory (#1681) * Add Stackdriver Logging GCP API for sinks * Add Stackdriver Logging methods to GCP ApiClient * Add logging sinks to the crawler * Add logging sinks to Modeller resources tables * Remove unnecessary split on name in FolderSinkIterator * Adding working changes for CSCC API integration (#1746) * Fix CSCC Notifier Test (#1750) Get Travis back to Green after merging the initial CSCC API PR * Fixing exemption typo issue#1643 (#1749) * Fixing exemption typo issue#1643 * Fixing exemption typo issue#1643 * Updated the dataset type_name to use dataset_policy/{dataset_id} format. (#1759) * Updated the dataset type_name to use dataset_policy/{dataset_id} format. * Removed extra post action call * Tidy-Up CSCC API (#1757) * Tidy up CSCC API * tweak * tweak * tweak * tweak * tweak * tweak * fix tests * add test * fix tests * add log message * tweak * fix test * Updated to not query again in the same session during yield_per() (#1763) * Updated to not query again in the same session during yield_per() * format updates * Added error checks * Get type_name from full_name * update iter_foo and fetch_foo issue#1702 (#1760) * Fix broken links on README.md (#1751) * Merge stable to dev (#1764) * Merge 2.0 release branch into stable branch. (#1732) * Updated hardcoded resource types in violation to use the resource types defined in the ResourceType class (#1665) * Updated hardcoded resource types in violation to use the resource type defined in the ResourceType class. * removed unused comments * Updated unit tests * Addressed PR comments * Updated unit test * Removed warning messages * updated installation instructions (#1689) * fix service account key scanner name in sample config (#1691) * update cloudsql naming (#1695) * Merge Dev into 2.0.0 release branch (#1727) * clean up the code to change inventory_index_id arg to type int64 everywhere except at the parser level (#1561) * fixes * Updated inventory_index_id in notifier * updates * updated docstring * updated unit tests * Updated notifier unit tests * pylint fixes * Changed notifier inventory_index_id default value to 0 * cli better grpc error handling (#1562) * Improved grpc error handling. * added unit test * Fix Missing Scopes in Local Environment (#1564) * Fix missing scope * fix test * CLI stacktrace clean up (#1584) * CLI stacktrace clean up * Addressed PR comment * pylint fixes * Updated firewall rule to not call the parse function when the time is empty (#1588) * Updated firewall rule to not call the parse function when the time is not passed in * Added comment * handle error from saving cscc (#1589) * Clarify required flags (#1592) * Clarify required flags * Clarify required flags * Clarify required flags * fix test * Update resource name to use hash of timestamp & org id (#1591) * Updated the resource identifier * Add identifier generation * pylint fixes * Update cloudsql instance after identifier is generated * Lower cased the identifier * Pass identifer to client if both client and server are installed at the same time * Addressed PR comments * Updated kubernetes violation to use full name instead of parent full name (#1594) * Added cluster name to kubernetes full name * Updated to use resource full name instead of parent full name * skip child resource when getting ancestors * Updated unit tests * Removed google.apputils usage in stubs (#1596) * Removed apputils usage * Added default args back for testing purpose * removed unused import * Updated lock file location (#1597) * updated lock file location * added sudo * Lock on README instead so we don't need to create any new file * added lock file * update_flag (#1602) * Disable scanners that have rules all commented out (#1603) * Update ke scanner rules (#1604) * Updated logs (#1606) * Added cloud function script to export inventory summary to bigquery (#1607) * Added cloud function script to export inventory summary to bigquery * Updates * Port Inventory email to 2.0 (#1598) * Port Inventory email to 2.0 * tweak * tweak * add timestamp * add timestamp * fix inventory summary tests * fix test * fix tests * update the config files * tweak * Remove default plain text output style (#1608) * Updated the inventory to output a viewmodel instead * Removed plain text output format option * updated show config method * Updated unit tests * Increased column size for inventory errors and message (#1615) * Increased column size for inventory errors * increased column size for message column in model * Updated output (#1714) * Fix scanner_iter() db session out of sync when used by multiple simultaneous queries. (#1718) * Fix db session out of sync * tweak * tweak * Remove additional db query which conflicts with existing db session (#1720) * Fix blacklist scanner to handle network intefaces that do not have external internet access. (#1721) * updates * updates * Fix blacklist scanner to handle network intefaces that do not have external internet access. (#1721) * updates * updates * Update build and code coverage status for 2.0 branches. (#1723) * Update travis status in README * tweak table * tweak * address comment * Handle the deprecated port field in backend service. (#1717) * Added error handling in scanners * Addressed PR comment * updates * updates * updates * updates * Addressed PR comments * Updates * updates * updates * updates * updates * updates * add blank line * Merge RC3 into Dev (#1726) * clean up the code to change inventory_index_id arg to type int64 everywhere except at the parser level (#1561) * fixes * Updated inventory_index_id in notifier * updates * updated docstring * updated unit tests * Updated notifier unit tests * pylint fixes * Changed notifier inventory_index_id default value to 0 * cli better grpc error handling (#1562) * Improved grpc error handling. * added unit test * Fix Missing Scopes in Local Environment (#1564) * Fix missing scope * fix test * CLI stacktrace clean up (#1584) * CLI stacktrace clean up * Addressed PR comment * pylint fixes * Updated firewall rule to not call the parse function when the time is empty (#1588) * Updated firewall rule to not call the parse function when the time is not passed in * Added comment * handle error from saving cscc (#1589) * Clarify required flags (#1592) * Clarify required flags * Clarify required flags * Clarify required flags * fix test * Update resource name to use hash of timestamp & org id (#1591) * Updated the resource identifier * Add identifier generation * pylint fixes * Update cloudsql instance after identifier is generated * Lower cased the identifier * Pass identifer to client if both client and server are installed at the same time * Addressed PR comments * Updated kubernetes violation to use full name instead of parent full name (#1594) * Added cluster name to kubernetes full name * Updated to use resource full name instead of parent full name * skip child resource when getting ancestors * Updated unit tests * Removed google.apputils usage in stubs (#1596) * Removed apputils usage * Added default args back for testing purpose * removed unused import * Updated lock file location (#1597) * updated lock file location * added sudo * Lock on README instead so we don't need to create any new file * added lock file * update_flag (#1602) * Disable scanners that have rules all commented out (#1603) * Update ke scanner rules (#1604) * Updated logs (#1606) * Added cloud function script to export inventory summary to bigquery (#1607) * Added cloud function script to export inventory summary to bigquery * Updates * Port Inventory email to 2.0 (#1598) * Port Inventory email to 2.0 * tweak * tweak * add timestamp * add timestamp * fix inventory summary tests * fix test * fix tests * update the config files * tweak * Remove default plain text output style (#1608) * Updated the inventory to output a viewmodel instead * Removed plain text output format option * updated show config method * Updated unit tests * Increased column size for inventory errors and message (#1615) * Increased column size for inventory errors * increased column size for message column in model * Fix scanner_iter() db session out of sync when used by multiple simultaneous queries. (#1718) * Fix db session out of sync * tweak * tweak * Remove additional db query which conflicts with existing db session (#1720) * Fix blacklist scanner to handle network intefaces that do not have external internet access. (#1721) * updates * updates * add blank line * Run cloudsqlproxy service as root (#1762) * Removed User= from cloudsqlproxy service * Addressed PR comment * Add Compute Engine disks to crawler and data model. (#1766) * Add Compute Engine disks to crawler and data model. * Updated broken urls (#1773) * Alpha sort the scanner maps issue#1654 (#1777) * Merge stable to dev (#1780) * Merge 2.0 release branch into stable branch. (#1732) * Updated hardcoded resource types in violation to use the resource types defined in the ResourceType class (#1665) * Updated hardcoded resource types in violation to use the resource type defined in the ResourceType class. * removed unused comments * Updated unit tests * Addressed PR comments * Updated unit test * Removed warning messages * updated installation instructions (#1689) * fix service account key scanner name in sample config (#1691) * update cloudsql naming (#1695) * Merge Dev into 2.0.0 release branch (#1727) * clean up the code to change inventory_index_id arg to type int64 everywhere except at the parser level (#1561) * fixes * Updated inventory_index_id in notifier * updates * updated docstring * updated unit tests * Updated notifier unit tests * pylint fixes * Changed notifier inventory_index_id default value to 0 * cli better grpc error handling (#1562) * Improved grpc error handling. * added unit test * Fix Missing Scopes in Local Environment (#1564) * Fix missing scope * fix test * CLI stacktrace clean up (#1584) * CLI stacktrace clean up * Addressed PR comment * pylint fixes * Updated firewall rule to not call the parse function when the time is empty (#1588) * Updated firewall rule to not call the parse function when the time is not passed in * Added comment * handle error from saving cscc (#1589) * Clarify required flags (#1592) * Clarify required flags * Clarify required flags * Clarify required flags * fix test * Update resource name to use hash of timestamp & org id (#1591) * Updated the resource identifier * Add identifier generation * pylint fixes * Update cloudsql instance after identifier is generated * Lower cased the identifier * Pass identifer to client if both client and server are installed at the same time * Addressed PR comments * Updated kubernetes violation to use full name instead of parent full name (#1594) * Added cluster name to kubernetes full name * Updated to use resource full name instead of parent full name * skip child resource when getting ancestors * Updated unit tests * Removed google.apputils usage in stubs (#1596) * Removed apputils usage * Added default args back for testing purpose * removed unused import * Updated lock file location (#1597) * updated lock file location * added sudo * Lock on README instead so we don't need to create any new file * added lock file * update_flag (#1602) * Disable scanners that have rules all commented out (#1603) * Update ke scanner rules (#1604) * Updated logs (#1606) * Added cloud function script to export inventory summary to bigquery (#1607) * Added cloud function script to export inventory summary to bigquery * Updates * Port Inventory email to 2.0 (#1598) * Port Inventory email to 2.0 * tweak * tweak * add timestamp * add timestamp * fix inventory summary tests * fix test * fix tests * update the config files * tweak * Remove default plain text output style (#1608) * Updated the inventory to output a viewmodel instead * Removed plain text output format option * updated show config method * Updated unit tests * Increased column size for inventory errors and message (#1615) * Increased column size for inventory errors * increased column size for message column in model * Updated output (#1714) * Fix scanner_iter() db session out of sync when used by multiple simultaneous queries. (#1718) * Fix db session out of sync * tweak * tweak * Remove additional db query which conflicts with existing db session (#1720) * Fix blacklist scanner to handle network intefaces that do not have external internet access. (#1721) * updates * updates * Fix blacklist scanner to handle network intefaces that do not have external internet access. (#1721) * updates * updates * Update build and code coverage status for 2.0 branches. (#1723) * Update travis status in README * tweak table * tweak * address comment * Handle the deprecated port field in backend service. (#1717) * Added error handling in scanners * Addressed PR comment * updates * updates * updates * updates * Addressed PR comments * Updates * updates * updates * updates * updates * updates * add blank line * Merge RC3 into Dev (#1726) * clean up the code to change inventory_index_id arg to type int64 everywhere except at the parser level (#1561) * fixes * Updated inventory_index_id in notifier * updates * updated docstring * updated unit tests * Updated notifier unit tests * pylint fixes * Changed notifier inventory_index_id default value to 0 * cli better grpc error handling (#1562) * Improved grpc error handling. * added unit test * Fix Missing Scopes in Local Environment (#1564) * Fix missing scope * fix test * CLI stacktrace clean up (#1584) * CLI stacktrace clean up * Addressed PR comment * pylint fixes * Updated firewall rule to not call the parse function when the time is empty (#1588) * Updated firewall rule to not call the parse function when the time is not passed in * Added comment * handle error from saving cscc (#1589) * Clarify required flags (#1592) * Clarify required flags * Clarify required flags * Clarify required flags * fix test * Update resource name to use hash of timestamp & org id (#1591) * Updated the resource identifier * Add identifier generation * pylint fixes * Update cloudsql instance after identifier is generated * Lower cased the identifier * Pass identifer to client if both client and server are installed at the same time * Addressed PR comments * Updated kubernetes violation to use full name instead of parent full name (#1594) * Added cluster name to kubernetes full name * Updated to use resource full name instead of parent full name * skip child resource when getting ancestors * Updated unit tests * Removed google.apputils usage in stubs (#1596) * Removed apputils usage * Added default args back for testing purpose * removed unused import * Updated lock file location (#1597) * updated lock file location * added sudo * Lock on README instead so we don't need to create any new file * added lock file * update_flag (#1602) * Disable scanners that have rules all commented out (#1603) * Update ke scanner rules (#1604) * Updated logs (#1606) * Added cloud function script to export inventory summary to bigquery (#1607) * Added cloud function script to export inventory summary to bigquery * Updates * Port Inventory email to 2.0 (#1598) * Port Inventory email to 2.0 * tweak * tweak * add timestamp * add timestamp * fix inventory summary tests * fix test * fix tests * update the config files * tweak * Remove default plain text output style (#1608) * Updated the inventory to output a viewmodel instead * Removed plain text output format option * updated show config method * Updated unit tests * Increased column size for inventory errors and message (#1615) * Increased column size for inventory errors * increased column size for message column in model * Fix scanner_iter() db session out of sync when used by multiple simultaneous queries. (#1718) * Fix db session out of sync * tweak * tweak * Remove additional db query which conflicts with existing db session (#1720) * Fix blacklist scanner to handle network intefaces that do not have external internet access. (#1721) * updates * updates * add blank line * Run cloudsqlproxy service as root (#1762) * Removed User= from cloudsqlproxy service * Addressed PR comment * Updated variable in the configs yaml file (#1781) * Updated variable in the configs yaml file * Updated org id in sample file * Updates * Updated version to 2.0.1 * Use absolute path to detech discovery_documents folder * Increased column size for data model * Added init file for discovery documents * Pre release patches- Increased column size, updated discovery_documents path and incremented release version (#1786) * Updated version to 2.0.1 * Use absolute path to detech discovery_documents folder * Increased column size for data model * Added init file for discovery documents * Updated version to 2.1 * Updated readme.md (#1794) * Shorten the violation id hash. (#1803) (#1809) * shorten the hash * shorten the hash when we send to CSCC, so that we can preserve the original full hash for other clients that rely on it * fix test * Restart the server at the beginning of the cronjob, temp fix to #1832 (#1892) * Restart the server before running the start of the cronjob, temp fix to issue #1832 * pin ruamel.yaml to 0.15.37 to avoid regression * Increment version to 2.2.0 * Added space * Improve CSCC usability (#1907) * Improve CSCC usability * tweak * tweak * tweak * tweak * tweak * tweak * add test * add test * tweak * tweak …

* Merge 2.0 release branch into stable branch. (#1732) * Updated hardcoded resource types in violation to use the resource types defined in the ResourceType class (#1665) * Updated hardcoded resource types in violation to use the resource type defined in the ResourceType class. * removed unused comments * Updated unit tests * Addressed PR comments * Updated unit test * Removed warning messages * updated installation instructions (#1689) * fix service account key scanner name in sample config (#1691) * update cloudsql naming (#1695) * Merge Dev into 2.0.0 release branch (#1727) * clean up the code to change inventory_index_id arg to type int64 everywhere except at the parser level (#1561) * fixes * Updated inventory_index_id in notifier * updates * updated docstring * updated unit tests * Updated notifier unit tests * pylint fixes * Changed notifier inventory_index_id default value to 0 * cli better grpc error handling (#1562) * Improved grpc error handling. * added unit test * Fix Missing Scopes in Local Environment (#1564) * Fix missing scope * fix test * CLI stacktrace clean up (#1584) * CLI stacktrace clean up * Addressed PR comment * pylint fixes * Updated firewall rule to not call the parse function when the time is empty (#1588) * Updated firewall rule to not call the parse function when the time is not passed in * Added comment * handle error from saving cscc (#1589) * Clarify required flags (#1592) * Clarify required flags * Clarify required flags * Clarify required flags * fix test * Update resource name to use hash of timestamp & org id (#1591) * Updated the resource identifier * Add identifier generation * pylint fixes * Update cloudsql instance after identifier is generated * Lower cased the identifier * Pass identifer to client if both client and server are installed at the same time * Addressed PR comments * Updated kubernetes violation to use full name instead of parent full name (#1594) * Added cluster name to kubernetes full name * Updated to use resource full name instead of parent full name * skip child resource when getting ancestors * Updated unit tests * Removed google.apputils usage in stubs (#1596) * Removed apputils usage * Added default args back for testing purpose * removed unused import * Updated lock file location (#1597) * updated lock file location * added sudo * Lock on README instead so we don't need to create any new file * added lock file * update_flag (#1602) * Disable scanners that have rules all commented out (#1603) * Update ke scanner rules (#1604) * Updated logs (#1606) * Added cloud function script to export inventory summary to bigquery (#1607) * Added cloud function script to export inventory summary to bigquery * Updates * Port Inventory email to 2.0 (#1598) * Port Inventory email to 2.0 * tweak * tweak * add timestamp * add timestamp * fix inventory summary tests * fix test * fix tests * update the config files * tweak * Remove default plain text output style (#1608) * Updated the inventory to output a viewmodel instead * Removed plain text output format option * updated show config method * Updated unit tests * Increased column size for inventory errors and message (#1615) * Increased column size for inventory errors * increased column size for message column in model * Updated output (#1714) * Fix scanner_iter() db session out of sync when used by multiple simultaneous queries. (#1718) * Fix db session out of sync * tweak * tweak * Remove additional db query which conflicts with existing db session (#1720) * Fix blacklist scanner to handle network intefaces that do not have external internet access. (#1721) * updates * updates * Fix blacklist scanner to handle network intefaces that do not have external internet access. (#1721) * updates * updates * Update build and code coverage status for 2.0 branches. (#1723) * Update travis status in README * tweak table * tweak * address comment * Handle the deprecated port field in backend service. (#1717) * Added error handling in scanners * Addressed PR comment * updates * updates * updates * updates * Addressed PR comments * Updates * updates * updates * updates * updates * updates * add blank line * Merge RC3 into Dev (#1726) * clean up the code to change inventory_index_id arg to type int64 everywhere except at the parser level (#1561) * fixes * Updated inventory_index_id in notifier * updates * updated docstring * updated unit tests * Updated notifier unit tests * pylint fixes * Changed notifier inventory_index_id default value to 0 * cli better grpc error handling (#1562) * Improved grpc error handling. * added unit test * Fix Missing Scopes in Local Environment (#1564) * Fix missing scope * fix test * CLI stacktrace clean up (#1584) * CLI stacktrace clean up * Addressed PR comment * pylint fixes * Updated firewall rule to not call the parse function when the time is empty (#1588) * Updated firewall rule to not call the parse function when the time is not passed in * Added comment * handle error from saving cscc (#1589) * Clarify required flags (#1592) * Clarify required flags * Clarify required flags * Clarify required flags * fix test * Update resource name to use hash of timestamp & org id (#1591) * Updated the resource identifier * Add identifier generation * pylint fixes * Update cloudsql instance after identifier is generated * Lower cased the identifier * Pass identifer to client if both client and server are installed at the same time * Addressed PR comments * Updated kubernetes violation to use full name instead of parent full name (#1594) * Added cluster name to kubernetes full name * Updated to use resource full name instead of parent full name * skip child resource when getting ancestors * Updated unit tests * Removed google.apputils usage in stubs (#1596) * Removed apputils usage * Added default args back for testing purpose * removed unused import * Updated lock file location (#1597) * updated lock file location * added sudo * Lock on README instead so we don't need to create any new file * added lock file * update_flag (#1602) * Disable scanners that have rules all commented out (#1603) * Update ke scanner rules (#1604) * Updated logs (#1606) * Added cloud function script to export inventory summary to bigquery (#1607) * Added cloud function script to export inventory summary to bigquery * Updates * Port Inventory email to 2.0 (#1598) * Port Inventory email to 2.0 * tweak * tweak * add timestamp * add timestamp * fix inventory summary tests * fix test * fix tests * update the config files * tweak * Remove default plain text output style (#1608) * Updated the inventory to output a viewmodel instead * Removed plain text output format option * updated show config method * Updated unit tests * Increased column size for inventory errors and message (#1615) * Increased column size for inventory errors * increased column size for message column in model * Fix scanner_iter() db session out of sync when used by multiple simultaneous queries. (#1718) * Fix db session out of sync * tweak * tweak * Remove additional db query which conflicts with existing db session (#1720) * Fix blacklist scanner to handle network intefaces that do not have external internet access. (#1721) * updates * updates * add blank line * Run cloudsqlproxy service as root (#1762) * Removed User= from cloudsqlproxy service * Addressed PR comment * Merge release-2.1 into stable (#1787) * Updated hardcoded resource types in violation to use the resource types defined in the ResourceType class (#1665) * Updated hardcoded resource types in violation to use the resource type defined in the ResourceType class. * removed unused comments * Updated unit tests * Addressed PR comments * Updated unit test * Removed warning messages * updated installation instructions (#1689) * fix service account key scanner name in sample config (#1691) * update cloudsql naming (#1695) * Updated output (#1714) * Fix blacklist scanner to handle network intefaces that do not have external internet access. (#1721) * updates * updates * Update build and code coverage status for 2.0 branches. (#1723) * Update travis status in README * tweak table * tweak * address comment * Handle the deprecated port field in backend service. (#1717) * Added error handling in scanners * Addressed PR comment * updates * updates * updates * updates * Addressed PR comments * Updates * updates * updates * updates * updates * updates * Merge RC3 into Dev (#1726) * clean up the code to change inventory_index_id arg to type int64 everywhere except at the parser level (#1561) * fixes * Updated inventory_index_id in notifier * updates * updated docstring * updated unit tests * Updated notifier unit tests * pylint fixes * Changed notifier inventory_index_id default value to 0 * cli better grpc error handling (#1562) * Improved grpc error handling. * added unit test * Fix Missing Scopes in Local Environment (#1564) * Fix missing scope * fix test * CLI stacktrace clean up (#1584) * CLI stacktrace clean up * Addressed PR comment * pylint fixes * Updated firewall rule to not call the parse function when the time is empty (#1588) * Updated firewall rule to not call the parse function when the time is not passed in * Added comment * handle error from saving cscc (#1589) * Clarify required flags (#1592) * Clarify required flags * Clarify required flags * Clarify required flags * fix test * Update resource name to use hash of timestamp & org id (#1591) * Updated the resource identifier * Add identifier generation * pylint fixes * Update cloudsql instance after identifier is generated * Lower cased the identifier * Pass identifer to client if both client and server are installed at the same time * Addressed PR comments * Updated kubernetes violation to use full name instead of parent full name (#1594) * Added cluster name to kubernetes full name * Updated to use resource full name instead of parent full name * skip child resource when getting ancestors * Updated unit tests * Removed google.apputils usage in stubs (#1596) * Removed apputils usage * Added default args back for testing purpose * removed unused import * Updated lock file location (#1597) * updated lock file location * added sudo * Lock on README instead so we don't need to create any new file * added lock file * update_flag (#1602) * Disable scanners that have rules all commented out (#1603) * Update ke scanner rules (#1604) * Updated logs (#1606) * Added cloud function script to export inventory summary to bigquery (#1607) * Added cloud function script to export inventory summary to bigquery * Updates * Port Inventory email to 2.0 (#1598) * Port Inventory email to 2.0 * tweak * tweak * add timestamp * add timestamp * fix inventory summary tests * fix test * fix tests * update the config files * tweak * Remove default plain text output style (#1608) * Updated the inventory to output a viewmodel instead * Removed plain text output format option * updated show config method * Updated unit tests * Increased column size for inventory errors and message (#1615) * Increased column size for inventory errors * increased column size for message column in model * Fix scanner_iter() db session out of sync when used by multiple simultaneous queries. (#1718) * Fix db session out of sync * tweak * tweak * Remove additional db query which conflicts with existing db session (#1720) * Fix blacklist scanner to handle network intefaces that do not have external internet access. (#1721) * updates * updates * add blank line * Added Service Account support for installing Forseti (#1706) * Added Support for running installer as Service Account * Added service account file credential path lookup * Fixed typo * Ignore domain check if using Service Account * Minor cleanup * Bug fix * Fix trailing whitespace * Fix linter errors * Fix linter errors * Fix lynter errors * Fix linter errors * Added --service-account-key-path support * Removed credentials lookup and added key_path support to activate_service_account * Fix key path * Fix key path * Fix linter errors * Fix linter errors * Update gcp_installer.py * Update config.py * Update gcloud.py * Update gcloud.py * Update gcloud.py * Update gcloud.py * Update gcp_installer.py * Update gcloud.py * Update gcloud.py * Update gcp_installer.py * Update gcp_installer.py * Update config.py * Update gcp_installer.py * Update forseti_installer.py * Update forseti_installer.py * Add Log Sinks (Exports) to the Forseti Inventory (#1681) * Add Stackdriver Logging GCP API for sinks * Add Stackdriver Logging methods to GCP ApiClient * Add logging sinks to the crawler * Add logging sinks to Modeller resources tables * Remove unnecessary split on name in FolderSinkIterator * Adding working changes for CSCC API integration (#1746) * Fix CSCC Notifier Test (#1750) Get Travis back to Green after merging the initial CSCC API PR * Fixing exemption typo issue#1643 (#1749) * Fixing exemption typo issue#1643 * Fixing exemption typo issue#1643 * Updated the dataset type_name to use dataset_policy/{dataset_id} format. (#1759) * Updated the dataset type_name to use dataset_policy/{dataset_id} format. * Removed extra post action call * Tidy-Up CSCC API (#1757) * Tidy up CSCC API * tweak * tweak * tweak * tweak * tweak * tweak * fix tests * add test * fix tests * add log message * tweak * fix test * Updated to not query again in the same session during yield_per() (#1763) * Updated to not query again in the same session during yield_per() * format updates * Added error checks * Get type_name from full_name * update iter_foo and fetch_foo issue#1702 (#1760) * Fix broken links on README.md (#1751) * Merge stable to dev (#1764) * Merge 2.0 release branch into stable branch. (#1732) * Updated hardcoded resource types in violation to use the resource types defined in the ResourceType class (#1665) * Updated hardcoded resource types in violation to use the resource type defined in the ResourceType class. * removed unused comments * Updated unit tests * Addressed PR comments * Updated unit test * Removed warning messages * updated installation instructions (#1689) * fix service account key scanner name in sample config (#1691) * update cloudsql naming (#1695) * Merge Dev into 2.0.0 release branch (#1727) * clean up the code to change inventory_index_id arg to type int64 everywhere except at the parser level (#1561) * fixes * Updated inventory_index_id in notifier * updates * updated docstring * updated unit tests * Updated notifier unit tests * pylint fixes * Changed notifier inventory_index_id default value to 0 * cli better grpc error handling (#1562) * Improved grpc error handling. * added unit test * Fix Missing Scopes in Local Environment (#1564) * Fix missing scope * fix test * CLI stacktrace clean up (#1584) * CLI stacktrace clean up * Addressed PR comment * pylint fixes * Updated firewall rule to not call the parse function when the time is empty (#1588) * Updated firewall rule to not call the parse function when the time is not passed in * Added comment * handle error from saving cscc (#1589) * Clarify required flags (#1592) * Clarify required flags * Clarify required flags * Clarify required flags * fix test * Update resource name to use hash of timestamp & org id (#1591) * Updated the resource identifier * Add identifier generation * pylint fixes * Update cloudsql instance after identifier is generated * Lower cased the identifier * Pass identifer to client if both client and server are installed at the same time * Addressed PR comments * Updated kubernetes violation to use full name instead of parent full name (#1594) * Added cluster name to kubernetes full name * Updated to use resource full name instead of parent full name * skip child resource when getting ancestors * Updated unit tests * Removed google.apputils usage in stubs (#1596) * Removed apputils usage * Added default args back for testing purpose * removed unused import * Updated lock file location (#1597) * updated lock file location * added sudo * Lock on README instead so we don't need to create any new file * added lock file * update_flag (#1602) * Disable scanners that have rules all commented out (#1603) * Update ke scanner rules (#1604) * Updated logs (#1606) * Added cloud function script to export inventory summary to bigquery (#1607) * Added cloud function script to export inventory summary to bigquery * Updates * Port Inventory email to 2.0 (#1598) * Port Inventory email to 2.0 * tweak * tweak * add timestamp * add timestamp * fix inventory summary tests * fix test * fix tests * update the config files * tweak * Remove default plain text output style (#1608) * Updated the inventory to output a viewmodel instead * Removed plain text output format option * updated show config method * Updated unit tests * Increased column size for inventory errors and message (#1615) * Increased column size for inventory errors * increased column size for message column in model * Updated output (#1714) * Fix scanner_iter() db session out of sync when used by multiple simultaneous queries. (#1718) * Fix db session out of sync * tweak * tweak * Remove additional db query which conflicts with existing db session (#1720) * Fix blacklist scanner to handle network intefaces that do not have external internet access. (#1721) * updates * updates * Fix blacklist scanner to handle network intefaces that do not have external internet access. (#1721) * updates * updates * Update build and code coverage status for 2.0 branches. (#1723) * Update travis status in README * tweak table * tweak * address comment * Handle the deprecated port field in backend service. (#1717) * Added error handling in scanners * Addressed PR comment * updates * updates * updates * updates * Addressed PR comments * Updates * updates * updates * updates * updates * updates * add blank line * Merge RC3 into Dev (#1726) * clean up the code to change inventory_index_id arg to type int64 everywhere except at the parser level (#1561) * fixes * Updated inventory_index_id in notifier * updates * updated docstring * updated unit tests * Updated notifier unit tests * pylint fixes * Changed notifier inventory_index_id default value to 0 * cli better grpc error handling (#1562) * Improved grpc error handling. * added unit test * Fix Missing Scopes in Local Environment (#1564) * Fix missing scope * fix test * CLI stacktrace clean up (#1584) * CLI stacktrace clean up * Addressed PR comment * pylint fixes * Updated firewall rule to not call the parse function when the time is empty (#1588) * Updated firewall rule to not call the parse function when the time is not passed in * Added comment * handle error from saving cscc (#1589) * Clarify required flags (#1592) * Clarify required flags * Clarify required flags * Clarify required flags * fix test * Update resource name to use hash of timestamp & org id (#1591) * Updated the resource identifier * Add identifier generation * pylint fixes * Update cloudsql instance after identifier is generated * Lower cased the identifier * Pass identifer to client if both client and server are installed at the same time * Addressed PR comments * Updated kubernetes violation to use full name instead of parent full name (#1594) * Added cluster name to kubernetes full name * Updated to use resource full name instead of parent full name * skip child resource when getting ancestors * Updated unit tests * Removed google.apputils usage in stubs (#1596) * Removed apputils usage * Added default args back for testing purpose * removed unused import * Updated lock file location (#1597) * updated lock file location * added sudo * Lock on README instead so we don't need to create any new file * added lock file * update_flag (#1602) * Disable scanners that have rules all commented out (#1603) * Update ke scanner rules (#1604) * Updated logs (#1606) * Added cloud function script to export inventory summary to bigquery (#1607) * Added cloud function script to export inventory summary to bigquery * Updates * Port Inventory email to 2.0 (#1598) * Port Inventory email to 2.0 * tweak * tweak * add timestamp * add timestamp * fix inventory summary tests * fix test * fix tests * update the config files * tweak * Remove default plain text output style (#1608) * Updated the inventory to output a viewmodel instead * Removed plain text output format option * updated show config method * Updated unit tests * Increased column size for inventory errors and message (#1615) * Increased column size for inventory errors * increased column size for message column in model * Fix scanner_iter() db session out of sync when used by multiple simultaneous queries. (#1718) * Fix db session out of sync * tweak * tweak * Remove additional db query which conflicts with existing db session (#1720) * Fix blacklist scanner to handle network intefaces that do not have external internet access. (#1721) * updates * updates * add blank line * Run cloudsqlproxy service as root (#1762) * Removed User= from cloudsqlproxy service * Addressed PR comment * Add Compute Engine disks to crawler and data model. (#1766) * Add Compute Engine disks to crawler and data model. * Updated broken urls (#1773) * Alpha sort the scanner maps issue#1654 (#1777) * Merge stable to dev (#1780) * Merge 2.0 release branch into stable branch. (#1732) * Updated hardcoded resource types in violation to use the resource types defined in the ResourceType class (#1665) * Updated hardcoded resource types in violation to use the resource type defined in the ResourceType class. * removed unused comments * Updated unit tests * Addressed PR comments * Updated unit test * Removed warning messages * updated installation instructions (#1689) * fix service account key scanner name in sample config (#1691) * update cloudsql naming (#1695) * Merge Dev into 2.0.0 release branch (#1727) * clean up the code to change inventory_index_id arg to type int64 everywhere except at the parser level (#1561) * fixes * Updated inventory_index_id in notifier * updates * updated docstring * updated unit tests * Updated notifier unit tests * pylint fixes * Changed notifier inventory_index_id default value to 0 * cli better grpc error handling (#1562) * Improved grpc error handling. * added unit test * Fix Missing Scopes in Local Environment (#1564) * Fix missing scope * fix test * CLI stacktrace clean up (#1584) * CLI stacktrace clean up * Addressed PR comment * pylint fixes * Updated firewall rule to not call the parse function when the time is empty (#1588) * Updated firewall rule to not call the parse function when the time is not passed in * Added comment * handle error from saving cscc (#1589) * Clarify required flags (#1592) * Clarify required flags * Clarify required flags * Clarify required flags * fix test * Update resource name to use hash of timestamp & org id (#1591) * Updated the resource identifier * Add identifier generation * pylint fixes * Update cloudsql instance after identifier is generated * Lower cased the identifier * Pass identifer to client if both client and server are installed at the same time * Addressed PR comments * Updated kubernetes violation to use full name instead of parent full name (#1594) * Added cluster name to kubernetes full name * Updated to use resource full name instead of parent full name * skip child resource when getting ancestors * Updated unit tests * Removed google.apputils usage in stubs (#1596) * Removed apputils usage * Added default args back for testing purpose * removed unused import * Updated lock file location (#1597) * updated lock file location * added sudo * Lock on README instead so we don't need to create any new file * added lock file * update_flag (#1602) * Disable scanners that have rules all commented out (#1603) * Update ke scanner rules (#1604) * Updated logs (#1606) * Added cloud function script to export inventory summary to bigquery (#1607) * Added cloud function script to export inventory summary to bigquery * Updates * Port Inventory email to 2.0 (#1598) * Port Inventory email to 2.0 * tweak * tweak * add timestamp * add timestamp * fix inventory summary tests * fix test * fix tests * update the config files * tweak * Remove default plain text output style (#1608) * Updated the inventory to output a viewmodel instead * Removed plain text output format option * updated show config method * Updated unit tests * Increased column size for inventory errors and message (#1615) * Increased column size for inventory errors * increased column size for message column in model * Updated output (#1714) * Fix scanner_iter() db session out of sync when used by multiple simultaneous queries. (#1718) * Fix db session out of sync * tweak * tweak * Remove additional db query which conflicts with existing db session (#1720) * Fix blacklist scanner to handle network intefaces that do not have external internet access. (#1721) * updates * updates * Fix blacklist scanner to handle network intefaces that do not have external internet access. (#1721) * updates * updates * Update build and code coverage status for 2.0 branches. (#1723) * Update travis status in README * tweak table * tweak * address comment * Handle the deprecated port field in backend service. (#1717) * Added error handling in scanners * Addressed PR comment * updates * updates * updates * updates * Addressed PR comments * Updates * updates * updates * updates * updates * updates * add blank line * Merge RC3 into Dev (#1726) * clean up the code to change inventory_index_id arg to type int64 everywhere except at the parser level (#1561) * fixes * Updated inventory_index_id in notifier * updates * updated docstring * updated unit tests * Updated notifier unit tests * pylint fixes * Changed notifier inventory_index_id default value to 0 * cli better grpc error handling (#1562) * Improved grpc error handling. * added unit test * Fix Missing Scopes in Local Environment (#1564) * Fix missing scope * fix test * CLI stacktrace clean up (#1584) * CLI stacktrace clean up * Addressed PR comment * pylint fixes * Updated firewall rule to not call the parse function when the time is empty (#1588) * Updated firewall rule to not call the parse function when the time is not passed in * Added comment * handle error from saving cscc (#1589) * Clarify required flags (#1592) * Clarify required flags * Clarify required flags * Clarify required flags * fix test * Update resource name to use hash of timestamp & org id (#1591) * Updated the resource identifier * Add identifier generation * pylint fixes * Update cloudsql instance after identifier is generated * Lower cased the identifier * Pass identifer to client if both client and server are installed at the same time * Addressed PR comments * Updated kubernetes violation to use full name instead of parent full name (#1594) * Added cluster name to kubernetes full name * Updated to use resource full name instead of parent full name * skip child resource when getting ancestors * Updated unit tests * Removed google.apputils usage in stubs (#1596) * Removed apputils usage * Added default args back for testing purpose * removed unused import * Updated lock file location (#1597) * updated lock file location * added sudo * Lock on README instead so we don't need to create any new file * added lock file * update_flag (#1602) * Disable scanners that have rules all commented out (#1603) * Update ke scanner rules (#1604) * Updated logs (#1606) * Added cloud function script to export inventory summary to bigquery (#1607) * Added cloud function script to export inventory summary to bigquery * Updates * Port Inventory email to 2.0 (#1598) * Port Inventory email to 2.0 * tweak * tweak * add timestamp * add timestamp * fix inventory summary tests * fix test * fix tests * update the config files * tweak * Remove default plain text output style (#1608) * Updated the inventory to output a viewmodel instead * Removed plain text output format option * updated show config method * Updated unit tests * Increased column size for inventory errors and message (#1615) * Increased column size for inventory errors * increased column size for message column in model * Fix scanner_iter() db session out of sync when used by multiple simultaneous queries. (#1718) * Fix db session out of sync * tweak * tweak * Remove additional db query which conflicts with existing db session (#1720) * Fix blacklist scanner to handle network intefaces that do not have external internet access. (#1721) * updates * updates * add blank line * Run cloudsqlproxy service as root (#1762) * Removed User= from cloudsqlproxy service * Addressed PR comment * Updated variable in the configs yaml file (#1781) * Updated variable in the configs yaml file * Updated org id in sample file * Updates * Updated version to 2.0.1 * Use absolute path to detech discovery_documents folder * Increased column size for data model * Added init file for discovery documents * Pre release patches- Increased column size, updated discovery_documents path and incremented release version (#1786) * Updated version to 2.0.1 * Use absolute path to detech discovery_documents folder * Increased column size for data model * Added init file for discovery documents * Updated version to 2.1 * Updated readme.md (#1794) * Shorten the violation id hash. (#1803) (#1809) * shorten the hash * shorten the hash when we send to CSCC, so that we can preserve the original full hash for other clients that rely on it * fix test * Restart the server at the beginning of the cronjob, temp fix to #1832 (#1892) * Restart the server before running the start of the cronjob, temp fix to issue #1832 * pin ruamel.yaml to 0.15.37 to avoid regression * Merge release-2.2.0 to stable (#1951) * Updated hardcoded resource types in violation to use the resource types defined in the ResourceType class (#1665) * Updated hardcoded resource types in violation to use the resource type defined in the ResourceType class. * removed unused comments * Updated unit tests * Addressed PR comments * Updated unit test * Removed warning messages * updated installation instructions (#1689) * fix service account key scanner name in sample config (#1691) * update cloudsql naming (#1695) * Updated output (#1714) * Fix blacklist scanner to handle network intefaces that do not have external internet access. (#1721) * updates * updates * Update build and code coverage status for 2.0 branches. (#1723) * Update travis status in README * tweak table * tweak * address comment * Handle the deprecated port field in backend service. (#1717) * Added error handling in scanners * Addressed PR comment * updates * updates * updates * updates * Addressed PR comments * Updates * updates * updates * updates * updates * updates * Merge RC3 into Dev (#1726) * clean up the code to change inventory_index_id arg to type int64 everywhere except at the parser level (#1561) * fixes * Updated inventory_index_id in notifier * updates * updated docstring * updated unit tests * Updated notifier unit tests * pylint fixes * Changed notifier inventory_index_id default value to 0 * cli better grpc error handling (#1562) * Improved grpc error handling. * added unit test * Fix Missing Scopes in Local Environment (#1564) * Fix missing scope * fix test * CLI stacktrace clean up (#1584) * CLI stacktrace clean up * Addressed PR comment * pylint fixes * Updated firewall rule to not call the parse function when the time is empty (#1588) * Updated firewall rule to not call the parse function when the time is not passed in * Added comment * handle error from saving cscc (#1589) * Clarify required flags (#1592) * Clarify required flags * Clarify required flags * Clarify required flags * fix test * Update resource name to use hash of timestamp & org id (#1591) * Updated the resource identifier * Add identifier generation * pylint fixes * Update cloudsql instance after identifier is generated * Lower cased the identifier * Pass identifer to client if both client and server are installed at the same time * Addressed PR comments * Updated kubernetes violation to use full name instead of parent full name (#1594) * Added cluster name to kubernetes full name * Updated to use resource full name instead of parent full name * skip child resource when getting ancestors * Updated unit tests * Removed google.apputils usage in stubs (#1596) * Removed apputils usage * Added default args back for testing purpose * removed unused import * Updated lock file location (#1597) * updated lock file location * added sudo * Lock on README instead so we don't need to create any new file * added lock file * update_flag (#1602) * Disable scanners that have rules all commented out (#1603) * Update ke scanner rules (#1604) * Updated logs (#1606) * Added cloud function script to export inventory summary to bigquery (#1607) * Added cloud function script to export inventory summary to bigquery * Updates * Port Inventory email to 2.0 (#1598) * Port Inventory email to 2.0 * tweak * tweak * add timestamp * add timestamp * fix inventory summary tests * fix test * fix tests * update the config files * tweak * Remove default plain text output style (#1608) * Updated the inventory to output a viewmodel instead * Removed plain text output format option * updated show config method * Updated unit tests * Increased column size for inventory errors and message (#1615) * Increased column size for inventory errors * increased column size for message column in model * Fix scanner_iter() db session out of sync when used by multiple simultaneous queries. (#1718) * Fix db session out of sync * tweak * tweak * Remove additional db query which conflicts with existing db session (#1720) * Fix blacklist scanner to handle network intefaces that do not have external internet access. (#1721) * updates * updates * add blank line * Added Service Account support for installing Forseti (#1706) * Added Support for running installer as Service Account * Added service account file credential path lookup * Fixed typo * Ignore domain check if using Service Account * Minor cleanup * Bug fix * Fix trailing whitespace * Fix linter errors * Fix linter errors * Fix lynter errors * Fix linter errors * Added --service-account-key-path support * Removed credentials lookup and added key_path support to activate_service_account * Fix key path * Fix key path * Fix linter errors * Fix linter errors * Update gcp_installer.py * Update config.py * Update gcloud.py * Update gcloud.py * Update gcloud.py * Update gcloud.py * Update gcp_installer.py * Update gcloud.py * Update gcloud.py * Update gcp_installer.py * Update gcp_installer.py * Update config.py * Update gcp_installer.py * Update forseti_installer.py * Update forseti_installer.py * Add Log Sinks (Exports) to the Forseti Inventory (#1681) * Add Stackdriver Logging GCP API for sinks * Add Stackdriver Logging methods to GCP ApiClient * Add logging sinks to the crawler * Add logging sinks to Modeller resources tables * Remove unnecessary split on name in FolderSinkIterator * Adding working changes for CSCC API integration (#1746) * Fix CSCC Notifier Test (#1750) Get Travis back to Green after merging the initial CSCC API PR * Fixing exemption typo issue#1643 (#1749) * Fixing exemption typo issue#1643 * Fixing exemption typo issue#1643 * Updated the dataset type_name to use dataset_policy/{dataset_id} format. (#1759) * Updated the dataset type_name to use dataset_policy/{dataset_id} format. * Removed extra post action call * Tidy-Up CSCC API (#1757) * Tidy up CSCC API * tweak * tweak * tweak * tweak * tweak * tweak * fix tests * add test * fix tests * add log message * tweak * fix test * Updated to not query again in the same session during yield_per() (#1763) * Updated to not query again in the same session during yield_per() * format updates * Added error checks * Get type_name from full_name * update iter_foo and fetch_foo issue#1702 (#1760) * Fix broken links on README.md (#1751) * Merge stable to dev (#1764) * Merge 2.0 release branch into stable branch. (#1732) * Updated hardcoded resource types in violation to use the resource types defined in the ResourceType class (#1665) * Updated hardcoded resource types in violation to use the resource type defined in the ResourceType class. * removed unused comments * Updated unit tests * Addressed PR comments * Updated unit test * Removed warning messages * updated installation instructions (#1689) * fix service account key scanner name in sample config (#1691) * update cloudsql naming (#1695) * Merge Dev into 2.0.0 release branch (#1727) * clean up the code to change inventory_index_id arg to type int64 everywhere except at the parser level (#1561) * fixes * Updated inventory_index_id in notifier * updates * updated docstring * updated unit tests * Updated notifier unit tests * pylint fixes * Changed notifier inventory_index_id default value to 0 * cli better grpc error handling (#1562) * Improved grpc error handling. * added unit test * Fix Missing Scopes in Local Environment (#1564) * Fix missing scope * fix test * CLI stacktrace clean up (#1584) * CLI stacktrace clean up * Addressed PR comment * pylint fixes * Updated firewall rule to not call the parse function when the time is empty (#1588) * Updated firewall rule to not call the parse function when the time is not passed in * Added comment * handle error from saving cscc (#1589) * Clarify required flags (#1592) * Clarify required flags * Clarify required flags * Clarify required flags * fix test * Update resource name to use hash of timestamp & org id (#1591) * Updated the resource identifier * Add identifier generation * pylint fixes * Update cloudsql instance after identifier is generated * Lower cased the identifier * Pass identifer to client if both client and server are installed at the same time * Addressed PR comments * Updated kubernetes violation to use full name instead of parent full name (#1594) * Added cluster name to kubernetes full name * Updated to use resource full name instead of parent full name * skip child resource when getting ancestors * Updated unit tests * Removed google.apputils usage in stubs (#1596) * Removed apputils usage * Added default args back for testing purpose * removed unused import * Updated lock file location (#1597) * updated lock file location * added sudo * Lock on README instead so we don't need to create any new file * added lock file * update_flag (#1602) * Disable scanners that have rules all commented out (#1603) * Update ke scanner rules (#1604) * Updated logs (#1606) * Added cloud function script to export inventory summary to bigquery (#1607) * Added cloud function script to export inventory summary to bigquery * Updates * Port Inventory email to 2.0 (#1598) * Port Inventory email to 2.0 * tweak * tweak * add timestamp * add timestamp * fix inventory summary tests * fix test * fix tests * update the config files * tweak * Remove default plain text output style (#1608) * Updated the inventory to output a viewmodel instead * Removed plain text output format option * updated show config method * Updated unit tests * Increased column size for inventory errors and message (#1615) * Increased column size for inventory errors * increased column size for message column in model * Updated output (#1714) * Fix scanner_iter() db session out of sync when used by multiple simultaneous queries. (#1718) * Fix db session out of sync * tweak * tweak * Remove additional db query which conflicts with existing db session (#1720) * Fix blacklist scanner to handle network intefaces that do not have external internet access. (#1721) * updates * updates * Fix blacklist scanner to handle network intefaces that do not have external internet access. (#1721) * updates * updates * Update build and code coverage status for 2.0 branches. (#1723) * Update travis status in README * tweak table * tweak * address comment * Handle the deprecated port field in backend service. (#1717) * Added error handling in scanners * Addressed PR comment * updates * updates * updates * updates * Addressed PR comments * Updates * updates * updates * updates * updates * updates * add blank line * Merge RC3 into Dev (#1726) * clean up the code to change inventory_index_id arg to type int64 everywhere except at the parser level (#1561) * fixes * Updated inventory_index_id in notifier * updates * updated docstring * updated unit tests * Updated notifier unit tests * pylint fixes * Changed notifier inventory_index_id default value to 0 * cli better grpc error handling (#1562) * Improved grpc error handling. * added unit test * Fix Missing Scopes in Local Environment (#1564) * Fix missing scope * fix test * CLI stacktrace clean up (#1584) * CLI stacktrace clean up * Addressed PR comment * pylint fixes * Updated firewall rule to not call the parse function when the time is empty (#1588) * Updated firewall rule to not call the parse function when the time is not passed in * Added comment * handle error from saving cscc (#1589) * Clarify required flags (#1592) * Clarify required flags * Clarify required flags * Clarify required flags * fix test * Update resource name to use hash of timestamp & org id (#1591) * Updated the resource identifier * Add identifier generation * pylint fixes * Update cloudsql instance after identifier is generated * Lower cased the identifier * Pass identifer to client if both client and server are installed at the same time * Addressed PR comments * Updated kubernetes violation to use full name instead of parent full name (#1594) * Added cluster name to kubernetes full name * Updated to use resource full name instead of parent full name * skip child resource when getting ancestors * Updated unit tests * Removed google.apputils usage in stubs (#1596) * Removed apputils usage * Added default args back for testing purpose * removed unused import * Updated lock file location (#1597) * updated lock file location * added sudo * Lock on README instead so we don't need to create any new file * added lock file * update_flag (#1602) * Disable scanners that have rules all commented out (#1603) * Update ke scanner rules (#1604) * Updated logs (#1606) * Added cloud function script to export inventory summary to bigquery (#1607) * Added cloud function script to export inventory summary to bigquery * Updates * Port Inventory email to 2.0 (#1598) * Port Inventory email to 2.0 * tweak * tweak * add timestamp * add timestamp * fix inventory summary tests * fix test * fix tests * update the config files * tweak * Remove default plain text output style (#1608) * Updated the inventory to output a viewmodel instead * Removed plain text output format option * updated show config method * Updated unit tests * Increased column size for inventory errors and message (#1615) * Increased column size for inventory errors * increased column size for message column in model * Fix scanner_iter() db session out of sync when used by multiple simultaneous queries. (#1718) * Fix db session out of sync * tweak * tweak * Remove additional db query which conflicts with existing db session (#1720) * Fix blacklist scanner to handle network intefaces that do not have external internet access. (#1721) * updates * updates * add blank line * Run cloudsqlproxy service as root (#1762) * Removed User= from cloudsqlproxy service * Addressed PR comment * Add Compute Engine disks to crawler and data model. (#1766) * Add Compute Engine disks to crawler and data model. * Updated broken urls (#1773) * Alpha sort the scanner maps issue#1654 (#1777) * Merge stable to dev (#1780) * Merge 2.0 release branch into stable branch. (#1732) * Updated hardcoded resource types in violation to use the resource types defined in the ResourceType class (#1665) * Updated hardcoded resource types in violation to use the resource type defined in the ResourceType class. * removed unused comments * Updated unit tests * Addressed PR comments * Updated unit test * Removed warning messages * updated installation instructions (#1689) * fix service account key scanner name in sample config (#1691) * update cloudsql naming (#1695) * Merge Dev into 2.0.0 release branch (#1727) * clean up the code to change inventory_index_id arg to type int64 everywhere except at the parser level (#1561) * fixes * Updated inventory_index_id in notifier * updates * updated docstring * updated unit tests * Updated notifier unit tests * pylint fixes * Changed notifier inventory_index_id default value to 0 * cli better grpc error handling (#1562) * Improved grpc error handling. * added unit test * Fix Missing Scopes in Local Environment (#1564) * Fix missing scope * fix test * CLI stacktrace clean up (#1584) * CLI stacktrace clean up * Addressed PR comment * pylint fixes * Updated firewall rule to not call the parse function when the time is empty (#1588) * Updated firewall rule to not call the parse function when the time is not passed in * Added comment * handle error from saving cscc (#1589) * Clarify required flags (#1592) * Clarify required flags * Clarify required flags * Clarify required flags * fix test * Update resource name to use hash of timestamp & org id (#1591) * Updated the resource identifier * Add identifier generation * pylint fixes * Update cloudsql instance after identifier is generated * Lower cased the identifier * Pass identifer to client if both client and server are installed at the same time * Addressed PR comments * Updated kubernetes violation to use full name instead of parent full name (#1594) * Added cluster name to kubernetes full name * Updated to use resource full name instead of parent full name * skip child resource when getting ancestors * Updated unit tests * Removed google.apputils usage in stubs (#1596) * Removed apputils usage * Added default args back for testing purpose * removed unused import * Updated lock file location (#1597) * updated lock file location * added sudo * Lock on README instead so we don't need to create any new file * added lock file * update_flag (#1602) * Disable scanners that have rules all commented out (#1603) * Update ke scanner rules (#1604) * Updated logs (#1606) * Added cloud function script to export inventory summary to bigquery (#1607) * Added cloud function script to export inventory summary to bigquery * Updates * Port Inventory email to 2.0 (#1598) * Port Inventory email to 2.0 * tweak * tweak * add timestamp * add timestamp * fix inventory summary tests * fix test * fix tests * update the config files * tweak * Remove default plain text output style (#1608) * Updated the inventory to output a viewmodel instead * Removed plain text output format option * updated show config method * Updated unit tests * Increased column size for inventory errors and message (#1615) * Increased column size for inventory errors * increased column size for message column in model * Updated output (#1714) * Fix scanner_iter() db session out of sync when used by multiple simultaneous queries. (#1718) * Fix db session out of sync * tweak * tweak * Remove additional db query which conflicts with existing db session (#1720) * Fix blacklist scanner to handle network intefaces that do not have external internet access. (#1721) * updates * updates * Fix blacklist scanner to handle network intefaces that do not have external internet access. (#1721) * updates * updates * Update build and code coverage status for 2.0 branches. (#1723) * Update travis status in README * tweak table * tweak * address comment * Handle the deprecated port field in backend service. (#1717) * Added error handling in scanners * Addressed PR comment * updates * updates * updates * updates * Addressed PR comments * Updates * updates * updates * updates * updates * updates * add blank line * Merge RC3 into Dev (#1726) * clean up the code to change inventory_index_id arg to type int64 everywhere except at the parser level (#1561) * fixes * Updated inventory_index_id in notifier * updates * updated docstring * updated unit tests * Updated notifier unit tests * pylint fixes * Changed notifier inventory_index_id default value to 0 * cli better grpc error handling (#1562) * Improved grpc error handling. * added unit test * Fix Missing Scopes in Local Environment (#1564) * Fix missing scope * fix test * CLI stacktrace clean up (#1584) * CLI stacktrace clean up * Addressed PR comment * pylint fixes * Updated firewall rule to not call the parse function when the time is empty (#1588) * Updated firewall rule to not call the parse function when the time is not passed in * Added comment * handle error from saving cscc (#1589) * Clarify required flags (#1592) * Clarify required flags * Clarify required flags * Clarify required flags * fix test * Update resource name to use hash of timestamp & org id (#1591) * Updated the resource identifier * Add identifier generation * pylint fixes * Update cloudsql instance after identifier is generated * Lower cased the identifier * Pass identifer to client if both client and server are installed at the same time * Addressed PR comments * Updated kubernetes violation to use full name instead of parent full name (#1594) * Added cluster name to kubernetes full name * Updated to use resource full name instead of parent full name * skip child resource when getting ancestors * Updated unit tests * Removed google.apputils usage in stubs (#1596) * Removed apputils usage * Added default args back for testing purpose * removed unused import * Updated lock file location (#1597) * updated lock file location * added sudo * Lock on README instead so we don't need to create any new file * added lock file * update_flag (#1602) * Disable scanners that have rules all commented out (#1603) * Update ke scanner rules (#1604) * Updated logs (#1606) * Added cloud function script to export inventory summary to bigquery (#1607) * Added cloud function script to export inventory summary to bigquery * Updates * Port Inventory email to 2.0 (#1598) * Port Inventory email to 2.0 * tweak * tweak * add timestamp * add timestamp * fix inventory summary tests * fix test * fix tests * update the config files * tweak * Remove default plain text output style (#1608) * Updated the inventory to output a viewmodel instead * Removed plain text output format option * updated show config method * Updated unit tests * Increased column size for inventory errors and message (#1615) * Increased column size for inventory errors * increased column size for message column in model * Fix scanner_iter() db session out of sync when used by multiple simultaneous queries. (#1718) * Fix db session out of sync * tweak * tweak * Remove additional db query which conflicts with existing db session (#1720) * Fix blacklist scanner to handle network intefaces that do not have external internet access. (#1721) * updates * updates * add blank line * Run cloudsqlproxy service as root (#1762) * Removed User= from cloudsqlproxy service * Addressed PR comment * Updated variable in the configs yaml file (#1781) * Updated variable in the configs yaml file * Updated org id in sample file * Updates * Pre release patches- Increased column size, updated discovery_documents path and incremented release version (#1786) * Updated version to 2.0.1 * Use absolute path to detech discovery_documents folder * Increased column size for data model * Added init file for discovery documents * Fix full_res_name value when the root_resource is a project or a folder (#1788) * Improved the keys of Memory.mem to avoid key conflicts. Some GCP resources share the same key (an Instance Group and its Instance Group Manager, for example). (#1796) * Add a scanner for Log Sinks (#1747) * Add Log Sink Rules Engine. * Improve test coverage, fix bugs * Add log sink engine * Add Log Sink scanner to scanne map and configs. * Use empty string instead of none for sink rule violation * Update examples in log_sink_rules.yaml * Shorten the violation id hash. (#1803) * shorten the hash * shorten the hash when we send to CSCC, so that we can preserve the original full hash for other clients that rely on it * fix test * Short the violation hash id to 32 characters for upload to CSCC (#1841) * increment version to 2.1.0 (#1842) * Fixes to gcloud.py and Service Account Support (#1815) * + Fixed silent gcloud errors that are generating errors in the logs + Removed service account name requirement from activate_service_account * Update gcloud.py * Update gcloud.py * Refactor server.py to move config classes into base/config.py. (#1854) * Clean up test dependencies (#1858) * [Issue 1848] Fix a test that is emitting errors and polluting the logs. (#1857) * Fixing copy and paste error in test description (#1867) * Added requriemodel decorator to the scanner run method, pin the version of rumael.yaml library (#1870) * Added requriemodel decorator to scanner run method * pin rumael.yaml version * Add Billing Account log sinks to Inventory (#1839) * Add list and get_iam_policy to Cloud Billing client * Fix lint errors * Add billing account methods to GCP ApiClient * Add billing accounts to crawler * Add billing account sinks to the crawler and model. * Code tidyup. * Add Compute client methods to insert, update and delete firewall rules. (#1872) * Add mixins for Insert, Update, Delete actions * Add support for Dry Run / read only mode to client * Add support to make async calls blocking until call completes * Switch API version for compute.firewalls to v1 instead of beta * Re-enable groups scanner test (#1873) * Re-enable groups scanner test * remove the direct member count * Fix iam scanner so that it audits allUsers correctly (#1878) * Fix iam scanner so that it audits allUsers correctly * update tests * fix line too long * address lint * fix spacing * Collapse apt layers in base dockerfile (#1883) * Restore VPC Support (#1874) * + Fixed silent gcloud errors that are generating errors in the logs + Removed service account name requirement from activate_service_account * Update gcloud.py * Update gcloud.py * add arguments for VPC support * Added VPC flags and debug lines * Added VPC support * Update gcp_installer.py * Update gcloud.py * Rename VPC network flags * Rename VPC variables in config.py * Rename VPC variables * Update VPC client schema variables * Rename VPC server variables * Rename VPC variable names * Rename VPC server schema variables * Rename VPC template variables * Rename VPC template variables * Update deploy-forseti-client.yaml.in * Update VPC variable names * Update forseti_client_installer.py * Rename VPC variables * Rename VPC variable names * Update config.py * Update gcloud.py * Update gcloud.py * Update config.py * Delete forseti-instance-server.py * Update bigquery_rules.yaml * Update group_rules.yaml * Update bucket_rules.yaml * Update iam_rules.yaml * Update log_sink_rules.yaml * Update iap_rules.yaml * Update iam_rules.yaml * Update firewall_rules.yaml * Update cloudsql_rules.yaml * Templatize server region and zone * Renamed check_network_host_project_id to set_network_host_projecT_id * Minor wording change * Revert template Reverting the template back to using refs.cloudsql.region. This is going to require a separate PR. * Update gcloud.py * Update gcloud.py * Update deploy-forseti-server.yaml.in * Update deploy-forseti-server.yaml.in * Update forseti_server_installer.py * Update docker_unittest_forseti.sh (#1886) * Fixing CrawlerTest to use Forseti Logging infrastructure (#1889) * Fixing CrawlerTest to use Forseti Logging infrastructure All logging should be done through the central Forseti logging infrastructure so that we can selectively control which logs go to the console or not in an effort to fix test log pollution. Help fix #1848 * [Issue #1848] Fixing logging to use Forsetting logging infrastructure. (#1890) * [Issue #1848] Fixing logging to use Forsetting logging infrastructure. * Remove cluster auth data, but keep keys (#1888) * update stacktrace in broad excepts issue#1797 (#1836) * update stacktrace in broad excepts issue#1797 * Adding more files to update stacktrace in broad excepts issue#1797 * update dao.py stacktrace in broad excepts issue#1797 * update dao.py logger stacktrace in broad excepts issue#1797 * update threadpool.py logger stacktrace in broad excepts issue#1797 * update threadpool & dao.py logger stacktrace in broad excepts issue#1797 * update and fix dao.py logger stacktrace in broad excepts issue#1797 * update and fix crawler.py logger stacktrace in broad excepts issue#1797 * update import order crawler.py logger stacktrace in broad excepts issue#1797 * fix import order crawler.py logger stacktrace in broad excepts issue#1797 * Added try catch before uploading files to gcs bucket. (#1895) * Added try catch before uploading files to gcs bucket. * Addressed PR comments * Add a flake8 test (pycodestyle) to check for pep8 related stye (#1896) * flake8 support and changes. * name change. * suggested change. * ignore *pb2 files. * stop flake8 and pylint fighting. * exclusion cleanup. * Updated logger to use exception() instead of error() when it's logging inside an except block. (#1897) * Updated LOGGER.error() to LOGGER.except() when it's logging inside an except block. * Fixed unit tests * pylint updates * Addressed PR comment * [Issue 1848] Mock out logger to fix pollution of test logs. (#1899) * [Issue 1848] Mock out logger to fix almost all remaining instances of pollution of test logs. Remaining issus involve server and will likely require some production refactoring. (#1903) * Fixes #1871, Update Enforcer to use the common gcp_api compute client. (#1904) * Update Enforcer to use the common gcp_api compute client. * Remove one off compute API implementation from Enforcer * Update the Compute Insert/Update/Delete firewall rules to take an optional retry argument * Update the common date_time library and remove the requirement for the google python datelib module. * Switch enforcement from running multiple simultaneous operations to running a single operation in blocking mode. This will reduce load on the back end and should improve reliability of operations. * Fix some merge conflicts * Test cleanup. * Fix retry test. * Add operation timeout test. * Inventory and model compute snapshots (#1893) * Implement compute#snapshot inventory * Add compute#snapshots to crawler * Create tests for compute#snapshot inventory * Implement importer for compute#snapshots for modelling * Update forseti-test.db * [Fixes #1859] Remove dependency on the _metadata server module from google.auth (#1860) * Remove dependency on the _metadata server module from google.auth * Fix getheader. * Fix dev installer (#1917) * Removed sample from actual rule names (#1916) * Templatize Forseti server region and Zone (#1887) * Templatized Forseti Server Region and Zone * Templatized Forseti Server Region and Zone * Removed rules directory from PR tracking * Update forseti-instance-server.py.schema * Readd rules now that branch is clean * Untrack rules directory * Readd rules * Revert rules files * [Issue 1848] Mock out server errors for invalid arguments to eliminate log pollution in tests. (#1919) * [Fixes #1865] Fix bigquery scanner to respect resources (#1884) * Pipe resources through scanner and resources and actually use them * Add test to ensure inapplicable resource rules are not matched * Fix lint * Fix lint #2... * Fix bigquery scanner hierarchy and resource struct * Create project resource from dao Resource * Document ValueError * Use type instead of type_name * Don't conver violations to set * rename resource to parent_project, fix full name in test, and rename gen to be clearer * Fix lint * Removed unused variable in required section when generating the deployment template. (#1924) * Updated the group scanner to avoid scanning members with no rule (#1905) * Updated group scanner logic to avoid scanning members with no rules and scanning the same member multiple times. * docstring updates * Updates * updates * Updated unit tests * updates * [Issue 1848] Fix for more log pollution of tests. (#1921) * Updated Installer with G Suite optional (#1934) * Removed sample from actual rule names * updated installer with G Suite optional * Updated Installer prompt that G Suite is optional (#1936) * Removed sample from actual rule names * Updated Installer code to not block if the field is empty * Updated Installer prompt that G Suite is Optional * Merge stable to dev (#1940) * Merge 2.0 release branch into stable branch. (#1732) * Updated hardcoded resource types in violation to use the resource types defined in the ResourceType class (#1665) * Updated hardco…

John Evans added 2 commits August 14, 2018 21:00

Templatized Forseti Server Region and Zone

76de619

Templatized Forseti Server Region and Zone

70f0e15

googlebot added the cla: yes label Aug 15, 2018

Merge branch 'dev' into johnrevans6-dev-templatize

89f5dbe

blueandgold reviewed Aug 17, 2018

View reviewed changes

forseti-security-waffle-bot added the status: in-progress label Aug 20, 2018

John Evans and others added 7 commits August 20, 2018 14:34

Removed rules directory from PR tracking

1e023a8

Update forseti-instance-server.py.schema

228a3c2

Readd rules now that branch is clean

76aadce

Untrack rules directory

ab93a18

Readd rules

218894a

Revert rules files

cee35aa

Merge branch 'dev' into johnrevans6-dev-templatize

aa5ff8d

blueandgold approved these changes Aug 20, 2018

View reviewed changes

blueandgold merged commit 088f46d into forseti-security:dev Aug 20, 2018

forseti-security-waffle-bot removed the status: in-progress label Aug 20, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Templatize Forseti server region and Zone #1887

Templatize Forseti server region and Zone #1887

johnrevans6 commented Aug 15, 2018

blueandgold left a comment

blueandgold Aug 17, 2018

johnrevans6 Aug 20, 2018

blueandgold Aug 20, 2018

blueandgold Aug 17, 2018

johnrevans6 Aug 20, 2018

blueandgold Aug 20, 2018

blueandgold Aug 17, 2018

johnrevans6 Aug 20, 2018

blueandgold Aug 20, 2018

blueandgold left a comment

blueandgold Aug 20, 2018

blueandgold Aug 20, 2018

blueandgold Aug 20, 2018

Templatize Forseti server region and Zone #1887

Templatize Forseti server region and Zone #1887

Conversation

johnrevans6 commented Aug 15, 2018

blueandgold left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

blueandgold left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment