diff --git a/.ruby-version b/.ruby-version new file mode 100644 index 0000000..aedc15b --- /dev/null +++ b/.ruby-version @@ -0,0 +1 @@ +2.5.3 diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..700f43f --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,17 @@ +# Change Log + +All notable changes to this project will be documented in this file. + +The format is based on [Keep a Changelog](http://keepachangelog.com/) and this +project adheres to [Semantic Versioning](http://semver.org/). + +## [Unreleased] + +## [0.1.0] - 2019-02-15 + +### Added + +- Initial release + +[Unreleased]: https://github.com/terraform-google-modules/terraform-google-event-function/compare/v0.1.0...HEAD +[0.1.0] \ No newline at end of file diff --git a/CODEOWNERS b/CODEOWNERS new file mode 100644 index 0000000..ded1cc7 --- /dev/null +++ b/CODEOWNERS @@ -0,0 +1 @@ +* @morgante @aaron-lane @adrienthebo diff --git a/Gemfile b/Gemfile new file mode 100644 index 0000000..cf3f389 --- /dev/null +++ b/Gemfile @@ -0,0 +1,19 @@ +# Copyright 2019 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +ruby '2.5.3' + +source 'https://rubygems.org/' do + gem 'kitchen-terraform', '~> 4.2' +end diff --git a/README.md b/README.md index 3ce2c00..2ace5da 100644 --- a/README.md +++ b/README.md @@ -1 +1,194 @@ -# terraform-google-event-function \ No newline at end of file +# terraform-google-event-function + +This module configures a system which responds to filtered Stackdriver +Logging events by invoking a Cloud Functions function. + +A project-level Stackdriver Logging export uses a provided filter to +identify events of interest and publish them to a dedicated Pub/Sub +topic. A Cloud Functions function subscribes to the topic and uses +provided source code to process each event. The source code is +retrieved from an archive which is created locally and stored in a +Storage bucket. + +## Usage + +The [examples directory](examples) contains tested references of how to +use this module. + +[^]: (autogen_docs_start) + +## Inputs + +| Name | Description | Type | Default | Required | +|------|-------------|:----:|:-----:|:-----:| +| function\_available\_memory\_mb | The amount of memory in megabytes allotted for the function to use. | string | `"256"` | no | +| function\_description | The description of the function. | string | `"Processes log export events provided through a Pub/Sub topic subscription."` | no | +| function\_entry\_point | The name of a method in the function source which will be invoked when the function is executed. | string | n/a | yes | +| function\_environment\_variables | A set of key/value environment variable pairs to assign to the function. | map | `` | no | +| function\_event\_trigger\_failure\_policy\_retry | A toggle to determine if the function should be retried on failure. | string | `"false"` | no | +| function\_labels | A set of key/value label pairs to assign to the function. | map | `` | no | +| function\_runtime | The runtime in which the function will be executed. | string | `"nodejs6"` | no | +| function\_source\_archive\_bucket\_labels | A set of key/value label pairs to assign to the function source archive bucket. | map | `` | no | +| function\_source\_archive\_bucket\_location | The Google Cloud Storage location in which to create the function source archive bucket. | string | `"US"` | no | +| function\_source\_directory | The contents of this directory will be archived and used as the function source. | string | n/a | yes | +| function\_timeout\_s | The amount of time in seconds allotted for the execution of the function. | string | `"60"` | no | +| log\_export\_filter | The filter to apply when exporting logs to the Pub/Sub topic. | string | n/a | yes | +| name | The name to apply to any nameable resources. | string | `"event-function"` | no | +| project\_id | The ID of the project to which resources will be applied. | string | n/a | yes | +| region | The region in which resources will be applied. | string | n/a | yes | + +[^]: (autogen_docs_end) + +## Requirements + +The following requirements must be met in order to invoke this module: + +1. [Software dependencies](#software-dependencies). +2. [IAM roles](#iam-roles). +3. [APIs](#apis). + +### Software Dependencies + +The following software dependencies must be installed on the system +from which this module will be invoked: + +- [Terraform][terraform-site] v0.11.x +- [Google Terraform provider][terraform-provider-google-site] v1.20.0 + +### IAM Roles + +The Service Account which will be used to invoke this module must have +the following IAM roles: + +- Cloud Functions Developer +- Compute Viewer +- Logs Configuration Writer +- Pub/Sub Admin +- Service Account User +- Storage Admin + +### APIs + +The project against which this module will be invoked must have the +following APIs enabled: + +- Cloud Functions API +- Cloud Pub/Sub API +- Google Cloud Storage + +The [Project Factory module][project-factory-module-site] can be used to +provision projects with specific APIs activated. + +## Testing + +The [fixtures directory](test/fixtures) and +[integration directory](test/integration) comprise Terraform +modules and InSpec tests used to verify the behaviour of this module. + +### Testing Software Dependencies + +The following software dependencies must be installed on the system +from which the tests will be invoked: + +- [Ruby][ruby-site] v2.5 +- [Bundler][bundler-site] v1.17 + +### Integration Tests + +Integration tests are invoked using [Kitchen][kitchen-site], +[Kitchen-Terraform][kitchen-terraform-site], and [InSpec][inspec-site]. + +Kitchen instances are configured in the +[Kitchen configuration file](kitchen.yml). The instances use the modules +in [fixtures directory](test/fixtures) to invoke identically named +modules in the [examples directory](examples) and test this module. + +#### Integration Tests Configuration + +Each Kitchen instance requires a variable file named `terraform.tfvars` +to be created and populated in the associated test fixture. For +convenience, a [sample variable file][sameple-variable-file] is +available. + +A key file for a Service Account with the required +[IAM roles](#iam-roles) must be downloaded from the GCP console and +placed in the root directory of this repository. The key file must be +renamed to `credentials.json`. + +#### Integration Tests Execution + +Run `make test_integration_docker` to execute all of the Kitchen +instances in a non-interactive manner within a Docker container. + +Alternatively, the Kitchen instances can be invoked interactively: + +1. Run `make docker_run` to start the Docker container. The root + directory of this repository will be mounted in the Docker container + at `/cft/workdir/`. +1. Run `kitchen create` to initialize all Kitchen instances, or run + `kitchen create ` to initialize a specific Kitchen + instance. +1. Run `kitchen converge` to apply all Kitchen instances, or run + `kitchen converge ` to apply a specific Kitchen + instance. +1. Run `kitchen verify` to test all Kitchen instances, or run + `kitchen verify ` to test a specific Kitchen instance. +1. Run `kitchen destroy` to destroy all Kitchen instances, or run + `kitchen destroy ` to destroy a specific Kitchen + instance. + +## Linting + +Linters are available for most of the filetypes in this repository. + +### Linting Software Dependencies + +The following software dependencies must be installed on the system +from which the linting will be invoked: + +- [flake8][flake8-site]. +- [ShellCheck][shellcheck-site]. +- [terrafom validate][terraform-validate-site]. + +### Linting Execution + +Run `make check --silent` to execute all of the linters. + +Alternatively, the linters can be invoked individually. + +- Run `make check_python` to lint Python files. +- Run `make check_shell` to lint Shell files. +- Run `make check_terraform` to lint Terraform files. + +## Documentation + +The documentation of inputs and outputs for modules in this repository +is automatically generated in each module's `README.md` based on the +contents of the relevant `.tf` files. + +### Documentation Software Dependencies + +The following software dependencies must be installed on the system +from which the documentation will be generated: + +- [terraform-docs][terraform-docs-site] v0.6.0 + +### Generation + +Run `make generate_docs` to update the documentation. + +[bundler-site]: https://bundler.io/ +[flake8-site]: https://pypi.org/project/flake8/ +[gofmt-site]: https://golang.org/cmd/gofmt/ +[hadolint-site]: https://github.com/hadolint/hadolint/ +[inspec-site]: https://inspec.io/ +[kitchen-site]: https://kitchen.ci/ +[kitchen-terraform-site]: https://github.com/newcontext-oss/kitchen-terraform/ +[project-factory-module-site]: https://github.com/terraform-google-modules/terraform-google-project-factory/ +[ruby-site]: https://ruby-lang.org/ +[sample-variable-file]: test/fixtures/shared/terraform.tfvars.sample +[shellcheck-site]: https://www.shellcheck.net/ +[terraform-docs-site]: https://github.com/segmentio/terraform-docs/releases/ +[terraform-provider-google-site]: https://github.com/terraform-providers/terraform-provider-google/ +[terraform-site]: https://www.terraform.io/ +[terraform-validate-site]: https://www.terraform.io/docs/commands/validate.html diff --git a/main.tf b/main.tf new file mode 100644 index 0000000..f429b34 --- /dev/null +++ b/main.tf @@ -0,0 +1,84 @@ +/** + * Copyright 2019 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +resource "google_pubsub_topic" "main" { + name = "${var.name}" + project = "${var.project_id}" +} + +resource "google_logging_project_sink" "main" { + name = "${var.name}" + destination = "pubsub.googleapis.com/${google_pubsub_topic.main.id}" + filter = "${var.log_export_filter}" + project = "${var.project_id}" + unique_writer_identity = true +} + +resource "google_pubsub_topic_iam_member" "main" { + topic = "${google_pubsub_topic.main.name}" + role = "roles/pubsub.publisher" + member = "${google_logging_project_sink.main.writer_identity}" + project = "${var.project_id}" +} + +resource "google_cloudfunctions_function" "main" { + name = "${var.name}" + source_archive_bucket = "${google_storage_bucket.main.name}" + source_archive_object = "${google_storage_bucket_object.main.name}" + description = "${var.function_description}" + available_memory_mb = "${var.function_available_memory_mb}" + timeout = "${var.function_timeout_s}" + entry_point = "${var.function_entry_point}" + + event_trigger { + event_type = "google.pubsub.topic.publish" + resource = "${google_pubsub_topic.main.name}" + + failure_policy { + retry = "${var.function_event_trigger_failure_policy_retry}" + } + } + + labels = "${var.function_labels}" + runtime = "${var.function_runtime}" + environment_variables = "${var.function_environment_variables}" + project = "${var.project_id}" + region = "${var.region}" +} + +data "archive_file" "main" { + type = "zip" + output_path = "${pathexpand("${var.function_source_directory}.zip")}" + source_dir = "${pathexpand("${var.function_source_directory}")}" +} + +resource "google_storage_bucket" "main" { + name = "${var.name}" + force_destroy = "true" + location = "${var.region}" + project = "${var.project_id}" + storage_class = "REGIONAL" + labels = "${var.function_source_archive_bucket_labels}" +} + +resource "google_storage_bucket_object" "main" { + name = "event_function.zip" + bucket = "${google_storage_bucket.main.name}" + source = "${data.archive_file.main.output_path}" + content_disposition = "attachment" + content_encoding = "gzip" + content_type = "application/zip" +} diff --git a/outputs.tf b/outputs.tf new file mode 100644 index 0000000..437465a --- /dev/null +++ b/outputs.tf @@ -0,0 +1,16 @@ +/** + * Copyright 2019 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + diff --git a/test/integration/automatic_labelling/inspec.yml b/test/integration/automatic_labelling/inspec.yml index c1f516e..91cdf9b 100644 --- a/test/integration/automatic_labelling/inspec.yml +++ b/test/integration/automatic_labelling/inspec.yml @@ -28,4 +28,3 @@ attributes: type: string required: true description: "The zone in which resources are applied." - \ No newline at end of file diff --git a/variables.tf b/variables.tf new file mode 100644 index 0000000..5e4e73f --- /dev/null +++ b/variables.tf @@ -0,0 +1,93 @@ +/** + * Copyright 2019 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +variable "function_available_memory_mb" { + type = "string" + default = "256" + description = "The amount of memory in megabytes allotted for the function to use." +} + +variable "function_description" { + type = "string" + default = "Processes log export events provided through a Pub/Sub topic subscription." + description = "The description of the function." +} + +variable "function_entry_point" { + type = "string" + description = "The name of a method in the function source which will be invoked when the function is executed." +} + +variable "function_environment_variables" { + type = "map" + default = {} + description = "A set of key/value environment variable pairs to assign to the function." +} + +variable "function_event_trigger_failure_policy_retry" { + type = "string" + default = "false" + description = "A toggle to determine if the function should be retried on failure." +} + +variable "function_labels" { + type = "map" + default = {} + description = "A set of key/value label pairs to assign to the function." +} + +variable "function_runtime" { + type = "string" + default = "nodejs6" + description = "The runtime in which the function will be executed." +} + +variable "function_source_archive_bucket_labels" { + type = "map" + default = {} + description = "A set of key/value label pairs to assign to the function source archive bucket." +} + +variable "function_source_directory" { + type = "string" + description = "The contents of this directory will be archived and used as the function source." +} + +variable "function_timeout_s" { + type = "string" + default = "60" + description = "The amount of time in seconds allotted for the execution of the function." +} + +variable "log_export_filter" { + type = "string" + description = "The filter to apply when exporting logs to the Pub/Sub topic." +} + +variable "name" { + type = "string" + description = "The name to apply to any nameable resources." +} + +variable "project_id" { + type = "string" + description = "The ID of the project to which resources will be applied." +} + +variable "region" { + type = "string" + description = "The region in which resources will be applied." +}