B. GCP

Full API doc at https://www.pulumi.com/docs/reference/pkg/gcp/

Table of contents

• Buckets
• Enable services
  • Standard GCP services
  • Firebase service
  • Identity Platform service
• Cloud Run
  • Basic Cloud Run example
  • Setting up public HTTPS access
  • Congiguring service-to-service communication
• Identity Platform
• Service Accounts

Buckets

const pulumi = require('@pulumi/pulumi')
const gcp = require('@pulumi/gcp')

if (!process.env.PROJECT)
	throw new Error('Missing required environment variable \'process.env.PROJECT\'')

const config = new pulumi.Config()

const { location } = config.requireObject('gcp_bucket')

const STACK_NAME = pulumi.getStack()
const RESOURCE_PREFIX = `${process.env.PROJECT}-${STACK_NAME}`
const FILE_BUCKET = `${RESOURCE_PREFIX}-storage-pb`
const PRIVATE_BUCKET = `${RESOURCE_PREFIX}-nosql-db`

// Create the public file storage
const publicFileBucket = new gcp.storage.Bucket(FILE_BUCKET, {
	name: FILE_BUCKET, // This seems redundant, but it is not. It forces Pulumi to not add a unique suffix on your bucket.
	bucketPolicyOnly: true, // Means the policy applies on the entire bucket rather than on a per object basis
	cors: [{
		maxAgeSeconds: 3600,
		methods: [ 'GET', 'OPTIONS', 'HEAD', 'POST', 'PUT', 'DELETE' ],
		origins: ['*'],
		responseHeaders: ['*'],
	}],
	location
})

// Create the private bucket
const privateBucket = new gcp.storage.Bucket(PRIVATE_BUCKET, {
	name: PRIVATE_BUCKET,
	location
})

module.exports = {
	publicFileBucket: {
		id: publicFileBucket.id,
		publicUrl: publicFileBucket.selfLink,
		url: publicFileBucket.url,
		storageClass: publicFileBucket.storageClass,
		location: publicFileBucket.location
	},
	privateBucket: {
		id: privateBucket.id,
		publicUrl: privateBucket.selfLink,
		url: privateBucket.url,
		storageClass: privateBucket.storageClass,
		location: privateBucket.location
	}
}

Enable services

Standard GCP services

require('@pulumi/pulumi')
const gcp = require('@pulumi/gcp')

if (!process.env.PROJECT)
	throw new Error('Missing required environment variable \'process.env.PROJECT\'')

const SERVICES = [
	'cloudbuild.googleapis.com',
	'containerregistry.googleapis.com',
	'run.googleapis.com',
	'secretmanager.googleapis.com'
]

const services = []

for(const service of SERVICES) {
	const { id } = new gcp.projects.Service(service, {
		project: process.env.PROJECT,
		service
	})

	services.push(id)
}

module.exports = {
	services
}

Firebase service

WARNING: Enabling Firebase on a Google project cannot be undone. I would suggest to not delete the Pulumi code that enable that service even if you which to not use Firebase in your project. You might think you just want to clean the Pulumi project, but the truth is that this will create issues as the Firebase project cannot be disabled.

Firebase is kind of a weird service. In essence, it is part of the GCP suite, but from a brand perspective, it is a separate product. Though there are a few Firebase services(1) that can be enabled in a GCP project the way it was explained in the previous section, this is not the way to enable Firebase on a Google project. The correct Pulumi API is the following:

const gcp = require('@pulumi/gcp')

const firebase = new gcp.firebase.Project('your-firebase-project-name', {
	project: 'your-gcp-project-id'
})

module.exports = {
	firebase: firebase.id
}

This above snippets as a few side-effects. It will provision the following:

• A few new firebase servcies are enabled. Some of those are listed in (1) below.
• A new service account called Firebase Admin SDK is added.

(1) The GCP Firebase services are:

• firebase.googleapis.com
• firebaseappdistribution.googleapis.com
• firebaseapptesters.googleapis.com
• firebasedynamiclinks.googleapis.com
• firebaseextensions.googleapis.com
• firebasehosting.googleapis.com
• firebaseinappmessaging.googleapis.com
• firebaseinstallations.googleapis.com
• firebaseml.googleapis.com
• firebasemods.googleapis.com
• firebasepredictions.googleapis.com
• firebaseremoteconfig.googleapis.com
• firebaserules.googleapis.com
• firebasestorage.googleapis.com
• firestore.googleapis.com

Identity Platform service

Unfortunately, as of August 2020, it is not possible to automate the enabling of that service via Pulumi because Identity Platform is an app in the Google Cloud Marketplace rather than a first class Google Cloud service.

To enable that service, manually log to the Google Cloud console here.

Cloud Run

Basic Cloud Run example

The following steps shows how to provision a Cloud Run service with the following aspects:

• Conventions:
  • The Cloud Run service name is built as follow: <PULUMI PROJECT NAME>-<STACK>. <PULUMI PROJECT NAME> is the name property in the Pulumi.yaml. For example, if the stack is called test, the service's name could be: yourproject-test.
  • The Docker image is tagged with the first 7 characters of the current git commit sha.
• Environment variables are passed to the container so the app can access them. Typically, those are secrets. More about this at the bottom of this section.
• Though it is not required, this sample creates a dedicated service account for the Cloud Run service. This is considered a best practice because it makes it easier to control IAM policies for service-to-service communination.
• That service cannot be accessed publicly via HTTPS. This is the default behavior. If you need to expose that service to the public, jump to the Setting up public HTTPS access section. To learn how to safely enable service-to-service communication without exposing them to the public, please refer to the Congiguring service-to-service communication section.

To use this sample, make sure to:

• Install the dependencies:

   npm i @pulumi/pulumi @pulumi/gcp @pulumi/docker
  

• Configure the Pulumi.<STACK NAME>.yaml so it contain at a minimum the following settings:

   config:
     your-project-name:memory: 512Mi
     gcp:project: your-gcp-project-id
     gcp:region: australia-southeast1

• Set up the following environment variables (e.g., use dotenv or your build server):
  • DB_USER
  • DB_PASSWORD
• Add the git helper module to get the current short commit sha. That module is documented here.
• Add your Cloud Run source-code under the app folder. It does not need any cloudbuild.yaml since the build is automated with Pulumi, but it still needs a Dockerfile as per usual.

const pulumi = require('@pulumi/pulumi')
const gcp = require('@pulumi/gcp')
const docker = require('@pulumi/docker')
const { git } = require('./utils')

// Validates that the environment variables are set up
const ENV_VARS = [
	'DB_USER',
	'DB_PASSWORD'
]

for (let varName of ENV_VARS)
	if (!process.env[varName])
		throw new Error(`Missing required environment variables 'process.env.${varName}'`)

const config = new pulumi.Config()

const STACK_NAME = pulumi.getStack()
const MEMORY = config.require('memory') || '512Mi'
const SHORT_SHA = git.shortSha()
const SERVICE_NAME = `${config.name}-${STACK_NAME}`
const IMAGE_NAME = `${SERVICE_NAME}-image`
const SERVICE_ACCOUNT_NAME = `${SERVICE_NAME}-cloudrun`

const SERVICE_ACCOUNT_NAME = `${config.name}-${STACK_NAME}-cloudrun`

if (!SHORT_SHA)
	throw new Error('This project is not a git repository')
if (!gcp.config.project)
	throw new Error(`Missing required 'gcp:project' in the '${STACK_NAME}' stack config`)
if (!gcp.config.region)
	throw new Error(`Missing required 'gcp:region' in the '${STACK_NAME}' stack config`)

// Enables the Cloud Run service (doc: https://www.pulumi.com/docs/reference/pkg/gcp/projects/service/)
const enableCloudRun = new gcp.projects.Service('run.googleapis.com', {
	service: 'run.googleapis.com'
})

const gcpAccessToken = pulumi.output(gcp.organizations.getClientConfig({}).then(c => c.accessToken))

// Uploads new Docker image with your app to Google Cloud Container Registry (doc: https://www.pulumi.com/docs/reference/pkg/docker/image/)
const dockerImage = new docker.Image(IMAGE_NAME, {
	imageName: pulumi.interpolate`gcr.io/${gcp.config.project}/${config.name}:${SHORT_SHA}`,
	build: {
		context: './app'
	},
	registry: {
		server: 'gcr.io',
		username: 'oauth2accesstoken',
		password: pulumi.interpolate`${gcpAccessToken}`
	}
})

// Creates a new service account for that Cloud Run service (doc: https://www.pulumi.com/docs/reference/pkg/gcp/serviceaccount/account/)
const serviceAccount = new gcp.serviceAccount.Account(SERVICE_ACCOUNT_NAME, {
	accountId: SERVICE_ACCOUNT_NAME, // This will automatically create the service account email as follow: <SERVICE_ACCOUNT_NAME>@<PROJECT ID>.iam.gserviceaccount.com
	displayName: SERVICE_ACCOUNT_NAME
})

// Deploys the new Docker image to Google Cloud Run (doc: https://www.pulumi.com/docs/reference/pkg/gcp/cloudrun/)
const cloudRunService = new gcp.cloudrun.Service(SERVICE_NAME, {
	name: SERVICE_NAME,
	location: gcp.config.region,
	template: {
		// doc: https://www.pulumi.com/docs/reference/pkg/gcp/cloudrun/service/#servicetemplatespec
		spec: {
			// doc: https://www.pulumi.com/docs/reference/pkg/gcp/cloudrun/service/#servicetemplatespeccontainer
			containers: [{
				envs: ENV_VARS.map(name => ({ name, value:process.env[name] })),
				image: dockerImage.imageName,
				// doc: https://www.pulumi.com/docs/reference/pkg/gcp/cloudrun/service/#servicetemplatespeccontainerresources
				resources: {
					limits: {
						memory: MEMORY // Available units are 'Gi', 'Mi' and 'Ki'
					},
				},
			}],
			serviceAccountName: serviceAccount.email, // This is optional. The default is the project's default service account
			containerConcurrency: 80, // 80 is the max. Above this limit, Cloud Run spawn another container.
		},
	},
}, { 
	dependsOn: [
		enableCloudRun 
	]
})

module.exports = {
	serviceAccount: {
		id: serviceAccount.id,
		name: serviceAccount.name,
		accountId: serviceAccount.accountId,
		email: serviceAccount.email,
		project: serviceAccount.project
	},
	cloudRunService: {
		id: cloudRunService.id,
		name: cloudRunService.name,
		project: cloudRunService.project,
		location: cloudRunService.location,
		url: cloudRunService.status.url,
		serviceAccount: cloudRunService.template.spec.serviceAccountName
	},
	dockerImage: dockerImage.imageName,
	enableCloudRun: enableCloudRun.id
}

What's interesting in this template:

• The environment variables are passed to the container via envs: ENV_VARS.map(name => ({ name, value:process.env[name] })). If your use case requires to pass some of those variables to the Docker image, please refer to the Passing environment variables to the Docker image rather than the Docker container section.
• To push the Docker image to a registry other than DockerHub (in our example gcr.io), we must add a registry property in the docker.Image instantiation. The Pulumi documentation on how to set this up for Google Cloud Container Registry was not really clear:
  • server: Must be hardcoded to gcr.io.
  • username: Must be hardcoded to oauth2accesstoken.
  • password: This is the short-lived OAuth2 access token retrieved based on your Google credentials. That token can retrieved with the gcp.organizations.getClientConfig({}).then(c => c.accessToken) API. However, because this is a Promise that resolves to a string, it must first be converted to an Output with pulumi.output. The string can finally be passed to the docker.Image instance with the pulumi.interpolation function.
• A new service account is created just for that Cloud Run:

   const serviceAccount = new gcp.serviceAccount.Account(...)
   ...
   const cloudRunService = new gcp.cloudrun.Service(SERVICE_NAME, {
   	template: {
   		spec: {
   			...
   			serviceAccountName: serviceAccount.email,
   			...
   		}
   	}
   })

  As mentioned earlier, this step is optional, but it is considered a best practice ot manage IAM policies betwene services. If the line serviceAccountName: serviceAccount.email is omitted, the Cloud Run service is associated to the project default service account.

Setting up public HTTPS access

By default, Cloud Run services are protected. This means that they cannot be access via HTTPS outside of your Google Clloud project's VPC. To enable HTTPS access to the public, add the following snippet at the bottom of the previous code snippet:

// Allows this service to be accessed via public HTTPS
const PUBLIC_MEMBER = `${SERVICE_NAME}-public-member`
const publicAccessMember = new gcp.cloudrun.IamMember(PUBLIC_MEMBER, {
	service: cloudRunService.name,
	location: cloudRunService.location,
	role: 'roles/run.invoker',
	member: 'allUsers'
})

Congiguring service-to-service communication

This section demonstrates how to create a Cloud Run service that can invoke another protected Cloud Run service.

It is considered a best practice to not expose your Cloud Run services publicly unless this is a business requirement (e.g., exposing a web API for a mobile or web app). This means that for service-to-service communication, roles must be explicitly configured to allow specific agents to interact with each other. The approach is quite straightforward:

1. Get the Pulumi stack of the protected Cloud Run service. We need three pieces of information from that stack:
   • name
   • project
   • location This means that those pieces of information must have been added to the stack's outputs.

   const otherProtectedStack = new pulumi.StackReference('your-other-protected-stack')

2. Add a new IAM binding on that protected Cloud Run service which associates the roles/run.invoker role to the current Cloud Run's service account.

   const binding = new gcp.cloudrun.IamBinding('your-new-binding-name', {
   	service: otherProtectedStack.outputs.cloudRunService.name,
   	location: otherProtectedStack.outputs.cloudRunService.location,
   	project: otherProtectedStack.outputs.cloudRunService.project,
   	role: 'roles/run.invoker',
   	members: [
   		pulumi.interpolate`serviceAccount:${serviceAccount.email}`
   	]
   })

   IMPORTANT: Notice the convention used to define the members:

   1. We need to use pulumi.interpolate because serviceAccount.email is an Output.
   2. We need to prefix the service account email with serviceAccount (careful, this is case-sensitive!), otherwise, a Error 400: The member ... is of an unknown type error is thrown.

Identity Platform

Manually enable the Identity Platform

1. Manually enable Identity Platform service
2. If you need to use the multi-tenants feature, manually enable it (as of August 2020, this cannot be automated yet):
   • Log in to your project's Identity Platform page.
   • Click on the Tenants in the menu.
   • Click on Settings, select the Security tab and then click on the Allow tenants button.

Create a new tenant

doc: https://www.pulumi.com/docs/reference/pkg/gcp/identityplatform/tenant/

const tenant = new gcp.identityplatform.Tenant('your-tenant-name', {
	allowPasswordSignup: true,
	displayName: 'your-tenant-name'
})

module.exports = {
	tenant: {
		id: tenant.id,
		tenantId: tenant.name // Value required in the client: firebase.auth().tenantId = tenantId
	}
}

Service accounts

There are no Pulumi APIs to list all the project's service accounts, but it is easy to call the official Google Cloud REST API to get that information. Convert that Promise into an Output with pulumi.output so you can use it with other resources.

const pulumi = require('@pulumi/pulumi')
const gcp = require('@pulumi/gcp')
const fetch = require('node-fetch')

/**
 * Selects service accounts in the current project. 
 * 
 * @param  {String} query.where.email
 * @param  {String} query.where.emailContains
 * 
 * @return {String} serviceAccounts[].description					
 * @return {String} serviceAccounts[].displayName					
 * @return {String} serviceAccounts[].email					
 * @return {String} serviceAccounts[].etag					
 * @return {String} serviceAccounts[].name					
 * @return {String} serviceAccounts[].oauth2ClientId					
 * @return {String} serviceAccounts[].projectId					
 * @return {String} serviceAccounts[].uniqueId	
 */
const select = async query => {
	const where = (query || {}).where || {}
	const { accessToken } = await gcp.organizations.getClientConfig({})

	const uri = `https://iam.googleapis.com/v1/projects/${gcp.config.project}/serviceAccounts`
	
	const data = await fetch(uri, {
		headers: {
			'Content-Type': 'application/json',
			Authorization: `Bearer ${accessToken}`
		}
	}).then(res => res.json())

	if (!data || !data.accounts || !data.accounts.length)
		return []

	const filters = []

	if (where.email)
		filters.push(account => account.email == where.email)		
	if (where.emailContains)
		filters.push(account => account.email.indexOf(where.emailContains) >= 0)	

	return data.accounts.filter(account => filters.every(f => f(account)))
}

const find = query => select(query).then(data => data[0])

module.exports = {
	select,
	find
}