Skip to content

Commit

Permalink
chore: small fixes in GCP log forwarding using Dataflow job
Browse files Browse the repository at this point in the history
  • Loading branch information
jsubirat committed Nov 15, 2021
1 parent d752a63 commit 4f906aa
Showing 1 changed file with 16 additions and 15 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ To install our Dataflow template, you will need the following tools to be availa
1. A Unix terminal (Linux, MacOS).
2. [Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git).
3. [Java JDK](https://adoptopenjdk.net/index.html) 8 or greater.
4. [Apache Maven 3.2 or greater](https://maven.apache.org/). Please note that earlier versions have been observed to fail during the compilation process.
4. [Apache Maven 3.2 or greater](https://maven.apache.org/). Please note that **earlier versions** have been observed to **fail** during the compilation process.
5. The [Google Cloud SDK (includes `gcloud` and `gsutil` command-line tools)](https://cloud.google.com/sdk/docs/install).

### Log into your Google Cloud Platform account
Expand All @@ -81,7 +81,7 @@ Run the following command and follow the prompt instructions to log into GCP and
gcloud init
```

During the wizard, you will select a cloud project to use, and you will be able to optionally select a default Compute Region and Zone for the resources you create using `gcloud` or `gsutil`. We will not preassume a default project, location or region for the following commands
During the wizard, you will select a cloud project to use, and you will be able to optionally select a default Compute Region and Zone for the resources you create using `gcloud` or `gsutil`. We will not presume a default project, location or region for the following commands

### Clone the DataflowTemplates GitHub repository

Expand All @@ -99,25 +99,26 @@ cd DataflowTemplates

### Set the required configuration to compile and run the Dataflow forwarder

Run the following commands. You only need to specify the `PROJECT_ID`, the `NR_LICENSE_KEY` and the `INPUT_SUBSCRIPTION_NAME`; all others can be left with their suggested values:
Run the following commands. You only need to specify the `PROJECT_ID`, the `BUCKET_NAME`, the `NR_LICENSE_KEY` and the `INPUT_SUBSCRIPTION_NAME`; all others can be left with their suggested values:

```bash
# The Google Cloud Platform project id where your logs are and where the Dataflow log forwarder will run
export PROJECT_ID=<your_project_id>
# Temporary bucket that will store intermediary files as a result of compiling the Dataflow template
export BUCKET_NAME=nr-dataflow-forwarder-storage
PROJECT_ID=<your_project_id>
# Temporary bucket that will store intermediary files as a result of compiling the Dataflow template. Its name must be unique.
BUCKET_NAME=<a_nonexisting_gcs_bucket_name>
# New Relic license key
NR_LICENSE_KEY=<your_newrelic_license_key>
# Input PubSub subscription to read logs from
INPUT_SUBSCRIPTION_NAME=<your_pubsub_input_subscription_name>

# Region where the created resources will be placed
export REGION=us-west1
REGION=us-west1
# Service account used to execute the Dataflow template
export SERVICE_ACCOUNT_NAME=nr-dataflow-forwarder-sa
SERVICE_ACCOUNT_NAME=nr-dataflow-forwarder-sa
# File name where the service account credentials will be stored
export SERVICE_ACCOUNT_KEY_FILENAME=service-account-key.json
# Input PubSub subscription to read logs from
export INPUT_SUBSCRIPTION_NAME=listen_to_logs-sub
# New Relic license key
export NR_LICENSE_KEY=<your_newrelic_license_key>
SERVICE_ACCOUNT_KEY_FILENAME=service-account-key.json
# The name your Dataflow log forwarder job will have
export JOB_NAME=nr-log-forwarder
JOB_NAME=nr-log-forwarder
```

### Create a bucket in GCP to hold the generated Dataflow template
Expand All @@ -135,7 +136,7 @@ Create the service account:
gcloud iam service-accounts create ${SERVICE_ACCOUNT_NAME}
```

Grant permissons to the service account:
Grant permissions to the service account:
```bash
gcloud projects add-iam-policy-binding ${PROJECT_ID} --member="serviceAccount:${SERVICE_ACCOUNT_NAME}@${PROJECT_ID}.iam.gserviceaccount.com" --role="roles/owner"
```
Expand Down

0 comments on commit 4f906aa

Please sign in to comment.