This is a simple microservice (to be used in conjunction with Project Helix) that sets up correct logging for a Project Helix-managed Fastly service config. It makes sure logs will be sent to Google BigQuery.
Send a POST request with following parameters to $ curl https://adobeioruntime.net/api/v1/web/helix/helix-services/logging@v1:
service: the Fastly service config ID of the service that should get updatedtoken: a Fastly API token that has write permissions on the service aboveversion: the version number of a checked out (draft) version of the service config above
Optional:
coralogixkey: A private key from Coralogixcoralogixapp: name of the Coralogix application to send logs to
Sending over these optional parameters will also enable logging to Coralogix.
$ curl -X POST -H "Content-Type: application/json" -d '{"service":"6E6ge7REhiWetPCqy9jht2","version":2,"token":"thisismysecret"}'
{
"dataset": "helix_logging_6E6ge7REhiWetPCqy9jht2",
"format": {
"client_as_name": "%{client.as.name}V",
"client_geo_city": "%{client.geo.city.utf8}V",
"client_geo_conn_speed": "%{client.geo.conn_speed}V",
"client_geo_continent_code": "%{client.geo.continent_code}V",
"client_geo_country_code": "%{client.geo.country_code}V",
"client_geo_gmt_offset": "%{client.geo.gmt_offset}V",
"client_geo_latitude": "%{client.geo.latitude}V",
"client_geo_longitude": "%{client.geo.longitude}V",
"client_geo_metro_code": "%{client.geo.metro_code}V",
"client_geo_postal_code": "%{client.geo.postal_code}V",
"client_geo_region": "%{client.geo.region}V",
"client_ip_hashed": "%{digest.hash_sha1(client.ip)}V",
"client_ip_masked": "%{client.ip}V",
"fastly_info_state": "%{fastly_info.state}V",
"req_http_Referer": "%{req.http.Referer}V",
"req_http_User_Agent": "%{req.http.User-Agent}V",
"req_http_X_CDN_Request_ID": "%{req.http.X-CDN-Request-ID}V",
"req_http_X_Host": "%{req.http.X-Host}V",
"req_http_X_Owner": "%{req.http.X-Owner}V",
"req_http_X_Ref": "%{req.http.X-Ref}V",
"req_http_X_Repo": "%{req.http.X-Repo}V",
"req_http_X_Static": "%{req.http.X-Static}V",
"req_http_X_Strain": "%{req.http.X-Strain}V",
"req_http_X_URL": "%{req.http.X-URL}V",
"req_http_host": "%v",
"req_url": "https://%{req.http.X-Host}V%{req.http.X-URL}V",
"resp_http_Content_Type": "%{resp.http.Content-Type}V",
"resp_http_X_Version": "%{req.http.X-Version}V",
"resp_http_x_openwhisk_activation_id": "%{resp.http.x-openwhisk-activation-id}V",
"server_datacenter": "%{server.datacenter}V",
"server_region": "%{server.region}V",
"service_config": "6E6ge7REhiWetPCqy9jht2",
"status_code": "%>s",
"time_elapsed_usec": "%{time.elapsed.usec}V",
"time_end_usec": "%{time.end.usec}V",
"time_start_usec": "%{time.start.usec}V",
"vcl_sub": "log-general"
},
"name": "helix-logging",
"project": "helix-225321",
"service": "6E6ge7REhiWetPCqy9jht2",
"version": "2"
}- the service creates or updates a Google Cloud Platform service account that corresponds to the service config
- the service creates or rotates the private keys for the service account
- the service creates or updates a Google BigQuery dataset with proper tables
- the service grants permission to the service user to write the Google BigQuery dataset
- the service creates or updates a log configuration in Fastly that sends logs to the above Google BigQuery dataset using the private key and service account from above
This service depends on three external services to operate:
- Fastly
- Adobe I/O Runtime (only for deployments)
- Google Cloud Platform
It is configured using a number of environment variables that are required for testing (tests that miss required variables will be skipped) and deployment (deployment will fail or be non-functional). These variables are required and this is how to set them up:
This is the email address associated with a Google Cloud Platform Service account. It looks like <name>@<project>.iam.gserviceaccount.com. You can create a proper service account following the instructions in the Google Cloud Platform documentation or this step-by-step guide:
- Log in to Google Cloud Platform Console
- Select menu → "IAM & admin" → "Service accounts" → "Create service account"
- Create the service account
- Add the following roles to the service account:
- BigQuery Admin
- Service Account Admin
- Service Account Key Admin
- Service Account Key Admin
- Create a private key in JSON format for the service account and download the key file
Note: The private key file and the value of the GOOGLE_CLIENT_EMAIL environment variable should be considered private and should never be checked in to source control.
The downloaded file will look something like this:
{
"type": "service_account",
"project_id": "project-12345678",
"private_key_id": "111122223333aaaabbbbccccdddd123412345",
"private_key": "-----BEGIN PRIVATE KEY-----\n…\n-----END PRIVATE KEY-----\n",
"client_email": "example-account@project-12345678.iam.gserviceaccount.com",
"client_id": "111122223333444456789",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/example-account%40project-12345678.iam.gserviceaccount.com"
}Copy the value of the client_email field (e.g. example-account@project-12345678.iam.gserviceaccount.com) and save it in the GOOGLE_CLIENT_EMAIL environment variable.
This is the private key associated with the Google Cloud Platform Service account created above. In order to retrieve the correct value, see Creating and Managing Service Account Keys in the Google Cloud Platform documentation or continue the step-by-step guide from above:
- Make sure you've followed all steps to get the value of
GOOGLE_CLIENT_EMAIL - Copy the value of the
private_keyproperty in the JSON file you've downloaded
Note: The private key and the value of the GOOGLE_PRIVATE_KEY environment variable should be considered private and should never be checked in to source control.
The private key is a multi-line value.
Note: Private keys created using an API typically have a short expiration time and need to be rotated in regular intervals. Even for private keys that have been created manually, regular rotation is a best practice.
This is the Google Cloud Platform project ID. It looks like project-12345678 and you will find it in lots of places in the Google Cloud Platform Console UI. In addition, you can just take the value of the project_id property in your downloaded key JSON file.
This property is only required for testing and development. It is the service config ID that you can retrieve from Fastly.
For testing, it is a good idea to use a separate, non-production service config, as the tests not only perform frequent updates, but they also rotate the private keys of the created Google Cloud Platform service accounts. As the tests don't activate the service config, this will lead to an invalid logging configuration in a short time.
This property is only required for testing and development. It is an API token for the Fastly API. Follow the instructions in the Fastly documentation to create a token.
The token needs to have global, i.e. write access to your service config.
Note: The API token and the value of the HLX_FASTLY_AUTH environment variable should be considered private and should never be checked in to source control.
This is the version number of a version of the service config above that has been checked out, but has never been activated. This means it is editable.
You need node>=8.0.0 and npm>=5.4.0. Follow the typical npm install, npm test workflow.
Contributions are highly welcome.
Deploying Helix Logging requires the wsk command-line client, authenticated to a namespace of your choice. For Project Helix, we use the helix namespace.
Run npm run deploy to do a one-shot deploment of Helix Logging. All commits to main that pass the testing will be deployed automatically.