logging endpoint support

[lukesneeringer]

From @chrishiestand on June 5, 2017 19:23

I need to be able to programmatically use the storage logging endpoints: https://cloud.google.com/storage/docs/access-logs

(It would be so much nicer to be able to make an API call to get storage usage per bucket rather than create a new bucket, add logging, parse logfiles and so on - but that is a GCP issue, not a google-cloud-node issue.)

I see you are already working on custom endpoint support: googleapis/google-cloud-node#1630

I don't know how far away custom endpoints are, but in any case it would be very useful to me to add google-cloud-node storage logging endpoint support in one form or another.

Copied from original issue: googleapis/google-cloud-node#2359

[lukesneeringer]

From @stephenplusplus on June 7, 2017 12:24

Thanks for the request! I wasn't familiar with Storage logging before this, so please excuse any incorrect assumptions I might make.

I believe you are requesting that we allow you to make requests to these endpoints-- #1630 actually is about allowing requests to go to a custom-defined host, but the URL path after would stay the same. However, we do have a method, request(), that expects the same arguments as the npm request module, but we handle the authentication and basic plumbing. It is not an officially documented method, however, so there isn't a guarantee it will always work.

I went over the doc, and tried to match the raw HTTP requests they're making to our Storage methods:

var gcs = require('@google-cloud/storage')(...)
var bucket = gcs.bucket('my-bucket')

// POST /storage/v1/b/example-logs-bucket/acl
// Host: www.googleapis.com
// {
// "entity": "group-cloud-storage-analytics@google.com",
// "role": "WRITER"
// }
bucket.acl.writers.addGroup('cloud-storage-analytics@google.com', function() {})

// PATCH /storage/v1/b/example-bucket
// Host: www.googleapis.com
// {
//   "logging": {
//    "logBucket": "example-logs-bucket",
//    "logObjectPrefix": "log_object_prefix"
//   }
// }
bucket.setMetadata({
  logging: {
    logBucket: bucket.name,
    logObjectPrefix: 'log_object_prefix'
  }
}, function(err, apiResponse) {})

// GET /storage/v1/b/example-bucket?fields=logging
// Host: www.googleapis.com
bucket.getMetadata(function(err, metadata) {
  // metadata.logging === logging configuration
})

// Getting logs:
bucket.getFiles({ prefix: 'log_object_prefix' }, function(err, files) {
  // files === log files
})

We could consider adding a method to make enabling logging easier, i.e. bucket.enableLogging({ prefix: 'custom-prefix' }). Did I miss anything you're looking for?

[lukesneeringer]

From @chrishiestand on June 16, 2017 0:56

Oops, guess I didn't look at these API calls closely enough to see that they map onto existing functions. Your solution looks like it will work just fine, thank you very much for taking the time to map those.

I don't think that it is necessary to add a convenience method, though if not it would be good to document these steps somewhere in the google-cloud-node docs https://googlecloudplatform.github.io/google-cloud-node/#/docs/google-cloud/v0.55.0/storage - and funny enough, the easiest way to document it might be to add a convenience method?

[fhinkel]

@lukesneeringer Can we close this?

[frankyn]

Hi thanks for pinging @JustinBeckwith,

The library supports this feature but isn't documented as a snippet.

However, the steps to enable Audit Logs requires multiple steps, I think it'd be helpful to create a small helper method to enables Access Logs. I'll file a FR request against the backlog to get it prioritized from the GCS team.

The helper method would be similar to what @stephenplusplus proposed with the small difference to allow a different log bucket to be used:

bucket.enableLogging({ logBucket: 'logs-bucket', prefix: 'custom-prefix' });

logBucket doesn't need to be the same bucket as bucket.name/example-bucket.

[stephenplusplus]

@frankyn did anything end up happening with the upstream feature request?

[frankyn]

On an internal list but not scheduled this quarter.