Add support for Google Cloud Functions #1510

Closed
eahefnawy opened this Issue Jul 8, 2016 · 9 comments

Comments

Projects
None yet
5 participants
@eahefnawy
Member

eahefnawy commented Jul 8, 2016

Google Cloud Functions work a little differently from AWS. However we'll try to have a common interface across provider whenever we can, which should be easier with the provider-per-service approach. Here's a basic outline of their offering and how we can support it:

Research:

  • Google Functions use triggers to trigger functions. It's just like our events concept, so we'll call it events for consistency.
  • Google Functions support HTTP, Bucket, Pub/Sub topic and direct invoke triggers. They don't yet support schedule events.
  • Google Functions can support additional triggers by using Pub/Sub topics as a proxy, for example Gmail push notifications and cloud logging.
  • Google Functions can be tested/invoked using the direct trigger event.
  • Google Functions can be deployed from either a Google Storage bucket, or a source control repo. We'll probably go with the bucket approach for consistency.
  • Google Functions are not yet supported in the SDK, and they don't provide a public REST API either. The only way to access Google Functions right now is via the CLI. So we'll have to have this dependency, at least for now.
  • Google Functions are not supported in Google Resource Manager either. So we won't be able to deploy via the resource manager and be consistent in that front.
  • Google Functions logs are accessed via Google StackDriver, the equivalent of AWS CloudWatch. So we'll be able to have logging functionality.
  • The entry point is called entryPoint but we should call this handler for consistency
  • Right now only Node.js is supported
  • Google thinks in terms of Projects we should keep Service
  • Two types of functions (foreground and background --> docs)

Questions:

  • Should we go full CLI since CloudFunctions can only be deployed this way or should we use the Resource Manager to e.g. deploy the serviceBucket?
  • How do we abstract the Pub/Sub topic support as an event source?

Helpful links:

@pmuens pmuens added this to the v1.0.0-alpha.3 milestone Jul 8, 2016

@pmuens

This comment has been minimized.

Show comment
Hide comment
@pmuens

pmuens Jul 8, 2016

Member

@eahefnawy thanks for that! Awesome!

Added a few more notes to your already awesome description!

Member

pmuens commented Jul 8, 2016

@eahefnawy thanks for that! Awesome!

Added a few more notes to your already awesome description!

@eahefnawy

This comment has been minimized.

Show comment
Hide comment
@eahefnawy

eahefnawy Jul 8, 2016

Member
Available Regions:
  • europe-west1
  • us-central1
  • us-east1
Default timeout:

60 seconds

Triggers:

  • Pub/Sub topic
  • Storage Bucket
  • HTTP: endpoint is autogenrated based on this format:
    .cloudfunctions.net/

Package Deployment:

  • Zip upload
  • Source control repo
  • Zip file from storage bucket. Note that even if you choose one of the two other choices above, you'll still need a bucket to store the uploaded zip.

Notes

  • It seems that you have to choose an event/trigger no matter what. However it seems that you always invoke the function for testing.
Member

eahefnawy commented Jul 8, 2016

Available Regions:
  • europe-west1
  • us-central1
  • us-east1
Default timeout:

60 seconds

Triggers:

  • Pub/Sub topic
  • Storage Bucket
  • HTTP: endpoint is autogenrated based on this format:
    .cloudfunctions.net/

Package Deployment:

  • Zip upload
  • Source control repo
  • Zip file from storage bucket. Note that even if you choose one of the two other choices above, you'll still need a bucket to store the uploaded zip.

Notes

  • It seems that you have to choose an event/trigger no matter what. However it seems that you always invoke the function for testing.

@pmuens pmuens modified the milestones: v1.0, v1.0.0-beta Jul 12, 2016

@jmdobry

This comment has been minimized.

Show comment
Hide comment
@jmdobry

jmdobry Oct 13, 2016

Google Functions are not yet supported in the SDK, and they don't provide a public REST API either. The only way to access Google Functions right now is via the CLI. So we'll have to have this dependency, at least for now.

var google = require('googleapis');
var discoveryURL = 'https://cloudfunctions.googleapis.com/$discovery/rest?version=v1beta2';

google.discoverAPI(discoveryURL, function(err, functions) {

  // For this to work, set these environment variables:
  // 
  // - GCLOUD_PROJECT - Your Google Cloud Platform Project id
  // - GOOGLE_APPLICATION_CREDENTIALS - The path to your Service Account key file
  google.auth.getApplicationDefault(function(err, authClient) {
    if (err) {
      throw err;
    }

    if (authClient.createScopedRequired && authClient.createScopedRequired()) {
      authClient = authClient.createScoped(['https://www.googleapis.com/auth/cloud-platform']);
    }

    // For example, retrieve a function's details
    functions.projects.locations.functions.get({
      auth: authClient,
      name: 'projects/YOUR_PROJECT_ID/locations/YOUR_REGION/functions/YOUR_FUNCTION_NAME'
    }, function(err, body) {
      if (err) {
        throw err;
      }

      console.log('Function details:');
      console.log(body);
    });
  });
});

gRPC (HTTP/2) client is in the works.

jmdobry commented Oct 13, 2016

Google Functions are not yet supported in the SDK, and they don't provide a public REST API either. The only way to access Google Functions right now is via the CLI. So we'll have to have this dependency, at least for now.

var google = require('googleapis');
var discoveryURL = 'https://cloudfunctions.googleapis.com/$discovery/rest?version=v1beta2';

google.discoverAPI(discoveryURL, function(err, functions) {

  // For this to work, set these environment variables:
  // 
  // - GCLOUD_PROJECT - Your Google Cloud Platform Project id
  // - GOOGLE_APPLICATION_CREDENTIALS - The path to your Service Account key file
  google.auth.getApplicationDefault(function(err, authClient) {
    if (err) {
      throw err;
    }

    if (authClient.createScopedRequired && authClient.createScopedRequired()) {
      authClient = authClient.createScoped(['https://www.googleapis.com/auth/cloud-platform']);
    }

    // For example, retrieve a function's details
    functions.projects.locations.functions.get({
      auth: authClient,
      name: 'projects/YOUR_PROJECT_ID/locations/YOUR_REGION/functions/YOUR_FUNCTION_NAME'
    }, function(err, body) {
      if (err) {
        throw err;
      }

      console.log('Function details:');
      console.log(body);
    });
  });
});

gRPC (HTTP/2) client is in the works.

@pmuens

This comment has been minimized.

Show comment
Hide comment
@pmuens

pmuens Oct 14, 2016

Member

Damn @jmdobry 💃 👯 right in time!

Thank you very much for that!

Member

pmuens commented Oct 14, 2016

Damn @jmdobry 💃 👯 right in time!

Thank you very much for that!

@pmuens

This comment has been minimized.

Show comment
Hide comment
@pmuens

pmuens Oct 25, 2016

Member

Hey @jmdobry thanks for the example!
I just started to implement this and got your example up and running.

However I'm struggling to create / update a function.

Is there any chance to get a real quick example how one can create a function with the help of the API? I saw that I need to specify the function itself in the request but I'm not sure how exactly this would look like.

Thanks in advance!

Member

pmuens commented Oct 25, 2016

Hey @jmdobry thanks for the example!
I just started to implement this and got your example up and running.

However I'm struggling to create / update a function.

Is there any chance to get a real quick example how one can create a function with the help of the API? I saw that I need to specify the function itself in the request but I'm not sure how exactly this would look like.

Thanks in advance!

@jmdobry

This comment has been minimized.

Show comment
Hide comment
@jmdobry

jmdobry Oct 25, 2016

Edit: This sample could probably be tweaked so that is doesn't write the zip file to disk, but instead streams it straight up to the GCS bucket. When the new library is ready it will do some of this stuff for you.

const archiver = require('archiver');
const async = require('async');
const fs = require('fs');
const google = require('googleapis');
const storage = require('@google-cloud/storage')();
const uuid = require('node-uuid');

const DISCOVERY_DOC_URL = 'https://cloudfunctions.googleapis.com/$discovery/rest?version=v1beta2';
const BUCKET_NAME = 'YOUR_STAGE_BUCKET';
const FUNCTION_NAME = 'helloWorld';
const ZIP_FILENAME = `us-central1-${FUNCTION_NAME}-${uuid.v4()}.zip`
const PROJECT_ID = process.env.GCLOUD_PROJECT;
const TOPIC_NAME = 'my-topic';

const file1 = '/path/to/index.js';
const file2 = '/path/to/package.json';

let functionsClient, authClient;

async.waterfall([
  // Prepare the zip file
  (cb) => {
    const archive = archiver.create('zip');
    const output = fs.createWriteStream(ZIP_FILENAME);
    archive.pipe(output);
    archive
      .append(fs.createReadStream(file1), { name: 'index.js' })
      .append(fs.createReadStream(file2), { name: 'package.json' })
      .finalize();
    output.on('close', cb);
    archive.on('error', cb);
  },
  // Upload the zip file to the stage bucket
  (cb) => storage.bucket(BUCKET_NAME).upload(ZIP_FILENAME, cb),
  // Build the Cloud Functions client
  (apiResponse, cb) => google.discoverAPI(DISCOVERY_DOC_URL, cb),
  // Acquire credentials
  (functions, cb) => {
    functionsClient = functions;
    google.auth.getApplicationDefault(cb);
  },
  (authClient, cb) => {
    if (authClient.createScopedRequired && authClient.createScopedRequired()) {
      authClient = authClient.createScoped(['https://www.googleapis.com/auth/cloud-platform']);
    }
    // Create the function (creates a long-running operation)
    functionsClient.projects.locations.functions.create({
      auth: authClient,
      location: `projects/${PROJECT_ID}/locations/us-central1`,
      // This example creates a background function triggered by a Pub/Sub topic
      resource: {
        gcsUrl: `gs://${BUCKET_NAME}/${ZIP_FILENAME}`,
        name: `projects/${PROJECT_ID}/locations/us-central1/functions/${FUNCTION_NAME}`,
        pubsubTrigger: `projects/${PROJECT_ID}/topics/${TOPIC_NAME}`
      }
    }, (err, operation) => {

      // Wait for the operation to finish
      function pollOperation () {
        functionsClient.operations.get({
          auth: authClient,
          name: operation.name
        }, (err, _operation) => {
          if (err) {
            cb(err);
          } else if (_operation.done) {
            cb();
          } else {
            process.stdout.write('.');
            setTimeout(() => pollOperation(), 500);
          }
        });
      }

      // Start polling
      pollOperation();
    });
  }
], (err, operation) => {
  if (err) {
    throw err;
  }
  console.log('done!');
});

jmdobry commented Oct 25, 2016

Edit: This sample could probably be tweaked so that is doesn't write the zip file to disk, but instead streams it straight up to the GCS bucket. When the new library is ready it will do some of this stuff for you.

const archiver = require('archiver');
const async = require('async');
const fs = require('fs');
const google = require('googleapis');
const storage = require('@google-cloud/storage')();
const uuid = require('node-uuid');

const DISCOVERY_DOC_URL = 'https://cloudfunctions.googleapis.com/$discovery/rest?version=v1beta2';
const BUCKET_NAME = 'YOUR_STAGE_BUCKET';
const FUNCTION_NAME = 'helloWorld';
const ZIP_FILENAME = `us-central1-${FUNCTION_NAME}-${uuid.v4()}.zip`
const PROJECT_ID = process.env.GCLOUD_PROJECT;
const TOPIC_NAME = 'my-topic';

const file1 = '/path/to/index.js';
const file2 = '/path/to/package.json';

let functionsClient, authClient;

async.waterfall([
  // Prepare the zip file
  (cb) => {
    const archive = archiver.create('zip');
    const output = fs.createWriteStream(ZIP_FILENAME);
    archive.pipe(output);
    archive
      .append(fs.createReadStream(file1), { name: 'index.js' })
      .append(fs.createReadStream(file2), { name: 'package.json' })
      .finalize();
    output.on('close', cb);
    archive.on('error', cb);
  },
  // Upload the zip file to the stage bucket
  (cb) => storage.bucket(BUCKET_NAME).upload(ZIP_FILENAME, cb),
  // Build the Cloud Functions client
  (apiResponse, cb) => google.discoverAPI(DISCOVERY_DOC_URL, cb),
  // Acquire credentials
  (functions, cb) => {
    functionsClient = functions;
    google.auth.getApplicationDefault(cb);
  },
  (authClient, cb) => {
    if (authClient.createScopedRequired && authClient.createScopedRequired()) {
      authClient = authClient.createScoped(['https://www.googleapis.com/auth/cloud-platform']);
    }
    // Create the function (creates a long-running operation)
    functionsClient.projects.locations.functions.create({
      auth: authClient,
      location: `projects/${PROJECT_ID}/locations/us-central1`,
      // This example creates a background function triggered by a Pub/Sub topic
      resource: {
        gcsUrl: `gs://${BUCKET_NAME}/${ZIP_FILENAME}`,
        name: `projects/${PROJECT_ID}/locations/us-central1/functions/${FUNCTION_NAME}`,
        pubsubTrigger: `projects/${PROJECT_ID}/topics/${TOPIC_NAME}`
      }
    }, (err, operation) => {

      // Wait for the operation to finish
      function pollOperation () {
        functionsClient.operations.get({
          auth: authClient,
          name: operation.name
        }, (err, _operation) => {
          if (err) {
            cb(err);
          } else if (_operation.done) {
            cb();
          } else {
            process.stdout.write('.');
            setTimeout(() => pollOperation(), 500);
          }
        });
      }

      // Start polling
      pollOperation();
    });
  }
], (err, operation) => {
  if (err) {
    throw err;
  }
  console.log('done!');
});
@pmuens

This comment has been minimized.

Show comment
Hide comment
@pmuens

pmuens Oct 26, 2016

Member

Thank you very much for this in-depth example @jmdobry! 👏 👍
That's exactly what I need.

I just deployed the first function this morning through the API 💃

Member

pmuens commented Oct 26, 2016

Thank you very much for this in-depth example @jmdobry! 👏 👍
That's exactly what I need.

I just deployed the first function this morning through the API 💃

@pmuens pmuens referenced this issue Oct 29, 2016

Closed

WIP: Add Google Cloud Functions support #2568

8 of 13 tasks complete
@flomotlik

This comment has been minimized.

Show comment
Hide comment
@flomotlik

flomotlik Nov 4, 2016

Contributor

closing here as this is now in a separate PR and will be moved to a separate repo

Contributor

flomotlik commented Nov 4, 2016

closing here as this is now in a separate PR and will be moved to a separate repo

@flomotlik flomotlik closed this Nov 4, 2016

@arycloud

This comment has been minimized.

Show comment
Hide comment
@arycloud

arycloud Nov 20, 2017

I'm also struggling to achieve this thing using python, not success till now!

I'm also struggling to achieve this thing using python, not success till now!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment