Add Serverless Run Plugin #4034

eahefnawy · 2017-08-03T14:12:43Z

What did you implement:

serverless run command that installs and/or downloads the event gateway and the local emulator, spins them up and deploys & registers functions to the local emulator and/or event gateway.

How did you implement it:

We first check whether the local emulator is running, if it is, we deploy functions to the local emulator. We then check whether the event gateway is running, if it is, we register functions/events to the event gateway. If they're not running, we check whether they're installed and install them if not. We then spin them up and deploy and register functions.

How can we verify it:

Basic Example

Clone the basic example from the examples repo
from inside the service directory, run sls run
After functions are deployed/registered, visit localhost:4000/e1. You should see the response from the function.

Todos

Is this ready for review?: YES
Is it a breaking change?: NO

rupakg · 2017-08-03T18:38:37Z

@eahefnawy @pmuens I gave it a spin and I got errors in installing the serverless-local-emulator. Looks like serverless-local-emulator is not yet published to npm.

npm http 404 https://registry.npmjs.org/serverless-local-emulator

Full trace:

root@0555e2cc096a:/home/prjtest/apps/email-service# sls run
 Serverless     |  Installing Local Emulator...
npm info it worked if it ends with ok
npm info using npm@3.10.10
npm info using node@v6.10.3
npm info attempt registry request try #1 at 6:33:19 PM
npm http request GET https://registry.npmjs.org/serverless-local-emulator
npm http 404 https://registry.npmjs.org/serverless-local-emulator
npm ERR! Linux 4.9.31-moby
npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "install" "-g" "serverless-local-emulator"
npm ERR! node v6.10.3
npm ERR! npm  v3.10.10
npm ERR! code E404

npm ERR! 404 Registry returned 404 for GET on https://registry.npmjs.org/serverless-local-emulator
npm ERR! 404
npm ERR! 404  'serverless-local-emulator' is not in the npm registry.
npm ERR! 404 You should bug the author to publish it (or use the name yourself!)
npm ERR! 404
npm ERR! 404 Note that you can also install from a
npm ERR! 404 tarball, folder, http url, or git url.

npm ERR! Please include the following file with any support request:
npm ERR!     /home/prjtest/apps/email-service/npm-debug.log

  Error --------------------------------------------------

  Command failed: npm install -g serverless-local-emulator
npm info it worked if it ends with ok
npm info using npm@3.10.10
npm info using node@v6.10.3
npm info attempt registry request try #1 at 6:33:19 PM
npm http request GET https://registry.npmjs.org/serverless-local-emulator
npm http 404 https://registry.npmjs.org/serverless-local-emulator
npm ERR! Linux 4.9.31-moby
npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "install" "-g" "serverless-local-emulator"
npm ERR! node v6.10.3
npm ERR! npm  v3.10.10
npm ERR! code E404

npm ERR! 404 Registry returned 404 for GET on https://registry.npmjs.org/serverless-local-emulator
npm ERR! 404
npm ERR! 404  'serverless-local-emulator' is not in the npm registry.
npm ERR! 404 You should bug the author to publish it (or use the name yourself!)
npm ERR! 404
npm ERR! 404 Note that you can also install from a
npm ERR! 404 tarball, folder, http url, or git url.

npm ERR! Please include the following file with any support request:
npm ERR!     /home/prjtest/apps/email-service/npm-debug.log


     For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.

  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Forums:        forum.serverless.com
     Chat:          gitter.im/serverless/serverless

  Your Environment Information -----------------------------
     OS:                     linux
     Node Version:           6.10.3
     Serverless Version:     1.19.0

pmuens · 2017-08-03T19:04:42Z

@rupakg thanks for testing! 👍

AFAIK it's now called @serverless/emulator and is already published to npm. @eahefnawy just needs to update the code here (which he already has on his Todo list for this PR AFAIK).

rupakg · 2017-08-03T19:09:15Z

@pmuens Maybe I can do a PR. But, I could not find it on npm.

pmuens · 2017-08-04T07:12:09Z

@pmuens Maybe I can do a PR. But, I could not find it on npm.

Thanks for that @rupakg 👍

🤔 maybe @eahefnawy knows more...

eahefnawy · 2017-08-04T10:53:40Z

local emulator not published yet, gonna publish it today

rupakg · 2017-08-04T19:27:43Z

@eahefnawy @pmuens Any word on resolving this issue?

serverless run makes it easy to use local-emulator and event-gateway altogether with the framework. Thanks for working on it. I am kinda stuck with this issue...

pmuens · 2017-08-05T08:58:29Z

@eahefnawy @pmuens Any word on resolving this issue?

serverless run makes it easy to use local-emulator and event-gateway altogether with the framework. Thanks for working on it. I am kinda stuck with this issue...

@rupakg the setup / config to auto-deploy the @serverless/emulator package to npm via Travis should be in place so I think it should be published in the beginning of the next week.

rupakg · 2017-08-07T16:00:25Z

@eahefnawy I got latest of the branch and tried it on a service I have. I get this error:

npm info ok
 Serverless     |  Installing Event Gateway...
 Serverless     |  Spinning Up the Local Emulator...
 Serverless     |  Spinning Up the Event Gateway...
 Local Emulator |  /usr/local/lib/node_modules/@serverless/emulator/node_modules/koa-bodyparser/index.js:72
  return async function bodyParser(ctx, next) {
               ^^^^^^^^
SyntaxError: Unexpected token function
    at createScript (vm.js:56:10)
    at Object.runInThisContext (vm.js:97:10)
    at Module._compile (module.js:542:28)
    at Object.Module._extensions..js (module.js:579:10)
    at Module.load (module.js:487:32)
    at tryModuleLoad (module.js:446:12)
    at Function.Module._load (module.js:438:3)
    at Module.require (module.js:497:17)
    at require (internal/module.js:20:19)
    at Object.<anonymous> (/usr/local/lib/node_modules/@serverless/emulator/dist/main.js:211:22)
 Event Gateway  |  INFO	Running in development mode with embedded etcd.
 Serverless     |  Registering Functions to the Event Gateway...
 Event Gateway  |  INFO	Function registered.	{"functionId": "email-service-send", "type": "http"}

  Error --------------------------------------------------

  Failed to register at least one function.


     For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.

  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Forums:        forum.serverless.com
     Chat:          gitter.im/serverless/serverless

  Your Environment Information -----------------------------
     OS:                     linux
     Node Version:           6.10.3
     Serverless Version:     1.19.0

Any ideas?

pmuens · 2017-08-08T07:54:06Z

It looks like the node_modules in the local-emulator (koa in that case) are not transpiled correctly so they won't work with early node versions.

/cc @eahefnawy

mthenw · 2017-08-08T16:25:49Z

lib/plugins/run/utils/installEventGateway.js

+const path = require('path');
+
+function installEventGateway(eventGatewayVersion) {
+  let eventGatewayDownloadUrl = `https://github.com/serverless/event-gateway/releases/download/${eventGatewayVersion}/event-gateway_${eventGatewayVersion}_darwin_386.tar.gz`;


can we download amd64 version for 64bit arch?

I think this could do it https://stackoverflow.com/a/37465917/837709

rupakg · 2017-08-08T20:29:10Z

@eahefnawy @pmuens I tried the run command with the latest update, and I am running into this issue:

npm info ok
 Serverless     |  Installing Event Gateway...
 Serverless     |  Spinning Up the Local Emulator...
 Serverless     |  Spinning Up the Event Gateway...
 Event Gateway  |  INFO	Running in development mode with embedded etcd.
 Local Emulator |  Tue, 08 Aug 2017 20:13:55 GMT koa deprecated Support for generators will be removed in v3. See the documentation for examples of how to convert old middleware https://github.com/koajs/koa/blob/master/docs/migration.md at ../../../../../usr/local/lib/node_modules/@serverless/emulator/dist/main.js:14:17
 Local Emulator |  Serverless Local Emulator Daemon listening at localhost:4002...
 Serverless     |  Deploying Functions to Local Emulator...
 Serverless     |  Functions Deployed to the Local Emulator!
 Serverless     |  Registering Functions to the Event Gateway...
 Event Gateway  |  DEBUG	Function registered.	{"functionId": "email-service-send", "type": "http"}

  Error --------------------------------------------------

  Failed to register at least one function.


     For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.

  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Forums:        forum.serverless.com
     Chat:          gitter.im/serverless/serverless

  Your Environment Information -----------------------------
     OS:                     linux
     Node Version:           6.10.3
     Serverless Version:     1.19.0

And, my serverless.yml fragment. It has a forward-slash in path.

functions:
  send:
    handler: handler.sendEmail
    events:
      - http:
          path: /email
          method: post

Highlighting the error:

Event Gateway  |  DEBUG	Function registered.	{"functionId": "email-service-send", "type": "http"}

  Error --------------------------------------------------

  Failed to register at least one function.

Another couple of things from the output:

Can we get rid of this deprecation message?

 Local Emulator |  Tue, 08 Aug 2017 20:13:55 GMT koa deprecated Support for generators will be removed in v3. See the documentation for examples of how to convert old middleware https://github.com/koajs/koa/blob/master/docs/migration.md at ../../../../../usr/local/lib/node_modules/@serverless/emulator/dist/main.js:14:17

I really like how the Local Emulator advertises the port it listen's on. I would like the Event Gateway to do the same. (Issue 246)

 Local Emulator |  Serverless Local Emulator Daemon listening at localhost:4002...

Maybe at the bottom, we can also show version info. for Local Emulator and Event Gateway as well:

  Your Environment Information -----------------------------
     OS:                     linux
     Node Version:           6.10.3
     Serverless Version:     1.19.0

PS: I was not sure if opening issues for these would confuse existing users as the feature is not merged in yet.

nikgraf · 2017-08-09T14:17:35Z

Shout out to @eahefnawy 🙌 The code is clean and easy to work with!

aheissenberger · 2017-09-20T19:47:24Z

I am using "serverless" installed local with the project and would like to do this with the emulator too.
But whenever I call ../../node_modules/.bin/sls run it tries to do a global install and fails on my system as I do not have admin rights:

$ ../../node_modules/.bin/sls run
 Serverless     Installing Emulator
npm WARN checkPermissions Missing write access to /usr/local/lib/node_modules
npm ERR! path /usr/local/lib/node_modules
npm ERR! code EACCES
npm ERR! errno -13
npm ERR! syscall access
npm ERR! Error: EACCES: permission denied, access '/usr/local/lib/node_modules'
npm ERR!  { Error: EACCES: permission denied, access '/usr/local/lib/node_modules'
npm ERR!   stack: 'Error: EACCES: permission denied, access \'/usr/local/lib/node_modules\'',
npm ERR!   errno: -13,
npm ERR!   code: 'EACCES',
npm ERR!   syscall: 'access',
npm ERR!   path: '/usr/local/lib/node_modules' }
npm ERR! 
npm ERR! Please try running this command again as root/Administrator.

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/ah/.npm/_logs/2017-09-20T19_38_14_739Z-debug.log
 
  Error --------------------------------------------------
 
  Command failed: npm install -g @serverless/emulator@0.1.18
npm WARN checkPermissions Missing write access to /usr/local/lib/node_modules
npm ERR! path /usr/local/lib/node_modules
npm ERR! code EACCES
npm ERR! errno -13
npm ERR! syscall access
npm ERR! Error: EACCES: permission denied, access '/usr/local/lib/node_modules'
npm ERR!  { Error: EACCES: permission denied, access '/usr/local/lib/node_modules'
npm ERR!   stack: 'Error: EACCES: permission denied, access \'/usr/local/lib/node_modules\'',
npm ERR!   errno: -13,
npm ERR!   code: 'EACCES',
npm ERR!   syscall: 'access',
npm ERR!   path: '/usr/local/lib/node_modules' }
npm ERR! 
npm ERR! Please try running this command again as root/Administrator.

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/ah/.npm/_logs/2017-09-20T19_38_14_739Z-debug.log

 
     For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.
 
  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Forums:        forum.serverless.com
     Chat:          gitter.im/serverless/serverless
 
  Your Environment Information -----------------------------
     OS:                     darwin
     Node Version:           8.5.0
     Serverless Version:     1.22.0

packages.json

"devDependencies": {
   "serverless": "^1.22.0",
    "@serverless/emulator": "0.1.18"
  }

pmuens · 2017-09-21T06:39:22Z

@aheissenberger thanks for commenting 👍

🤔 nice catch! It looks like the Emulator and Event Gateway support is only available for global installs (but local Serverless Framework installs are also pretty common).

Could you please open up a separate issue so that we can track and discuss this in isolation? Thanks in advance!

eahefnawy added 6 commits August 3, 2017 20:40

first draft of serverless run

52f743b

connect with local emulator

997d458

local emulator updates

6443bf3

serverless run: refactoring

a63ee7a

installing latest eventgateway

da91c5a

updated FDK

564abd8

pmuens added the stage/in-progress label Aug 3, 2017

pmuens self-requested a review August 3, 2017 15:28

updated http events

670b955

eahefnawy and others added 3 commits August 8, 2017 20:19

added temp dirs for event gateway

a811534

Merge branch 'master' into serverless-run

9e45094

fix linting error

bdde712

mthenw reviewed Aug 8, 2017

View reviewed changes

upgrade fdk to the latest version

1c0da07

nikgraf and others added 3 commits August 9, 2017 12:59

Merge branch 'master' into serverless-run

213fb74

introduce logEventGateway to prepare for upcoming log format

41c01b6

install local emulator if functions not deployed

e84bc39

eahefnawy and others added 2 commits August 10, 2017 17:31

update eventgateway and local emulator

824a2dc

improve logging

8730645

nikgraf and others added 21 commits August 14, 2017 18:21

fix test-suite by disabling color matching

2bcd836

simplify trigger function

b7aa82c

minor tweaks to formatting

cfd6b21

minor tweak to output styling

09a8d4f

Add separation between platform and framework commands

91d746d

serverless run ui improvements

fca3015

serverless run ui modifications

35499a5

minor indentation tweak

1720e2d

run without functions and lock versions

e176dc8

Merge branch 'master' into serverless-run

909fae7

fix tests

2c722c2

Merge branch 'master' into serverless-run

c7e972c

updated lockfile

082f1bd

Fix Node.js 4.x compatibility

8fc34f6

color emulator errors

699b9bc

Update to use new Emulator and Event Gateway versions

73953d0

Fix noFunctions detection bug

111dea3

Add detection of processor arch for Event Gateway binary downloads

23cff7f

add event gateway config info to logs

c0d53b2

remove unnecessary console log

94318ae

upgrade to latest event-gateway

b1ae5a5

pmuens approved these changes Aug 16, 2017

View reviewed changes

pmuens added stage/in-review and removed stage/in-progress labels Aug 16, 2017

pmuens added this to the 1.20 milestone Aug 16, 2017

pmuens merged commit 5cb791f into master Aug 16, 2017

pmuens deleted the serverless-run branch August 16, 2017 15:23

aheissenberger mentioned this pull request Sep 29, 2017

local install / no admin rights / sls run fails because it tries a global install of the emulator serverless/emulator#133

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Serverless Run Plugin #4034

Add Serverless Run Plugin #4034

eahefnawy commented Aug 3, 2017 •

edited

rupakg commented Aug 3, 2017 •

edited

pmuens commented Aug 3, 2017

rupakg commented Aug 3, 2017 •

edited

pmuens commented Aug 4, 2017

eahefnawy commented Aug 4, 2017

rupakg commented Aug 4, 2017

pmuens commented Aug 5, 2017 •

edited

rupakg commented Aug 7, 2017

pmuens commented Aug 8, 2017

mthenw Aug 8, 2017

nikgraf Aug 8, 2017

rupakg commented Aug 8, 2017 •

edited

nikgraf commented Aug 9, 2017

aheissenberger commented Sep 20, 2017

pmuens commented Sep 21, 2017 •

edited

Add Serverless Run Plugin #4034

Add Serverless Run Plugin #4034

Conversation

eahefnawy commented Aug 3, 2017 • edited

What did you implement:

How did you implement it:

How can we verify it:

Basic Example

Todos

rupakg commented Aug 3, 2017 • edited

pmuens commented Aug 3, 2017

rupakg commented Aug 3, 2017 • edited

pmuens commented Aug 4, 2017

eahefnawy commented Aug 4, 2017

rupakg commented Aug 4, 2017

pmuens commented Aug 5, 2017 • edited

rupakg commented Aug 7, 2017

pmuens commented Aug 8, 2017

mthenw Aug 8, 2017

Choose a reason for hiding this comment

nikgraf Aug 8, 2017

Choose a reason for hiding this comment

rupakg commented Aug 8, 2017 • edited

nikgraf commented Aug 9, 2017

aheissenberger commented Sep 20, 2017

pmuens commented Sep 21, 2017 • edited

eahefnawy commented Aug 3, 2017 •

edited

rupakg commented Aug 3, 2017 •

edited

rupakg commented Aug 3, 2017 •

edited

pmuens commented Aug 5, 2017 •

edited

rupakg commented Aug 8, 2017 •

edited

pmuens commented Sep 21, 2017 •

edited