Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HttpEndpoint fails when importing other node packages (TypeScript) #249

Closed
rgwood opened this issue Jun 24, 2018 · 10 comments

Comments

@rgwood
Copy link

commented Jun 24, 2018

I'm trying to get started with Pulumi by creating a very simple REST API using TypeScript and pulumi-aws, but am running into some issues. My repro code is available here: https://github.com/rgwood/pulumi-import-repro-ts

I've started with a pretty simple architecture based on the pulumi new typescript template:
my business logic in lambda.ts, plus Pulumi code in index.ts (which imports lambda.ts and calls the exported function).

It all works great until I import a few Node packages and use them in lambda.ts. I'm using qs to do some query string parsing, and uuid to generate a unique ID.

At first, I tried importing lambda.ts in index.ts like so: import * as lambda from "./lambda";. However, this fails when running pulumi update with the following error:

Diagnostics:
  global: global
    error: Error serializing '(ev, ctx, cb) => { let body; ...': api.js(234,106)

    '(ev, ctx, cb) => { let body; ...': api.js(234,106): captured
      variable 'route' which indirectly referenced
        '(req, res) => { // this // const lam ...': index.js(6,18): which captured
          module './bin/lambda.js' which indirectly referenced
            function 'testFunc': lambda.js(5,17): which captured
              module './node_modules/uuid/index.js' which indirectly referenced
                function 'v4': v4.js(4,11): which captured
                  module './node_modules/uuid/lib/rng.js' which indirectly referenced
                    function 'nodeRNG': rng.js(6,33): which captured
                      module 'crypto' which indirectly referenced
                        function 'randomBytes': random.js(52,20): which referenced
                          function 'assertSize': random.js(33,19): which captured
                            'ERR_INVALID_ARG_TYPE', a function defined at
                              function 'NodeError': errors.js(156,15): which referenced
                                function 'getMessage': errors.js(208,19): which captured
                                  variable 'messages' which indirectly referenced
                                    function 'get': which could not be serialized because
                                      it was a native code function.

    Function code:
      function get() { [native code] }

    Capturing modules can sometimes cause problems.
    Consider using import('./bin/lambda.js') or require('./bin/lambda.js') inside '(req, res) => { // this // const lam ...': index.js(6,18)

    error: an unhandled error occurred: Program exited with non-zero exit code: 1

As suggested, I tried requiring the compiled JavaScript using require('./bin/lambda.js') but then that fails at runtime because it cannot load qs for some reason. From my pulumi logs output:

hello-world4c238266] Error: Cannot find module 'qs'
    at Function.Module._resolveFilename (module.js:547:15)
    at Function.Module._load (module.js:474:25)
    at Module.require (module.js:596:17)
    at require (internal/module.js:11:18)
    at Object.<anonymous> (/var/task/bin/lambda.js:3:12)
    at Module._compile (module.js:652:30)
    at Object.Module._extensions..js (module.js:663:10)
    at Module.load (module.js:565:32)
    at tryModuleLoad (module.js:505:12)
    at Function.Module._load (module.js:497:3)

I'm probably doing something incorrect here, but am not sure what. Any help (or a pointer to example code that successfully imports+uses other Node libraries in an HttpEndpoint) would be greatly appreciated.

Thanks for your time, and Pulumi looks really neat! I'm excited to start using it.

@rgwood

This comment has been minimized.

Copy link
Author

commented Jun 30, 2018

Oh, and I should probably mention that I was using Pulumi v0.14 on macOS. I've just tried again using v0.14.1 on Windows, and now both the import and require approaches fail at runtime with the Cannot find module 'qs' error. (edit: I may have been forgetting to build, disregard my comments about v0.14.1)

Now that I understand Pulumi a bit better, it seems like the problem is just that node dependencies are not being deployed with the Lambda function. I've looked through the documentation but I don't know how to make that happen - any suggestions would be much appreciated.

@joeduffy

This comment has been minimized.

Copy link
Member

commented Jun 30, 2018

@CyrusNajmabadi @lukehoban Any idea what's wrong here?

@joeduffy

This comment has been minimized.

Copy link
Member

commented Jun 30, 2018

@rgwood It appears we aren't automatically detecting the qs and uuid dependencies. Not sure why, since our automatic capture analysis should have caught this. I filed pulumi/pulumi#1588 to track down root causing and fixing this.

In the meantime, you can fix this issue by running

$ pulumi config set cloud-aws:functionIncludePackages qs,uuid

I have verified locally using your repro that this will work, and submitted a PR to your repro repo.

@joeduffy joeduffy closed this Jun 30, 2018

@rgwood

This comment has been minimized.

Copy link
Author

commented Jun 30, 2018

Fantastic, thanks! I can confirm that the cloud-aws:functionIncludePackages config solves the issue for now.

By the way, is requiring the compiled JS output like require('./bin/lambda'); the recommended approach for multi-file TypeScript projects like this? Not the end of the world but it feels a little awkward to reference compiled output instead of importing the Typescript code like import * as lambda from "./lambda"; (when I do that I still get a serialization error).

@joeduffy

This comment has been minimized.

Copy link
Member

commented Jun 30, 2018

No, we should be letting you do import * as lambda from "./lambda", that's definitely the preferred, idiomatic ES6 style. I'm going to move discussion over to pulumi/pulumi#1588 because I think all of this is actually related ...

@CyrusNajmabadi

This comment has been minimized.

Copy link
Contributor

commented Jun 30, 2018

Taking a look. :)

@CyrusNajmabadi

This comment has been minimized.

Copy link
Contributor

commented Jun 30, 2018

Ok. So this is definitely an area that can cause confusion and trouble (as you've run into yourself). I will let @lukehoban have the final say here as he has recently made some changes in how modules work. However, i'm fairly confident in saying that the appropriate way to code things up here is as follows:

  1. When referencing your own code, or referencing modules that you expect to use at deployment time (i.e. the code that will execute when run pulumi update), use normal 'imports' outside of your functions. i.e.:

index.ts:

import * as pulumi from "@pulumi/pulumi";
import * as cloud from "@pulumi/cloud-aws";

import * as lambda from "./lambda";

let endpoint = new cloud.HttpEndpoint("hello-world");

endpoint.get("/", async (req, res) => {
    await lambda.testFunc(req, res);
});

exports.endpoint = endpoint.publish().url;

So the right thing here is to have your 'lambda' import outside the function.

  1. For functions you expect to run inside the cloud at runtime (i.e. things that will run in AWS in response to certain events, like 'testFunc') then you should require/await import the modules those function need inside of them. i.e.:
export async function testFunc(req: Request, res: Response): Promise<void> {
    const qs = await import("qs");
    const uuid = await import("uuid")

    console.log('in testFunc');

    let random = uuid.v4();
    console.log(`random: ${random}`);

    let qsResult = qs.parse('hello=world');
    res.status(200).json({qs: qsResult, rand: random});
}

I have tested this and this all works fine. My next post will try to clarify what's going on and explain why the above pattern is appropriate given how pulumi works.

@CyrusNajmabadi

This comment has been minimized.

Copy link
Contributor

commented Jun 30, 2018

Ok: deeper dive into what's going on. When you write a TS/JS function that is converted by 'pulumi' into an AWS-lambda, we have to do a deep analysis on your code to figure out what's going on, so we can best convert this function-object (which is actually created while you're running pulumi update) into something suitable to be executed an some arbitrary point in the future by AWS.

One of the things we do is we actually introspect this function and we see all the values it 'captures' from outside of itself. i.e. if you have:

var x = GetX();
endpoint.get("/", () => { console.log(x); });  //<-- you're capturing 'x' here.

In order to make this work we need to actually grab the value of 'x' and 'serialize' it into some form so that later on when your AWS lambda runs, we can inject that value so it can be used by that 'console.log' call.

--

Now, during the development of pulumi, we went through a lot of designs on what it would mean to 'capture' a module. i.e. if you had code like you wrote:

import * as qs from "qs";

endpoint.get("/", () => { /* use qs here */ });

Without digressing too much, i'll just say that many of the approaches we took turned out to have significant issues associated with themselves. So we finally settled on a simpler (but sometimes more awkward) solution to things. Instead of your code actually importing a module 'outside' the function-to-serialize. You should instead just import the module (using require or await import) directly inside your function.

This has two benefits:

  1. it sidesteps the question of serialization entirely. Because this is not a value that is 'captured' by the function, we don't need to even ask the question 'how do we serialize this value such that you can use it at cloud-runtime?'.
  2. it more accurately represents the logic of what is going to happen at runtime. Specifically, having an outside-import makes it certainly feel like "oh... this module is just going to be imported once... and i can use that single instance over all calls to this function at runtime". But that's not really what's going on. Each time your http-endpoint is hit, that callback you provided may be loaded fresh, and will have to import the module. Placing the import 'inside' the function helps make it clearer this work that will be done at runtime.

--

I'm happy to expand and clarify anything you'd like here. I don't want to overload too much with a huge dump of information. So please let me know if this made sense and has helped get you through this hump. Otherwise, feel free to pester me more with questions, and i'll def provide whatever additional detail i can!

Thanks!

@rgwood

This comment has been minimized.

Copy link
Author

commented Jun 30, 2018

That makes a lot of sense, thank you very much for the detailed reply! This should be enough for me to finish converting some of my Serverless Framework functions to Pulumi.

@CyrusNajmabadi

This comment has been minimized.

Copy link
Contributor

commented Jun 30, 2018

@rgwood That's great to hear. I looked and couldn't find any easily available documentation on ths topic. So it's something we def need to make and help explain to people. It's definitely a place you can trip up, and that's never a good thing!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.