AWS profile cannot be loaded when the `--useWorkerThreads` option is used #1430

MiyamotoTa · 2022-05-18T08:32:26Z

Bug Report

Current Behavior

When the --useWorkerThreads option is enabled, the value of AWS_PROFILE cannot be referenced and valid credentials cannot be retrieved. Therefore, the following error occurs.

❯ AWS_PROFILE=myprofile npx serverless offline --useWorkerThreads

Starting Offline at stage dev (us-east-1)

Offline [http for lambda] listening on http://localhost:3002
Function names exposed for local invocation by aws-sdk:
           * hello: serverless-offline-credentials-dev-hello

   ┌─────────────────────────────────────────────────────────────────────────┐
   │                                                                         │
   │   POST | http://localhost:3000/dev/hello                                │
   │   POST | http://localhost:3000/2015-03-31/functions/hello/invocations   │
   │                                                                         │
   └─────────────────────────────────────────────────────────────────────────┘

Server ready: http://localhost:3000 🚀

Enter "rp" to replay the last request


POST /dev/hello (λ: hello)
✖ Uncaught exception
Environment: darwin, node 14.18.0, framework 3.17.0 (local), plugin 6.2.2, SDK 4.3.2
Docs:        docs.serverless.com
Support:     forum.serverless.com
Bugs:        github.com/serverless/serverless/issues

Error:
/Users/tak/projects/repro-sls-offline-aws_profile/node_modules/@aws-sdk/credential-provider-node/dist-cjs/defaultProvider.js:13
    throw new property_provider_1.CredentialsProviderError("Could not load credentials from any providers", false);
          ^

CredentialsProviderError: Could not load credentials from any providers
    at null.<anonymous> (/Users/tak/projects/repro-sls-offline-aws_profile/node_modules/@aws-sdk/credential-provider-node/dist-cjs/defaultProvider.js:13:11)
    at null.<anonymous> (/Users/tak/projects/repro-sls-offline-aws_profile/node_modules/@aws-sdk/property-provider/dist-cjs/chain.js:11:28)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
    at null.coalesceProvider (/Users/tak/projects/repro-sls-offline-aws_profile/node_modules/@aws-sdk/property-provider/dist-cjs/memoize.js:14:24)
    at SignatureV4.credentialProvider (/Users/tak/projects/repro-sls-offline-aws_profile/node_modules/@aws-sdk/property-provider/dist-cjs/memoize.js:33:24)
    at SignatureV4.signRequest (/Users/tak/projects/repro-sls-offline-aws_profile/node_modules/@aws-sdk/signature-v4/dist-cjs/SignatureV4.js:84:29)
    at null.<anonymous> (/Users/tak/projects/repro-sls-offline-aws_profile/node_modules/@aws-sdk/middleware-signing/dist-cjs/middleware.js:13:18)
    at StandardRetryStrategy.retry (/Users/tak/projects/repro-sls-offline-aws_profile/node_modules/@aws-sdk/middleware-retry/dist-cjs/StandardRetryStrategy.js:51:46)
    at null.<anonymous> (/Users/tak/projects/repro-sls-offline-aws_profile/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:6:22)
    at hello (/Users/tak/projects/repro-sls-offline-aws_profile/src/functions/hello/handler.ts:29:16)

Sample Code

Please prepare a profile that can run AWS Comprehend as a preliminary step.
(I've not checked other AWS clients, but I expect this to occur for other than Comprehend.)

https://github.com/MiyamotoTa/repro-sls-offline-aws_profile

Check out the above repository and execute the following command and send the request.
Also, the client's region is hard-coded, so please change it if necessary.

❯ npx serverless offline --useWorkerThreads

❯ curl -X POST -H "Content-Type: application/json" -d '{"phrase":"How do I use AWS comprehend?"}' localhost:3000/dev/hello

Expected behavior/code

As with the case without the --useWorkerThreads option, a valid credential should be obtained and no errors should occur.

Environment

serverless version: 3.0.0
serverless-offline version: 8.8.0
node.js version: 14.18.0
OS: macOS 12.3.1

Possible Solution

N/A

Additional context/Screenshots

N/A

The text was updated successfully, but these errors were encountered:

dnalborczyk · 2022-06-30T02:01:56Z

thank you for the repro @MiyamotoTa ! I'll try to have a look soon. does this apply to worker threads only?

MiyamotoTa · 2022-06-30T04:14:02Z

@dnalborczyk Yeah, it worked removing that option.

dnalborczyk · 2022-07-01T01:37:01Z

looking into this again, I remember now that using worker threads is closer to the behavior of lambda functions ("containers") within node.js. when you run serverless-offline in-process (the default in v8.8.0). process.env is being shared across functions as well as with serverless and serverless-offline. that's not the case with worker threads. we should probably point out the differences between all run-modes in the documentation.

in order for you to use a local env variable with worker threads you would have to pass it explicitly to your function, in the same way you would have to do with just serverless, either at the provider level or the function level:

provider:
    # either at the provider level
    environment:
        FOO: ${env:FOO}

functions:
    foo:
        # or the function level
        environment:
            FOO: ${env:FOO}

edit, fixed example to make it more clear to lift the local env variable

MiyamotoTa · 2022-07-02T13:11:23Z

@dnalborczyk Great, I'll try it the way.

dnalborczyk · 2022-07-05T20:50:04Z

added some additional tests @MiyamotoTa b5559cc#diff-6f01d26e6c42c1c26923cd15ccdfd64a3054931dfbd7b8f957f6dcf2b41dfed3R1

we could potentially copy over the env from the main process, but not sure if that's desired or helpful? if we do that, it should be done only behind a flag.

nicwise · 2022-07-05T23:13:34Z

Hi. I'm running into the same problem, but I can't solve it this way.

We use aws-vault and AWS SSO, so we have NO creds on the machine, ever, except temporary session creds in environment variables.

I noticed that worker threads were going to be default, and we were having trouble with child processes, so thought it was a good time to switch. (that one is in InvocationController where it's trying to split on an undefined variable if you return a non-success from the lambda. Old bug.... :( )

Now I get this error:

  Error: connect EHOSTDOWN 169.254.169.254:80 - Local (10.50.10.141:51048)
[service]         at internalConnect (node:net:953:16)
[service]         at defaultTriggerAsyncIdScope (node:internal/async_hooks:465:18)
[service]         at node:net:1044:9
[service]         at processTicksAndRejections (node:internal/process/task_queues:78:11) {
[service]       message: 'Missing credentials in config, if using AWS_CONFIG_FILE, set AWS_SDK_LOAD_CONFIG=1',
[service]       locations: [Array],
[service]       path: [Array]
[service]     }
[service]   ],
[service]   errorCode: undefined,
[service]   errorStrings: 'Missing credentials in config, if using AWS_CONFIG_FILE, set AWS_SDK_LOAD_CONFIG=1',

This is a false error, kinda. Its not found AWS creds in the environment (sound familiar?) so it's looking for the ECS/EC2 endpoint, and not finding it.

I can sort of fake that by using aws-vault exec xxx --server -- but thats problematic for scripting as I have 8 serverless projects, and I can only run one --server per machine.

The problem, I guess, is that you're not copying environment variables from the main process to the worker thread, as per:

we could potentially copy over the env from the main process, but not sure if that's desired or helpful? if we do that, it should be done only behind a flag.

So, if you're not doing that, then anyone using ONLY AWS environment variables will get this error, and not be able to use SLS with worker threads (and these days if you have creds in ~/.aws/credentials you are a security risk. Don't do that.)

Hope that helps @dnalborczyk

nicwise · 2022-07-05T23:16:28Z

If you wanted to limit the items you are copying over, I think these should be enough, tho AWS_* might also cover it

AWS_DEFAULT_REGION
AWS_REGION
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_SESSION_TOKEN
AWS_SECURITY_TOKEN
AWS_SESSION_EXPIRATION

dnalborczyk · 2022-07-07T13:51:47Z

Hi. I'm running into the same problem, but I can't solve it this way.

We use aws-vault and AWS SSO, so we have NO creds on the machine, ever, except temporary session creds in environment variables.

I assume you are using this tool: https://github.com/99designs/aws-vault ? I have not used it myself yet.

I noticed that worker threads were going to be default, and we were having trouble with child processes, so thought it was a good time to switch. (that one is in InvocationController where it's trying to split on an undefined variable if you return a non-success from the lambda. Old bug.... :( )

unfortunately I'm not sure what you are referring to, could you open a new issue for this?

Now I get this error:

  Error: connect EHOSTDOWN 169.254.169.254:80 - Local (10.50.10.141:51048)
[service]         at internalConnect (node:net:953:16)
[service]         at defaultTriggerAsyncIdScope (node:internal/async_hooks:465:18)
[service]         at node:net:1044:9
[service]         at processTicksAndRejections (node:internal/process/task_queues:78:11) {
[service]       message: 'Missing credentials in config, if using AWS_CONFIG_FILE, set AWS_SDK_LOAD_CONFIG=1',
[service]       locations: [Array],
[service]       path: [Array]
[service]     }
[service]   ],
[service]   errorCode: undefined,
[service]   errorStrings: 'Missing credentials in config, if using AWS_CONFIG_FILE, set AWS_SDK_LOAD_CONFIG=1',

is this the aws-sdk throwing?

This is a false error, kinda. Its not found AWS creds in the environment (sound familiar?) so it's looking for the ECS/EC2 endpoint, and not finding it.

I can sort of fake that by using aws-vault exec xxx --server -- but thats problematic for scripting as I have 8 serverless projects, and I can only run one --server per machine.

The problem, I guess, is that you're not copying environment variables from the main process to the worker thread, as per:

the intention of this is that the "lambda" is encapsulated like the real container lambda.

currently, when you deploy with serverless, how are your AWS creds created and consumed on the AWS side-of-things?

just noticed, that my example from above wasn't quite clear:

// cc @MiyamotoTa

this "lifts" any local env vars to serverless and in turn serverless-offline (worker-thead):

provider:
    # either at the provider level
    environment:
        FOO: ${env:FOO}

functions:
    foo:
        # or the function level
        environment:
            FOO: ${env:FOO}

we could potentially copy over the env from the main process, but not sure if that's desired or helpful? if we do that, it should be done only behind a flag.

So, if you're not doing that, then anyone using ONLY AWS environment variables will get this error, and not be able to use SLS with worker threads (and these days if you have creds in ~/.aws/credentials you are a security risk. Don't do that.)

Hope that helps @dnalborczyk

it always helps to hear other opinions 😉

dnalborczyk · 2022-07-07T14:05:21Z

If you wanted to limit the items you are copying over, I think these should be enough, tho AWS_* might also cover it
AWS_DEFAULT_REGION
AWS_REGION
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_SESSION_TOKEN
AWS_SECURITY_TOKEN
AWS_SESSION_EXPIRATION

if we still think this is needed, we should just copy (clone) over everything, with a specific order of precedence. (clone -> provider -> function).

ZebraFlesh · 2022-07-23T03:54:31Z

Copying over the AWS_* environment variables would be a great solution here. I've had to resort to the --useInProcess flag to successfully upgrade to v9. (We're heavily leveraging CloudMap for service discovery, which requires credentials, which can be supplied in a myriad of ways -- so I can't generically say "copy the AWS_PROFILE value" because not everyone uses profiles.)

dnalborczyk · 2022-07-23T17:50:48Z

labeling as feature. I can see the need for this. Tho I think we should copy over everything instead of just restricting it to AWS_ values. I also think this should be implemented behind a flag, and be done for worker threads and possibly docker. in-process picks local envs up automatically, I have to double check how child process is handling local envs.

dnalborczyk · 2022-07-25T02:04:58Z

added with: #1513

dnalborczyk · 2022-07-27T18:18:45Z

what I ended up implementing is the automatic copying of any AWS_xxx environment variables (without the need for a flag) as well as a new option --localEnvironment to copy all environment variables. it's released as part of v9.1.

please let me know if that fixes your issue (ideally without the flag).

dnalborczyk self-assigned this Jun 30, 2022

dnalborczyk added the question label Jul 18, 2022

dnalborczyk added feature and removed question labels Jul 23, 2022

This was referenced Jul 23, 2022

8.8.1 -> 9.0.0.. AWS config environment variables no longer available #1510

Closed

feat: add local environment variables flag #1513

Merged

dnalborczyk closed this as completed Jul 25, 2022

rizowski mentioned this issue Jul 26, 2022

Missing AWS_PROFILE when running with new worker threads #1516

Closed

aq17 mentioned this issue Nov 14, 2022

Error validating resource with policy mfa-enabled-for-iam-console-access pulumi/pulumi-policy-aws#91

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

AWS profile cannot be loaded when the `--useWorkerThreads` option is used #1430

AWS profile cannot be loaded when the `--useWorkerThreads` option is used #1430

MiyamotoTa commented May 18, 2022

dnalborczyk commented Jun 30, 2022

MiyamotoTa commented Jun 30, 2022

dnalborczyk commented Jul 1, 2022 •

edited

Loading

MiyamotoTa commented Jul 2, 2022

dnalborczyk commented Jul 5, 2022

nicwise commented Jul 5, 2022

nicwise commented Jul 5, 2022

dnalborczyk commented Jul 7, 2022 •

edited

Loading

dnalborczyk commented Jul 7, 2022

ZebraFlesh commented Jul 23, 2022

dnalborczyk commented Jul 23, 2022

dnalborczyk commented Jul 25, 2022

dnalborczyk commented Jul 27, 2022

AWS profile cannot be loaded when the --useWorkerThreads option is used #1430

AWS profile cannot be loaded when the --useWorkerThreads option is used #1430

Comments

MiyamotoTa commented May 18, 2022

Bug Report

dnalborczyk commented Jun 30, 2022

MiyamotoTa commented Jun 30, 2022

dnalborczyk commented Jul 1, 2022 • edited Loading

MiyamotoTa commented Jul 2, 2022

dnalborczyk commented Jul 5, 2022

nicwise commented Jul 5, 2022

nicwise commented Jul 5, 2022

dnalborczyk commented Jul 7, 2022 • edited Loading

dnalborczyk commented Jul 7, 2022

ZebraFlesh commented Jul 23, 2022

dnalborczyk commented Jul 23, 2022

dnalborczyk commented Jul 25, 2022

dnalborczyk commented Jul 27, 2022

AWS profile cannot be loaded when the `--useWorkerThreads` option is used #1430

AWS profile cannot be loaded when the `--useWorkerThreads` option is used #1430

dnalborczyk commented Jul 1, 2022 •

edited

Loading

dnalborczyk commented Jul 7, 2022 •

edited

Loading