Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Serverless null error when using SSM variables that not exist in region #5110

Closed
crash7 opened this issue Jul 10, 2018 · 18 comments
Closed

Serverless null error when using SSM variables that not exist in region #5110

crash7 opened this issue Jul 10, 2018 · 18 comments

Comments

@crash7
Copy link
Contributor

crash7 commented Jul 10, 2018

This is a Bug Report

Description

Serverless throws an error when using SSM variables that not exist in the AWS region.

For bug reports:

  • What went wrong?
    When the variable doesn't exists in the configured region, I get a null error in serverless
$ /Users/chris/prj/fb/lambda-service/node_modules/.bin/sls package --environment development

  Serverless Error ---------------------------------------


  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Issues:        forum.serverless.com

  Your Environment Information -----------------------------
     OS:                     darwin
     Node Version:           8.10.0
     Serverless Version:     1.28.0

error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
  • What did you expect should have happened?
    sls script runs normally

  • What was the config you used?

service: service
provider:
  name: aws
  runtime: nodejs8.10
  region: ${file(./config/${opt:environment}.json):aws.region}
  stage: ${opt:environment}
  profile: myprofile
  memorySize: 128
  timeout: 30
  environment:
    MY_VARIABLE: ${ssm:/myvariable, '${file(./config/${opt:environment}.json):myvariable}'}
  • What stacktrace or error message from your provider did you see?
    I got this after adding a console log in getValueFromSsm, seems that the promise is rejected because something changed in the error message expected in this case
      .catch((err) => {
        // console.log(err);
        const expectedErrorMessage = `Parameter ${param} not found.`;
        if (err.message !== expectedErrorMessage) {
          return BbPromise.reject(new this.serverless.classes.Error(err.message));
        }
        return BbPromise.resolve(undefined);
      });
{ ServerlessError: Parameter /myvariable not found.
    at BbPromise.fromCallback.catch.err (/Users/chris/prj/fb/lambda-service/node_modules/serverless/lib/plugins/aws/provider/awsProvider.js:270:11)
    at tryCatcher (/Users/chris/prj/fb/lambda-service/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/Users/chris/prj/fb/lambda-service/node_modules/bluebird/js/release/promise.js:512:31)
    at Promise._settlePromise (/Users/chris/prj/fb/lambda-service/node_modules/bluebird/js/release/promise.js:569:18)
    at Promise._settlePromise0 (/Users/chris/prj/fb/lambda-service/node_modules/bluebird/js/release/promise.js:614:10)
    at Promise._settlePromises (/Users/chris/prj/fb/lambda-service/node_modules/bluebird/js/release/promise.js:689:18)
    at Async._drainQueue (/Users/chris/prj/fb/lambda-service/node_modules/bluebird/js/release/async.js:133:16)
    at Async._drainQueues (/Users/chris/prj/fb/lambda-service/node_modules/bluebird/js/release/async.js:143:10)
    at Immediate.Async.drainQueues (/Users/chris/prj/fb/lambda-service/node_modules/bluebird/js/release/async.js:17:14)
    at runCallback (timers.js:794:20)
    at tryOnImmediate (timers.js:752:5)
    at processImmediate [as _immediateCallback] (timers.js:729:5)
  name: 'ServerlessError',
  statusCode: 400,
  providerError:
   { ParameterNotFound: Parameter /myvariable not found.
    at Request.extractError (/Users/chris/prj/fb/lambda-service/node_modules/serverless/node_modules/aws-sdk/lib/protocol/json.js:48:27)
    at Request.callListeners (/Users/chris/prj/fb/lambda-service/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:105:20)
    at Request.emit (/Users/chris/prj/fb/lambda-service/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
    at Request.emit (/Users/chris/prj/fb/lambda-service/node_modules/serverless/node_modules/aws-sdk/lib/request.js:683:14)
    at Request.transition (/Users/chris/prj/fb/lambda-service/node_modules/serverless/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/Users/chris/prj/fb/lambda-service/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /Users/chris/prj/fb/lambda-service/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/Users/chris/prj/fb/lambda-service/node_modules/serverless/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/Users/chris/prj/fb/lambda-service/node_modules/serverless/node_modules/aws-sdk/lib/request.js:685:12)
    at Request.callListeners (/Users/chris/prj/fb/lambda-service/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:115:18)
    at Request.emit (/Users/chris/prj/fb/lambda-service/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
    at Request.emit (/Users/chris/prj/fb/lambda-service/node_modules/serverless/node_modules/aws-sdk/lib/request.js:683:14)
    at Request.transition (/Users/chris/prj/fb/lambda-service/node_modules/serverless/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/Users/chris/prj/fb/lambda-service/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /Users/chris/prj/fb/lambda-service/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/Users/chris/prj/fb/lambda-service/node_modules/serverless/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/Users/chris/prj/fb/lambda-service/node_modules/serverless/node_modules/aws-sdk/lib/request.js:685:12)
    at Request.callListeners (/Users/chris/prj/fb/lambda-service/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:115:18)
    at callNextListener (/Users/chris/prj/fb/lambda-service/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:95:12)
    at IncomingMessage.onEnd (/Users/chris/prj/fb/lambda-service/node_modules/serverless/node_modules/aws-sdk/lib/event_listeners.js:269:13)
    at emitNone (events.js:111:20)
    at IncomingMessage.emit (events.js:208:7)
    at endReadableNT (_stream_readable.js:1064:12)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickDomainCallback (internal/process/next_tick.js:218:9)
     cause:
      { ParameterNotFound: Parameter /myvariable not found.
    at Request.extractError (/Users/chris/prj/fb/lambda-service/node_modules/serverless/node_modules/aws-sdk/lib/protocol/json.js:48:27)
    at Request.callListeners (/Users/chris/prj/fb/lambda-service/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:105:20)
    at Request.emit (/Users/chris/prj/fb/lambda-service/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
    at Request.emit (/Users/chris/prj/fb/lambda-service/node_modules/serverless/node_modules/aws-sdk/lib/request.js:683:14)
    at Request.transition (/Users/chris/prj/fb/lambda-service/node_modules/serverless/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/Users/chris/prj/fb/lambda-service/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /Users/chris/prj/fb/lambda-service/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/Users/chris/prj/fb/lambda-service/node_modules/serverless/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/Users/chris/prj/fb/lambda-service/node_modules/serverless/node_modules/aws-sdk/lib/request.js:685:12)
    at Request.callListeners (/Users/chris/prj/fb/lambda-service/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:115:18)
    at Request.emit (/Users/chris/prj/fb/lambda-service/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
    at Request.emit (/Users/chris/prj/fb/lambda-service/node_modules/serverless/node_modules/aws-sdk/lib/request.js:683:14)
    at Request.transition (/Users/chris/prj/fb/lambda-service/node_modules/serverless/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/Users/chris/prj/fb/lambda-service/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /Users/chris/prj/fb/lambda-service/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/Users/chris/prj/fb/lambda-service/node_modules/serverless/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/Users/chris/prj/fb/lambda-service/node_modules/serverless/node_modules/aws-sdk/lib/request.js:685:12)
    at Request.callListeners (/Users/chris/prj/fb/lambda-service/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:115:18)
    at callNextListener (/Users/chris/prj/fb/lambda-service/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:95:12)
    at IncomingMessage.onEnd (/Users/chris/prj/fb/lambda-service/node_modules/serverless/node_modules/aws-sdk/lib/event_listeners.js:269:13)
    at emitNone (events.js:111:20)
    at IncomingMessage.emit (events.js:208:7)
    at endReadableNT (_stream_readable.js:1064:12)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickDomainCallback (internal/process/next_tick.js:218:9)
        message: 'Parameter /myvariable not found.',
        code: 'ParameterNotFound',
        time: 2018-07-10T12:56:54.931Z,
        requestId: 'cda55b82-7fa9-4cdc-8b99-6e27a06f3eb0',
        statusCode: 400,
        retryable: false,
        retryDelay: 44.30403493043325 },
     isOperational: true,
     code: 'ParameterNotFound',
     time: 2018-07-10T12:56:54.931Z,
     requestId: 'cda55b82-7fa9-4cdc-8b99-6e27a06f3eb0',
     statusCode: 400,
     retryable: false,
     retryDelay: 44.30403493043325 } }

Additional Data

  • Serverless Framework Version you're using: 1.28.0
  • Operating System: macos
  • Stack Trace: added above
  • Provider Error messages: ParameterNotFound
@gcphost
Copy link
Contributor

gcphost commented Jul 10, 2018

Same issue here, randomly appeared after working most of the day.

Using serverless-offline.

Editing the same function as above but returning early so it wont trigger trigger the rejection and I can finally start my application.

I was also able to start it by using a region I've never used before.

I've also downgraded to 1.27.x and still have the issue.

To me it seems like a random AWS problem since it was ok after the update then randomly broke no matter the version.

Subbed for a proper resolution.

<3 serverless!

@crash7
Copy link
Contributor Author

crash7 commented Jul 10, 2018

@gcphost yep, I thought the same after checking the latest releases, seems that AWS changed something since the project was working the before.

@tjwilder
Copy link

Got the same problem with a null error. However, I actually found it was non-deterministic. Most of the time it would show the null error, but about 1/3 of the time it would instead show me a warning:

Serverless Warning --------------------------------------
    A valid SSM parameter to satisfy the declaration 'ssm:/myvariable~true' could not be found.

@nullflux
Copy link

nullflux commented Jul 10, 2018

Encountering this issue both with serverless-offline and deploying. It seems to occur any time Serverless encounters an SSM parameter that does not exist, even if there is a default to fallback to. Temporarily worked around this issue by commenting out

        const expectedErrorMessage = `Parameter ${param} not found.`;
        if (err.message !== expectedErrorMessage) {
          return BbPromise.reject(new this.serverless.classes.Error(err.message));
        }

in classes/Variables.js:695. Perhaps the exact shape of AWS's error message has changed?

Additional note: Affects all versions from 1.26 through 1.28

@gcphost
Copy link
Contributor

gcphost commented Jul 11, 2018

I decided to check this AM and simply put, err.message is null.

expectedErrorMessage is only used once and is a waste to define as a const so I did this:

        if (err.message && err.message !== `Parameter ${param} not found.`) {
          return BbPromise.reject(new this.serverless.classes.Error(err.message));
        }
        return BbPromise.resolve(undefined);

I'm uncertain if the error message will ever equal Parameter ${param} not found. again, but in case it does I have left it.

@fivepapertigers
Copy link
Contributor

Can also confirm that err.message is null. I'm also seeing that the error message is missing in the AWS CLI and python's boto3, so it seems like AWS removed the error message entirely from that SSM endpoint.

Upon further inspection of the error that's being raised in the JS, it seems like there's a lot of other information contained in that error object that could be used to determine if the error is in fact a ParameterNotFound error. It's not clear why the choice was made to do a string comparison on the message itself over, say, checking the HTTP status code, but that would probably to allow "missing param" errors to pass through as a warning.

@tiagogoncalves89
Copy link

You can do this:

        const errorCode = _.get(err, 'providerError.cause.code');
        const expectedCode = 'ParameterNotFound';
        if (errorCode !== expectedCode) {
          return BbPromise.reject(new this.serverless.classes.Error(err.message));
        }
        return BbPromise.resolve(undefined);

interisti added a commit to interisti/serverless that referenced this issue Jul 13, 2018
@jacobevans
Copy link

Hey y'all, I'm new to using SSM and I'm having the same issue--quick question:

Is the issue at stake that there is no error message printed? Or is the issue that having an unset ssm variable triggers an error?

To me, it should be the latter, so that I can pull from SSM, but if not set, use a default value. (i.e. for env-specific configs). Is this right?

@crash7
Copy link
Contributor Author

crash7 commented Jul 16, 2018

@jacobevans yes, the latter, it should pull the default value without triggering the error.
The error being empty it's a side effect.

@jacobevans
Copy link

@crash7 thanks yeah, thats what I thought/hoped. You doing anything as a workaround in the meantime?

@crash7
Copy link
Contributor Author

crash7 commented Jul 17, 2018

@jacobevans yes, editing the code I mentioned in the issue, in the file serverless/lib/classes/Variables.js.
There is also a PR linked with this that solves the problem but since the tests are failing, it won't be merged soon 😢

@gcphost
Copy link
Contributor

gcphost commented Jul 17, 2018

PR is now passing. The tests were sending a 123 error code instead of a 400.

@marcfielding1
Copy link

I know there's a PR up for this but I thought it'd be worth mentioning. I got this because I use an AWS credential switcher for numerous accounts I work with and forgot to switch back to right account so sls offline will bug out because as you rightly say the credentials don't exist.

I'm not suggesting that you make any changes but hopefully, future readers will have the facepalm moment I did when they read this. :-)

horike37 added a commit that referenced this issue Aug 6, 2018
#5110 null error on undefined AWS SSM variables
@nfantone
Copy link

nfantone commented Aug 23, 2018

On a related note, when the ssm.${region}.amazonaws.com domain cannot be reached, sls errors with:

  Serverless Error ---------------------------------------

  Inaccessible host: `ssm.us-west-1.amazonaws.com'. This service may not be available in the `us-west-1' region.

If a default is provided (by means of configuring vars like so ${ssm:/lambda/services/foo, ' http://localhost:4001/foo'}), I think it should be fair to expect sls to pick it up instead of exploding. This effectively renders projects useless if offline.

@dagumak
Copy link

dagumak commented Nov 18, 2018

I am using serverless 1.33.1 and I deployed my SSM with:

aws ssm put-parameter --name circleApiUserToken --type String --value REDACTED_VALUE --profile awsProfile  --overwrite --region eu-west-2

And when I try to invoke the lambda locally, I am getting this error:

A valid SSM parameter to satisfy the declaration 'ssm:circleApiUserToken' could not be found.

Does anyone know what I should do to fix this? Is it a permission issue?

@pmuens pmuens added the bug label Jan 18, 2019
@pmuens pmuens added this to the 1.46.0 milestone Jun 5, 2019
@medikoo medikoo modified the milestones: 1.46.0, 1.47.0 Jun 26, 2019
@pmuens pmuens modified the milestones: 1.47.0, 1.48.0 Jun 26, 2019
@okarlsson
Copy link

I am using serverless 1.33.1 and I deployed my SSM with:

aws ssm put-parameter --name circleApiUserToken --type String --value REDACTED_VALUE --profile awsProfile  --overwrite --region eu-west-2

And when I try to invoke the lambda locally, I am getting this error:

A valid SSM parameter to satisfy the declaration 'ssm:circleApiUserToken' could not be found.

Does anyone know what I should do to fix this? Is it a permission issue?

@dagumak Make sure you have a valid url for your ssm parameter, starting with a slash eg. ${ssm:/my-service/stage/dbPassword~true}

@marcfielding1
Copy link

marcfielding1 commented Jul 10, 2019

I get this when I use either the wrong AWS profile from ~/.aws/credentials(Mac) or if I have the wrong region selected as above @okarlsson make sure you've got the right format.

    dbUser: ${ ssm:DbUser-${self:custom.currentStage}~true}

the above includes the stage name since I have multiple stages in one account.

Remember by default serverless will use the [default] profile from your AWS creds unless you have env vars specified(I think?)

In short I don't think this is a bug - it's a local config issue.

@pmuens pmuens modified the milestones: 1.48.0, 1.49.0, 1.50.0 Jul 18, 2019
@medikoo
Copy link
Contributor

medikoo commented Apr 12, 2021

Closing, as it's solved with new variable resolver, which crashes on not existing parameter only if fallback wasn't provided

@medikoo medikoo closed this as completed Apr 12, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests