Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to stream models to ElasticSearch #2385

Closed
usmansbk opened this issue Sep 21, 2019 · 20 comments
Closed

Unable to stream models to ElasticSearch #2385

usmansbk opened this issue Sep 21, 2019 · 20 comments

Comments

@usmansbk
Copy link

@usmansbk usmansbk commented Sep 21, 2019

What is the current behavior?
When I create a mutation, the lambda function that streams dynamodb models to elasticsearch fails and logs Unable to import module 'python_streaming_function': cannot import name 'BotocoreHTTPSession'.
I haven't edited any of the code as it is automatically generated by the GraphQL Transform

What is the expected behavior?
I should be able to search my models in AWS elasticsearch

Which versions and which environment (browser, react-native, nodejs) / OS are affected by this issue? Did this work in previous versions?

    "aws-amplify": "^1.1.40",
    "aws-amplify-react-native": "^2.1.19",
    "aws-appsync": "^2.0.0",
    "aws-appsync-react": "^2.0.0",
    "react": "^16.9.0",
    "react-apollo": "2.5.8",
    "react-native": "^0.60.5",

     aws-cli/1.16.243 Python/3.6.0 Windows/10 botocore/1.12.233
@sollipse

This comment has been minimized.

Copy link

@sollipse sollipse commented Sep 23, 2019

+1 We have started seeing this issue in an extremely similar timeframe.

Has the version of Botocore made available to AWS Lambda changed? Alternatively, have the amplify team made any changes to the underlying zipfile housing this Streaming code?

Please treat this as a high severity issue. As Amplify does not offer a way to "refresh" entries in ES, the only remedy to missing ES records at them moment is a full table rewrite.

@manueliglesias manueliglesias transferred this issue from aws-amplify/amplify-js Sep 23, 2019
@kaustavghosh06

This comment has been minimized.

Copy link
Contributor

@kaustavghosh06 kaustavghosh06 commented Sep 23, 2019

We're looking into it. I'll update on this as soon as we have a fix for it. As a workaround please take a look at - #2353 (comment) for a workaround until we push the fix.

@kaustavghosh06

This comment has been minimized.

Copy link
Contributor

@kaustavghosh06 kaustavghosh06 commented Sep 23, 2019

This issue should be fixed from the Lambda service side as they rolled-back their changes - causing this behavior. Please feel free to comment on this thread if you're still seeing this issue.

@usmansbk

This comment has been minimized.

Copy link
Author

@usmansbk usmansbk commented Sep 24, 2019

@kaustavghosh06 still not fixed.

@kaustavghosh06

This comment has been minimized.

Copy link
Contributor

@kaustavghosh06 kaustavghosh06 commented Sep 24, 2019

@usmansbk We dug into the issue and after talking to the Lambda team, we understood the reason for the change and errors. We're working on a fix in the CLI that will enable you to re-deploy your function as a part of the ampliy push command to avoid seeing the Lambda errors. For the time being, here's an explanation and the workaround provided by the Lambda team to prevent your app from seeing errors:

AWS Lambda is updating the default boto3 and botocore SDKs for the Python 2.7, Python 3.6 and Python 3.7 runtimes. This update changes the version of boto3 from 1.7.74 to 1.9.221 for the Python 2.7 and Python 3.6 runtimes and from 1.9.42 to 1.9.221 for the Python 3.7 runtime. The update contains SDK enhancements and updated APIs. One notable example of an internal class that underwent change is the BotocoreHTTPSessions class which is no longer supported starting with botocore version 1.11.0.

Since the CLI used the BotocoreHTTPSessions class, that's the reason why you're seeing these errors.

As a workaround, you can update your DdbToEsFn-xxxxxxfunction in the AWS Console to use a Lambda layer that contains the previous version of the SDK. For your convenience, the AWS Lambda team already published these layers for you to use. You can find additional information on Lambda layers out here - https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html

Here are the list of lambda layer ARN's which you can use based on the region you are in:_

ap-northeast-1: arn:aws:lambda:ap-northeast-1:249908578461:layer:AWSLambda-Python-AWS-SDK:1
us-east-1: arn:aws:lambda:us-east-1:668099181075:layer:AWSLambda-Python-AWS-SDK:1
ap-southeast-1: arn:aws:lambda:ap-southeast-1:468957933125:layer:AWSLambda-Python-AWS-SDK:1
eu-west-1: arn:aws:lambda:eu-west-1:399891621064:layer:AWSLambda-Python-AWS-SDK:1
us-west-1: arn:aws:lambda:us-west-1:325793726646:layer:AWSLambda-Python-AWS-SDK:1
ap-east-1: arn:aws:lambda:ap-east-1:118857876118:layer:AWSLambda-Python-AWS-SDK:1
ap-northeast-2: arn:aws:lambda:ap-northeast-2:296580773974:layer:AWSLambda-Python-AWS-SDK:1
ap-northeast-3: arn:aws:lambda:ap-northeast-3:961244031340:layer:AWSLambda-Python-AWS-SDK:1
ap-south-1:631267018583: arn:aws:lambda:ap-south-1:631267018583:layer:AWSLambda-Python-AWS-SDK:1
ap-southeast-2: arn:aws:lambda:ap-southeast-2:817496625479:layer:AWSLambda-Python-AWS-SDK:1
ca-central-1: arn:aws:lambda:ca-central-1:778625758767:layer:AWSLambda-Python-AWS-SDK:1
eu-central-1: arn:aws:lambda:eu-central-1:292169987271:layer:AWSLambda-Python-AWS-SDK:1
eu-north-1: arn:aws:lambda:eu-north-1:642425348156:layer:AWSLambda-Python-AWS-SDK:1
eu-west-2: arn:aws:lambda:eu-west-2:142628438157:layer:AWSLambda-Python-AWS-SDK:1
eu-west-3: arn:aws:lambda:eu-west-3:959311844005:layer:AWSLambda-Python-AWS-SDK:1
sa-east-1: arn:aws:lambda:sa-east-1:640010853179:layer:AWSLambda-Python-AWS-SDK:1us-us-east-2: arn:aws:lambda:us-east-2:259788987135:layer:AWSLambda-Python-AWS-SDK:1
us-west-2: arn:aws:lambda:us-west-2:420165488524:layer:AWSLambda-Python-AWS-SDK:1
cn-north-1: arn:aws-cn:lambda:cn-north-1:683298794825:layer:AWSLambda-Python-AWS-SDK:1
cn-northwest-1: arn:aws-cn:lambda:cn-northwest-1:382066503313:layer:AWSLambda-Python-AWS-SDK:1
us-gov-west-1:: arn:aws-us-gov:lambda:us-gov-west-1:556739011827:layer:AWSLambda-Python-AWS-SDK:1
us-gov-east-1: arn:aws-us-gov:lambda:us-gov-east-1:138526772879:layer:AWSLambda-Python-AWS-SDK:1

@sacrampton

This comment has been minimized.

Copy link

@sacrampton sacrampton commented Sep 24, 2019

I am also seeing this problem - I created a new type with searchable and no index is created in ElasticSearch.

The instructions for the work around are a bit opaque to me - assuming you have just used standard amplify/appsync to sync DynamoDB and ElasticSearch, what are the specific set of steps/commands we need to execute to "unbreak" this whilst we are waiting for the fix?

@usmansbk

This comment has been minimized.

Copy link
Author

@usmansbk usmansbk commented Sep 24, 2019

@sacrampton

  • Open the generated lambda function configuration in the AWS console -> Lambda, which is typically named DdbToEsFn-XXXXXX.

  • Select layers
    Screenshot (17)_LI

  • Then click the Add a layer button and select the ARN option.

  • Paste one of the ARNs above based on your region

@sacrampton

This comment has been minimized.

Copy link

@sacrampton sacrampton commented Sep 24, 2019

Thanks for the above - I could add the layer - pushed to deploy again - but still no data streaming to ElasticSearch. I updated amplify CLI to latest version being pushing.

@tezqa

This comment has been minimized.

Copy link

@tezqa tezqa commented Sep 24, 2019

@kaustavghosh06 Thanks for the workaround! I hope the AWS team will not do breaking changes like this one!

@ericclemmons

This comment has been minimized.

Copy link
Member

@ericclemmons ericclemmons commented Sep 24, 2019

For anyone finding this from Google because their GraphQL API queries (e.g. searchPhotos(filter: { description: { match: "foo } })) is returning Elasticsearch:UserIllegalArgumentException errors with the message:

Elasticsearch responded with an error: Not Found

This bug appears to be related. Subscribe to this thread & try out the fix listed here: #2385 (comment).

@cylim

This comment has been minimized.

Copy link

@cylim cylim commented Sep 25, 2019

The issues should be related to boto/botocore#1544

@babu-upcomer

This comment has been minimized.

Copy link

@babu-upcomer babu-upcomer commented Sep 25, 2019

@kaustavghosh06 Why is it amplify CLI, JS, Cognito and Lambda teams are working out of sync? There has been hundreds of issues referring one another in the github itself. I do not want to highlight the feedback I get from the AWS support. To make the amplify a complete framework to serverless cloud problems you guys should work in sync.

Back to the issue, this issue has halted our ES indexing for the last 1 week and we got complaints from users that the search is no more working. Unfortunately this happened at our production env. In future, what level of guarantee we'll get to make sure that this doesn't happen again?

@bernhardt1

This comment has been minimized.

Copy link

@bernhardt1 bernhardt1 commented Sep 26, 2019

@babu-upcomer thats rough.

I'm in beta right now and have a similar story (luckily didn't have more than a few dozen people's experience hurt). It kind of undermines my trust in AWS in general to have something like this happen.

@cylim

This comment has been minimized.

Copy link

@cylim cylim commented Sep 27, 2019

I think lambda should warned developers who using build-in libraries from AWS, this is not only affecting Amplify + Elasticsearch users, but everyone who used deprecated library.

In the meantime, what Amplify can do is to specify version for each dependencies.

@kaustavghosh06

This comment has been minimized.

Copy link
Contributor

@kaustavghosh06 kaustavghosh06 commented Sep 29, 2019

In the latest version of the CLI - v3.10.0, we've added support for the above mentioned (#2353 (comment)) Lambda Layer in to avoid breaking of the Lambda functions due to the Python Lambda runtime change. Please let us know if you're still seeing this issue.

@ArjunAhuja95

This comment has been minimized.

Copy link

@ArjunAhuja95 ArjunAhuja95 commented Oct 8, 2019

@kaustavghosh06 I am still seeing this issue in lamda function.

@avespoli-bf

This comment has been minimized.

Copy link

@avespoli-bf avespoli-bf commented Oct 10, 2019

@kaustavghosh06 still seeing the issue even after trying the workaround

@nikhname

This comment has been minimized.

Copy link
Contributor

@nikhname nikhname commented Oct 10, 2019

@ArjunAhuja95 @avespoli-bf which version of the cli are you using?

@avespoli-bf

This comment has been minimized.

Copy link

@avespoli-bf avespoli-bf commented Oct 10, 2019

@nikhname 3.11.0

@avespoli-bf

This comment has been minimized.

Copy link

@avespoli-bf avespoli-bf commented Oct 10, 2019

@nikhname @kaustavghosh06 I didn't add the layer correctly to my lambda, it's working now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.