EDSC-3612, make search results export asynchronous #1588
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…sts__/handler.test.js
…-sqs, adding scripts for installing and running sqs and s3 simulations
…t S3 buckets, used by search exporting
…ss-configs/aws-resources.yml
…ocking of randomUUID
…onnections while running tests in parallel
…ng local development
DanielJDufour
changed the title
DanielJDufour
commented
Feb 27, 2023
| @@ -130,6 +130,50 @@ | |||
| environment: | |||
| updateOrderStatusStateMachineArn: ${self:resources.Outputs.UpdateOrderStatusWorkflow.Value} | |||
|
|
|||
| exportSearchRequest: | |||
| handler: serverless/src/exportSearchRequest/handler.default | |||
| timeout: 300 # this lambda isn't restricted to 30s because it's triggered by SQS not API Gateway | |||
There was a problem hiding this comment.
move comment down
DanielJDufour
commented
Feb 27, 2023
| exportSearchRequest: | ||
| handler: serverless/src/exportSearchRequest/handler.default | ||
| timeout: 300 # this lambda isn't restricted to 30s because it's triggered by SQS not API Gateway | ||
| memorySize: 256 |
There was a problem hiding this comment.
keep memory default
DanielJDufour
commented
Feb 27, 2023
| Properties: | ||
| QueueName: ${self:custom.siteName}-SearchExportQueue | ||
| ReceiveMessageWaitTimeSeconds: 20 | ||
| VisibilityTimeout: 300 |
There was a problem hiding this comment.
make larger than lambda timeout
DanielJDufour
commented
Feb 27, 2023
| @@ -31,8 +31,7 @@ const edlOptionalAuthorizer = async (event) => { | |||
| if (!jwtToken || jwtToken === '') { | |||
| const authOptionalPaths = [ | |||
| '/autocomplete', | |||
| '/opensearch/granules', | |||
| '/collections/export' | |||
There was a problem hiding this comment.
put /collections/export back in
DanielJDufour
commented
Feb 27, 2023
| const MOCK_KEY = '00000000-0000-0000-0000-000000000000' // see /__mocks__/crypto.js | ||
|
|
||
| // over-rides randomUUID to return the MOCK_KEY | ||
| jest.mock('crypto') |
There was a problem hiding this comment.
remove, for uuid package
DanielJDufour
commented
Feb 27, 2023
| @@ -0,0 +1,124 @@ | |||
| import { randomUUID } from 'crypto' // if we do 'node:crypto', jest.mock won't work | |||
There was a problem hiding this comment.
use older library
DanielJDufour
commented
Feb 27, 2023
| const earthdataEnvironment = determineEarthdataEnvironment(headers) | ||
|
|
||
| const jwt = getJwtToken(event) | ||
| if (!jwt) throw Error("missing jwt") |
There was a problem hiding this comment.
make jwt optional
DanielJDufour
commented
Feb 27, 2023
|
|
||
| const { id: userId } = getVerifiedJwtToken(jwt, earthdataEnvironment) | ||
|
|
||
| if (!userId) throw Error("failed getting userId from jwt") |
There was a problem hiding this comment.
auth is optional now
DanielJDufour
commented
Feb 27, 2023
| variables | ||
| } | ||
|
|
||
| const key = randomUUID() |
DanielJDufour
commented
Feb 27, 2023
|
|
||
| const dbConnection = await getDbConnection() | ||
|
|
||
| await dbConnection('exports').insert({ |
DanielJDufour
commented
Feb 27, 2023
| const searchExportQueueUrl = getSearchExportQueueUrl() | ||
| console.log('searchExportQueueUrl:', searchExportQueueUrl) | ||
|
|
||
| await sqs.sendMessage({ |
DanielJDufour
commented
Feb 27, 2023
| QueueUrl: searchExportQueueUrl, | ||
| MessageBody: messageBody | ||
| }).promise() | ||
| console.log('posted to search export queue') |
There was a problem hiding this comment.
prefix with lambda name, proper caps
There was a problem hiding this comment.
and requestId!
DanielJDufour
commented
Feb 27, 2023
| statusCode: 200, | ||
| headers: { | ||
| ...defaultResponseHeaders, | ||
| 'jwt-token': jwt |
DanielJDufour
commented
Feb 27, 2023
| @@ -1,125 +1,189 @@ | |||
| import crypto from 'node:crypto' | |||
There was a problem hiding this comment.
uuid lib instead
DanielJDufour
commented
Feb 27, 2023
| * @param {Object} event Details about the HTTP request that it received | ||
| */ | ||
| const exportSearch = async (event, context) => { | ||
| // https://stackoverflow.com/questions/49347210/why-aws-lambda-keeps-timing-out-when-using-knex-js | ||
| // eslint-disable-next-line no-param-reassign | ||
| context.callbackWaitsForEmptyEventLoop = false | ||
|
|
||
| const { body, headers } = event | ||
| if (process.env.IS_OFFLINE || process.env.JEST_WORKER_ID) { |
DanielJDufour
commented
Feb 27, 2023
| if (!filename) throw new Error("missing filename") | ||
| if (!key) throw new Error("missing key") | ||
| if (!requestId) throw new Error("missing requestId") | ||
| if (!userId) throw new Error("missing userId") |
DanielJDufour
commented
Feb 27, 2023
|
|
||
| const { data, requestId } = JSON.parse(body) | ||
| const updateState = (state) => dbConnection('exports').where({ user_id: userId, key }).update({ state }) |
There was a problem hiding this comment.
userId optional
macrouch
reviewed
Feb 27, 2023
| "s3": "EXTRA_CORS_ALLOWED_ORIGINS=http://localhost:5000 MOTO_ALLOW_NONEXISTENT_REGION=True moto_server -H 0.0.0.0", | ||
| "s3:install": "brew install moto", | ||
| "s3:reset": "curl -X POST http://localhost:5000/moto-api/reset", | ||
| "sqs": "java -jar elasticmq-server-1.3.9.jar", |
There was a problem hiding this comment.
I'd rather have sqs offline set up the same way as other projects our team manages, with the docker compose file.
I hate the idea of running extra processes in order to run EDSC locally. This PR for serverless offline sqs shows that it is possible to disable sqs offline. So we'd get the benefit of queues locally, but only if the dev wants them. I don't think that is happening, but it is easy enough to open a new PR that is up to date and install the forked version of the plugin
|
superseded by #1630 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overview
What is the feature?
Please summarize the feature or fix.
What is the Solution?
Summarize what you changed.
What areas of the application does this impact?
List impacted areas.
Testing
Reproduction steps
Attachments
Please include relevant screenshots or files that would be helpful in reviewing and verifying this change.
Checklist