Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add error catching for s3-pull batch insert #2250

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

crayolakat
Copy link
Collaborator

Fixes #2236

Description

Originally, I wanted to add tests for this change, but then I found that there currently aren't any existing tests for the s3-pull feature. I started trying to write tests for it, and discovered that it's going to be a complicated task due to the need to mock the AWS.S3 function: https://github.com/MoveOnOrg/Spoke/blob/2e8c479ef7e4071bc903bdf882334488b50f2910/src/extensions/contact-loaders/s3-pull/index.js#L92

This PR is a small change that adds a line of error catching, so I don't think a test is necessary for this. I created a new feature request to add tests to the s3-pull extension: https://github.com/MoveOnOrg/Spoke/issues/2249

Checklist:

  • I have manually tested my changes on desktop and mobile
  • The test suite passes locally with my changes
  • If my change is a UI change, I have attached a screenshot to the description section of this pull request
  • My change is 300 lines of code or less, or has a documented reason in the description why it’s longer
  • I have made any necessary changes to the documentation
  • I have added tests that prove my fix is effective or that my feature works
  • My PR is labeled [WIP] if it is in progress

@crayolakat crayolakat changed the title Add catch Add error catching for s3-pull batch insert Dec 16, 2022
@lastminutediorama
Copy link
Contributor

Just curious -- have we ever considered adding localstack to our docker-compose dev containers? It's pretty handy for local s3 testing, especially.

@lastminutediorama
Copy link
Contributor

I just have two questions:
Is there any reason we might want to rethrow the error, so that any calling function knows this import has failed? For example, it looks like this function is handling possible errors that could be thrown by this file.

And (just for my curiosity) is there a convention in Spoke to use console.log for logging errors instead of console.error?

@crayolakat
Copy link
Collaborator Author

@lastminutediorama No, we haven't considered using localstack before but should, it looks pretty handy! I'll add this note to the issue I created about adding tests for the S3 Pull feature. There is also no convention for using console.log instead of console.error, I updated the code to use console.error

The try/catch here: https://github.com/MoveOnOrg/Spoke/blob/fc267eafbbc07d6d383ddab4bbc17810d7c75883/src/workers/job-processes.js#L150-L166

Would catch error in the loadContactS3PullProcessFile function only if it was called within the loadContactS3PullProcessFileJob function, right? But there are also instances where it is called outside of the loadContactS3PullProcessFileJob function, such as here: https://github.com/MoveOnOrg/Spoke/blob/2e8c479ef7e4071bc903bdf882334488b50f2910/src/extensions/contact-loaders/s3-pull/index.js#L360

In which case it wouldn't catch the error. Also, in the try/catch, it will check if eventCallback is defined and then run eventCallback, which I guess is supposed to catch the error. But I was digging more into this, and eventCallback is undefined in our production Spoke instance, so if an error was caught nothing would happen anyway. Also, it's not clear to me how to define eventCallback, since the loadContactS3PullProcessFileJob function is called as a command into the sendJobToAWSLambda function like so: https://github.com/MoveOnOrg/Spoke/blob/2e8c479ef7e4071bc903bdf882334488b50f2910/src/extensions/contact-loaders/s3-pull/index.js#L240-L248

Probably have to dig more into the aws-sdk library to figure it out.

Thank you for being so thorough and advising in your PR reviews!

@crayolakat
Copy link
Collaborator Author

@lastminutediorama will add unit tests before merging

@crayolakat crayolakat added the S-ready for stage-main (qa) Status (ADMINS ONLY): PR label for those ready to be added for stage: Approved, tests, etc label Dec 8, 2023
@crayolakat crayolakat mentioned this pull request Dec 8, 2023
@crayolakat crayolakat added S-testing-on-stage Status (ADMINS ONLY): PR label for presence on a staging instance and removed S-ready for stage-main (qa) Status (ADMINS ONLY): PR label for those ready to be added for stage: Approved, tests, etc labels Dec 8, 2023
@crayolakat crayolakat mentioned this pull request Jan 12, 2024
@crayolakat crayolakat mentioned this pull request Mar 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
S-testing-on-stage Status (ADMINS ONLY): PR label for presence on a staging instance
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Feature Request: Add error catching for s3-pull batchInsert
2 participants