-
Notifications
You must be signed in to change notification settings - Fork 3
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Mike feature/awscompatible #17
Conversation
Added dynamo support, fixed list out of index bug if no events are in a month. Added uuid to the data objects.
ShadowSeals is now writing to Dynamo. Added UUID to shadowseals.
Changed ofa Found link to Event. Made more sense for other scrapers which did not share a unique URL. Added proper logging for SS
Location was missing from both OFA and ShadowSeals
Log file reqiured to write to package a log to write to cloudwatch.
Same as previous commit except for OFA.
for proper naming convetions
To Test - Requires a AWS credential in your environmental variables which are applicable for our class AWS. This is for dynamo. If dynamo is inactive then it should simply create a log file. To Test ON AWS - go to our lambda functions |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It worked great Mike, the Lambda trigger the EC2 to run both the crawler scripts, the log output worked as expected. The EC2 stopped after running the scripts as expected. I approve this PR.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Everything worked as expected. Great work Mike!
I ran the test again several times and I found an error in the OFA scraper. |
This pull requests contains the OFAScraper and SSScraper as it appears on the EC2. code writes to dynamo and prints to a log file which is uploaded to cloudwatch if it appears on the EC2.