Issues I found attempting to follow along

[brianwestfall1971]

First let me say thank you for this! I found this extremely helpful to get started using AWS for a project similar to yours. I would like to give back a little by making it easier for others to implement this by highlighting the following issues I encountered.

1. Suggested Implementation Plan - Step 5 - Hyperlink for aws-lambda-functions incorrectly maps to aws-step-function.

2. s3-trigger-image-processing instructions, upload to AWS section - Step 2. Instead of "/upload" it needs to be "upload". The "/" prevents the trigger from working.

3. You have hardcoded your AWS region into index.js files for both nodemailer-send-notification and nodemailer-error-handler. The description doesn't indicate that this needs to be changed. Maybe using another Environment Variable to hold the region?

Thanks again...

[markwest1972]

@brianwestfall1971 Many thanks for your feedback! It's much appreciated!

I've fixed # 1 and # 2. # 3 is also fixable, but I wonder if one really needs to specify the region when instantiating AWS Helper classes. I see also that I don't always specify the version, which could cause problems if a later version of the AWS Helper classes change the API behavior.

In any case I'll look into this and update the code accordingly.

Once again many thanks for your feedback!

[markwest1972]

I have now set the Region to that of the calling Lambda Function using a process variable. I'll also specified all the API Versions when instantiating the AWS Helper Classes.

[NickSutton]

Hi Mark,
I enjoyed your blog on the smart security camera very much, it just so happened I'd finished something similar with Motion not long ago and so was interested in following your work.

I've never used AWS before and got myself completely lost, despite the detail and level of care in your instructions. So far, I have made the Pi run the script to upload an image to my S3 bucket... The rest of it I have no clue! I've read the notes for Lambda, and tried some examples but must be missing something!!

Is there any help you can offer?
Thanks,
Nick

[markwest1972]

@NickSutton I have limited time, but will try to help you as much as I can.

First off, will you be creating NodeJS or Java based AWS Lambda Functions? If you don't have a preference, have no programming experience, or no Java experience, then I would suggest going the NodeJS direction, as it requires less tools.

Once you have decided, try following the following instructions to create your first Lambda Function:

• NodeJS
• Java

This Lambda Function will take an image on S3 and send it to AWS Rekognition for analysis.

Try to get as far as you can by googling / using the AWS Documentation. If you get stuck, create a new issue on GitHub and explain your problem.

[NickSutton]

Thanks very much, I'll take another look tonight. I was surprised at how easy it was to get the image in to the S3 bucket, and then I was stumped!

Hopefully better luck tonight.
Cheers

[MatthiasGemelli]

Hello Nick,
I implemented a simplified Version (without Step functions), just S3 and then a simple Lambda. In Python. https://github.com/MatthiasGemelli/IntelliCam

Caution I just racked up 44 US$ Rekognition charges for October alone... Oops.
I guess I should upload fewer Pictures...

[markwest1972]

@NickSutton Mattias' version is a lot quicker and easier to implement if you are looking for a quick fix :)

@MatthiasGemelli Have you tried tweaking Motions parameters to generate less pics?

[MatthiasGemelli]

@markwest1972 - nope, I just learned about the REKO bill yesterday, luckily it was on my corporate LAB account (not my personal one). Tweaking Motion config is top on my todo list.
I still have to optimize my Code to remember an Alarm state for a few seconds. And to summarize/thumbnail the Images (so I don't get notified on each individual Alarm pic).

[NickSutton]

Thanks Gents,

I'll have a go at the simplified version, but - since I might be paying for it - I'd like to understand the full capabilities of AWS, and how it can benefit projects like this!

Cheers

[NickSutton]

Hi,

So I found the following link in Matthias' page really helpful and explains a lot: https://aws.amazon.com/de/blogs/ai/capture-and-analyze-customer-demographic-data-using-amazon-rekognition-amazon-athena/

As it stands, it's doing something... But I've got a bit of a mix of scripts that's probably going to cause a problem somewhere....

Mark's script is correctly uploading images from my pi to a bucket album called 'upload'
The bucket is correctly triggering because I can see a Cloud Watch log, but not sure if it means it was successful or not
04:28:59 Loading function 04:28:59 START RequestId: 2f6afc3b-ba06-11e7-b57e-354013fa4373 Version: $LATEST
04:29:02 END RequestId: 2f6afc3b-ba06-11e7-b57e-354013fa4373 04:29:02 REPORT RequestId: 2f6afc3b-ba06-11e7-b57e-354013fa4373 Duration: 3003.15 ms Billed
Duration: 3000 ms Memory Size: 128 MB Max Memory Used: 34 MB 04:29:02 2017-10-26T04:29:02.144Z 2f6afc3b-ba06-11e7-b57e-354013fa4373 Task timed out after 3.00 seconds

It looks like it is timing out, I'm not sure why. Do all my services need to be using the same region?

[MatthiasGemelli]

@NickSutton - yes, most Code assumes that Lambda and S3 are in the same Region.
You can also specify the Region when establishing the Boto Client - see snippet below.
For Troubleshooting I recommend you sprinkle your Code with print ('Nick was here') Messages, so you know where exactly the issue is (e.g. before/after the Client is established). Check my Lambda Code snippets in my repo for examples in Python.

client = boto3.client('ec2') 
     region_iterator = client.describe_regions()['Regions'] 
          for region in region_iterator: 
               #print ('Looking for EC2 instances in: ' + str(region['RegionName'])) 
               ec2 = boto3.resource('ec2', region_name=region['RegionName'])  

[NickSutton]

Holy shit. It works!!

Thanks very much guys

[NickSutton]

Hi Mark,

So I took a few hours and finally made sense of AWS, and carefully followed your instructions.

I'm getting a 'succeeded' message in Step Functions, but the log is showing a Lamba fail message on the Nodemailer-send-notification function.

On investigation the CloudWatch logs give me the following:

2017-10-31T08:23:01.392Z b3eebb57-be14-11e7-8ba8-a52776b694af Error: Invalid status code 404 at ClientRequest.<anonymous> (/var/task/node_modules/nodemailer-fetch/lib/fetch.js:177:36) at emitOne (events.js:96:13) at ClientRequest.emit (events.js:188:7) at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:473:21) at HTTPParser.parserOnHeadersComplete (_http_comm

2017-10-31T08:23:01.392Z b3eebb57-be14-11e7-8ba8-a52776b694af Error: Invalid status code 404 at ClientRequest.<anonymous> (/var/task/node_modules/nodemailer-fetch/lib/fetch.js:177:36) at emitOne (events.js:96:13) at ClientRequest.emit (events.js:188:7) at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:473:21) at HTTPParser.parserOnHeadersComplete (_http_common.js:99:23) at TLSSocket.socketOnData (_http_client.js:362:20) at emitOne (events.js:96:13) at TLSSocket.emit (events.js:188:7) at readableAddChunk (_stream_readable.js:176:18) at TLSSocket.Readable.push (_stream_readable.js:134:10)

I did have a few warnings during NPM install but no fails. Any pointers as I feel I'm at the final step!

The error handler is correctly sending an email but it only says that an unknown error occurred and Lambda didn't return an error type.

Thanks for your continued help with this!

[markwest1972]

My guess is that the function fails when it tries to attach the image to your email. This is probably because the image address is incorrect (404 means "not found").

Try printing out the following:

process.env.S3_URL_PREFIX + bucket + '/'+ filename

To see if the URL is being correctly created. Perhaps the URL prefix is being incorrectly specified?

[NickSutton]

Thanks, found a missing '/' and got rid of the 404, but getting a 403 now instead...

[markwest1972]

@NickSutton Good! I'm sure we can fix the 403 error. This is because the lambda function isn't allowed to access the URL.

What you can try is to replace this code (line 47 of nodemailer-send-notification/index.js):
path: process.env.S3_URL_PREFIX + bucket + '/'+ filename

With this code:
path: s3.getSignedUrl('getObject', { Bucket: bucket, Key: filename, Expires: 360})

Hopefully this will help!

[NickSutton]

Between that and correctly setting AWS_Region variable - which I'd hard coded. It now works!

Some tidying up to do with the archiving function, but pleased with the progress! Image comparison functions next!
Thanks Mark!

[markwest1972]

Marvellous! Just remember that you only get 5000 free Rekognition calls a month under the free their. After that I think it's $1 per 1000 calls, which isn't expensive but adds up over time.

[NickSutton]

Yes, thanks again Mark.

Do you have any plans to implement Rekognition comparison, to further reduce alerts when, say, a member of your family or even yourself triggers the camera?

[markwest1972]

I am no longer actively working on this project and have no plans for further development. But this kind of enhancement is possible with Rekognition, as long as the pictures contain a good enough facial image.