IoT Ingest Large-Sized Payloads

Securely ingest large sized payloads from IoT devices using AWS.

Description

This project is mainly based on this AWS Blog which leverages the power of S3 pre-signed URLs, MQTT and IoT Core to upload large files. Once the object is uploaded, the S3 Notification triggers a Lambda which sends an email to view the file.

Getting Started

Dependencies

• An AWS Account
  • Generate the certificate needed to authenticate the IoT device(s). The following are needed:
    • Amazon Root CA certificate file (Amazon-root-CA-1.pem)
    • Private key file (private.pem.key)
    • A certificate for this thing (device.pem.crt)
  • Obtain the device-data-endpoint. Looks similar to account-specific-prefix.jobs.iot.aws-region.amazonaws.com.
  • Register your email address for the SES to be able to send emails.
• An IoT device (eg. a Raspberry Pi).
• Terraform

Terraform setup

• Clone this repository on the laptop/desktop/server

git clone https://github.com/vikramaditya91/iot-ingest-large-size.git

• Get the ARN of the certificate and set it in the terraform/variables.tf
• Adjust other variables in terraform/variables.tf as required
• Apply the terraform configuration:

cd terraform
terraform apply

Set-up on IoT device

• Copy the certificates onto the IoT device
• Clone this repository on the IoT device

git clone https://github.com/vikramaditya91/iot-ingest-large-size.git

• Create a virtual environment of Python3+
• Install the dependencies listed in pip_requirements.txt
• Set the correct values in send_large_file.py
  • paths to the certificates
  • bucket-name as set in variables.tf
  • end-point url as obtained above
  • request url topic as set in variables.tf
• Run the script python send_large_file.py <full_path_to_file_to_be_uploaded>

Additional information

• This project was originally meant for a motion detection system for when my cat Doudou wanted to be let in
• Motion was detected using the Motion Project

Authors

Vikramaditya Gaonkar