This component downloads files from an Amazon S3 bucket by using Amazon S3 Transfer Manager. It has the following features:
- Download orders are sent from AWS IoT Core to the edge devices
- Parallel download of multiple files
- Pause/resume downloads
- Write resume files to disk to continue from
- Send download progress to AWS IoT Core
This component specifically aims to address situations where downloading large files is necessary but network conditions are unreliable.
This component has the following versions:
- 1.0.0
This component is a generic component. The Greengrass nucleus runs the component's lifecycle scripts.
For more information, see component types
This component will download a file from an S3 Bucket to the local disk. You need to modify the Token Exchange Role that is used by Greengrass and give authorization to the S3 bucket that you want to use to download files from. See Authorize core devices to interact with AWS services for the details. It is also possible to use the artifacts bucket that Greengrass already has permission to deploy components.
An example policy can be seen below:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetObject"
],
"Resource": "arn:aws:s3:::DOWNLOADER-BUCKET/download-folder/*"
}
]
}You can read more about best practices of creating IAM policies in this documentation.
This component requires the following dependencies.
- Token Exchange Service >=2.0.3
Run the following command to build the component:
gdk component build
Before publishing the component, you need to update the gdk-config.json file by replacing the bucket and region parameters.
"publish": {
"bucket": "<S3 BUCKET PREFIX>",
"region": "<REGION>"
}gdk will use -- as the bucket name. If it doesn't exist, an S3 bucket will be created with this name.
To be able to use this component in your devices, you need to publish it to AWS IoT Core.
gdk component publish
Go to AWS IoT Core->Greengrass devices->Components in AWS console to check if the aws.greengrass.labs.S3FileDownloader component has been published successfully.
Now you can deploy the component by creating a deployment for your devices.
When a Greengrass component is deployed, it uses ggc_user system user by default. It is possible to change the system user by configuring the component during the deployment. You can set the system user parameter as a separate user in the Advanced configuration of the component. This is the recommended way of running the S3FileDownloader component, since the files written to the destination folder will only be accessed by this component or an authorized user. The user must be created in the operating system first in order to let the component to use this user.
This component has the following configuration:
- DOWNLOAD_TOPIC: MQTT topic which the component subscribes to receive download tasks
- PAUSE_TOPIC: MQTT topic to receive pause commands
- JOB_FEEDBACK_TOPIC: MQTT topic which the component publishes the download progress
- DESTINATION_FOLDER: the destination folder which the files are written
The topic structure uses {iot:thingName} recipe variable in the following form things/{iot:thingName}/+ , in order to send task to individual devices.
To be able to use recipe variables, you need to set interpolateComponentConfiguration for Greengrass nucleus. You can find more information in the Greengrass documentation.
- To view the download progress in AWS Console, go to AWS IoT Core->MQTT Test Client, then in the Subscribe to a topic tab, enter
things/<your-thing-name>/updateas the topic filter where<your-thing-name>is the name of the Greengrass core thing name and click Subscribe.
-
Copy a file to an S3 bucket which the Greengrass core has access. To test parallel file download, you can copy multiple files to the bucket.
-
In the MQTT Test Client, switch to Publish to a topic tab and fill it with the following information :
Topic name : things/<your-thing-name>/download
{
"s3Bucket": "<testbucket>",
"key": "file.zip",
"jobId": "filedownload"
}- You can view the download progress in the Subscription window of the MQTT Test Client.
- To test parallel download function, you should have multiple files in the S3 bucket. You can publish the payload with different key and jobId. As in the following example:
Topic name : things/<your-thing-name>/download
{
"s3Bucket": "<testbucket>",
"key": "file2.zip",
"jobId": "filedownload2"
}- By using MQTT Test Client, publish any payload to
things/<your-thing-name>/pausetopic. This will pause all the downloads in the device. You can also view the pause feedback in the Subscription window of the MQTT Test Client.
- To resume a download process, publish the same payload to
things/<your-thing-name>/downloadtopic. The component will continue to download this particular file from where it was left.
Topic name : things/<your-thing-name>/download
{
"s3Bucket": "<testbucket>",
"key": "file.zip",
"jobId": "filedownload"
}This component does not have any mechanism to prevent filling up the system disk due to large files. It is recommended to have a separate partition for the destination folder. By this way, the rest of the system will be protected from the failures due to limited disk space.
This component publishes the download progress updates to the device specific topic things/<your-thing-name>/update. For auditing purposes, you can forward these payloads to another service such as Amazon CloudWatch, in order to audit the historical download tasks of the devices.
This component writes the logs to <GreengrassInstallPath>/logs/aws.greengrass.labs.S3FileDownloader.log file. It is also possible to configure Greengrass to send logs to Amazon CloudWatch.
The following table describes the changes in each version of the component.
| Version | Changes |
|---|---|
| 1.0.0 | Initial version |
This project is licensed under the Apache-2.0 License.