Media Optimizer for AWS S3

Automatically optimize your images & videos hosted on AWS S3 when an S3 event is detected

This project is inspired by image-optimizer & Lambda ECS Worker Pattern

Architecture

A tag is also applied to the optimized file so this file will be ignored in subsequent event

How it works ?

• S3
  • S3 is configured to trigger a specific lambda on objectCreate event
• Lambda
  • catch the S3 Event
  • push the event to a SQS queue
• EC2
  • constantly listens to messages from SQS queue
  • when a message arrives :
    • get the S3 Event from the message
    • get information about S3 object (content-type)
    • download the S3 object
    • executes optimization chain on object (one or more command)
    • upload the optimized object to S3

Run on AWS

In AWS dashboard, go to cloudformation & create a new stack from the template file cloudformation.yml

In the parameters, change the bucket name & the subnet value (a private subnet is fine)

When the stack is up, go to the created S3 bucket & upload some images. After some moment, you will notice they will be automatically optimized

You can check that the new optimized image has the tag optimizer_ignore :

In CloudWatch you can get the logs of your Lambda & your docker container running the optimization program

Optimization configuration

The optimization configuration is located in optimizer.json file. In the actual version, the JSON file need to be at the same level as the media-optimizer executable

Current configuration

Currently using all the optimizers from image-optimizer project + ffmpeg for mp4 video :

• image/png
  • pngquant --output [output_file] -f [intput_file]
  • optipng -out [output_file] -i0 -o2 -clobber [intput_file]
• image/jpeg
  • jpegoptim -d [output_directory] -m85 --strip-all --all-progressive [intput_file]
• image/svg+xml
  • svgo -o [output_file] --disable={cleanupIDs,removeViewBox} [intput_file]
• image/gif
  • gifsicle -o [output_file] -b -O3 [intput_file]
• image/webp
  • cwebp -o [output_file] -m 6 -pass 10 -mt -q 80 [intput_file]
• video/mp4
  • ffmpeg -i [input_file] -vcodec libx264 -crf 24 -y [output_file]

You can add more format & more commands to the config file optimizer.json

Configuration Format

{	
	"optimizers": [{
		"contentType": "image/png",
		"exec": [{
			"binary": "pngquant",
			"outputFile": "--output",
			"params": ["-f"]
		},{
			"binary":"optipng",
			"outputFile": "-out",
			"params": ["-i0","-o2", "-clobber"]
		}]
	.....
}

• binary is the command to execute
• params are the command parameters
• outputFile is the parameter for the output file
• outputDirectory is the parameter for the output directory if outputFile is not specified
• inputFile is the parameter for the input file if neither outputFile nor outputDirectory are specified. In case inputFile is specified the outputFile value will be appended to the parameters (at the end)

Environment variables

Those variables configure S3 Object configuration & metadata :

name	default value
OBJECT_ACL	private
OBJECT_CACHE_CONTROL	max-age=15552000
OBJECT_STORAGE_CLASS	STANDARD
OBJECT_SERVER_SIDE_ENCRYPTION	none

Note that if AWS::Region is not specified in your ~/.aws, the variable AWS_REGION should be set (see cloudformation.yml)

Run locally

make install
make run

Run with docker (locally)

docker build . -t media-optimizer
docker run -v $HOME/.aws:/root/.aws -it media-optimizer

Run with docker (DockerHub)

docker pull bertrandmartel/media-optimizer:latest
docker run -v $HOME/.aws:/root/.aws -it bertrandmartel/media-optimizer:latest

Dependencies

• https://github.com/aws/aws-sdk-go
• https://github.com/aws/aws-lambda-go