Skip to content
AWS Lambda Layers for Python
Python Shell HCL Dockerfile
Branch: master
Clone or download
Latest commit 105e8a3 Sep 15, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
arns 09/16/2019_01:10 Sep 16, 2019
binaries added new arch diagram Sep 5, 2019
documentation added new arch diagram Sep 5, 2019
infra added correct order of non_key attributes Aug 17, 2019
packages 09/16/2019_01:10 Sep 16, 2019
scripts docker deploy Aug 18, 2019
serverless added aws x-ray Aug 18, 2019
.gitignore removed binaries Aug 18, 2019
.pyup.yml Update pyup.yml Aug 5, 2019
LICENSE Create License Jun 18, 2019
README.MD corrected typo on builds Sep 5, 2019

README.MD

Keith's Layers (Klayers)

Python 3 Language grade: Python

A collection of AWS Lambda Layers for Python3.7

Status of layers

Updates

Click on badge above to see the specific package version of the latest builds.

Layers are built with the latest package version (available on PyPI) every Monday at 1:00am (UTC)

Layer List

All layers are publicly available with the following ARN pattern:

arn:aws:lambda:<region>:113088814899:layer:<Layer Name>:<version>

Full list of publicly available layers by region are here.

Python Packages

All packages below were built with the Python 3.7 runtime. May (or may not) work with Python 3.6. View here for detail list of package versions.

Package Layer Name
aiobotocore Klayers-python37-aiobotocore
aiohttp Klayers-python37-aiohttp
ansible Klayers-python37-ansible
arrow Klayers-python37-arrow
aws-xray-sdk Klayers-python37-aws-xray-sdk
bcrypt Klayers-python37-bcrypt
beautifulsoup4 Klayers-python37-beautifulsoup4
boltons Klayers-python37-boltons
boto3 Klayers-python37-boto3
construct Klayers-python37-construct
elasticsearch Klayers-python37-elasticsearch
envelopes Klayers-python37-envelopes
ffmpeg-python Klayers-python37-ffmpeg-python
flashtext Klayers-python37-flashtext
google-auth Klayers-python37-google-auth
google-auth-oauthlib Klayers-python37-google-auth-oauthlib
grpcio Klayers-python37-grpcio
gspread Klayers-python37-gspread
idna Klayers-python37-idna
itsdangerous Klayers-python37-itsdangerous
kafka-python Klayers-python37-kafka-python
nltk Klayers-python37-nltk
numpy Klayers-python37-numpy
opencv-python-headless Klayers-python37-opencv-python-headless
openpyxl Klayers-python37-openpyxl
pandas Klayers-python37-pandas
Pillow Klayers-python37-Pillow
PyJWT Klayers-python37-PyJWT
pymongo Klayers-python37-pymongo
PyMUPDF Klayers-python37-PyMUPDF
PyMySQL Klayers-python37-PyMySQL
PyNaCl Klayers-python37-PyNaCl
pyOpenSSL Klayers-python37-pyOpenSSL
pyparsing Klayers-python37-pyparsing
pytesseract Klayers-python37-pytesseract
python-docx Klayers-python37-python-docx
python-Levenshtein Klayers-python37-python-Levenshtein-wheels
pytz Klayers-python37-pytz
PyYAML Klayers-python37-PyYAML
requests Klayers-python37-requests
scipy Klayers-python37-scipy
simplejson Klayers-python37-simplejson
SQLAlchemy Klayers-python37-SQLAlchemy
tinydb Klayers-python37-tinydb
tldextract Klayers-python37-tldextract

Using the Layers

You can either:

  • Set any of the ARNs below (for your region) to be a layer for your function.
  • Use the Get Layer Version by ARN in python or bash command which will provide an S3 location to download the layer as a zip

Currently only the following regions have the layers deployed click the links for the full list of layer arns:

Layer expiry

Some layers will have a time_to_live field. This is the UNIX epoch time for when that layer will be deleted, typically this will be 30 days after a new version is deployed. If you use the latest version of a package, it'll be up for at least 30 days before deletion.

Binaries

Special hand-crafted binaries for layers. These layers are not automatically built, and hence slower update cycles.

Package ARN Version
tesseract arn:aws:lambda:<region>:113088814899:layer:Klayers--tesseract:2 4.1.0-rc11
pip arn:aws:lambda:<region>:113088814899:layer:Klayers-python37-pip:2 19.2.1
netcat arn:aws:lambda:<region>:113088814899:layer:Klayers-python37-netcat:2 0.7.12

1 Includes leptonica-1.76.0, libjpeg 6b (libjpeg-turbo 1.2.90) : libpng 1.2.49 : libtiff 4.0.3 : zlib 1.2.8 : libwebp 0.3.0

2 netcat-traditional with the -e option -- compiled for 64-bit to run on AWS Lambda

Layer specific notes

For pip you'll need to set your PYTHONPATH to /opt/python to override the existing pip inside the lambda environment.

There is an old spacy layer -- this has been deprecated.

To DO

  • Cleanup script to delete all lamba layers that are not the latest version (and more than x days old) Done using TTL on DynamoDB
  • Schedule this to run everyday for production Currently runs once a week on Monday
  • Package openssl v1.1.x for better encryption of KEK
  • Blog post and architecture review (WIP: here)

Architecture Diagram

Diagram is still pretty draft will add more details later!

Screenshot

Special Thanks

Chahna107 for adding tesseract config files into the tesseract layer.

You can’t perform that action at this time.