TA-Lib on AWS Lambda

[MrTeale]

I've been trying to get TA-Lib working on AWS Lambda utilising a Docker image.

Below is my current Dockerfile:

# AWS Lambda execution environment is based on Amazon Linux 1
FROM amazonlinux:1

# Install Python 3.6
RUN yum -y install python36 python36-pip
RUN curl -s https://bootstrap.pypa.io/get-pip.py | python3
RUN pip3 install numpy setuptools

# Install your dependencies
RUN yum -y install python36-devel gcc
RUN curl -SLO http://downloads.sourceforge.net/project/ta-lib/ta-lib/0.4.0/ta-lib-0.4.0-src.tar.gz && \
    tar -xvzf ta-lib-0.4.0-src.tar.gz && \
    cd ./ta-lib/ && \
    ./configure --prefix=/usr && \
    make && \
    make install

RUN rm -R ta-lib-0.4.0-src.tar.gz ta-lib
ENV LD_LIBRARY_PATH="/usr/lib:$LD_LIBRARY_PATH" 

# Set the same WORKDIR as default image
RUN mkdir /var/task
WORKDIR /var/task 

# BUILD WITH docker build -t lambdadocker .

I'm utilising serverless-python-requirements to package all the necessary requirements from requirements.txt into the docker image before uploading it to lambda.

I'm receiving the following traceback from Cloudwatch Logs:

libta_lib.so.0: cannot open shared object file: No such file or directory: ImportError
Traceback (most recent call last):
File "/var/task/analysis.py", line 13, in main
module = importlib.import_module('TA.' + analysis_name)
File "/var/lang/lib/python3.6/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 978, in _gcd_import
File "<frozen importlib._bootstrap>", line 961, in _find_and_load
File "<frozen importlib._bootstrap>", line 950, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 678, in exec_module
File "<frozen importlib._bootstrap>", line 205, in _call_with_frames_removed
File "/var/task/TA/macd.py", line 1, in <module>
import talib
File "/var/task/talib/__init__.py", line 43, in <module>
from ._ta_lib import (
ImportError: libta_lib.so.0: cannot open shared object file: No such file or directory

Does anyone have any ideas whether I'm just not building the image correctly? or whether I'm missing an environment variable?

Thanks heaps in advance, been racking my brain on this for a solid week.

[fmichaud]

@MrTeale : Hello from Paris,

I built a Docker image using your Dockerfile without difficulties and I could run perfectly the container.

I think this post could help you : https://forum.serverless.com/t/aws-python-function-dependencies-load/451/4

Hoping to have helped you,
François.

[AwolDes]

The latest comment on this issue helped me figure it out - #205 (comment)

I needed to include the TA-Lib binary in lib/, and instead of pip installing the talib package using serverless-python-requirements, I just included the talib package in my root directory.

Edit: only downside to this is the package size becomes ~40mb

[MrTeale]

As per @AwolDes' comment: I needed to include the TA-Lib binary in lib/, and instead of pip installing the talib package using serverless-python-requirements, I just included the talib package in my root directory. Unlike that comment though, I didn't include numpy manually downloaded instead leaving it within the serverless-python-requirements.

My total package size is only 2mb more than previously (23mb total)

See: #205 for solution