Run into Could not start SASL: Error in sasl_client_start (-4) SASL(-4) when using AWS Lambda

[liusztc09]

I'm trying to use impyla with aws lambda function. Everything worked well when I run the code on the ec2 instance running Linux. However, when I build the deployment package which include sasl, thrift_sasl and impyla modules and test my code in lambda, it gave me the error: Could not start SASL: Error in sasl_client_start (-4) SASL(-4): no mechanism available: No worthy mechs found
I notice a lot of people reporting this error but it works for me on linux without issues. Since lambda requires supplying all the libraries in the deployment package, I think there must be something missing if I just supply the three module listed above. What libs dependencies does sasl require? I need to rebuild the module with the missing dependencies and make them part of the package.

Thanks!

[wesm]

Can you show what you have tried? (code)

[liusztc09]

Hi Wes,

Thanks for your reply. Here is my code.

from impala.dbapi import connect

conn = connect('ec2-54-152-186-202.compute-1.amazonaws.com', port=10000, auth_mechanism='PLAIN')
cursor = conn.cursor()

def handler(event, context):
cursor.execute('show tables')
results = cursor.fetchall()
print(results)

It runs well on my mac and ec2 instance, but not AWS Lambda

I installed:
sudo yum install gcc-c++
sudo yum install cyrus-sasl-devel
And I include the following into the lambda deployment package
thrift_sasl,sasl,impyla

It looks like under the /usr/lib64/ of the ec2 instance there is a sasl2 and libsasl2.so which should be part of the deployment package. I should I set the path for this, looks like the error comes from thrift_sasl. How can I build it with all the dependencies on linux(EC2 for my case)?

Thanks!

[liusztc09]

Here's an example. https://github.com/jkehler/awslambda-psycopg2
For the popular module pyscopg2 to work with AWS Lambda, we need to make a custom build with static link to libpq. I would assume we need to do a similar to thrift_sasl, or all three of thrift_sasl, sasl and impyla to make it work. If so, what are the libraries and dependencies I should build with?

[alexgibs]

I've just spent a fair bit of time getting this working in AWS Lambda, so I'm updating here for future reference.

I encountered the same error from thrift_sasl: "Could not start SASL: Error in sasl_client_start (-4) SASL(-4): no mechanism available: No worthy mechs found"

We do not have to statically link these libraries. We need to make sure the shared objects are packaged with our Lambda function, so copy both libsasl2.so and the contents of /usr/lib64/sasl2/ to a folder named 'lib' in the root of your Lambda function package. LD_LIBRARY_PATH is set in Lambda to look in this location when the function runs.

libsasl2.so will still try to look for libraries in the /usr/lib64/sasl2 location. In order to have it look in the lib folder where our Lambda function exists, we need to set SAML_PATH. This can be done at the top of your function code before importing thrift_sasl (or importing the dependent module):

  import os
  os.environ['SASL_PATH'] = os.path.join(os.getcwd(), 'lib')
  from pyhive import hive

Note: make sure you are copying the libraries from the same ami that Lambda uses: http://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html

[moshir]

Hi,

I had the same issue, and followed your procedure with no success.
Is it SASL_PATH like in your code, or SAML_PATH like in your comment ?
(Tried both with no success)
Additionally, libsasl2 is already available in Lamdba, as found here, but under /usr/lib64/libsasl2.so.2 :
https://gist.github.com/gojko/aaea440be880d0162767639f8a6681f9
Any thoughts ?

[srhopkins]

I was able to resolve this using @alexgibs #201 (comment) solution but I needed to add additional packages before copying the contents of /usr/lib64/sasl2/. I skipped copying libsasl2.so as it is already baked into Amazon Linux images.

bash-4.2# # FYI: cyrus-sasl-lib should already be installed on amazon linux
bash-4.2# yum install cyrus-sasl-lib cyrus-sasl-gssapi cyrus-sasl-md5 

For the env var I used SASL_PATH.

[gaurang101197]

If you are using ldap as a authentication then install library using sudo yum install cyrus-sasl-ldap and then copying /usr/lib64/sasl2/ to your lambda package. And set os.environ['SASL_PATH'] = os.path.join(os.getcwd(), 'Path to content of /usr/lib64/sasl2/ in your lambda package.'). I did silly mistake while setting SASL_PATH, I have copied sasl2 folder to lib folder in my lambda, so in my lambda, folder structure was like lib/sasl2/content and I was setting os.path.join(os.getcwd(), 'lib') as path instead os.path.join(os.getcwd(), 'lib/sasl2'). After installing ldap dependencies and setting right SASL_PATH path in lambda, I am able to connect hive server2 with ldap authentication using pyhive library. Thanks @srhopkins and @alexgibs

[Bhupendra8992]

I followed below step but still I am unable to establish connection to Hive from Lambda

1. I spin up my EC2 instance and ran below command to install all the required modules

• sudo yum update -y
• sudo yum install -y python38
• sudo yum install -y gcc
• sudo yum install -y gcc-c++
• sudo yum install -y python38-devel.x86_64
• sudo yum install -y cyrus-sasl-devel.x86_64
• sudo pip-3.8 install boto3
• sudo pip-3.8 install pyhive==0.6.2
• sudo pip-3.8 install thrift==0.13.0
• sudo pip-3.8 install sasl==0.2.1
• sudo pip-3.8 install thrift_sasl==0.4.2

2. After successful completion of above commands, I took all the files from /usr/lib64/sasl2/ (from EC2) and set os.environ['SASL_PATH'] = os.path.join(os.getcwd(), ‘lib’) (lib is a folder at root level in my lambda function)
   Below are all the libraries kept inside lib folder in lambda

Refer attached screenshot sasl library files kept inside lib folder.png file

3. Post this when I try to run Lambda it gives me error as
   libsasl2.so.3: cannot open shared object file: No such file or directory

4. Currently this file is kept in lib folder in lambda at root level, If I keep it outside lib folder along with one more file libpython2.7.so.1.0 then error mentioned in point 3 goes away and lambda then gives another error on execution
   dynamic module does not define module export function (PyInit_saslwrapper)

Couple of things to note here
• Even though I have kept all SASL libraries inside lib folder in lambda still I have created a separate sasl folder at root level in lambda that contains all code files

Refer the attached screenshot sasl code files kept inside sasl folder.png for reference

Can someone please guide me how can I resolve above errors and establish connection to Hive from Lambda