SNOW-143578: Unable to use on AWS Lambda

[valkolovos]

Please answer these questions before submitting your issue. Thanks!

1. What version of Python are you using (python --version)? 3.8

2. What operating system and processor architecture are you using (python -c 'import platform; print(platform.platform())')? AWS Lambda Python3.8 runtime

3. What are the component versions in the environment (pip list)?

Package         Version
--------------- -------
boto3           1.10.34
botocore        1.13.34
docutils        0.15.2
jmespath        0.9.4
pip             19.2.3
python-dateutil 2.8.0
rapid-client    0.0.0
s3transfer      0.2.1
setuptools      41.2.0
six             1.13.0
urllib3         1.25.7

4. What did you do?
   Created a lambda. Tried to execute SQL using snowflake connector (created connector, got cursor from connector, executed SQL).

5. What did you expect to see? Results of SQL query

6. What did you see instead?

[ERROR] OSError: [Errno 38] Function not implemented
Traceback (most recent call last):
  File "/var/task/app.py", line 34, in lambda_handler
    load_student_transactions(txn_students),
  File "/var/task/app.py", line 61, in load_student_transactions
    cs.execute(
  File "/var/task/snowflake/connector/cursor.py", line 580, in execute
    self._init_result_and_meta(data, _use_ijson)
  File "/var/task/snowflake/connector/cursor.py", line 633, in _init_result_and_meta
    self._result = self._json_result_class(data, self, use_ijson)
  File "/var/task/snowflake/connector/json_result.py", line 23, in __init__
    self._init_from_meta(raw_response, use_ijson)
  File "/var/task/snowflake/connector/json_result.py", line 59, in _init_from_meta
    self._chunk_downloader = self._connection._chunk_downloader_class(
  File "/var/task/snowflake/connector/chunk_downloader.py", line 101, in __init__
    self._pre_init(chunks, connection, cursor, qrmk, chunk_headers,
  File "/var/task/snowflake/connector/chunk_downloader.py", line 89, in _pre_init
    self._pool = ThreadPool(self._effective_threads)
  File "/var/lang/lib/python3.8/multiprocessing/pool.py", line 922, in __init__
    Pool.__init__(self, processes, initializer, initargs)
  File "/var/lang/lib/python3.8/multiprocessing/pool.py", line 196, in __init__
    self._change_notifier = self._ctx.SimpleQueue()
  File "/var/lang/lib/python3.8/multiprocessing/context.py", line 113, in SimpleQueue
    return SimpleQueue(ctx=self.get_context())
  File "/var/lang/lib/python3.8/multiprocessing/queues.py", line 336, in __init__
    self._rlock = ctx.Lock()
  File "/var/lang/lib/python3.8/multiprocessing/context.py", line 68, in Lock
    return Lock(ctx=self.get_context())
  File "/var/lang/lib/python3.8/multiprocessing/synchronize.py", line 162, in __init__
    SemLock.__init__(self, SEMAPHORE, 1, 1, ctx=ctx)
  File "/var/lang/lib/python3.8/multiprocessing/synchronize.py", line 57, in __init__
    sl = self._semlock = _multiprocessing.SemLock(

7. Can you set logging to DEBUG and collect the logs?

Problem appears to be lack of support for multiprocessing.Pool in AWS Lambda execution environment - see https://blog.ruanbekker.com/blog/2019/02/19/parallel-processing-on-aws-lambda-with-python-using-multiprocessing/

[Jopepato]

Hello @valkolovos . I have the same problem and also tried changing the version of the package, but it did not work either: https://community.snowflake.com/s/article/Python-connector-2-0-0-incompatible-with-AWS-Lambda

[slopyjoe]

Hey, I'm seeing the same issue as @valkolovos and @Jopepato. Any idea when this will be resolved or if there is an older version of the connector we can go back to?

[keller00]

Please note that version >=2.0.0 is not compatible for a different reason. You should not mix the 2 issues up (however the next release will have this unrelated problem fixed).

I will look into what's wrong with multiprocessing on AWS lambda

[JosefCzanko]

Hi @keller00 , we are experiencing the same problem and we would like to use python connector in Lambda. Any estimate when the solution could be available?

[keller00]

There's no estimate as of now. The thing is that we don't use multiprocessing pool directly. We use a ThreadPool for uploading/downloading and this uses internally multiprocessing. So we might need to rework quite a bit of the connector to get this working.

[asnir]

Hey @keller00 , Is there any update?

[jberkenbilt]

@keller00 Take a look at #304 and see if it's okay. I added code to fall back to a thread-based pool instead of a process-based pool if the process pool is broken, but in fact, the process pool is used in only two places and not used that heavily. It would be easy to replace multiprocessing.pool.ThreadPool with concurrent.futures.ThreadPoolExecutor, but there are differences between process and thread pools in terms of shared memory, and the behavior of the ThreadPoolExecutor is slightly difference from ThreadPool as it doesn't implement the terminate method. In any case, for my use case, with the patch in #304, I am able to do my work with lambda. The patch is relative to master, which at this moment, is also 2.2.5.

[itamar-analyst]

Hi @keller00 , any update? we have the same issue. tried many versions 1.9.1, 2.0.0, and above.
it will be great to know if a fix planned or we should find another workaround.

[sfc-gh-mkeller]

I merged my fix for this issue, if everything goes well then this will be included in the next release.
Please someone close this issue when you verified that the new version works on Lambda.

[jberkenbilt]

@sfc-gh-mkeller What will the version number of the release with the fix be?

[jberkenbilt]

and will a comment be made here when it is released, or if not, is there a way to be notified? Thanks.

[sfc-gh-mkeller]

I think so far it's 2.2.7, but this might change.

and will a comment be made here when it is released, or if not, is there a way to be notified? Thanks.

Sure, I can let you know!

[sfc-gh-mkeller]

But anyone can subscribe notification about releases through this site for free: https://libraries.io/pypi/snowflake-connector-python

[jberkenbilt]

I see 2.2.7 is out and includes the changes. I've redeployed my lambda functions with 2.2.7 and without my patch and will monitor it for a few hours. Thanks!

[jberkenbilt]

Running for several hours with zero errors. :-)

[sfc-gh-mkeller]

That is great to hear! Thanks again for you help with this @jberkenbilt !
Please let me know if anything is not working for anyone, I'm going to close this issue for now.