SciPy module not found

[alfonsomhc]

Context

I have these settings:

"slim_handler": true,
"use_precompiled_packages": true

My application requires scipy, and it is installed in the virtual environment. When I deploy my application I see this:

Calling deploy for stage dev..
Downloading and installing dependencies..
 - scipy==0.19.0: Using locally cached manylinux wheel
File is not a zip file

Expected Behavior

After zappa deploy dev it says "Deployment complete" in green letters, so I would expect the application to run correctly (e.g. answer curl requests correctly).

Actual Behavior

If I send a curl request, I get

HTTP/1.1 500 Internal Server Error

and the error message ends with

"TypeError: 'NoneType' object is not callable\n"]

If I run zappa tail, I see this:

[1505391412036] No module named 'scipy': ModuleNotFoundError
Traceback (most recent call last):
  File "/var/task/handler.py", line 505, in lambda_handler
  return LambdaHandler.lambda_handler(event, context)
  File "/var/task/handler.py", line 239, in lambda_handler
  handler = cls()
  File "/var/task/handler.py", line 131, in __init__
  self.app_module = importlib.import_module(self.settings.APP_MODULE)
  File "/var/lang/lib/python3.6/importlib/__init__.py", line 126, in import_module
  return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 978, in _gcd_import
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load
  File "<frozen importlib._bootstrap>", line 950, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 205, in _call_with_frames_removed
  File "/tmp/channel-recomme/channel_recommender/webapp/webapp.py", line 24, in <module>
  from channel_recommender.recommender.loader import service_loader
  File "/tmp/channel-recomme/channel_recommender/recommender/loader.py", line 7, in <module>
  from sklearn.externals import joblib
  File "/tmp/channel-recomme/sklearn/__init__.py", line 134, in <module>
  from .base import clone
  File "/tmp/channel-recomme/sklearn/base.py", line 10, in <module>
  from scipy import sparse
ModuleNotFoundError: No module named 'scipy'

If I undeploy, disable using precompiled packages, i.e. I set this value:
"use_precompiled_packages": false
and then redeploy, then everything works as expected, i.e. the application answers curl correctly.

This is a bit strange, because initially (two weeks ago more or less) my application required scipy (I used scikit-learn, which required numpy and scipy) and this issue was not present (i.e. I could deploy the application without including use_precompiled_packages in my settings (i.e. it got the default value which is true).

However, a newer version of my application requires both scikit-learn and pandas. Therefore, I added pandas to the virtual environment. After that, the next time that I deployed I found the issue that I described.

I have tried uninstalling the packages involved and installing them again, but the issue is still there. I have tried undeploying the application and deploying it again, without any change. The only thing that seems to work is to set use_precompiled_packages to false, but as I mentioned earlier, this was initially not necessary.

[SongYunSeop]

@Miserlou

I had same problem.

When creating zip file for lambda, Zappa get a manylinux cached wheel file from local directory.

But When Zappa reading cached wheel file, sometimes raise BadZipfile exception.

I don't know why exactly, But I just know wheel file was broken.

So If I delete wheel file, and Re-deploy, Zappa re-downloading wheel file that is not broken.

I think this line is problem.

If BadZipfile exception raise, just print error message(File is not a zip file)

This is clearly error. So I think in this situation, Zappa should cancel deploy(packaging) or fix this error.

I changed code for this problem. Can it be help @Miserlou?

[Miserlou]

Related: #78

Which Python version are you guys using?

[Miserlou]

@SongYunSeop - feel free to submit a PR to improve that exception!

[alfonsomhc]

@Miserlou , I use Python 3.6.2

[Miserlou]

I have a feeling this is a P3-only bug for now.

There are two bugs here: one for the zip failing (which may be an upstream problem, IDK), and one that the failure isn't halting the package deployment.

[alfonsomhc]

@SongYunSeop, where did you delete the wheel file? I have deleted my folder
~/.cache/pip
but next time I run zappa update dev, I got

Using locally cached manylinux wheel

so zappa must be using a different cache folder...

[SongYunSeop]

@Miserlou I use Python 2.7.13.

OK. I will send PR!

[SongYunSeop]

@alfonsomhc

In my case, It is /var/folders/57/ffm0h96j1szfjzp6qr83w67h0000gn/T/cached_wheels.

You can get this directory.

import tempfile
print(tempfile.gettempdir() + "cached_wheels")

[alfonsomhc]

@SongYunSeop , thanks a lot for your answer. I found that my folder was
/tmp/cached_wheels
and there I found the scipy package:
0 Sep 13 18:04 scipy-0.19.0-cp36m-manylinux1_x86_64.whl
where 0 was the size. So as you said, the cached package had an issue (in my case it was empty!).

[Miserlou]

Ahhhhhhh. Excellent find. We could add a basic integrity check to the .whl here.

[SongYunSeop]

@Miserlou I send PR!
Thanks.

[Miserlou]

Merged, does this address the full issue though? Should the behavior at https://github.com/Miserlou/Zappa/blob/master/zappa/core.py#L560 changed before this can be closed?

[SongYunSeop]

@Miserlou
I think if any exception raise, Zappa should cancel all behavior.

The problem is that we do not know what exceptions will occur.

Why not print an error message like "Please report issue on https://github.com/Miserlou/Zappa/issues/new" and exit when an unknown exception occurs?

How do you think about this?

[alfonsomhc]

For reference, the next bump in my adventure is #1108

[alfonsomhc]

I updated zappa after #1107 and verified that zappa update dev downloaded a new scipy whl when the cached whl was empty.

[SongYunSeop]

@Miserlou #1110 can remove some more exception.

But we must fix https://github.com/Miserlou/Zappa/blob/master/zappa/core.py#L560.

[Miserlou]

Can this be closed?

[alfonsomhc]

@Miserlou , my problem was solved so I close the issue.

[loscil06]

I propose this issue to be reopened, it's happening again.

[cyrillay]

Ran into the same issue, spent a while figuring out what was wrong !

Thanks a lot @alfonsomhc for the "use_precompiled_packages": false, this solved it for me.

[dpmccabe]

Unfortunately, use_precompiled_packages isn't a viable option if you're anywhere near the Lambda size limit...