New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Only *.py in modules are uploaded #79

Closed
ooq opened this Issue Mar 1, 2017 · 3 comments

Comments

Projects
None yet
2 participants
@ooq
Collaborator

ooq commented Mar 1, 2017

This python program (https://gist.github.com/ooq/48c23aec420c682359d6a43cfc9cdd82) of mine throws Exception: [Errno 2] No such file or directory on remote but runs fine locally. Also it was working yesterday (was the runtime updated?) I suspect this could be a runtime issue because my local runtime uses a different boto3 botocore version.

I'm going to debug more today by trying these two things:

  1. upload my runtime
  2. pull the remote runtime and setup locally
@ooq

This comment has been minimized.

Show comment
Hide comment
@ooq

ooq Mar 2, 2017

Collaborator

Okay, problem identified. Boto3 will automatically detect certifi and use it for (maybe) some server authentication. This means that if certifi sits in my local env, Pywren will copy it to remote as a dependent module. However, certifi contains a bunch of .pem files that are ignored, as we only copy .py files:

files = glob2.glob(os.path.join(m, "**/*.py"))

This is why I'm getting the No such file or directory error, when certifi can't find those .pem files.

@ericmjonas

Collaborator

ooq commented Mar 2, 2017

Okay, problem identified. Boto3 will automatically detect certifi and use it for (maybe) some server authentication. This means that if certifi sits in my local env, Pywren will copy it to remote as a dependent module. However, certifi contains a bunch of .pem files that are ignored, as we only copy .py files:

files = glob2.glob(os.path.join(m, "**/*.py"))

This is why I'm getting the No such file or directory error, when certifi can't find those .pem files.

@ericmjonas

@ooq

This comment has been minimized.

Show comment
Hide comment
@ooq

ooq Mar 2, 2017

Collaborator

I'll push a quick fix that puts certifi on the preinstalled list.
We should rebuild the runtimes to include certifi (more security always seems better)

Collaborator

ooq commented Mar 2, 2017

I'll push a quick fix that puts certifi on the preinstalled list.
We should rebuild the runtimes to include certifi (more security always seems better)

@ooq

This comment has been minimized.

Show comment
Hide comment
@ooq

ooq Mar 2, 2017

Collaborator

For:
files = glob2.glob(os.path.join(m, "**/*.py"))
A better approach is to include all files but exclude *.pyc files?

Collaborator

ooq commented Mar 2, 2017

For:
files = glob2.glob(os.path.join(m, "**/*.py"))
A better approach is to include all files but exclude *.pyc files?

ooq added a commit that referenced this issue Mar 2, 2017

Merge pull request #80 from pywren/certifi-bug
[Issue #79] Add certifi in preinstalls.py

@ooq ooq changed the title from Runtime compatibility to Only *.py in modules are uploaded Mar 2, 2017

@ericmjonas ericmjonas modified the milestone: v0.2 Mar 9, 2017

@ericmjonas ericmjonas closed this Mar 22, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment