Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lambda toobig seatbelts #172

Merged
merged 7 commits into from Aug 26, 2017
Merged

Lambda toobig seatbelts #172

merged 7 commits into from Aug 26, 2017

Conversation

@ericmjonas
Copy link
Collaborator

@ericmjonas ericmjonas commented Aug 25, 2017

This is a series of fixes to handle:

  1. if the runtime is too big, communicate to the user and execute proper cleanup on the worker so the worker isn't forever broken. (issue #105)

  2. If downloading the func or the data is too large for the remaining space, error as well.

@ericmjonas
Copy link
Collaborator Author

@ericmjonas ericmjonas commented Aug 25, 2017

This brings up the question: Should we download the runtime before the user data/func pickles?

@@ -32,6 +32,8 @@

PROCESS_STDOUT_SLEEP_SECS = 2

TMP_MIN_FREE_SPACE_BYTES = 10000000

This comment has been minimized.

@shivaram

shivaram Aug 25, 2017
Collaborator

nit - can we add a comment above this "Leave at least 10MB free space" or something like that

This comment has been minimized.

@apengwin

apengwin Aug 25, 2017
Contributor

This could be 1e8 instead?

condatar.extractall(runtime_etag_dir)
try:
condatar.extractall(runtime_etag_dir)
except:

This comment has been minimized.

@shivaram

shivaram Aug 25, 2017
Collaborator

Is there a particular exception that indicates the tarball was too big ? I guess there could be other errors like corrupt runtimes etc.

This comment has been minimized.

@ericmjonas

ericmjonas Aug 25, 2017
Author Collaborator

Good question, I will dig further!


func_key_size = get_key_size(s3_client, s3_bucket, func_key)

free_disk_bytes = free_disk_space("/tmp")

This comment has been minimized.

@shivaram

shivaram Aug 25, 2017
Collaborator

is /tmp hardcoded elsewhere as well or is there some variable we can use here cc @apengwin

This comment has been minimized.

@apengwin

apengwin Aug 25, 2017
Contributor

/tmp is hardcoded in current wrenhandler. The portability PR abstracts this out to a generic TEMP variable that's set at the beginning, depending on what OS is running

This comment has been minimized.

@ericmjonas

ericmjonas Aug 25, 2017
Author Collaborator

It is actually hardcoded in the runtimes as well. Unfortunately there are some absolute path dependencies that continue to be a problem in the way conda is packaged.

ericmjonas added 4 commits Aug 26, 2017
Copy link
Collaborator

@shivaram shivaram left a comment

Thanks. One last question: is the catch all except at the end to handle python3 or is to handle other unforeseen exceptions ?

@ericmjonas
Copy link
Collaborator Author

@ericmjonas ericmjonas commented Aug 26, 2017

So remember right now we only support running under the python2 runtime for the handler; it's for all the exceptions that I wouldn't otherwise catch, to make sure we have a way to clean up the extracted runtime.

@shivaram
Copy link
Collaborator

@shivaram shivaram commented Aug 26, 2017

Ah got it. I forgot the handler was python2 specific. Sounds good.

@ericmjonas ericmjonas merged commit d528f10 into master Aug 26, 2017
2 checks passed
2 checks passed
continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details
@ericmjonas ericmjonas mentioned this pull request Aug 26, 2017
@ericmjonas ericmjonas deleted the lambda-toobig-seatbelts branch Sep 10, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

3 participants
You can’t perform that action at this time.