Skip to content

Lambda toobig seatbelts #172

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Aug 26, 2017
Merged

Lambda toobig seatbelts #172

merged 7 commits into from
Aug 26, 2017

Conversation

ericmjonas
Copy link
Collaborator

This is a series of fixes to handle:

  1. if the runtime is too big, communicate to the user and execute proper cleanup on the worker so the worker isn't forever broken. (issue Too big of runtime #105)

  2. If downloading the func or the data is too large for the remaining space, error as well.

@ericmjonas
Copy link
Collaborator Author

This brings up the question: Should we download the runtime before the user data/func pickles?

@@ -32,6 +32,8 @@

PROCESS_STDOUT_SLEEP_SECS = 2

TMP_MIN_FREE_SPACE_BYTES = 10000000
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit - can we add a comment above this "Leave at least 10MB free space" or something like that

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could be 1e8 instead?

condatar.extractall(runtime_etag_dir)
try:
condatar.extractall(runtime_etag_dir)
except:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a particular exception that indicates the tarball was too big ? I guess there could be other errors like corrupt runtimes etc.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question, I will dig further!


func_key_size = get_key_size(s3_client, s3_bucket, func_key)

free_disk_bytes = free_disk_space("/tmp")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is /tmp hardcoded elsewhere as well or is there some variable we can use here cc @apengwin

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/tmp is hardcoded in current wrenhandler. The portability PR abstracts this out to a generic TEMP variable that's set at the beginning, depending on what OS is running

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is actually hardcoded in the runtimes as well. Unfortunately there are some absolute path dependencies that continue to be a problem in the way conda is packaged.

Copy link
Collaborator

@shivaram shivaram left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. One last question: is the catch all except at the end to handle python3 or is to handle other unforeseen exceptions ?

@ericmjonas
Copy link
Collaborator Author

So remember right now we only support running under the python2 runtime for the handler; it's for all the exceptions that I wouldn't otherwise catch, to make sure we have a way to clean up the extracted runtime.

@shivaram
Copy link
Collaborator

Ah got it. I forgot the handler was python2 specific. Sounds good.

@ericmjonas ericmjonas merged commit d528f10 into master Aug 26, 2017
@ericmjonas ericmjonas mentioned this pull request Aug 26, 2017
@ericmjonas ericmjonas deleted the lambda-toobig-seatbelts branch September 10, 2017 13:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants