New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Issue#108] Refactor Storage API #119

merged 22 commits into from Jul 2, 2017

Address comments.

  • Loading branch information...
ooq committed Apr 17, 2017
commit 4e61199169a6be7926c37cc05a7fab41345495ec
@@ -5,11 +5,9 @@
import as storage
import pywren.wrenconfig as wrenconfig
# FIXME separate runtime code with S3
def get_runtime_info(runtime_config, storage_handler = None):
Download runtime information from S3 at deserialize
Download runtime information from storage at deserialize
if storage_handler is None:
storage_config = wrenconfig.extract_storage_config(wrenconfig.default())
@@ -27,6 +27,7 @@ def put_object(self, key, data):
Put an object in S3.
:param key: key of the object.
:param data: data of the object
:type data: str/bytes
:return: None
self.s3client.put_object(Bucket=self.s3_bucket, Key=key, Body=data)
@@ -36,6 +37,7 @@ def get_object(self, key):
Get object from S3 with a key.
:param key: key of the object
:return: Data of the object

This comment has been minimized.


shivaram Apr 17, 2017


Can we specify the type of the return object ? i.e. is it a byte array or an iterator etc ?

This comment has been minimized.


ooq Apr 17, 2017



:rtype: str/bytes
r = self.s3client.get_object(Bucket = self.s3_bucket, Key = key)
data = r['Body'].read()
@@ -67,43 +67,6 @@ def remote_executor(config=None, job_max_runtime=3600):
standalone_executor = remote_executor
def save_futures_to_string(futures):
Saving a list of futures to string.
:param futures: A list of futures to save.
:return: A serialized string.
return pickle.dumps(list(futures))
def load_futures_from_string(fut_str):
Load futures from a string.
:param fut_str: A string.
:return: A list of futures.
return list(pickle.loads(fut_str))
def save_futures(futures, filename):
Save a list of futures into a file.
:param futures: A list of futures
:param filename: file name
:return: None
pickle.dump(list(futures), open(filename, "wb"))
def load_futures(filename):
Load futures from file
:param filename: file name to load from
:return: A list of futures
return list(pickle.load(open(filename, "rb")))
def get_all_results(fs):
Take in a list of futures and block until they are repeated,
@@ -212,8 +212,8 @@ def plus_one(x):
x = np.arange(N)
futures_original =, x)
futures_str = pywren.save_futures_to_string(futures_original)
futures = pywren.load_futures_from_string(futures_str)
futures_str = pickle.dumps(futures_original)
futures = pickle.loads(futures_str)
result_count = 0
while result_count < N:
ProTip! Use n and p to navigate between commits in a pull request.