Add support for running python functions with wheel file #2593
Conversation
*Motivation* docker orgnization is missing for building test image. so the build will be failing with `-Pdocker`. *Changes* Move the docker organization parameter to root pom file.
|
|
||
| if function_details.runtime == Function_pb2.FunctionDetails.Runtime.Value("PYTHON_WHEEL"): | ||
| try: | ||
| util.install_wheel(str(args.py)) |
There was a problem hiding this comment.
I am not familiar with python. but isn't that manipulating the environment, and doesn't it require sudos and such?
There was a problem hiding this comment.
I've redone that part with installing using virtualenv to minimize impact on the system and needing permissions
|
@jerrypeng Have removed the string convertor. |
| @@ -229,6 +229,10 @@ void processArguments() throws Exception { | |||
| description = "Path to the main Python file for the function (if the function is written in Python)", | |||
| listConverter = StringConverter.class) | |||
| protected String pyFile; | |||
| @Parameter( | |||
| names = "--pywheel", | |||
There was a problem hiding this comment.
Instead of having a new switch, could we reuse the --py and then disambiguate based on the file extension?
|
@merlimat I have removed the new runtime. Please take a look again. Thanks! |
| @@ -74,6 +73,11 @@ def main(): | |||
| args = parser.parse_args() | |||
| function_details = Function_pb2.FunctionDetails() | |||
| json_format.Parse(args.function_details, function_details) | |||
|
|
|||
| if os.path.splitext(str(args.py))[1] == '.whl': | |||
| os.system("unzip -d %s -o %s" % (os.path.dirname(str(args.py)), str(args.py))) | |||
There was a problem hiding this comment.
unzip is a new dependency.
There was a problem hiding this comment.
where is this unzipped to?
There was a problem hiding this comment.
it is unzipped into the same tmp directory the .whl file is copied to. The -d option controls that.
There was a problem hiding this comment.
unzip is indeed a new dependency.
There was a problem hiding this comment.
unzip might not be always installed (especially in Docker images where everything is stripped to bone).
Python has a native way to deal with zip files, without invoking the CLI command: https://docs.python.org/3/library/zipfile.html#zipfile.ZipFile.extract
There was a problem hiding this comment.
I have removed the unzip dependency by switching over to zipfile.
|
run java8 tests |
|
run java8 tests |
6 similar comments
|
run java8 tests |
|
run java8 tests |
|
run java8 tests |
|
run java8 tests |
|
run java8 tests |
|
run java8 tests |
|
run cpp tests |
|
rum java8 tests |
|
run java8 tests |
2 similar comments
|
run java8 tests |
|
run java8 tests |
|
rerun tests |
|
run cpp tests |
|
run cpp tests |
Motivation
Currently running python functions require all function logic to be contained within one python file. This pr allows python wheel files to be submitted to be run as functions, thereby greatly increasing flexibility
Modifications
Describe the modifications you've done.
Result
After your change, what will change.