Make python docker interpreter work using py4j #16
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Really appreciate for backing up! 👍 |
|
@astroshim Done! |
|
In my testing, I got following error when i try to run |
|
Can you try with image |
|
I also tried with The error |
|
In my case, when i activate docker interpreter then python process killed and doesn't open new process. |
jongyoul
reviewed
Mar 18, 2017
| } | ||
|
|
||
| @Override | ||
| public InterpreterResult interpret(String st, InterpreterContext context) { |
There was a problem hiding this comment.
@Leemoonsoo Can you check this method? I think you miss actual running part of executing code. you'd check only "st == null", "activate", "deactivate" except "running code"
There was a problem hiding this comment.
PythonDockerInterpreter does not run any python code. It just replace python command for PythonInterpreter from 'python' to 'docker -i run ... 'python'.
So I think this method is okay.
There was a problem hiding this comment.
But... in case you try to run "print(1)", that method always return error, isn't it?
astroshim
pushed a commit
that referenced
this pull request
Mar 26, 2017
### What is this PR for? I've been testing the python interpreter and I found at least 4 major issues in the current python interpreter. 1. not working streaming output. - https://issues.apache.org/jira/browse/ZEPPELIN-2225 2. printed "..." when there is indent in the python code. - https://issues.apache.org/jira/browse/ZEPPELIN-1929 3. very slow output of matplotlib - https://issues.apache.org/jira/browse/ZEPPELIN-1894 - https://issues.apache.org/jira/browse/ZEPPELIN-1360 4. Unexpected output of matplotlib. - https://issues.apache.org/jira/browse/ZEPPELIN-2107 so I changed python interpreter to use py4j based on pyspark interpreter and would be fixed above issues. and I am going to recreate conda, docker for python interpreter ASAP. ### What type of PR is it? Bug Fix | Hot Fix | Refactoring ### How should this be tested? 1. not working streaming output. ``` import time for x in range(0, 5): print x time.sleep(1) ``` 2. printed "..." when there is indent in the python code. ``` def fn(): print("hi") fn() ``` 3. very slow output of matplotlib. ``` import matplotlib import sys import matplotlib.pyplot as plt plt.plot([1,2,3]) ``` 4. Unexpected output of matplotlib. ``` import matplotlib.pyplot as plt import matplotlib as mpl # Make a figure and axes with dimensions as desired. fig = plt.figure(figsize=(8, 3)) ax1 = fig.add_axes([0.05, 0.80, 0.9, 0.15]) ax2 = fig.add_axes([0.05, 0.475, 0.9, 0.15]) ax3 = fig.add_axes([0.05, 0.15, 0.9, 0.15]) # Set the colormap and norm to correspond to the data for which # the colorbar will be used. cmap = mpl.cm.cool norm = mpl.colors.Normalize(vmin=5, vmax=10) # ColorbarBase derives from ScalarMappable and puts a colorbar # in a specified axes, so it has everything needed for a # standalone colorbar. There are many more kwargs, but the # following gives a basic continuous colorbar with ticks # and labels. cb1 = mpl.colorbar.ColorbarBase(ax1, cmap=cmap, norm=norm, orientation='horizontal') cb1.set_label('Some Units') # The second example illustrates the use of a ListedColormap, a # BoundaryNorm, and extended ends to show the "over" and "under" # value colors. cmap = mpl.colors.ListedColormap(['r', 'g', 'b', 'c']) cmap.set_over('0.25') cmap.set_under('0.75') # If a ListedColormap is used, the length of the bounds array must be # one greater than the length of the color list. The bounds must be # monotonically increasing. bounds = [1, 2, 4, 7, 8] norm = mpl.colors.BoundaryNorm(bounds, cmap.N) cb2 = mpl.colorbar.ColorbarBase(ax2, cmap=cmap, norm=norm, # to use 'extend', you must # specify two extra boundaries: boundaries=[0] + bounds + [13], extend='both', ticks=bounds, # optional spacing='proportional', orientation='horizontal') cb2.set_label('Discrete intervals, some other units') # The third example illustrates the use of custom length colorbar # extensions, used on a colorbar with discrete intervals. cmap = mpl.colors.ListedColormap([[0., .4, 1.], [0., .8, 1.], [1., .8, 0.], [1., .4, 0.]]) cmap.set_over((1., 0., 0.)) cmap.set_under((0., 0., 1.)) bounds = [-1., -.5, 0., .5, 1.] norm = mpl.colors.BoundaryNorm(bounds, cmap.N) cb3 = mpl.colorbar.ColorbarBase(ax3, cmap=cmap, norm=norm, boundaries=[-10] + bounds + [10], extend='both', # Make the length of each extension # the same as the length of the # interior colors: extendfrac='auto', ticks=bounds, spacing='uniform', orientation='horizontal') cb3.set_label('Custom extension lengths, some other units') plt.show() ``` ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: astroshim <hsshim@zepl.com> Author: Lee moon soo <moon@apache.org> Author: HyungSung <hsshim@nflabs.com> Closes apache#2106 from astroshim/py4jPythonInterpreter and squashes the following commits: c9b195b [HyungSung] Merge pull request #16 from Leemoonsoo/py4jdocker e511ebe [Lee moon soo] add PythonDockerInterpreter to interpreter-setting.json a76b0d8 [Lee moon soo] fix test on python3 2eb5de7 [Lee moon soo] Fix PythonDockerInterpreterTest.java test 9fcf144 [Lee moon soo] Make python docker interpreter work using py4j 8a016c9 [astroshim] Merge branch 'master' into py4jPythonInterpreter aad7ee8 [astroshim] fix testcase ac92cdb [astroshim] fix python interpreter testcase e8570d2 [astroshim] fix ci for pandassql be5db4d [astroshim] fix pandas sql testcase f8e19be [astroshim] fix matplotlib testcase 046db88 [astroshim] add testcase e49ad24 [astroshim] add pandas 60e9820 [astroshim] bug fix about copying library 574bd21 [astroshim] fix interpreter-setting error a48df58 [astroshim] Merge branch 'master' into py4jPythonInterpreter 3c9585f [astroshim] update interpreter-setting.json a50179e [astroshim] add conda interpreter cbbc15c [astroshim] fix py4j path 5ae5120 [astroshim] fix interpreter-setting f17bff4 [astroshim] fix testcase failure. af097ac [astroshim] add testcase c3f5b78 [astroshim] Merge branch 'master' of https://github.com/apache/zeppelin into py4jPythonInterpreter 1395875 [astroshim] removed unnecessary code. 276011e [astroshim] add py4j lib 7304919 [astroshim] initialize python interpreter using py4j
astroshim
pushed a commit
that referenced
this pull request
Sep 12, 2017
### What is this PR for? I've been testing the python interpreter and I found at least 4 major issues in the current python interpreter. 1. not working streaming output. - https://issues.apache.org/jira/browse/ZEPPELIN-2225 2. printed "..." when there is indent in the python code. - https://issues.apache.org/jira/browse/ZEPPELIN-1929 3. very slow output of matplotlib - https://issues.apache.org/jira/browse/ZEPPELIN-1894 - https://issues.apache.org/jira/browse/ZEPPELIN-1360 4. Unexpected output of matplotlib. - https://issues.apache.org/jira/browse/ZEPPELIN-2107 so I changed python interpreter to use py4j based on pyspark interpreter and would be fixed above issues. and I am going to recreate conda, docker for python interpreter ASAP. ### What type of PR is it? Bug Fix | Hot Fix | Refactoring ### How should this be tested? 1. not working streaming output. ``` import time for x in range(0, 5): print x time.sleep(1) ``` 2. printed "..." when there is indent in the python code. ``` def fn(): print("hi") fn() ``` 3. very slow output of matplotlib. ``` import matplotlib import sys import matplotlib.pyplot as plt plt.plot([1,2,3]) ``` 4. Unexpected output of matplotlib. ``` import matplotlib.pyplot as plt import matplotlib as mpl # Make a figure and axes with dimensions as desired. fig = plt.figure(figsize=(8, 3)) ax1 = fig.add_axes([0.05, 0.80, 0.9, 0.15]) ax2 = fig.add_axes([0.05, 0.475, 0.9, 0.15]) ax3 = fig.add_axes([0.05, 0.15, 0.9, 0.15]) # Set the colormap and norm to correspond to the data for which # the colorbar will be used. cmap = mpl.cm.cool norm = mpl.colors.Normalize(vmin=5, vmax=10) # ColorbarBase derives from ScalarMappable and puts a colorbar # in a specified axes, so it has everything needed for a # standalone colorbar. There are many more kwargs, but the # following gives a basic continuous colorbar with ticks # and labels. cb1 = mpl.colorbar.ColorbarBase(ax1, cmap=cmap, norm=norm, orientation='horizontal') cb1.set_label('Some Units') # The second example illustrates the use of a ListedColormap, a # BoundaryNorm, and extended ends to show the "over" and "under" # value colors. cmap = mpl.colors.ListedColormap(['r', 'g', 'b', 'c']) cmap.set_over('0.25') cmap.set_under('0.75') # If a ListedColormap is used, the length of the bounds array must be # one greater than the length of the color list. The bounds must be # monotonically increasing. bounds = [1, 2, 4, 7, 8] norm = mpl.colors.BoundaryNorm(bounds, cmap.N) cb2 = mpl.colorbar.ColorbarBase(ax2, cmap=cmap, norm=norm, # to use 'extend', you must # specify two extra boundaries: boundaries=[0] + bounds + [13], extend='both', ticks=bounds, # optional spacing='proportional', orientation='horizontal') cb2.set_label('Discrete intervals, some other units') # The third example illustrates the use of custom length colorbar # extensions, used on a colorbar with discrete intervals. cmap = mpl.colors.ListedColormap([[0., .4, 1.], [0., .8, 1.], [1., .8, 0.], [1., .4, 0.]]) cmap.set_over((1., 0., 0.)) cmap.set_under((0., 0., 1.)) bounds = [-1., -.5, 0., .5, 1.] norm = mpl.colors.BoundaryNorm(bounds, cmap.N) cb3 = mpl.colorbar.ColorbarBase(ax3, cmap=cmap, norm=norm, boundaries=[-10] + bounds + [10], extend='both', # Make the length of each extension # the same as the length of the # interior colors: extendfrac='auto', ticks=bounds, spacing='uniform', orientation='horizontal') cb3.set_label('Custom extension lengths, some other units') plt.show() ``` ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: astroshim <hsshim@zepl.com> Author: Lee moon soo <moon@apache.org> Author: HyungSung <hsshim@nflabs.com> Closes apache#2106 from astroshim/py4jPythonInterpreter and squashes the following commits: c9b195b [HyungSung] Merge pull request #16 from Leemoonsoo/py4jdocker e511ebe [Lee moon soo] add PythonDockerInterpreter to interpreter-setting.json a76b0d8 [Lee moon soo] fix test on python3 2eb5de7 [Lee moon soo] Fix PythonDockerInterpreterTest.java test 9fcf144 [Lee moon soo] Make python docker interpreter work using py4j 8a016c9 [astroshim] Merge branch 'master' into py4jPythonInterpreter aad7ee8 [astroshim] fix testcase ac92cdb [astroshim] fix python interpreter testcase e8570d2 [astroshim] fix ci for pandassql be5db4d [astroshim] fix pandas sql testcase f8e19be [astroshim] fix matplotlib testcase 046db88 [astroshim] add testcase e49ad24 [astroshim] add pandas 60e9820 [astroshim] bug fix about copying library 574bd21 [astroshim] fix interpreter-setting error a48df58 [astroshim] Merge branch 'master' into py4jPythonInterpreter 3c9585f [astroshim] update interpreter-setting.json a50179e [astroshim] add conda interpreter cbbc15c [astroshim] fix py4j path 5ae5120 [astroshim] fix interpreter-setting f17bff4 [astroshim] fix testcase failure. af097ac [astroshim] add testcase c3f5b78 [astroshim] Merge branch 'master' of https://github.com/apache/zeppelin into py4jPythonInterpreter 1395875 [astroshim] removed unnecessary code. 276011e [astroshim] add py4j lib 7304919 [astroshim] initialize python interpreter using py4j (cherry picked from commit 287ffd5) Signed-off-by: Jongyoul Lee <jongyoul@apache.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Bringing PythonDockerInterpreter back!