Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for %%spark magic command in python #8189

Open
jaroslawmalekcodete opened this issue Nov 28, 2019 · 0 comments
Open

Add support for %%spark magic command in python #8189

jaroslawmalekcodete opened this issue Nov 28, 2019 · 0 comments
Assignees
@ildipo

Comments

@jaroslawmalekcodete
Copy link
Contributor

No description provided.

@jaroslawmalekcodete jaroslawmalekcodete self-assigned this Nov 28, 2019
jaroslawmalekcodete added a commit that referenced this issue Nov 28, 2019
@jaroslawmalekcodete
#8189: Spark magic command for python
98a2f91
piorek added a commit that referenced this issue Dec 19, 2019
@piorek
#8189 WIP - proof of concept
126977e
LeeTZ pushed a commit that referenced this issue Jan 27, 2020
@jaroslawmalekcodete @LeeTZ
#8189: Spark magic command for python (#8190)
40f0817
piorek added a commit that referenced this issue Feb 5, 2020
@piorek
#8189 WIP
db777c9
piorek added a commit that referenced this issue Feb 5, 2020
@piorek
#8189 WIP
a8ddfac
piorek added a commit that referenced this issue Feb 5, 2020
@piorek
#8189 WIP
4f29a9b
piorek added a commit that referenced this issue Feb 5, 2020
@piorek
#8189 WIP
2ea426b
piorek added a commit that referenced this issue Feb 5, 2020
@piorek
#8189 WIP
ca9681a
piorek added a commit that referenced this issue Feb 5, 2020
@piorek
#8189 WIP
27267aa
piorek added a commit that referenced this issue Feb 5, 2020
@piorek
#8189 WIP
9e11d29
piorek added a commit that referenced this issue Feb 5, 2020
@piorek
#8189 WIP
e092e21
piorek added a commit that referenced this issue Feb 5, 2020
@piorek
#8189 WIP
4307a69
piorek added a commit that referenced this issue Feb 5, 2020
@piorek
#8189 WIP
46b6332
piorek added a commit that referenced this issue Feb 5, 2020
@piorek
#8189 WIP
65dfbbe
piorek added a commit that referenced this issue Feb 5, 2020
@piorek
#8189 WIP
804b845
piorek added a commit that referenced this issue Feb 5, 2020
@piorek
#8189 WIP
9d0338f
piorek added a commit that referenced this issue Feb 11, 2020
@piorek
#8189 WIP
12619e0
piorek added a commit that referenced this issue Feb 13, 2020
@piorek
#8189 WIP
a36793a
piorek added a commit that referenced this issue Feb 13, 2020
@piorek
#8189 WIP
dd189d9
piorek added a commit that referenced this issue Feb 13, 2020
@piorek
#8189 WIP
9c47977
piorek added a commit that referenced this issue Feb 13, 2020
@piorek
#8189 WIP
940d731
piorek added a commit that referenced this issue Feb 13, 2020
@piorek
#8189 WIP
f0567d0
piorek added a commit that referenced this issue Feb 13, 2020
@piorek
#8189 WIP
bcc2942
piorek added a commit that referenced this issue Feb 14, 2020
@piorek
#8189 WIP
da94ab9
piorek added a commit that referenced this issue Feb 17, 2020
@piorek
#8189 fix - work around ts limitation
f844b9c
jaroslawmalekcodete added a commit that referenced this issue Feb 17, 2020
@jaroslawmalekcodete
#8189: run widget SparkUI from spark python magic command
63c0e29
piorek added a commit that referenced this issue Feb 18, 2020
@piorek
#8189 Send start event to backend
e821310
piorek added a commit that referenced this issue Feb 18, 2020
@piorek
#8189 fix add property; send stop message
828f890
jaroslawmalekcodete added a commit that referenced this issue Feb 18, 2020
@jaroslawmalekcodete
#8189: spark python magic command -> start and stop event
edc68d2
piorek added a commit that referenced this issue Mar 6, 2020
@piorek
#8189 %%spark --start handling at front
76f43e7
piorek added a commit that referenced this issue Mar 6, 2020
@piorek
#8189 fix tests
2ded1c0
piorek added a commit that referenced this issue Mar 6, 2020
@piorek
#8189 fix npm WARN npm npm does not support Node.js vX.Y.Z
8428e55
jaroslawmalekcodete added a commit that referenced this issue Mar 10, 2020
@jaroslawmalekcodete
#8189: [python] auto start
74ac9ad
piorek added a commit that referenced this issue Mar 11, 2020
@piorek
#8189 improve error handling
062deba
jaroslawmalekcodete added a commit that referenced this issue Mar 12, 2020
@jaroslawmalekcodete
#8189: [python] --noUI
4a5b408
jaroslawmalekcodete added a commit that referenced this issue Mar 17, 2020
@jaroslawmalekcodete
#8189: [python] reflect user spark conf in SparkUIForm
4b0c93b
piorek added a commit that referenced this issue Mar 17, 2020
@piorek
#8189 add spinner; improve styles
f7f0bfd
piorek added a commit that referenced this issue Mar 17, 2020
@piorek
#8189 more style improvements
f9f4662
piorek added a commit that referenced this issue Mar 17, 2020
@piorek
#8189 user_spark_conf handling
b2ce15c
jaroslawmalekcodete added a commit that referenced this issue Mar 17, 2020
@jaroslawmalekcodete
#8189: [java] reflect user spark conf in SparkUIForm
bb59221
jaroslawmalekcodete added a commit that referenced this issue Mar 17, 2020
@jaroslawmalekcodete
#8189: [python] reflect user spark conf in SparkUIForm update the exa…
1b67c29 
…mple
piorek added a commit that referenced this issue Mar 19, 2020
@piorek
#8189 split spinner into own file
0824d89
piorek added a commit that referenced this issue Mar 19, 2020
@piorek
#8189 make error color red
7b32c20
piorek added a commit that referenced this issue Mar 19, 2020
@piorek
#8189 fix enable hive support checkout
a472569
jaroslawmalekcodete added a commit that referenced this issue Mar 20, 2020
@jaroslawmalekcodete
#8189: [python] --yarn
6d5955b
piorek added a commit that referenced this issue Mar 20, 2020
@piorek
#8189 disable/enable executor inputs depending on master url
84ece6f
piorek added a commit that referenced this issue Mar 20, 2020
@piorek
#8189 wip
14453d6
jaroslawmalekcodete added a commit that referenced this issue Mar 24, 2020
@jaroslawmalekcodete
#8189: [python] single spark session
d54ec21
piorek added a commit that referenced this issue Mar 25, 2020
@piorek
#8189 fix - wait for spark comm to be ready
dd95085
jaroslawmalekcodete added a commit that referenced this issue Mar 25, 2020
@jaroslawmalekcodete
#8189: [python] send stop comm msg when spark.stop()
4b6e70a
jaroslawmalekcodete added a commit that referenced this issue Mar 25, 2020
@jaroslawmalekcodete
#8189: [python] send stop comm msg with context
86179be
piorek added a commit that referenced this issue Mar 25, 2020
@piorek
#8189 making spark.stop() working corrently on frontend side
3759967
piorek added a commit that referenced this issue Mar 26, 2020
@piorek
#8189 fix styles for jupyterlab
b208e69
jaroslawmalekcodete added a commit that referenced this issue Mar 26, 2020
@jaroslawmalekcodete
#8189: [python] send stop comm msg with context- add tests
a33c5d6
jaroslawmalekcodete added a commit that referenced this issue Mar 26, 2020
@jaroslawmalekcodete
#8189: [java] send stop comm msg with context
3d01383
jaroslawmalekcodete added a commit that referenced this issue Mar 31, 2020
@jaroslawmalekcodete
#8189: check master url and setup properties accordingly for yarn opt…
e6bbe6c 
…ion during starting spark
ildipo pushed a commit that referenced this issue Apr 3, 2020
@jaroslawmalekcodete @piorek
lukasz-jarek/8189: support for %%spark magic command in python + new …
86b50c9 
…SparkUIForm widget (#8257)

* #8189 WIP - proof of concept

* #8189 WIP

* #8189 WIP

* #8189 WIP

* #8189 WIP

* #8189 WIP

* #8189 WIP

* #8189 WIP

* #8189 WIP

* #8189 WIP

* #8189 WIP

* #8189 WIP

* #8189 WIP

* #8189 WIP

* #8189 WIP

* #8189 WIP

* #8189 WIP

* #8189 WIP

* #8189 WIP

* #8189 WIP

* #8189 WIP

* #8189 WIP

* #8189 fix - work around ts limitation

* #8189: run widget SparkUI from spark python magic command

* #8189 Send start event to backend

* #8189 fix add property; send stop message

* #8189: spark python magic command -> start and stop event

* #8189: spark python magic command -> start and stop event tests

* #8189: spark python magic command -> stop event tests

* #8189 WIP

* #8189 WIP

* #8189 WIP

* #8189 WIP

* #8189 WIP

* #8189: spark python magic command -> profiles

* #8189: spark python magic command -> profiles LOAD on init

* #8189: spark python magic command -> save current profile name

* #8189 change start message format

* #8189: spark python magic command -> save  profiles

* #8189: spark python magic command -> save  profiles

* #8189: spark python magic command -> profiles in comm_open message

* #8189: spark python magic command -> profiles in comm_open message

* #8189: spark python magic command -> profiles in comm_open message

* #8189: spark python magic command -> pretty print profiles to file

* #8189 fix listener beeing attached to wrong element

* #8189 WIP

* #8189 set input to checked if there is spark.sql.catalogImplementation === hive property

* #8189 removing profiles and some cleanup

* #8189: fix starting spark with properties

* #8189 WIP - cleanup and some refactor

* #8189 more cleanup and refactor

* #8189 more cleanup and refactor

* #8189 remove unnecessary logging and methods

* #8189 fix

* #8189 fix initial selection of configuration based on information from backend

* #8189 error handling

* #8189: new SparkUIForm widget in java

* #8189: [jvm] sending appId and web url when spark started

* #8189 dependencies update

* #8189 implement spark stats emitting; implement error emitting

* #8189 display and update stats in session widget

* #8189: [jvm] auto start

* #8189: [python] sending appId and web url when spark started

* #8189 %%spark --start handling at front

* #8189 fix tests

* #8189 fix `npm WARN npm npm does not support Node.js vX.Y.Z`

* #8189: [python] auto start

* #8189 improve error handling

* #8189: [python] --noUI

* #8189 add spinner; improve styles

* #8189: [python] reflect user spark conf in SparkUIForm

* #8189 more style improvements

* #8189 ```user_spark_conf``` handling

* #8189: [java] reflect user spark conf in SparkUIForm

* #8189: [python] reflect user spark conf in SparkUIForm update the example

* #8189 split spinner into own file

* #8189 make error color red

* #8189 fix enable hive support checkout

* #8189 disable/enable executor inputs depending on master url

* #8189: [python] --yarn

* #8189 wip

* #8189: [python] single spark session

* #8189 fix - wait for spark comm to be ready

* #8189: [python] send stop comm msg when spark.stop()

* #8189: [python] send stop comm msg with context

* #8189 making `spark.stop()` working corrently on frontend side

* #8189 fix styles for jupyterlab

* #8189: [python] send stop comm msg with context- add tests

* #8189: [java] send stop comm msg with context

* #8189: check master url and setup properties accordingly for yarn option during starting spark

Co-authored-by: Łukasz Pior <piorek82@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ildipo ildipo

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ildipo @jaroslawmalekcodete