add magic for matplotlib display(pyspark or pyspark3)

[wangqiaoshi]

now sparkmagic must be pass data to local then matplot.
it is expensive if the data is big

[Tagar]

This is a great improvement .

A screenshot of an example usage would be nice.. or sparkmagic documentation updated.

Thanks!

[wangqiaoshi]

This is a great improvement .

A screenshot of an example usage would be nice.. or sparkmagic documentation updated.

Thanks!

ok,i add screenshot.

[Tagar]

Thanks @wangqiaoshi

@apetresc could you please help review and hopefully merge this?

It seems to be a great improvement for larger visualizations - so viz can happen on Spark driver and then only visualization itself will have to travel over network to Jupyter.

[brookwc]

@Tagar @apetresc hey! This diff is quite useful to us. Has it been merged into trunk?

[Tagar]

@itamarst can this be part of the newer release you're preparing ?
thank you

[itamarst]

Merging to branch so I can iterate on this.

[itamarst]

Hi, could you all try the version in #552? Adds support for usage in normal notebooks, not just the PySpark kernel.