[SPARK-13848][SPARK-5185] Update to Py4J 0.9.2 in order to fix classl…

…oading issue

This patch upgrades Py4J from 0.9.1 to 0.9.2 in order to include a patch which modifies Py4J to use the current thread's ContextClassLoader when performing reflection / class loading. This is necessary in order to fix [SPARK-5185](https://issues.apache.org/jira/browse/SPARK-5185), a longstanding issue affecting the use of `--jars` and `--packages` in PySpark.

In order to demonstrate that the fix works, I removed the workarounds which were added as part of [SPARK-6027](https://issues.apache.org/jira/browse/SPARK-6027) / #4779 and other patches.

Py4J diff: py4j/py4j@0.9.1...0.9.2

/cc zsxwing tdas davies brkyvz

Author: Josh Rosen <joshrosen@databricks.com>

Closes #11687 from JoshRosen/py4j-0.9.2.

streaming-mqtt/python/mqtt.py

@@ -38,18 +38,15 @@ def createStream(ssc, brokerUrl, topic,
         :param storageLevel:  RDD storage level.
         :return: A DStream object
         """
-        jlevel = ssc._sc._getJavaStorageLevel(storageLevel)
-
         try:
-            helperClass = ssc._jvm.java.lang.Thread.currentThread().getContextClassLoader() \
-                .loadClass("org.apache.spark.streaming.mqtt.MQTTUtilsPythonHelper")
-            helper = helperClass.newInstance()
-            jstream = helper.createStream(ssc._jssc, brokerUrl, topic, jlevel)
-        except Py4JJavaError as e:
-            if 'ClassNotFoundException' in str(e.java_exception):
+            helper = ssc._jvm.org.apache.spark.streaming.mqtt.MQTTUtilsPythonHelper()
+        except TypeError as e:
+            if str(e) == "'JavaPackage' object is not callable":
                 MQTTUtils._printErrorMsg(ssc.sparkContext)
             raise
 
+        jlevel = ssc._sc._getJavaStorageLevel(storageLevel)
+        jstream = helper.createStream(ssc._jssc, brokerUrl, topic, jlevel)
         return DStream(jstream, ssc, UTF8Deserializer())
 
     @staticmethod