You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Method public org.apache.spark.sql.streaming.DataStreamReader org.apache.spark.sql.SQLContext.readStream() is not whitelisted on class class org.apache.spark.sql.SQLContext
#9
Open
jgranadosH opened this issue
Apr 2, 2020
· 3 comments
Hello I'm having problems reading in streams using evenhubs inside azure databricks, the read works using an interactive cluster but when i try to use the same code inside of a high currency cluster with the role access control enabled (only accepts sql and python code),i got the following error:
py4j.security.Py4JSecurityException: Method public org.apache.spark.sql.streaming.DataStreamReader org.apache.spark.sql.SQLContext.readStream() is not whitelisted on class class org.apache.spark.sql.SQLContext
Featured used :
Runtime 6.4(scala 2.11, Spark 2.4.5)
I was struggling with this issue, it was because the cluster I was using was too new, I downgraded my cluster to 5.5 LTS (includes Apache Spark 2.4.3, Scala 2.11) and structured streaming started working again, I don't think it is an issue with the version of scala but rather with Apache Spark.
Hello I'm having problems reading in streams using evenhubs inside azure databricks, the read works using an interactive cluster but when i try to use the same code inside of a high currency cluster with the role access control enabled (only accepts sql and python code),i got the following error:
py4j.security.Py4JSecurityException: Method public org.apache.spark.sql.streaming.DataStreamReader org.apache.spark.sql.SQLContext.readStream() is not whitelisted on class class org.apache.spark.sql.SQLContext
Featured used :
Runtime 6.4(scala 2.11, Spark 2.4.5)
Spark configuration:
spark.databricks.cluster.profile serverless
spark.databricks.io.cache.maxMetaDataCache 1g
spark.databricks.io.cache.maxDiskUsage 80g
spark.databricks.acl.dfAclsEnabled true
spark.databricks.delta.preview.enabled true
spark.databricks.io.cache.compression.enabled false
spark.databricks.repl.allowedLanguages python,sql
Library installed :
![image](https://user-images.githubusercontent.com/59688734/78262946-9c8b1880-7501-11ea-8836-b9b2ca54e1da.png)
com.microsoft.azure:azure-eventhubs-spark_2.11:2.3.6
The text was updated successfully, but these errors were encountered: