Spark 2.3 app in k8s cluster: parquet->COS throws exception #200
Comments
|
@dkazakevich thanks. A bit strange, are you sure the same code works for CSV and fails for Parquet? The exception in the log is "Could not initialize class sun.security.ssl.SSLSessionImpl", which is related SSL and has nothing to do to Stocator. |
|
@dkazakevich a short update; I tested your code on Spark 2.3 and the same code works perfectly for me. So i need more details ( see my previous msg ) in order to proceed with it |
|
@gilv Thank you for response. It works for JSON, but failed for parquet. I haven’t check fo CSV, will try it tomorrow. |
|
@dkazakevich I want to be sure i understand correctly. Is the following correct?
Do you use exactly the same code and the same path when access JSON and Parquet? SSL issues can't be related file types, so i try to figure out what else is different in your code |
|
@gilv, yes, you are correct. I have the same behavior:
|
|
@dkazakevich @bahdzevich thanks. what JDK version you are using? I also wonder, what is the value of "fs.cos.service.endpoint" ? |
|
@gilv We are using JDK 1.8.0_172 |
|
@dkazakevich have you installed or do you need to install a cert on Kubernetes? |
|
@paul-carron I haven't install certs on minikube and blumix k8s cluster. Also don't know is need cert to put data into COS. |
|
@dkazakevich can you read some existing Parquet file from COS on this cluster? |
|
@gilv reading Parquet file from COS and writing it back into COS works fine. |
|
@dkazakevich so all is working, except when you create 2 records size Parquet file in Spark and write it into COS? This sounds a bit strange.. and it fails with SSL.. ` val squaresDF = spark.sparkContext.makeRDD(1 to 5).map(i => (i, i * i)).toDF("value", "square") squaresDF.write.format(“parquet”).save("cos://yourbucket.myCOS/data.parquet") Will this work? |
|
@gilv Initially we faced with the problem in process of reading data from DB2 and writing into COS. The |
|
@dkazakevich you can connect spark-shell to the exiting cluster, by using --master spark://master_host:master_port |
|
@gilv Because we don't have k8s spark cluster with running master and executors (that necessary for spark-shell), and uses spark-submit tool that automatically creates master and indicated number of executors for a spark job (feature of spark 2.3 for k8s), is it ok to create a scala application, generate a .jar file for the spark-submit and run it for the above experiment? |
|
@dkazakevich I think it's the same to create jar with scala and use spark-submit |
|
@gilv Tried the experiment and got the same exception: Also FYI: I'm not sure that it could be the problem reason, but I tried to load data from some DB2 databases and write parquet into COS and found out that:
|
|
@dkazakevich if you can't write a simple Parquet with this Scala example, then it's not related DB2 i guess.. and the exception you getting is related an SSL connection.. but you can read from COS, write JSON without hitting SSL issues.. |
|
@dkazakevich please open also a ticket on IBM Bluemix Support. You can link them this issue that you opened against Stocator. You should see support in the console.bluemix.net and create a ticket |
|
@dkazakevich At the don't believe this is be an issue with the Java SDK as I’m unable to recreate it in a stand alone or cluster environment. My hunch at the minute is that its some sort of cert issue between Spark and Kubernetes although you haven’t installed certs on their minicube cluster. Unfortunately I’m not familiar with Kubernetes so don’t know what might be required. It might be worth having somebody with Kubernetes knowledge look at this. |
Created a sample Spark 2.3 application that runs in k8s cluster. The application creates a sample Dataset and trying to put it into Bluemix COS using stocator:
But I'm getting exception:
Writing the same data to COS json file works fine:
The text was updated successfully, but these errors were encountered: