Removed Spark Logs coming from Java

[2pushkaraj3]

issue:Remove Spark Logs coming from Java from the Pipeline Logs #38
solution: imported Sparksession from pyspark.sql and utilised setLogLevel for 'WARN'
<img width="682" alt="Screenshot 2023-01-19 at 3 32 21 PM" src="https://user-images.githubusercontent.com/85059599/2134178

67-3cfa82b0-8282-497d-b9b4-b5a8b4cb65bd.png">

[thepushkarp]

Hey @2pushkaraj3, thanks for opening this PR and working on #38

I ran your branch locally. Comparing it to the dev branch, I can see that the logs from java are still being printed to the console. Since you initialized the spark session in the radarpipeline.py file which is being run at the beginning of a pipeline run, the logs are now printed at the start instead of when the spark session was created before computation. I have attached the screenshots below.

Can you please check if there's something else we might need to do to prevent these logs, or instead show them through our custom logger?

Logs from 2pushkaraj3:main

Log from dev

[2pushkaraj3]

sorry for the late reply. I have tried with the highest priority of setLogLevel in both logger.py and radarpipeline but still, but there are some logs that are still present in the terminal. I have tried using the log4j.properties file with different levels of the log but still, they are getting printed. Is there any other way to approach this issue?

[thepushkarp]

Hi @2pushkaraj3

Yeah, somehow these methods don't seem to work. I've been trying to figure out something but no luck 😕

Will tinker some more next week and let you know if I find anything.

[thepushkarp]

Hey, @2pushkaraj3

I tried looking for a bunch of stuff but unfortunately couldn't get something that would work :(

[Hsankesara]

Sorry for getting late into the conversation. I agree, seems quite tricky to remove the spark logs. I think they are trivial for now, so we can tolerate it. @2pushkaraj3 thanks for working on it. If you're still interested in working on radar-pipeline, I can suggest #58 to you.

[2pushkaraj3]

Thank you for clarifying.will surely work on #58