Tune down the logging level for callback server messages

[agronholm]

INFO:py4j.java_gateway:Received command c on object id p0

This is not useful information unless one is debugging the callback server. It's flooding my logs and it's not easily filtered out if you want the useful messages too. This message should be logged on the DEBUG level.

[bartdag]

Just to make sure that I don't decrease the level of something useful, what information do you need at the INFO level?

[agronholm]

INFO:py4j.java_gateway:Closing down connection
INFO:py4j.java_gateway:Callback Server Starting
INFO:py4j.java_gateway:Socket listening on ('127.0.0.1', 25334)

At least server/gateway startup/shutdown messages should be on the INFO level. I'm undecided on callback server connection logging.

[bartdag]

Ok, the Java logging API is a bit weirder than other logging APIs, but I did not want to rely on an external one so here is what I proposed after having reviewed all the logging statements on the Java side:

INFO: socket and state related (gateway started, connection received, connection closed, socket listening on...)
FINE: (below info): Commands received.
FINER: (below fine): What the command is doing (e.g., getting field...) Useful to debug a particular command.
FINEST: (below fine): Raw characters received and sent. Useful to debug the protocol/socket/network

I might also revisit certain WARNING and SEVERE in the light of issue #93.

I'll try to do something similar on the Python side.

[bartdag]

I'll close this issue. Please open a new issue if there are other messages that need to be tuned down.

[hshteingart]

Question: how do i suppress this messsage on pyspark?
Answer: logging.getLogger("py4j").setLevel(logging.ERROR)