Various logging improvements #2899
Conversation
tillrohrmann
left a comment
tillrohrmann
left a comment
There was a problem hiding this comment.
Changes look good to me @uce. I had only some minor comments. After addressing them +1 for merging.
| # change the log levels here. | ||
| log4j.logger.akka=INFO | ||
| log4j.logger.org.apache.kafka=INFO | ||
| log4j.logger.org.apache.hadoop=INFO |
There was a problem hiding this comment.
Does this also work for the shaded hadoop dependencies?
There was a problem hiding this comment.
No, at runtime they have a different package. Therefore, we need to change this here, good catch I think ;) @rmetzger Can you confirm?
There was a problem hiding this comment.
We are not relocating Hadoop, so this configuration line is correct.
As you can see from our logs (the first line usually):
org.apache.hadoop.util.NativeCodeLoader
We are relocating some Hadoop dependencies (like Guava), and rewrite the Hadoop code to call them, but we are not relocating the Hadoop code.
There was a problem hiding this comment.
Ah, now I remember ;) This is indeed shady business. But good to know that we can keep it as is.
| long stateSize = 0L; | ||
| for (SubtaskState subtaskState : subtaskStates.values()) { | ||
| stateSize += subtaskState.getStateSize(); | ||
| } |
There was a problem hiding this comment.
can't we use the getStateSize method here?
| LOG.debug("Adding " + possibleHadoopConfPath + "/core-site.xml to hadoop configuration"); | ||
| } | ||
| } else { | ||
| LOG.debug("File " + possibleHadoopConfPath + "/core-site.xml not found."); |
| LOG.debug("Adding " + possibleHadoopConfPath + "/hdfs-site.xml to hadoop configuration"); | ||
| } | ||
| } else { | ||
| LOG.debug("File " + possibleHadoopConfPath + "/hdfs-site.xml not found."); |
|
We should also set the received and handled message in |
|
Added |
3 participants
I would like to include the following fixes for the 1.1.4 release. In general, I try to improve the debugging experience with this.
/cc @tillrohrmann