[SPARK-12757][CORE] lower "block locks were not released" log to info level #16513
Conversation
|
Test build #71067 has started for PR 16513 at commit |
|
Jenkins, retest this please |
|
Test build #71075 has finished for PR 16513 at commit
|
|
I think https://issues.apache.org/jira/browse/SPARK-13796 downgraded to warning from error; it might make sense to lower it further, but I'm not clear whether it should happen in normal operation? if not then it might be useful to keep the message until it's clear why this is happening reproducibly? |
|
@srowen completely agree. that was what I asked in https://issues.apache.org/jira/browse/SPARK-12757?focusedCommentId=15786726 and I'd defer to @JoshRosen who suggested lowering the log level |
|
merged to master. thanks! |
… level ## What changes were proposed in this pull request? lower "block locks were not released" log to info level, as it is generating a lot of warnings in running ML, graph calls, as pointed out in the JIRA. Author: Felix Cheung <felixcheung_m@hotmail.com> Closes apache#16513 from felixcheung/blocklockswarn.
… level ## What changes were proposed in this pull request? lower "block locks were not released" log to info level, as it is generating a lot of warnings in running ML, graph calls, as pointed out in the JIRA. Author: Felix Cheung <felixcheung_m@hotmail.com> Closes apache#16513 from felixcheung/blocklockswarn.
|
I'm seeing the same WARN when running LDA in Spark 2.1 in both the millib and ml libraries. When I run the LDA mllib implementation in Spark 1.6 I don't see the same error. For further information: LDA has two optimizers: "em" and "online" the error does not occur when running the "online" version. Is this the correct place to report this issue? |
|
I am still seeing the warning as well. I am using "0.5.0-spark2.1-s_2.11". Should we create a new issue or re-open this one? |
|
Please open a JIRA
|
|
as @lujangus noted, I also see this as a WARN after upgrading from Spark 1.6 -> 2.1 using mllib's LDA implementation. I actually get so many warnings that my stack eventually overflows, crashing the program. Is there a way to suppress these warnings? |
|
This is only fixed in 2.2+
So 2.1 is still affected by this issue. You should be able to set the logging level as described here
https://spark.apache.org/docs/latest/configuration.html#configuring-logging
|
What changes were proposed in this pull request?
lower "block locks were not released" log to info level, as it is generating a lot of warnings in running ML, graph calls, as pointed out in the JIRA.