New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

scala.tools.nsc.interpreter.IMain.TranslatingClassLoader is not thread-safe #10045

Open
scabug opened this Issue Nov 12, 2016 · 3 comments

Comments

Projects
None yet
2 participants
@scabug
Copy link

scabug commented Nov 12, 2016

scala.tools.nsc.interpreter.IMain.TranslatingClassLoader calls a non-thread-safe method translateSimpleResource (this method calls SymbolTable.enteringPhase), which makes it become non-thread-safe.

However, a ClassLoader must be thread-safe since the class can be loaded in arbitrary thread.

@scabug

This comment has been minimized.

Copy link
Author

scabug commented Nov 12, 2016

Imported From: https://issues.scala-lang.org/browse/SI-10045?orig=1
Reporter: Shixiong Zhu (zsxwing)
Affected Versions: 2.10.6, 2.11.8, 2.12.0

@scabug

This comment has been minimized.

Copy link
Author

scabug commented Nov 12, 2016

Shixiong Zhu (zsxwing) said (edited on Nov 14, 2016 6:10:26 PM UTC):
Sorry that I don't have a simple reproducer. Right now I can only reproduce this issue using very complicated Spark SQL codes in Spark shell. The REPL will throw some assertions when compiling codes. The issue will be gone if I use some hacky codes to make scala.tools.nsc.interpreter.IMain.TranslatingClassLoader thread-safe.

@scabug scabug added this to the Backlog milestone Apr 7, 2017

@savvyyabby

This comment has been minimized.

Copy link

savvyyabby commented Jan 17, 2019

I think I found an easy way to reproduce this bug with high reliability.
On Windows 10 and Windows Server 2016 I did some experiments to run Apache Spark from a command line using the scala REPL that is included with that. If you simply do a standard install, open up the spark-shell and then :quit (without doing anything else) then the cleanup process of Spark will try to delete any jar files or classes that were loaded to serve the REPL.

Since Windows (unlike Linux) locks the files there is an exception thrown within Spark that shows the offending class as this one in scala REPL. I am interpreting this behaviour as meaning the scala REPL classloader was not closed (by Spark's shutdown manager) maybe because that action can only happen within the scala REPL itself (i.e. the global.close() action is maybe not catching everything).

scala.tools.nsc.interpreter.IMain.TranslatingClassLoader

See the issue over on the Spark thread where a number of Windows developers discuss temp files that are not unloading correctly. I think this may be because they were files loaded by the REPL, and by this particular class. That is guesswork but it may help in isolating what the bug is.

The Spark thread is here:
https://issues.apache.org/jira/browse/SPARK-12216
Windows is just more fussy than Linux on the final step in the spark command shell shutdown process. Since the scala.tools.nsc.interpreter.IMain.TranslatingClassLoader is flagged here as being non threadsafe and it is this classloader that seems to load any class created in the spark-shell REPL I am guessing that this may be the cause of the whole problem, and therefore a good test case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment