You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
It's sort of implemented, but was never tested. Also trees aren't synchronized at all, because it was only recently when they've been converted to abstract types.
NOTE that this issue mostly affects runtime reflection, not reflection used in macros. Compiler itself has never been thread-safe, so noone uses it from multiple threads, therefore issues might only arise if a macro spawns multiple threads, which do certain operations simultaneously.
@xeno-by said:
Most symbols aren't synchronized right now, because synchronization is implemented by decoration, and decorators aren't applied to almost anything: scala/scala#1380
Kartik Subramanian (selfification) said:
How thread safe is reflection as it stands right now? If it is not really thread-safe, is there a way to globally synchronize all calls into the API?
I ask this because I have code that tries to use reflection to fetch modules that are a subtype of some given type. During testing, if I run my tests in parallel (using sbt test), I get various failures like NPEs, "scala.reflect.internal.Symbols$CyclicReference: illegal cyclic reference involving value", unsupported operation exception for "tail of empty list", all of them coming from reflect.internal.SymbolTable or from reflect.internal.Symbols$Symbol.lock. It's highly non-reproducible and I don't know enough about reflection to reduce it to a minimal test case. If I run the tests linearly (using sbt test-only name-of-test), each of them individually pass. If you guys are already aware that Symbol is not thread-safe, I don't want to spend time trying to simplify my issue to a bug you are already aware of.
Just as an aside, even when running the tests serially, I get printouts like:
[] ?$2 setInst scala.List[scala.Int]
[] ?$2 setInst scala.List[scala.Int]
and
[] !!! MyClass[String].transform(MyClass MyClass), but tparams.isEmpty and args=
[] !!! MyClass[T].transform(MyClass MyClass), but tparams.isEmpty and args=
I don't know if this is relevant to the thread-safety issue. I am trying to reflectively acquire objects with types that look like MyClass[_] - that may be triggering some code path that isn't thread-safe yet?
Let me know if you want more info or if you want to see the source.
I am using scala in the eclipse plugin version 2.1.0.m2 and I am getting problems when using the reflection API in a concurrent setting.
I have a big framework using MPJ-Express as the backend and running applications in multi-core mode. When I use the reflection api, I get random occurences of the runtime eror displayed below:
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethod)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at runtime.starter.MulticoreStarter$1.run(MulticoreStarter.java:289)
at java.lang.Thread.run(Thread.java:722)
Causedby: scala.reflect.runtime.ReflectError: value Predef is not a package
at scala.reflect.runtime.JavaMirrors$JavaMirror.scala$reflect$runtime$JavaMirrors$$makeScalaPackage(JavaMirrors.scala:893)
at scala.reflect.runtime.JavaMirrors$class.missingHook(JavaMirrors.scala:1256)
at scala.reflect.runtime.JavaUniverse.missingHook(JavaUniverse.scala:12)
at scala.reflect.internal.Mirrors$RootsBase.universeMissingHook(Mirrors.scala:77)
at scala.reflect.internal.Mirrors$RootsBase.missingHook(Mirrors.scala:79)
at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:48)
at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:61)
at scala.reflect.internal.Mirrors$RootsBase.staticModuleOrClass(Mirrors.scala:72)
at scala.reflect.internal.Mirrors$RootsBase.staticModule(Mirrors.scala:161)
at scala.reflect.internal.Mirrors$RootsBase.staticModule(Mirrors.scala:21)
at it.vigtig.thesis.test.NGridTest$$typecreator2$1.apply(NGridTest.scala:103)
at scala.reflect.api.TypeTags$TypeTag$.apply(TypeTags.scala:302)
at it.vigtig.thesis.test.NGridTest$.testGrid(NGridTest.scala:103)
at it.vigtig.thesis.test.NGridTest$$anonfun$main$1.apply$mcV$sp(NGridTest.scala:16)
at it.vigtig.thesis.env.package$$anonfun$parallelize$1.apply$mcV$sp(package.scala:68)
at it.vigtig.thesis.env.comm.mpj.MPJDistributorTrait$class.execute(MPJDistributorTrait.scala:17)
at it.vigtig.thesis.env.comm.mpj.MPJDistributor$.execute(MPJDistributor.scala:3)
at it.vigtig.thesis.env.package$.parallelize(package.scala:59)
at it.vigtig.thesis.test.NGridTest$.main(NGridTest.scala:14)
at it.vigtig.thesis.test.NGridTest.main(NGridTest.scala)
... 6 more
I am not sharing any reflection knowledge or objects between threads, so this behaviour must be a bug. The parts I am using from reflection are TypeTags and ClassTags as implicit arguments.
@paulp said:
"value Predef is not a package" suggests that it isn't even blocking for Definitions.init() to complete, is that true? If that's the case then people had better put their concurrent reflection aspirations on ice until at least 2.10.1.
@xeno-by said (edited on Dec 25, 2012 10:29:23 PM UTC):
To quote Paul Butcher,
Actually - I take that back. I have one test of this form for the old non-type-macro-based version of mocks and one for the typemacro-based version. And they're in different suites, so they might be running parallel with each other.
But if this is the problem, then I suspect we really need to reconsider the thread safety of reflection. If just adding a TypeTag context bound to a method is enough to make it non-threadsafe, then it's bound to catch people out.
> [info] - cope with context bounds ***FAILED***> [info] java.lang.NullPointerException:> [info] at scala.reflect.internal.Types$TypeRef.computeHashCode(Types.scala:2332)
> [info] at scala.reflect.internal.Types$UniqueType.<init>(Types.scala:1274)
> [info] at scala.reflect.internal.Types$TypeRef.<init>(Types.scala:2315)
> [info] at scala.reflect.internal.Types$NoArgsTypeRef.<init>(Types.scala:2107)
> [info] at scala.reflect.internal.Types$ModuleTypeRef.<init>(Types.scala:2078)
> [info] at scala.reflect.internal.Types$PackageTypeRef.<init>(Types.scala:2095)
> [info] at scala.reflect.internal.Types$TypeRef$.apply(Types.scala:2516)
> [info] at scala.reflect.internal.Types$class.typeRef(Types.scala:3577)
> [info] at scala.reflect.internal.SymbolTable.typeRef(SymbolTable.scala:13)
> [info] at scala.reflect.internal.Symbols$TypeSymbol.newTypeRef(Symbols.scala:2754)
Felix Palludan Hargreaves (hejfelix) said:
Doesn't work for me. I get the following:
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at runtime.starter.MulticoreStarter$1.run(MulticoreStarter.java:289)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.UnsupportedOperationException: tail of empty list
at scala.collection.immutable.Nil$.tail(List.scala:330)
at scala.collection.immutable.Nil$.tail(List.scala:325)
at scala.reflect.internal.SymbolTable.popPhase(SymbolTable.scala:166)
at scala.reflect.internal.pickling.UnPickler$Scan$LazyTypeRef.complete(UnPickler.scala:856)
at scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:1217)
at scala.reflect.internal.SymbolTable.openPackageModule(SymbolTable.scala:238)
at scala.reflect.internal.SymbolTable.openPackageModule(SymbolTable.scala:294)
at scala.reflect.runtime.SymbolLoaders$LazyPackageType.complete(SymbolLoaders.scala:88)
at scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:1217)
at scala.reflect.internal.Types$TypeRef.thisInfo(Types.scala:2364)
at scala.reflect.internal.Types$TypeRef.baseClasses(Types.scala:2369)
at scala.reflect.internal.Types$Type.findMember(Types.scala:1138)
at scala.reflect.internal.Types$Type.memberBasedOnName(Types.scala:687)
at scala.reflect.internal.Types$Type.member(Types.scala:645)
at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:43)
at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:40)
at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:61)
at scala.reflect.internal.Mirrors$RootsBase.staticModuleOrClass(Mirrors.scala:72)
at scala.reflect.internal.Mirrors$RootsBase.staticClass(Mirrors.scala:119)
at scala.reflect.internal.Mirrors$RootsBase.staticClass(Mirrors.scala:21)
at it.vigtig.thesis.test.MatrixBench$$typecreator2$1.apply(MatrixBench.scala:149)
at scala.reflect.api.TypeTags$TypeTag$.apply(TypeTags.scala:302)
at it.vigtig.thesis.test.MatrixBench$.mul(MatrixBench.scala:149)
at it.vigtig.thesis.test.MatrixBench$.bench(MatrixBench.scala:139)
at it.vigtig.thesis.test.MatrixBench$$anonfun$main$1$$anonfun$apply$mcV$sp$1$$anonfun$apply$1.apply$mcVI$sp(MatrixBench.scala:94)
at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:142)
at it.vigtig.thesis.test.MatrixBench$$anonfun$main$1$$anonfun$apply$mcV$sp$1.apply(MatrixBench.scala:93)
at it.vigtig.thesis.test.MatrixBench$$anonfun$main$1$$anonfun$apply$mcV$sp$1.apply(MatrixBench.scala:90)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1156)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at it.vigtig.thesis.test.MatrixBench$$anonfun$main$1.apply$mcV$sp(MatrixBench.scala:90)
at it.vigtig.thesis.env.package$$anonfun$parallelize$1.apply$mcV$sp(package.scala:72)
at it.vigtig.thesis.env.comm.mpj.MPJDistributorTrait$class.execute(MPJDistributorTrait.scala:17)
at it.vigtig.thesis.env.comm.mpj.MPJDistributor$.execute(MPJDistributor.scala:3)
at it.vigtig.thesis.env.package$.parallelize(package.scala:63)
at it.vigtig.thesis.test.MatrixBench$.main(MatrixBench.scala:35)
at it.vigtig.thesis.test.MatrixBench.main(MatrixBench.scala)
... 6 more
@xeno-by said:
Could you please tell more about your scenario? Probably the best way to communicate would be email - just google me, and there'll also be my email.
@eengbrec said:
If you checkout this branch of this relatively small project, comment out "parallelExecution in Test := false " in build.sbt, then run the tests you should see non-deterministic failures. Just run test repeatedly and watch.
@retronym said:
Erasure isn't to blame here, in fact it is rather crucial for a language like Scala to be able to innovate in its type system. Compare this to F#, for example, which hasn't been able to add type constructor polymorphism due to the assumptions about generics built into the VM.
scala.reflect is experimental. Making it thread safe without killing performance is nearly impossible. I was against even trying. The workaround is to synchronize access by the user, or not use it at all
Just to be clear, we do not intend to provide (full) thread-safety for scala.reflect -- I don't think it's feasible
on the other hand, improvements to thread safety do sometimes still land, e.g. @retronym's recent scala/scala#8433 . (I believe this was work for a customer.) and we're certainly open to merging contributed fixes in this area
still-open bugs in this area include #10598 and #10431; perhaps there are others
Activity
scabug commentedon Aug 15, 2012
Imported From: https://issues.scala-lang.org/browse/SI-6240?orig=1
Reporter: @xeno-by
Affected Versions: 2.10.0
scabug commentedon Sep 6, 2012
@xeno-by said:
Also synchronize WellKnownSymbols
scabug commentedon Sep 26, 2012
@xeno-by said:
Most symbols aren't synchronized right now, because synchronization is implemented by decoration, and decorators aren't applied to almost anything: scala/scala#1380
scabug commentedon Nov 15, 2012
Kartik Subramanian (selfification) said:
How thread safe is reflection as it stands right now? If it is not really thread-safe, is there a way to globally synchronize all calls into the API?
I ask this because I have code that tries to use reflection to fetch modules that are a subtype of some given type. During testing, if I run my tests in parallel (using sbt test), I get various failures like NPEs, "scala.reflect.internal.Symbols$CyclicReference: illegal cyclic reference involving value", unsupported operation exception for "tail of empty list", all of them coming from reflect.internal.SymbolTable or from reflect.internal.Symbols$Symbol.lock. It's highly non-reproducible and I don't know enough about reflection to reduce it to a minimal test case. If I run the tests linearly (using sbt test-only name-of-test), each of them individually pass. If you guys are already aware that Symbol is not thread-safe, I don't want to spend time trying to simplify my issue to a bug you are already aware of.
Just as an aside, even when running the tests serially, I get printouts like:
[] ?$2 setInst scala.List[scala.Int]
[] ?$2 setInst scala.List[scala.Int]
and
[] !!! MyClass[String].transform(MyClass MyClass), but tparams.isEmpty and args=
[] !!! MyClass[T].transform(MyClass MyClass), but tparams.isEmpty and args=
I don't know if this is relevant to the thread-safety issue. I am trying to reflectively acquire objects with types that look like MyClass[_] - that may be triggering some code path that isn't thread-safe yet?
Let me know if you want more info or if you want to see the source.
scabug commentedon Dec 6, 2012
@xeno-by said:
Another case study by hejfelix (https://github.com/scala/scala/issues/1723):
I am using scala in the eclipse plugin version 2.1.0.m2 and I am getting problems when using the reflection API in a concurrent setting.
I have a big framework using MPJ-Express as the backend and running applications in multi-core mode. When I use the reflection api, I get random occurences of the runtime eror displayed below:
I am not sharing any reflection knowledge or objects between threads, so this behaviour must be a bug. The parts I am using from reflection are TypeTags and ClassTags as implicit arguments.
scabug commentedon Dec 6, 2012
@paulp said:
"value Predef is not a package" suggests that it isn't even blocking for Definitions.init() to complete, is that true? If that's the case then people had better put their concurrent reflection aspirations on ice until at least 2.10.1.
scabug commentedon Dec 6, 2012
@xeno-by said:
Looks like it's not blocking indeed.
scabug commentedon Dec 25, 2012
@xeno-by said (edited on Dec 25, 2012 10:29:23 PM UTC):
To quote Paul Butcher,
Actually - I take that back. I have one test of this form for the old non-type-macro-based version of mocks and one for the typemacro-based version. And they're in different suites, so they might be running parallel with each other.
But if this is the problem, then I suspect we really need to reconsider the thread safety of reflection. If just adding a TypeTag context bound to a method is enough to make it non-threadsafe, then it's bound to catch people out.
Raising the priority.
upd. Discussion: http://groups.google.com/group/scala-internals/browse_thread/thread/58d62991c21ba2f8.
scabug commentedon Jan 2, 2013
@OlegYch said:
isn't there any way to init that upfront before normal program run?
scabug commentedon Jan 2, 2013
@xeno-by said:
There is. Just calling
scala.reflect.runtime.universeshould be enough.scabug commentedon Jan 2, 2013
Felix Palludan Hargreaves (hejfelix) said:
Doesn't work for me. I get the following:
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at runtime.starter.MulticoreStarter$1.run(MulticoreStarter.java:289)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.UnsupportedOperationException: tail of empty list
at scala.collection.immutable.Nil$.tail(List.scala:330)
at scala.collection.immutable.Nil$.tail(List.scala:325)
at scala.reflect.internal.SymbolTable.popPhase(SymbolTable.scala:166)
at scala.reflect.internal.pickling.UnPickler$Scan$LazyTypeRef.complete(UnPickler.scala:856)
at scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:1217)
at scala.reflect.internal.SymbolTable.openPackageModule(SymbolTable.scala:238)
at scala.reflect.internal.SymbolTable.openPackageModule(SymbolTable.scala:294)
at scala.reflect.runtime.SymbolLoaders$LazyPackageType.complete(SymbolLoaders.scala:88)
at scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:1217)
at scala.reflect.internal.Types$TypeRef.thisInfo(Types.scala:2364)
at scala.reflect.internal.Types$TypeRef.baseClasses(Types.scala:2369)
at scala.reflect.internal.Types$Type.findMember(Types.scala:1138)
at scala.reflect.internal.Types$Type.memberBasedOnName(Types.scala:687)
at scala.reflect.internal.Types$Type.member(Types.scala:645)
at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:43)
at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:40)
at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:61)
at scala.reflect.internal.Mirrors$RootsBase.staticModuleOrClass(Mirrors.scala:72)
at scala.reflect.internal.Mirrors$RootsBase.staticClass(Mirrors.scala:119)
at scala.reflect.internal.Mirrors$RootsBase.staticClass(Mirrors.scala:21)
at it.vigtig.thesis.test.MatrixBench$$typecreator2$1.apply(MatrixBench.scala:149)
at scala.reflect.api.TypeTags$TypeTag$.apply(TypeTags.scala:302)
at it.vigtig.thesis.test.MatrixBench$.mul(MatrixBench.scala:149)
at it.vigtig.thesis.test.MatrixBench$.bench(MatrixBench.scala:139)
at it.vigtig.thesis.test.MatrixBench$$anonfun$main$1$$anonfun$apply$mcV$sp$1$$anonfun$apply$1.apply$mcVI$sp(MatrixBench.scala:94)
at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:142)
at it.vigtig.thesis.test.MatrixBench$$anonfun$main$1$$anonfun$apply$mcV$sp$1.apply(MatrixBench.scala:93)
at it.vigtig.thesis.test.MatrixBench$$anonfun$main$1$$anonfun$apply$mcV$sp$1.apply(MatrixBench.scala:90)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1156)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at it.vigtig.thesis.test.MatrixBench$$anonfun$main$1.apply$mcV$sp(MatrixBench.scala:90)
at it.vigtig.thesis.env.package$$anonfun$parallelize$1.apply$mcV$sp(package.scala:72)
at it.vigtig.thesis.env.comm.mpj.MPJDistributorTrait$class.execute(MPJDistributorTrait.scala:17)
at it.vigtig.thesis.env.comm.mpj.MPJDistributor$.execute(MPJDistributor.scala:3)
at it.vigtig.thesis.env.package$.parallelize(package.scala:63)
at it.vigtig.thesis.test.MatrixBench$.main(MatrixBench.scala:35)
at it.vigtig.thesis.test.MatrixBench.main(MatrixBench.scala)
... 6 more
scabug commentedon Jan 2, 2013
@xeno-by said:
Could you please tell more about your scenario? Probably the best way to communicate would be email - just google me, and there'll also be my email.
scabug commentedon Jan 3, 2013
@eengbrec said:
If you checkout this branch of this relatively small project, comment out "parallelExecution in Test := false " in build.sbt, then run the tests you should see non-deterministic failures. Just run test repeatedly and watch.
https://github.com/eengbrec/optional/tree/scala210-apachecli
scabug commentedon Jan 3, 2013
@xeno-by said:
Thanks! This will be one of the first things I'll get to after my vacation
scabug commentedon Jan 8, 2013
@xeno-by said:
Put up the reproductions at dedicated repos at github:
40 remaining items
scabug commentedon Feb 2, 2015
@retronym said:
Erasure isn't to blame here, in fact it is rather crucial for a language like Scala to be able to innovate in its type system. Compare this to F#, for example, which hasn't been able to add type constructor polymorphism due to the assumptions about generics built into the VM.
SethTisue commentedon Jun 13, 2020
https://docs.scala-lang.org/overviews/reflection/thread-safety.html says "Thread safety issues have been fixed in Scala 2.11.0-RC1", with a link to this ticket.
@nogurenn asked on Gitter so I did a little research about where things stand today.
the most negative thing I found is #10766, particularly @adriaanm's remark at #10766 (comment) :
and then at #10766 (comment) :
on the other hand, improvements to thread safety do sometimes still land, e.g. @retronym's recent scala/scala#8433 . (I believe this was work for a customer.) and we're certainly open to merging contributed fixes in this area
still-open bugs in this area include #10598 and #10431; perhaps there are others
SethTisue commentedon Jun 13, 2020
uh, weird, I didn't intend to, and I don't understand why I can't undo the change, either :-/
[SPARK-57371][CONNECT] Fix data race in ArrowDeserializers
[SPARK-57371][CONNECT] Fix data race in ArrowDeserializers
[SPARK-57371][CONNECT] Fix data race in ArrowDeserializers
[SPARK-57371][CONNECT] Fix data race in ArrowDeserializers
[SPARK-57371][CONNECT] Fix data race in ArrowDeserializers