REPL can't find staging if started from sbt

[Katrix]

minimized code

https://gist.github.com/Blaisorblade/c05f9f6f994732892caa24cae2eb6c3c

Works fine when starting from a REPL, not when invoking <some-project>/console

expectation

It should work the same

Related dicussion
https://gitter.im/lampepfl/dotty?at=5de164a3d75ad3721d239f4e

[letalvoj]

Still present.

I have a the following in build.sbt

      "ch.epfl.lamp"    %% "dotty-staging"  % ScalaVersion,
      "ch.epfl.lamp"    %% "dotty-library"  % ScalaVersion, // this sure contains scalaShadowing

Yet:

sbt:loonyssh> console

scala> import scala.quoted._
     | import scala.tasty._

scala> import scala.quoted.staging._

scala> def fn[T : Type](v : T) = println("ok")
def fn[T](v: T)(using evidence$1: quoted.Type[T]): Unit

scala> given scala.quoted.staging.Toolbox = scala.quoted.staging.Toolbox.make(getClass.getClassLoader)                                                             
def given_Toolbox: quoted.staging.Toolbox

scala> withQuoteContext { fn("foo") }
dotty.tools.dotc.MissingCoreLibraryException: Could not find package scalaShadowing from compiler core libraries.
Make sure the compiler core libraries are on the classpath.
   
        at dotty.tools.dotc.core.Denotations$DenotationsBase.select$1(Denotations.scala:1212)
        at dotty.tools.dotc.core.Denotations$DenotationsBase.recurSimple$1(Denotations.scala:1240)
        at dotty.tools.dotc.core.Denotations$DenotationsBase.recur$1(Denotations.scala:1242)
        at dotty.tools.dotc.core.Denotations$DenotationsBase.staticRef(Denotations.scala:1244)
        at dotty.tools.dotc.core.Contexts$ContextBase.staticRef(Contexts.scala:664)
        at dotty.tools.dotc.core.Symbols.requiredPackage(Symbols.scala:367)
        at dotty.tools.dotc.core.Contexts$Context.requiredPackage(Contexts.scala:79)
        at dotty.tools.dotc.core.Definitions.ScalaShadowingPackage(Definitions.scala:253)
        ...

funny enough the following executes without an error

scala> import _root_.scalaShadowing

[nicolasstucki]

In 3.0.0-M3 the error became

scala> import scala.quoted._                                                                                                                                                                                                     

scala> import scala.quoted.staging._                                                                                                                                                                                             

scala> def fn[T : Type](v : T) = println("ok")                                                                                                                                                                                   
def fn[T](v: T)(implicit evidence$1: quoted.Type[T]): Unit

scala> given scala.quoted.staging.Toolbox = scala.quoted.staging.Toolbox.make(getClass.getClassLoader)                                                                                                                           
lazy val given_Toolbox: quoted.staging.Toolbox

scala> withQuotes { fn("foo") }                                                                                                                                                                                            
dotty.tools.dotc.MissingCoreLibraryException: Could not find package scala from compiler core libraries.
Make sure the compiler core libraries are on the classpath.
   
  at dotty.tools.dotc.core.Denotations$.select$1(Denotations.scala:1234)
  at dotty.tools.dotc.core.Denotations$.recurSimple$1(Denotations.scala:1262)
  at dotty.tools.dotc.core.Denotations$.recur$1(Denotations.scala:1264)
  at dotty.tools.dotc.core.Denotations$.staticRef(Denotations.scala:1266)
  at dotty.tools.dotc.core.Symbols$.requiredPackage(Symbols.scala:850)
  at dotty.tools.dotc.core.Definitions.ScalaPackageVal(Definitions.scala:202)
  at dotty.tools.dotc.core.Definitions.ScalaPackageClass(Definitions.scala:205)
  at dotty.tools.dotc.core.Definitions.AnyClass(Definitions.scala:260)
  at dotty.tools.dotc.core.Definitions.syntheticScalaClasses(Definitions.scala:1693)
  at dotty.tools.dotc.core.Definitions.syntheticCoreClasses(Definitions.scala:1709)
  at dotty.tools.dotc.core.Definitions.init(Definitions.scala:1725)
  at dotty.tools.dotc.core.Contexts$ContextBase.initialize(Contexts.scala:836)
  at dotty.tools.dotc.core.Contexts$Context.initialize(Contexts.scala:548)
  at dotty.tools.dotc.Run.rootContext(Run.scala:67)
  at dotty.tools.dotc.Run.<init>(Run.scala:85)
  at scala.quoted.staging.QuoteCompiler$ExprRun.<init>(QuoteCompiler.scala:105)
  at scala.quoted.staging.QuoteCompiler.newRun(QuoteCompiler.scala:48)
  at scala.quoted.staging.QuoteDriver.run(QuoteDriver.scala:39)
  at scala.quoted.staging.Toolbox$$anon$1.run(Toolbox.scala:38)
  at scala.quoted.staging.package$.withQuotes(staging.scala:42)
  ... 38 elided

The issue is probably in ClasspathFromClassloader. We are probably not getting the full classpath from the classloader.

[erikerlandson]

is there a known fix for this? I'm also getting similar:

Welcome to Scala 3.1.1 (11.0.14, Java OpenJDK 64-Bit Server VM).
Type in expressions for evaluation. Or try :help.
                                                                                                                                            
scala> import scala.quoted.*
                                                                                                                                            
scala> given staging.Compiler = staging.Compiler.make(getClass.getClassLoader)
lazy val given_Compiler: quoted.staging.Compiler
                                                                                                                                            
scala> staging.run { coulomb.infra.meta.test2("coulomb") }
dotty.tools.dotc.MissingCoreLibraryException: Could not find package scala from compiler core libraries.
Make sure the compiler core libraries are on the classpath.

[som-snytt]

scala> val s = getClass.getClassLoader.getParent.toString
val s: String = ClasspathFilter(
  parent = URLClassLoader with NativeCopyLoader with RawResources(
  urls = List(/home/.../snips/i7647/target/scala-3.1.1/classes, /home/.../.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-library_3/3.1.1/scala3-library_3-3.1.1.jar, /home/.../.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.6/scala-library-2.13.6.jar, /home/.../.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-compiler_3/3.1.1/scala3-compiler_3-3.1.1.jar, /home/.../.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-interfaces/3.1.1/scala3-interfaces-3.1.1.jar, /home/.../.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/tasty-core_3/3.1.1/tasty-core_3-3.1.1.jar, /home/.../.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/modules/scala-asm/9.1.0-scala-1/scala-asm-9.1.0-scala-1.jar, /home/.../.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-sbt/compiler- ... large output truncated, print value to show all

scala> s.indexOf("stag")
val res3: Int = -1

Whoever said "All the world's a staging" was wrong.

[som-snytt]

Also now

scala> import scala.quoted.*, staging.*
-- [E006] Not Found Error: -----------------------------------------------------
1 |import scala.quoted.*, staging.*
  |                       ^^^^^^^
  |                       Not found: staging
  |
  | longer explanation available when compiling with `-explain`
1 error found

scala> import scala.quoted.*

scala> given scala.quoted.staging.Compiler = staging.Compiler.make(getClass.getClassLoader)
-- [E046] Cyclic Error: --------------------------------------------------------
1 |given scala.quoted.staging.Compiler = staging.Compiler.make(getClass.getClassLoader)
  |                   ^
  |                   Cyclic reference involving val given_Compiler
  |
  | longer explanation available when compiling with `-explain`
-- [E006] Not Found Error: -----------------------------------------------------
1 |given scala.quoted.staging.Compiler = staging.Compiler.make(getClass.getClassLoader)
  |                                      ^^^^^^^
  |                                      Not found: staging

[griggt]

Also now

scala> import scala.quoted.*, staging.*
-- [E006] Not Found Error: -----------------------------------------------------
1 |import scala.quoted.*, staging.*
  |                       ^^^^^^^
  |                       Not found: staging

I think you need

libraryDependencies += "org.scala-lang" %% "scala3-staging" % scalaVersion.value

[erikerlandson]

I have the following:
libraryDependencies += "org.scala-lang" %% "scala3-staging" % scalaVersion.value

it seems to find that, but is not finding scala ? Which seems slightly strange, I certainly use other scala... stuff in the repl all the time

Like I need to add something else to the class loader that I give to the runtime compiler?

the bit about adding to lib dependencies is here:
https://docs.scala-lang.org/scala3/reference/metaprogramming/staging.html

[erikerlandson]

The classpath seems... reasonable:
https://gist.github.com/erikerlandson/5f0519664059b00e98cb4a5d2e6dfd50

[som-snytt]

A CL might have C and D on the CP, but it may get loaded by delegation to parent. If C depends on D, the parent may not have D.

I'll try to get some traction later. I've already done today's sudoku.

[erikerlandson]

just to make it unambiguous what I'm doing, here's a self-contained example:

Welcome to Scala 3.1.1 (11.0.14, Java OpenJDK 64-Bit Server VM).
Type in expressions for evaluation. Or try :help.
                                                                                                                                                                                               
scala> import scala.quoted.*
                                                                                                                                                                                               
scala> given staging.Compiler = staging.Compiler.make(getClass.getClassLoader)
lazy val given_Compiler: quoted.staging.Compiler
                                                                                                                                                                                               
scala> def test(using Quotes): Expr[String] = Expr("hi!")
def test(using x$1: quoted.Quotes): quoted.Expr[String]
                                                                                                                                                                                               
scala> staging.run { test }
dotty.tools.dotc.MissingCoreLibraryException: Could not find package scala from compiler core libraries.
Make sure the compiler core libraries are on the classpath.

[som-snytt]

scala> classOf[scala.Predef.type].getClassLoader
val res1: ClassLoader = CachedClassloader {
  parent: JLineLoader( ...

scala> classOf[scala.quoted.staging.Compiler].getClassLoader.getParent
val res2: ClassLoader = WrappedClassLoader(CachedClassloader {
  parent: JLineLoader( ...

scala> res2.getClass
val res3: Class[? <: ClassLoader] = class sbt.internal.classpath.WrappedLoader

[erikerlandson]

@som-snytt should I be using res2 (or res3?)

[som-snytt]

I don't know anything yet.

[erikerlandson]

bump

[smarter]

I don't think there's a bug here, you need to pass in a classloader which contains the whole classpath. getClass.getClassLoader doesn't work from the sbt repl because sbt uses a layered class loader by default. You can turn that off by setting ThisBuild / classLoaderLayeringStrategy := ClassLoaderLayeringStrategy.Flat in your build, or by finding the correct classloader that wraps everything you're using.
In particular, using classOf[staging.Compiler].getClassLoader rather than getClass.getClassLoader seems to work:

scala> import scala.quoted.*

scala> given staging.Compiler = staging.Compiler.make(classOf[staging.Compiler].getClassLoader)
lazy val given_Compiler: quoted.staging.Compiler

scala> def test(using Quotes): Expr[String] = Expr("hi!")
def test(using x$1: quoted.Quotes): quoted.Expr[String]

scala> staging.run { test }
val res0: String = hi!

[erikerlandson]

@smarter thanks! Using classOf[staging.Compiler].getClassLoader is working for me. FWIW, setting ThisBuild / classLoaderLayeringStrategy does not work.

[warn] there's a key that's not used by any other settings/tasks:
[warn]  
[warn] * ThisBuild / classLoaderLayeringStrategy
[warn]   +- /home/eje/git/coulomb/build.sbt:24