Plugin migration to 0.11.8

[Quafadas]

This is the child of a quick chat on discord ... about some confusion I have for a plugin upgrade from 0.11.7 -> 0.11.8.

Please note, This plugin is not important and I would not wish it to absorb a lot of time. The rules are; prison rules. There is a userbase of one (me), who owns this plugin and all projects which rely on it, so I can simply push everything forwards. I'm not worried about bincompat or breaking stuff - let's smash it up.

The symptom is that the plugin apparently worked on 0.11.7. On 0.11.8 I've started getting errors I don't understand ... the issue is likely that the plugin tampers with classpaths and stuff in an effort to get mdoc working. That it is brittle is... not surprising.

I'm happy, that I can reproduce test failures with ill. 0.11.8
https://github.com/Quafadas/millSite/actions/runs/9675334982/job/26692645876

But I'm not 100% clear, what to do about them!

[lefou]

@Quafadas Please post the essential output here, since github workflow logs will be deleted after some time. (Also, I couldn't spot the java.lang.AbstractMethodError you posted on Discord in the linked logs. Is this the same issue?)

[Quafadas]

Let me try and unconfuse myself here too.

Discovery
As currently published, (i.e. plugin v0.0.22) it relies on mill 0.11.5. If I attempt to use that plugin, from a project which has been upgraded to mill 0.11.8, then I get this error. This is what was posted on discord.

java.lang.AbstractMethodError: Receiver class io.github.quafadas.millSite.SiteModule$$anon$1 does not define or inherit an implementation of the resolved method 'abstract mill.define.Target mill$scalalib$JavaModule$$super$runClasspath()' of interface mill.scalalib.JavaModule.
    mill.scalalib.JavaModule.$anonfun$runClasspath$1(JavaModule.scala:554)
    scala.collection.mutable.HashMap.getOrElseUpdate(HashMap.scala:469)
    mill.moduledefs.Cacher.cachedTarget(Cacher.scala:10)
    mill.moduledefs.Cacher.cachedTarget$(Cacher.scala:9)
    mill.define.Module$BaseClass.cachedTarget(Module.scala:58)
    mill.scalalib.JavaModule.runClasspath(JavaModule.scala:553)
    mill.scalalib.JavaModule.runClasspath$(JavaModule.scala:553)
    io.github.quafadas.millSite.SiteModule$$anon$1.runClasspath(site.plugin.scala:22)
    io.github.quafadas.millSite.SiteJSModule.$anonfun$jsclasspath$1(site.jsplugin.scala:45)
    scala.collection.mutable.HashMap.getOrElseUpdate(HashMap.scala:469)
    mill.moduledefs.Cacher.cachedTarget(Cacher.scala:10)
    mill.moduledefs.Cacher.cachedTarget$(Cacher.scala:9)
    mill.define.Module$BaseClass.cachedTarget(Module.scala:58)
    io.github.quafadas.millSite.SiteJSModule.$init$(site.jsplugin.scala:44)
    io.github.quafadas.millSite.SiteModule$$anon$1.<init>(site.plugin.scala:22)
    io.github.quafadas.millSite.SiteModule.$init$(site.plugin.scala:22)
    millbuild.build$site$.<init>(build.sc:103)
    millbuild.build.site$lzycompute$1(build.sc:124)
    millbuild.build.site(build.sc:124)
    java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
    java.base/java.lang.reflect.Method.invoke(Method.java:580)
    mill.resolve.ResolveCore$.$anonfun$resolveDirectChildren0$12(ResolveCore.scala:371)

My (naive) attempt to fix this was to simply push forwards and upgrade the plugin to depend on mill 0.11.8.

I think you are right that this is therefore a different error. I think that this error would therefore be the one I want to fix, as it is what I imagine will work in the future. The error below comes from the test suite, when I tell the plugin it depends on mill 0.11.8

Compiling compiler interface...
[info] compiling 3 Scala sources to /Users/simon/Code/mill_scala3_mdoc_site/out/itest/test.dest/01-simple/out/mill-build/compile.dest/classes ...
[info] done compiling
java.lang.NoClassDefFoundError: mill/scalalib/WithZincWorker
    io.github.quafadas.millSite.SiteModule.$init$(site.plugin.scala:22)
    millbuild.build$simples$.<init>(build.sc:13)
    millbuild.build.simples$lzycompute$1(build.sc:35)
    millbuild.build.simples(build.sc:35)
    millbuild.build.verify(build.sc:20)
    millbuild.MiscInfo_build$.$anonfun$millBaseModuleInfo$68(build.sc:14)
    mainargs.Invoker$.$anonfun$invoke0$2(Invoker.scala:68)
    mainargs.Result.flatMap(Result.scala:13)
    mainargs.Result.flatMap$(Result.scala:12)
    mainargs.Result$Success.flatMap(Result.scala:23)
    mainargs.Invoker$.invoke0(Invoker.scala:67)
    mainargs.Invoker$.invoke(Invoker.scala:77)
    mill.resolve.Resolve$.$anonfun$invokeCommand0$11(Resolve.scala:158)
    mainargs.Result.flatMap(Result.scala:13)
    mainargs.Result.flatMap$(Result.scala:12)
    mainargs.Result$Success.flatMap(Result.scala:23)
java.lang.ClassNotFoundException: mill.scalalib.WithZincWorker
    java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445)
    java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:593)
    java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526)
    io.github.quafadas.millSite.SiteModule.$init$(site.plugin.scala:22)
    millbuild.build$simples$.<init>(build.sc:13)
    millbuild.build.simples$lzycompute$1(build.sc:35)
    millbuild.build.simples(build.sc:35)
    millbuild.build.verify(build.sc:20)
    millbuild.MiscInfo_build$.$anonfun$millBaseModuleInfo$68(build.sc:14)
    mainargs.Invoker$.$anonfun$invoke0$2(Invoker.scala:68)
    mainargs.Result.flatMap(Result.scala:13)
    mainargs.Result.flatMap$(Result.scala:12)
    mainargs.Result$Success.flatMap(Result.scala:23)
    mainargs.Invoker$.invoke0(Invoker.scala:67)
    mainargs.Invoker$.invoke(Invoker.scala:77)
    mill.resolve.Resolve$.$anonfun$invokeCommand0$11(Resolve.scala:158)
    mainargs.Result.flatMap(Result.scala:13)
    mainargs.Result.flatMap$(Result.scala:12)
    mainargs.Result$Success.flatMap(Result.scala:23)

[lefou]

Regarding the AbstractMethodError:

Just to get it said again: we effectively broke binary compatatibility in Mill 0.11.8 and probably also before, but assumed that all ways to break in the wild are unusual enough. Maybe we were wrong.

The best way to narrow it down would be to go to that version of your plugin (which depends on Mill 0.11.5) and add Mill 0.11.8 to the otherwise successful integration test matrix. It should reproduce the issue and can be a good starting point (for me) to analyze and work-around.

[Quafadas]

I've made a couple of changes which enable this.

As a part of this, I realised that the test suite encoded the version of mill rather poorly - and I hadn't changed it. As soon as I told the test suite to use mill 0.11.8, and the plugin to use mill 0.11.8 - it worked. So the second error I posted above is then a product of my own sloppy code. This means that I can indeed push forward to mill 0.11.8, and this would solve the issue to my own satisfaction.

If however, it was desirable to check what happened on the bincompat side, I pushed a branch 0.11.8 which does exactly as you suggest, and tests the plugin (compiled against mill 0.11.7) against mill 0.11.7 AND mill 0.11.8. I believe this sets up exactly the situation you asked for above.

https://github.com/Quafadas/millSite/actions/runs/9677594713/job/26699589363

Run ./millw itest[__].test
Publishing Artifact(io.github.quafadas,millSite_mill0.11_2.13,0.0.23-5-617964) to ivy repo /home/runner/work/millSite/millSite/out/itest/0.11.7/test.dest/ivyRepo/local
Starting integration test [1/4]: 01-simple
Invoking integration test [1/4]: 01-simple: Targets(targets=List(verify),expectedExitCode=0,env=Map(),noServer=true)
Succeeded integration test [1/4]: 01-simple
Starting integration test [2/4]: 02-recommended
Invoking integration test [2/4]: 02-recommended: Targets(targets=List(verify),expectedExitCode=0,env=Map(),noServer=true)
Succeeded integration test [2/4]: 02-recommended
Starting integration test [3/4]: 03-noMdoc
Invoking integration test [3/4]: 03-noMdoc: Targets(targets=List(verify),expectedExitCode=0,env=Map(),noServer=true)
Succeeded integration test [3/4]: 03-noMdoc
Starting integration test [4/4]: 04-mdocJS
Invoking integration test [4/4]: 04-mdocJS: Targets(targets=List(verify),expectedExitCode=0,env=Map(),noServer=true)
Succeeded integration test [4/4]: 04-mdocJS
Integration tests: 4, 4 succeeded, 0 skipped, 0 failed
Publishing Artifact(io.github.quafadas,millSite_mill0.11_2.13,0.0.23-5-617964) to ivy repo /home/runner/work/millSite/millSite/out/itest/0.11.8/test.dest/ivyRepo/local
Starting integration test [1/4]: 01-simple
Invoking integration test [1/4]: 01-simple: Targets(targets=List(verify),expectedExitCode=0,env=Map(),noServer=true)
Failed integration test [1/4]: 01-simple
Starting integration test [2/4]: 02-recommended
Invoking integration test [2/4]: 02-recommended: Targets(targets=List(verify),expectedExitCode=0,env=Map(),noServer=true)
Failed integration test [2/4]: 02-recommended
Starting integration test [3/4]: 03-noMdoc
Invoking integration test [3/4]: 03-noMdoc: Targets(targets=List(verify),expectedExitCode=0,env=Map(),noServer=true)
Failed integration test [3/4]: 03-noMdoc
Starting integration test [4/4]: 04-mdocJS
Invoking integration test [4/4]: 04-mdocJS: Targets(targets=List(verify),expectedExitCode=0,env=Map(),noServer=true)
Failed integration test [4/4]: 04-mdocJS

Details for 4 failed tests: 
Output of failed test: 01-simple
---------- Output for Invocation: TestInvocationResult(Targets(targets=List(verify),expectedExitCode=0,env=Map(),noServer=true),Failed,Vector(),Vector(),Some(/home/runner/work/millSite/millSite/out/itest/0.11.8/test.dest/01-simple/out-7792489995905545682.log))
[info] compiling 3 Scala sources to /home/runner/work/millSite/millSite/out/itest/0.11.8/test.dest/01-simple/out/mill-build/compile.dest/classes ...
Integration tests: 4, 0 succeeded, 0 skipped, 4 failed
[info] done compiling
java.lang.AbstractMethodError: Receiver class io.github.quafadas.millSite.SiteModule$$anon$1 does not define or inherit an implementation of the resolved method 'abstract mill.define.Target mill$scalalib$JavaModule$$super$runClasspath()' of interface mill.scalalib.JavaModule.
    mill.scalalib.JavaModule.$anonfun$runClasspath$1(JavaModule.scala:554)
    scala.collection.mutable.HashMap.getOrElseUpdate(HashMap.scala:469)
    mill.moduledefs.Cacher.cachedTarget(Cacher.scala:10)
    mill.moduledefs.Cacher.cachedTarget$(Cacher.scala:9)
    mill.define.Module$BaseClass.cachedTarget(Module.scala:58)
    mill.scalalib.JavaModule.runClasspath(JavaModule.scala:553)
    mill.scalalib.JavaModule.runClasspath$(JavaModule.scala:553)
    io.github.quafadas.millSite.SiteModule$$anon$1.runClasspath(site.plugin.scala:22)
    io.github.quafadas.millSite.SiteJSModule.$anonfun$jsclasspath$1(site.jsplugin.scala:45)
    scala.collection.mutable.HashMap.getOrElseUpdate(HashMap.scala:469)
    mill.moduledefs.Cacher.cachedTarget(Cacher.scala:10)
    mill.moduledefs.Cacher.cachedTarget$(Cacher.scala:9)
    mill.define.Module$BaseClass.cachedTarget(Module.scala:58)
    io.github.quafadas.millSite.SiteJSModule.$init$(site.jsplugin.scala:44)
    io.github.quafadas.millSite.SiteModule$$anon$1.<init>(site.plugin.scala:22)
    io.github.quafadas.millSite.SiteModule.$init$(site.plugin.scala:22)
    millbuild.build$simples$.<init>(build.sc:13)
    millbuild.build.simples$lzycompute$1(build.sc:34)
    millbuild.build.simples(build.sc:34)
    millbuild.build.verify(build.sc:20)
    millbuild.MiscInfo_build$.$anonfun$millBaseModuleInfo$71(build.sc:13)
    mainargs.Invoker$.$anonfun$invoke0$2(Invoker.scala:77)
    mainargs.Result.flatMap(Result.scala:13)
    mainargs.Result.flatMap$(Result.scala:12)
    mainargs.Result$Success.flatMap(Result.scala:23)
    mainargs.Invoker$.invoke0(Invoker.scala:76)
    mainargs.Invoker$.invoke(Invoker.scala:86)
    mill.resolve.Resolve$.$anonfun$invokeCommand0$12(Resolve.scala:159)
    mainargs.Result.flatMap(Result.scala:13)
    mainargs.Result.flatMap$(Result.scala:12)
    mainargs.Result$Success.flatMap(Result.scala:23)

---------- End of output

I do want to say again, that working through your suggestions has solved this to my own satisfaction - thank you for the tips!

[lefou]

Nice to hear that you found a solution. Thank you for preparing the reproducer! I'm still interested in understanding the situation, but since there is no pressure from your side and I already have a tight schedule this week, I will look at it somewhen later.