Fatal compilation error when using scala-pickles/boopickle libraries (probably macros-related)

[dkomanov]

Environment:

Ubuntu 16.04

bazel version 
Build label: 0.4.3rc6
Build target: bazel-out/local-fastbuild/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Wed Dec 21 15:53:43 2016 (1482335623)
Build timestamp: 1482335623
Build timestamp as int: 1482335623

In WORKSPACE:

git_repository(
    name = "io_bazel",
    remote = "git://github.com/bazelbuild/bazel.git",
    tag = "0.4.3",
)
git_repository(
    name = "io_bazel_rules_scala",
    remote = "https://github.com/bazelbuild/rules_scala.git",
    commit = "4f3fc159d64711f2b28d18b507cfe25c3348e4d5",
)
load("@io_bazel_rules_scala//scala:scala.bzl", "scala_repositories")
scala_repositories()

maven_jar(
    name = "me_chrons_boopickle_2_11",
    artifact = "me.chrons:boopickle_2.11:1.2.4",
)

In third_party/BUILD:

java_library(
    name = "me_chrons_boopickle_2_11",
    visibility = ["//visibility:public"],
    exports = [
        "@me_chrons_boopickle_2_11//jar",
    ],
)

SimpleCaseClass.scala:

package com.komanov.bazeltest

import boopickle.Default._

case class SimpleCaseClass(name: String)

object SimpleCaseClass {
  override def toByteArray(s: SimpleCaseClass) = {
    Pickle.intoBytes(s)
  }
}

Its BUILD file:

scala_library(
    name = "bazeltest",
    srcs = ["SimpleCaseClass.scala"],
    deps = ["//third_party:me_chrons_boopickle_2_11"],
)

When I try to build it:

bazel build --verbose_failures --worker_verbose //src/com/komanov/bazeltest
INFO: Found 1 target...
WARNING: Scalac worker (id 39) can no longer be used, because its process terminated itself or got killed.
INFO: Destroying Scalac worker (id 39)
INFO: Created new non-sandboxed Scalac worker (id 40), logging to .../.cache/bazel/_bazel_user/8284de460f2a76fffe4edd44c16c874d/bazel-workers/worker-40-Scalac.log
ERROR: .../src/com/komanov/bazeltest/BUILD:1:1: scala //src/com/komanov/bazeltest:bazeltest failed: Worker process did not return a correct WorkResponse. This is probably caused by a bug in the worker, writing unexpected other data to stdout.
Target //src/com/komanov/bazeltest:bazeltest failed to build
INFO: Elapsed time: 2.200s, Critical Path: 2.09s

worker-40-Scalac.log.txt

Looks like macros expansion related (non-macros code is working fine with this library).

[johnynek]

Yeah, you need to use a //file dependency, not //jar for targets that have macros. I wish there was a more robust way to detect this but currently it is manual.

…

On Sun, Jan 1, 2017 at 11:09 Dmitry Komanov ***@***.***> wrote: Environment: Ubuntu 16.04 bazel version Build label: 0.4.3rc6 Build target: bazel-out/local-fastbuild/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar Build time: Wed Dec 21 15:53:43 2016 (1482335623) Build timestamp: 1482335623 Build timestamp as int: 1482335623 In WORKSPACE: git_repository( name = "io_bazel", remote = "git://github.com/bazelbuild/bazel.git", tag = "0.4.3", ) git_repository( name = "io_bazel_rules_scala", remote = "https://github.com/bazelbuild/rules_scala.git", commit = "4f3fc159d64711f2b28d18b507cfe25c3348e4d5", ) ***@***.***_bazel_rules_scala//scala:scala.bzl", "scala_repositories") scala_repositories() maven_jar( name = "me_chrons_boopickle_2_11", artifact = "me.chrons:boopickle_2.11:1.2.4", ) In third_party/BUILD: java_library( name = "me_chrons_boopickle_2_11", visibility = ["//visibility:public"], exports = [ ***@***.***_chrons_boopickle_2_11//jar", ], ) SimpleCaseClass.scala: package com.komanov.bazeltest import boopickle.Default._ case class SimpleCaseClass(name: String) object SimpleCaseClass { override def toByteArray(s: SimpleCaseClass) = { Pickle.intoBytes(s) } } Its BUILD file: scala_library( name = "bazeltest", srcs = ["SimpleCaseClass.scala"], deps = ["//third_party:me_chrons_boopickle_2_11"], ) When I try to build it: bazel build --verbose_failures --worker_verbose //src/com/komanov/bazeltest INFO: Found 1 target... WARNING: Scalac worker (id 39) can no longer be used, because its process terminated itself or got killed. INFO: Destroying Scalac worker (id 39) INFO: Created new non-sandboxed Scalac worker (id 40), logging to .../.cache/bazel/_bazel_user/8284de460f2a76fffe4edd44c16c874d/bazel-workers/worker-40-Scalac.log ERROR: .../src/com/komanov/bazeltest/BUILD:1:1: scala //src/com/komanov/bazeltest:bazeltest failed: Worker process did not return a correct WorkResponse. This is probably caused by a bug in the worker, writing unexpected other data to stdout. Target //src/com/komanov/bazeltest:bazeltest failed to build INFO: Elapsed time: 2.200s, Critical Path: 2.09s worker-40-Scalac.log.txt <https://github.com/bazelbuild/rules_scala/files/680181/worker-40-Scalac.log.txt> Looks like macros expansion related (non-macros code is working fine with this library). — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#118>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEJdpHxvNCiceofzxluQN40TYCRvpkyks5rOBX4gaJpZM4LYxne> .

[ittaiz]

@johnynek is this due to bazelbuild/bazel#990?

[johnynek]

Yes. Somewhat. The ticket was a bit incorrect on my part. The Maven rule just makes targets. Now it makes a java jar target and a file target for the jar. If you use java jar, we assume (not anything in bazel) that we should run ijar. If we have a file target we don't (just a choice we made in the scala.bzl code). Thinking about it more, we really probably never need ijar on maven artifacts if we assume they change rarely. So, preferring file is probably the easiest way to go.

…

On Tue, Jan 10, 2017 at 23:20 Ittai Zeidman ***@***.***> wrote: @johnynek <https://github.com/johnynek> is this due to bazelbuild/bazel#990 <bazelbuild/bazel#990>? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#118 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEJdlyCDqq3u31H7QF_xFervjawF7sTks5rRJ7FgaJpZM4LYxne> .

[ittaiz]

Yeah that sounds really good. Because the main reason for ijars is to reduce unneeded churn and as far as I understand that is less likely with maven dependencies On Wed, 11 Jan 2017 at 18:13 P. Oscar Boykin <notifications@github.com> wrote:

…

Yes. Somewhat. The ticket was a bit incorrect on my part. The Maven rule just makes targets. Now it makes a java jar target and a file target for the jar. If you use java jar, we assume (not anything in bazel) that we should run ijar. If we have a file target we don't (just a choice we made in the scala.bzl code). Thinking about it more, we really probably never need ijar on maven artifacts if we assume they change rarely. So, preferring file is probably the easiest way to go. On Tue, Jan 10, 2017 at 23:20 Ittai Zeidman ***@***.***> wrote: > @johnynek <https://github.com/johnynek> is this due to > bazelbuild/bazel#990 <bazelbuild/bazel#990>? > > > > > — > You are receiving this because you were mentioned. > > > Reply to this email directly, view it on GitHub > < #118 (comment) >, > or mute the thread > < https://github.com/notifications/unsubscribe-auth/AAEJdlyCDqq3u31H7QF_xFervjawF7sTks5rRJ7FgaJpZM4LYxne > > . > — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#118 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABUIF9VfKNvXvzynlTy8a7OmNzzMbvrbks5rRP-UgaJpZM4LYxne> .

[ittaiz]

@johnynek going back to this again- where do we generate an ijar for maven jars? from the code it seems to me, probably mistakenly, that we generate it only on targets we create. Hence sending it to ScalaCInvoker and such. Can you point me to the offender?
Thanks!

[johnynek]

We don't run ijar on any inputs. Ijar is run to produce compile jar outputs. So by the time we see a maven dependency it is already a java (jar import) target. If we could distinguish jar imports from things compiled from source we could depend on the full jar for jar imports. But I don't know if that is possible from the API exposed to skylark rules.

…

On Sat, Jan 14, 2017 at 19:41 Ittai Zeidman ***@***.***> wrote: @johnynek <https://github.com/johnynek> going back to this again- where do we generate an ijar for maven jars? from the code it seems to me, probably mistakenly, that we generate it only on targets we create. Hence sending it to ScalaCInvoker and such. Can you point me to the offender? Thanks! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#118 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEJdj5F2bmRQtQ6x6ldd7J1CGbIYfzcks5rSbGNgaJpZM4LYxne> .

[ittaiz]

Two (hopefully last) questions:

1. Bazel itself does run ijar on maven_jars when you use //jar, no? That's why adding a //file solved Ignore scalatest in unused dependency checker #990 by allowing consumers to sidestep it, right?
2. when you say "preferring file is probably the easiest way to go" you mean that users of rules_scala should just manually use //file across the board and so avoid this problem?

[johnynek]

1. Bazel runs ijar on all java targets I believe. 2. Yes. I just mean as a work around, users would depend on /file from scala targets.

…

On Sat, Jan 14, 2017 at 20:55 Ittai Zeidman ***@***.***> wrote: Two (hopefully last) questions: 1. Bazel itself does run ijar on maven_jars when you use //jar, no? That's why adding a //file solved #990 by allowing consumers to sidestep it, right? 2. when you say "preferring file is probably the easiest way to go" you mean that users of rules_scala should just manually use //file across the board and so avoid this problem? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#118 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEJdgOkyszNiWuEyIUMXBBM7C0VqH-Sks5rScLPgaJpZM4LYxne> .

[johnynek]

Unless, of course as I said, we could find a way to distinguish jar import from other types of java targets and we wanted to assume that a jar import just won't change much. On Sat, Jan 14, 2017 at 21:02 P. Oscar Boykin <oscar.boykin@gmail.com> wrote:

…

1. Bazel runs ijar on all java targets I believe. 2. Yes. I just mean as a work around, users would depend on /file from scala targets. On Sat, Jan 14, 2017 at 20:55 Ittai Zeidman ***@***.***> wrote: > Two (hopefully last) questions: > > 1. Bazel itself does run ijar on maven_jars when you use //jar, no? > That's why adding a //file solved #990 by allowing consumers to sidestep > it, right? > 2. when you say "preferring file is probably the easiest way to go" > you mean that users of rules_scala should just manually use //file across > the board and so avoid this problem? > > — > You are receiving this because you were mentioned. > > > Reply to this email directly, view it on GitHub > <#118 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AAEJdgOkyszNiWuEyIUMXBBM7C0VqH-Sks5rScLPgaJpZM4LYxne> > . >

[ittaiz]

let's pull in @damienmg and ask if he knows if we have a way to make this distinction.
If not and it's not likely to get in we should probably update the readme

[damienmg]

What are you trying to do exactly? I believe aspect can let you investigate these information. /cc @dslomov

[ittaiz]

We to bypass the ijar for maven jars. This is because the ijar for some scala maven jars which contains macros breaks the build. Seeing as people depend on maven jars in the following practice

java_library(
    name = "me_chrons_boopickle_2_11",
    visibility = ["//visibility:public"],
    exports = [
        "@me_chrons_boopickle_2_11//jar",
    ],
)

bazel runs ijar on the export. We'd like to get the "raw" jar.
What @johnynek said was that we need a way to know if the dependency is a maven/jar import and so try to take the full jar and not the ijar.

[damienmg]

I still do not understand exactly how you want to do that, e.g. who is the consumer.

Also, why not fix ijar for it?

[ittaiz]

good questions and I'm not sure actually so I'll let @johnynek take a stab at them while I mull them over. Thanks for your guidance!

[johnynek]

"Fixing" ijar is highly non trivial. Macros are just code that runs at compile time. So your job is to determine what code could possibly be called by a macro method and not remove that code. With inheritance this may not even be statically knowable. Secondly, jars just don't change much, so rather than worry about getting this wrong, we want to consider not depend on ijar targets for external jars and only use it for jars built in the repo. Also, note, ijar does not work great for scala anyway (it works okay, not great). The scala compiler folks are a bit skeptical it is possible to make it work much better without changing the scala compiler. So even if you solve the first problem (figure out which methods could possibly be called), the payoff is not so great. Now the manual work around is for users to make their maven jar targets use /file.

…

On Sun, Jan 15, 2017 at 23:46 Ittai Zeidman ***@***.***> wrote: good questions and I'm not sure actually so I'll let @johnynek <https://github.com/johnynek> take a stab at them while I mull them over. Thanks for your guidance! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#118 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEJdowJ8wvx0co3YG3szul_uIiTMB-nks5rSzxzgaJpZM4LYxne> .

[ittaiz]

@johnynek What we'd like to do is to analyze the dependencies on every invocation of a scala_* target type and if we see deps which arrive from jar import to use the jar and not ijar, right?

[johnynek]

That is one technique for making macros more reliable.

…

On Mon, Jan 16, 2017 at 07:29 Ittai Zeidman ***@***.***> wrote: @johnynek <https://github.com/johnynek> What we'd like to do is to analyze the dependencies on every invocation of a scala_* target type and if we see deps which arrive from jar import to use the jar and not ijar, right? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#118 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEJdgLXFmdYJg35t5kY_2LC8N9T44YJks5rS6j-gaJpZM4LYxne> .