Skip to content

Better caching for Scala 3 (via ijar or tasty inspection) #1657

Open
@susliko

Description

@susliko

Bazel includes the ijar tool, which strips regular JAR files down to their interfaces. Target hashes are computed based on the resulting ijars, ensuring that only the necessary targets are rebuilt when changes do not affect public interfaces (e.g., modifications to implementations or private method signatures).

However, this approach often falls short in practice. For Scala 2, as noted in this issue, scalac adds a ScalaSignature annotation to classes, causing cache misses and unnecessary rebuilds.

For Scala 3, ijar generation is explicitly disabled.

Is the situation with ijars similar for Scala 3?
If ijars are not a viable solution, would analyzing TASTy files for interface-level changes be a feasible alternative?

Activity

jjudd

jjudd commented on Dec 3, 2024

@jjudd
Contributor

I agree that it would be great to improve ijar support for Scala or to use some other mechanism to improve detection of interface vs private implementation changes.

We have a discussion started in #1658 that includes a proposal for ijar (as well as many other things). Our current thoughts are to improve the ijar tool to be more Scala aware. If the Bazel folks are not interested in that, then we could explore other alternatives - perhaps there is some way to leverage the Zinc analysis file to accomplish this? Haven't thought much through that.

Part of the proposal covering ijar: https://docs.google.com/document/d/12BmUPpJpworA9ep2P_J00TxXwFRBZDQttUCsoYo0Yao/edit?tab=t.0#heading=h.z65373pwhv7g

jadenPete

jadenPete commented on Dec 23, 2024

@jadenPete
Contributor

Is the situation with ijars similar for Scala 3?

We've enabled ijar for Scala 3 targets in lucidsoftware/rules_scala haven't encountered any issues, so I think enabling it for Scala 3 targets should be fine. We have received the following warning:

ijar: skipping unknown attribute: "TASTY".

but I believe this should no longer be an issue, now that the TASTY attribute is explicitly included in ijar.

From what I understand, though, TASTy markes a major shift in how type information is stored between compiler invocations from Scala 2.13's "Pickle" format in that it includes a completely typed representation of the program's AST. I'm of the opinion that this defeats the purpose of using ijar, since most changes to a program's source code will now induce changes in its TASTy files, thus changing the generated ijar and invalidating reverse dependencies' cached compilation actions.

When you consider that and the fact that "scalac doesn’t have to read their .class files anymore; it can read their .tasty files", it may make more sense to develop an alternative to ijar specially for Scala 3 targets, as opposed to modifying ijar. That way, we can strip out information from TASTy files not needed for compilation (e.g. positional information, documentation, private members, etc.), similar to what @susliko described.

susliko

susliko commented on Dec 26, 2024

@susliko
Author

I'm of the opinion that this defeats the purpose of using ijar, since most changes to a program's source code will now induce changes in its TASTy files, thus changing the generated ijar and invalidating reverse dependencies' cached compilation actions.

Honestly, I don't think I've ever seen ijar work with Scala

jadenPete

jadenPete commented on Dec 26, 2024

@jadenPete
Contributor

Honestly, I don't think I've ever seen ijar work with Scala

Huh, that's odd; it's worked fine in Scala 2 for me. If you add a comment or some whitespace to a Scala file, or change the implementation of a function, do you find that ijar prevents recompilation?

jadenPete

jadenPete commented on Dec 26, 2024

@jadenPete
Contributor

When you consider that and the fact that "scalac doesn’t have to read their .class files anymore; it can read their .tasty files", it may make more sense to develop an alternative to ijar specially for Scala 3 targets, as opposed to modifying ijar.

Just an FYI for those following this issue: I'm going to try implementing this. I'll report back on my progress.

linked a pull request that will close this issue on Jan 15, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      Participants

      @jjudd@jadenPete@susliko

      Issue actions

        Better caching for Scala 3 (via ijar or tasty inspection) · Issue #1657 · bazel-contrib/rules_scala