Add a `MillBuildModule` and improve BSP for `build.sc` #1972

lefou · 2022-07-27T09:07:32Z

The idea is to have a Module that can read and build Mill projects.

With such a module, we can model the build files compilation with Mill instead of Ammonite.

This also opens the door for further improvements. Incremental compilation and better BSP support comes to mind.

Fix [RFC] Create a module that can compile and run Ammonite and scala-cli scripts #1675

I experimentally also included support for using directives as comments, as they are used in Scala CLI. This is to play with it, but also to eventually replace the Ammonite-specific import syntax.

Tasks:

Translate file and line numbers in compile error messages
Write some tests

Caveats:

Ammonite imports are currently parsed with simple regular expressions, which definitely don't catch all legal combinations (but only the most common form). We can clearly improve them or integrate a real parser (that's not my home playground, PRs welcome). Instead, I think we should focus on supporting using directives.

Try it out

Additional to the trait, I also created an external module mill.scalalib.buildfile.MillBuildModule, so you can easily test it with any project you want.

$ mill -i dev.run ../mill-bsp -i showNamed "mill.scalalib.buildfile.MillBuildModule/{millVersion,sources,allSourceFiles,wrappedSourceFiles,millBuildSourceFiles,includedSourceFiles,compile,semanticDbData}"
[40/844] de.tobiasroeser.mill.vcs.version.VcsVersion.vcsState.overridden.de.tobiasroeser.mill.vcs.version.VcsVersion.vcsState 
[844/844] dev.run 
[1/1] showNamed 
[1/1] showNamed > [47/52] mill.scalalib.buildfile.MillBuildModule.semanticDbVersion 
{
  "mill.scalalib.buildfile.MillBuildModule.millVersion": "0.10.7",
  "mill.scalalib.buildfile.MillBuildModule.sources": [
    "ref:cb2806e2:/home/lefou/work/opensource/mill-bsp/.mill-version",
    "ref:c984eca8:/home/lefou/work/opensource/mill-bsp/.config/mill-version",
    "ref:c984eca8:/home/lefou/work/opensource/mill-bsp/.mill-jvm-opts",
    "ref:e0965e05:/home/lefou/work/opensource/mill-bsp/build.sc",
    "ref:60f41993:/home/lefou/work/opensource/mill-bsp/ci/shared.sc",
    "ref:5b62e47c:/home/lefou/work/opensource/mill-bsp/ci/upload.sc"
  ],
  "mill.scalalib.buildfile.MillBuildModule.allSourceFiles": [
    "ref:e0965e05:/home/lefou/work/opensource/mill-bsp/build.sc",
    "ref:60f41993:/home/lefou/work/opensource/mill-bsp/ci/shared.sc",
    "ref:5b62e47c:/home/lefou/work/opensource/mill-bsp/ci/upload.sc"
  ],
  "mill.scalalib.buildfile.MillBuildModule.wrappedSourceFiles": [
    {
      "orig": "ref:e0965e05:/home/lefou/work/opensource/mill-bsp/build.sc",
      "wrapped": [
        "ref:3f526faa:/home/lefou/work/opensource/mill-bsp/out/mill/scalalib/buildfile/MillBuildModule/wrappedSourceFiles.dest/build.sc.scala"
      ]
    },
    {
      "orig": "ref:60f41993:/home/lefou/work/opensource/mill-bsp/ci/shared.sc",
      "wrapped": [
        "ref:a59621d9:/home/lefou/work/opensource/mill-bsp/out/mill/scalalib/buildfile/MillBuildModule/wrappedSourceFiles.dest/shared.sc.scala"
      ]
    },
    {
      "orig": "ref:5b62e47c:/home/lefou/work/opensource/mill-bsp/ci/upload.sc",
      "wrapped": [
        "ref:03e8d55c:/home/lefou/work/opensource/mill-bsp/out/mill/scalalib/buildfile/MillBuildModule/wrappedSourceFiles.dest/upload.sc.scala"
      ]
    }
  ],
  "mill.scalalib.buildfile.MillBuildModule.millBuildSourceFiles": [
    "ref:e0965e05:/home/lefou/work/opensource/mill-bsp/build.sc",
    "ref:60f41993:/home/lefou/work/opensource/mill-bsp/ci/shared.sc",
    "ref:5b62e47c:/home/lefou/work/opensource/mill-bsp/ci/upload.sc"
  ],
  "mill.scalalib.buildfile.MillBuildModule.includedSourceFiles": [
    "ref:60f41993:/home/lefou/work/opensource/mill-bsp/ci/shared.sc",
    "ref:5b62e47c:/home/lefou/work/opensource/mill-bsp/ci/upload.sc"
  ],
  "mill.scalalib.buildfile.MillBuildModule.compile": {
    "analysisFile": "/home/lefou/work/opensource/mill-bsp/out/mill/scalalib/buildfile/MillBuildModule/compile.dest/zinc",
    "classes": "ref:c4b58a29:/home/lefou/work/opensource/mill-bsp/out/mill/scalalib/buildfile/MillBuildModule/compile.dest/classes"
  },
  "mill.scalalib.buildfile.MillBuildModule.semanticDbData": "ref:87bdfae1:/home/lefou/work/opensource/mill-bsp/out/mill/scalalib/buildfile/MillBuildModule/semanticDbData.dest/classes"
}

lefou · 2022-09-22T15:22:52Z

New release of using_directives is out. https://github.com/VirtusLab/using_directives/releases/tag/v0.0.9

# Conflicts: # build.sc # Conflicts: # build.sc

Using directives parser is currently not able to handle larger files, see VirtusLab/using_directives#41 Also improved BSP support. BSP mill-build modules should now be able to compile build files. # Conflicts: # scalalib/src/mill/scalalib/bsp/BspModule.scala

But use wrapped files in compilatin and semanticDb generation. Still need to figure out, how to properly match those to the original files.

kubukoz · 2023-01-20T17:26:51Z

@lefou my team is quite affected by Metals not working in build.sc (we're forced to use intellij, which has its well-known issues). Is there something I/we can do to help get this over the line?

lefou · 2023-01-21T10:07:44Z

@lefou my team is quite affected by Metals not working in build.sc (we're forced to use intellij, which has its well-known issues). Is there something I/we can do to help get this over the line?

This PR is currently not suitable for merging. I think I made some mistake in the latest rebase, so it fails most CI jobs. Also, the wrapping of the *.sc isn't correct.

I currently use IntelliJ via GenIdea/idea for all of my Mill-based projects, and I'm quite happy with it. What issues do you have? Are they reported? The only real issue I'm aware of is bad support for Cross modules, which use the same source files. You can only have one of each, which I typically control with the skipIdea target.

kubukoz · 2023-01-22T04:10:53Z

What issues do you have? Are they reported?

if you're asking about IDEA itself...

it's not Metals ;) personal preference, bloat etc.
still fails to type some things, that's been reported years ago I believe.

GenIdea/idea works fine.

lefou · 2023-01-29T09:14:27Z

@kubukoz I created PR #2291 which should improve the editing experience in Metals.

kubukoz · 2023-02-02T23:26:39Z

I tried that (after a somewhat confusing process of getting the locally published version work in the launcher), but it didn't seem to have much of an effect on our build :/

lefou · 2023-02-03T11:38:13Z

I tried that (after a somewhat confusing process of getting the locally published version work in the launcher), but it didn't seem to have much of an effect on our build :/

Oh, sorry to hear that. If I cut a 0.11.0-M4, would you be able to test with it more easily? You could then open a new discussion with the issues you see and maybe some details about your setup.

kubukoz · 2023-02-03T19:36:13Z

Yes, a release would help (I had issues getting millw to work with snapshots) :) I will then try to minimize the things that make it not work.

lefou · 2023-02-03T22:26:52Z

We already have a released version 0.11.0-M3-22-ac0ea4 which includes PR #2291.

kubukoz · 2023-02-05T16:46:07Z

okay, tried that and it still didn't work - posted a minimal reproduction in #2304.

lefou · 2023-02-19T09:33:02Z

BSP support is already improved by

BSP: Improve support for mill-build module and the build.sc #2291

This PR needs more work, including a rebase, so I'm closing it for now.

lefou mentioned this pull request Jul 29, 2022

SemanticDB support to be used by BSP (and others) #1977

Merged

5 tasks

lefou assigned lefou and unassigned lefou Sep 2, 2022

lefou mentioned this pull request Sep 9, 2022

Piggyback on ammonite's BSP implementation for build files #971

Closed

lefou force-pushed the mill-build-module branch from 3a04e86 to 628117c Compare September 22, 2022 20:47

lolgab changed the title ~~Add a MillBUildModule and improve BSP for build.sc~~ Add a MillBuildModule and improve BSP for build.sc Oct 4, 2022

lefou force-pushed the mill-build-module branch 2 times, most recently from 2dbe780 to 1a12a56 Compare October 12, 2022 08:51

lefou force-pushed the mill-build-module branch from a5ce916 to a605a47 Compare November 6, 2022 10:03

lefou mentioned this pull request Nov 19, 2022

Foreign modules in T.workspace are not stored in out/foreign-modules/ #2130

Closed

lefou force-pushed the mill-build-module branch from a2b2813 to 11259e1 Compare December 16, 2022 11:28

lefou added 18 commits January 8, 2023 11:39

Drafted a MillBuildModule and a new build.sc parser

61c26da

# Conflicts: # build.sc # Conflicts: # build.sc

Improved using parser and added ammonite include parser

3506533

Added ScalaModule.scalaMixedCompileTask and improved BSP support

b9b03b5

Added some scala doc

7ebfbb3

Removed mima config from buildfile module

cb60599

Update using_directives to 0.0.9

2f3a25a

Improved MillBuildModule, read and use project Mill version

f94c0a9

Refactored file layout and clean up

6c1b71a

Return wrapped files from allSourceFiles to match JavaModule semantics

3c9b0ce

Switched back to unwrapped sources in allSourceFiles

94b69c9

But use wrapped files in compilatin and semanticDb generation. Still need to figure out, how to properly match those to the original files.

Fixed some compiler warnings

d19f965

Added more @experimental annotations

7c91e08

Added first simple integration test for MillBuildModule

6109df9

Fixed regex for pathref cleaner

75956c1

Tweaked test

f40bf7b

Added temporary fix for test

345601a

Version bumps

543d158

lefou force-pushed the mill-build-module branch from 1d97b0e to 543d158 Compare January 8, 2023 11:00

lefou closed this Feb 19, 2023

lefou mentioned this pull request Mar 9, 2023

Target-level incremental re-building when build.sc changes #2348

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a `MillBuildModule` and improve BSP for `build.sc` #1972

Add a `MillBuildModule` and improve BSP for `build.sc` #1972

lefou commented Jul 27, 2022 •

edited

lefou commented Sep 22, 2022

kubukoz commented Jan 20, 2023

lefou commented Jan 21, 2023

kubukoz commented Jan 22, 2023

lefou commented Jan 29, 2023

kubukoz commented Feb 2, 2023

lefou commented Feb 3, 2023 •

edited

kubukoz commented Feb 3, 2023

lefou commented Feb 3, 2023

kubukoz commented Feb 5, 2023

lefou commented Feb 19, 2023

Add a MillBuildModule and improve BSP for build.sc #1972

Add a MillBuildModule and improve BSP for build.sc #1972

Conversation

lefou commented Jul 27, 2022 • edited

Try it out

lefou commented Sep 22, 2022

kubukoz commented Jan 20, 2023

lefou commented Jan 21, 2023

kubukoz commented Jan 22, 2023

lefou commented Jan 29, 2023

kubukoz commented Feb 2, 2023

lefou commented Feb 3, 2023 • edited

kubukoz commented Feb 3, 2023

lefou commented Feb 3, 2023

kubukoz commented Feb 5, 2023

lefou commented Feb 19, 2023

Add a `MillBuildModule` and improve BSP for `build.sc` #1972

Add a `MillBuildModule` and improve BSP for `build.sc` #1972

lefou commented Jul 27, 2022 •

edited

lefou commented Feb 3, 2023 •

edited