{"payload":{"feedbackUrl":"https://github.com/orgs/community/discussions/53140","repo":{"id":151247156,"defaultBranch":"main","name":"mill","ownerLogin":"lefou","currentUserCanPush":false,"isFork":true,"isEmpty":false,"createdAt":"2018-10-02T11:59:48.000Z","ownerAvatar":"https://avatars.githubusercontent.com/u/1321393?v=4","public":true,"private":false,"isOrgOwned":false},"refInfo":{"name":"","listCacheKey":"v0:1718989427.0","currentOid":""},"activityList":{"items":[{"before":"4916d61f1f614f46eb25884366b88dfa3fd7b9d8","after":"c8884e648df8e947272aa2aafe3f2505962bc184","ref":"refs/heads/main","pushedAt":"2024-06-21T17:03:47.000Z","pushType":"push","commitsCount":26,"pusher":{"login":"lefou","name":"Tobias Roeser","path":"/lefou","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1321393?s=80&v=4"},"commit":{"message":"Prepared release 0.11.8 (#3229)\n\nPull request: https://github.com/com-lihaoyi/mill/pull/3229","shortMessageHtmlLink":"Prepared release 0.11.8 (com-lihaoyi#3229)"}},{"before":"2bb9c12e71d7f649edb9d6ae73f8469b3a08f210","after":"4916d61f1f614f46eb25884366b88dfa3fd7b9d8","ref":"refs/heads/main","pushedAt":"2024-05-29T11:00:34.000Z","pushType":"push","commitsCount":11,"pusher":{"login":"lefou","name":"Tobias Roeser","path":"/lefou","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1321393?s=80&v=4"},"commit":{"message":"Add `TestModule.discoveredTestClasses` target to ease test inspection (#3191)\n\nExample usage:\r\n\r\n```\r\n> mill show __.discoverTestClasses\r\n{\r\n \"foo.test.discoverTestClasses\": [\r\n \"foo.tests.FooTests\",\r\n \"foo.tests.FoobarTests\"\r\n ]\r\n \"bar.test.discoverTestClasses\": [\r\n \"foo.tests.BarTests\"\r\n ]\r\n}\r\n```\r\n\r\nFix https://github.com/com-lihaoyi/mill/issues/3169\r\n\r\nPull request: https://github.com/com-lihaoyi/mill/pull/3191","shortMessageHtmlLink":"Add TestModule.discoveredTestClasses target to ease test inspection (…"}},{"before":"17dea9c9a858abf61014f39f58959bfe0da15653","after":"2bb9c12e71d7f649edb9d6ae73f8469b3a08f210","ref":"refs/heads/main","pushedAt":"2024-05-22T16:16:09.000Z","pushType":"push","commitsCount":83,"pusher":{"login":"lefou","name":"Tobias Roeser","path":"/lefou","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1321393?s=80&v=4"},"commit":{"message":"Fix compiler bridge build setup and build essential versions in CI (#3179)\n\nFixed the build setup to build compiler brigdes. There were missing\r\ndependencies. I added all transitive dependencies of the source package\r\nto the compile classpath.\r\n\r\nAlso extended the CI setup to build essential compiler bridges with each\r\nCI run to detect any issues as early as possible.\r\n\r\nFix https://github.com/com-lihaoyi/mill/issues/2576\r\n\r\nPull request: https://github.com/com-lihaoyi/mill/pull/3179","shortMessageHtmlLink":"Fix compiler bridge build setup and build essential versions in CI (c…"}},{"before":"3335d2a2f7f33766a680e30df6a7d0dc0fbe08b3","after":"17dea9c9a858abf61014f39f58959bfe0da15653","ref":"refs/heads/main","pushedAt":"2024-02-20T08:38:36.000Z","pushType":"push","commitsCount":14,"pusher":{"login":"lefou","name":"Tobias Roeser","path":"/lefou","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1321393?s=80&v=4"},"commit":{"message":"Update semanticdb-scalac from 4.8.15 to 4.9.0 (#3038)\n\nPull request: https://github.com/com-lihaoyi/mill/pull/3038","shortMessageHtmlLink":"Update semanticdb-scalac from 4.8.15 to 4.9.0 (com-lihaoyi#3038)"}},{"before":"cc332c97de159b18b6d70f3f7c6637ad489e5b03","after":"3335d2a2f7f33766a680e30df6a7d0dc0fbe08b3","ref":"refs/heads/main","pushedAt":"2024-02-08T21:21:45.000Z","pushType":"push","commitsCount":10,"pusher":{"login":"lefou","name":"Tobias Roeser","path":"/lefou","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1321393?s=80&v=4"},"commit":{"message":"Update coursier from 2.1.8 to 2.1.9 (#2971)\n\nPull request: https://github.com/com-lihaoyi/mill/pull/2971","shortMessageHtmlLink":"Update coursier from 2.1.8 to 2.1.9 (com-lihaoyi#2971)"}},{"before":"59075270383d4ecec14893d8e813e95adc391c93","after":null,"ref":"refs/heads/wrapper","pushedAt":"2024-02-08T21:21:27.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"lefou","name":"Tobias Roeser","path":"/lefou","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1321393?s=80&v=4"}},{"before":"137735683c748c9e6309999699f1aff00e87fa3c","after":"cc332c97de159b18b6d70f3f7c6637ad489e5b03","ref":"refs/heads/main","pushedAt":"2024-01-29T08:54:10.000Z","pushType":"push","commitsCount":22,"pusher":{"login":"lefou","name":"Tobias Roeser","path":"/lefou","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1321393?s=80&v=4"},"commit":{"message":"Print `done` message from test framework to stdout (#2993)\n\nFixes #2992 \r\n\r\nWe are returning the doneMessage from the test framework as value of the\r\ntest task, but we are not printing it. Moreover, the output is\r\nshortcircuited when a task fails, so it's not passed to the `test`\r\ncommand.\r\nThis PR uses `ctx.log.outputStream.println(doneMessage)` to print the\r\nmessage before returning it so users can clearly see it in case of both\r\nsuccess and failure.\r\nThis was discovered because of the way weaver works on Scala.js Since\r\nthe `TestRunnerTests` are on `scalalib` and can't use Scala.js code, I\r\nimplemented some dummy `sbt.testing.Framework`s which just return a\r\nfixed `done` message.\r\n\r\nPull Request: https://github.com/com-lihaoyi/mill/pull/2993","shortMessageHtmlLink":"Print done message from test framework to stdout (com-lihaoyi#2993)"}},{"before":"b21ef92555495b03304e6594d59b86febaa1779e","after":"137735683c748c9e6309999699f1aff00e87fa3c","ref":"refs/heads/main","pushedAt":"2024-01-10T17:08:03.000Z","pushType":"push","commitsCount":33,"pusher":{"login":"lefou","name":"Tobias Roeser","path":"/lefou","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1321393?s=80&v=4"},"commit":{"message":"Update jline from 3.24.1 to 3.25.0 (#2962)\n\nPull request: https://github.com/com-lihaoyi/mill/pull/2962","shortMessageHtmlLink":"Update jline from 3.24.1 to 3.25.0 (com-lihaoyi#2962)"}},{"before":"223ae8f5304196d8ad76a4493cd2d2e295c60e5a","after":"b21ef92555495b03304e6594d59b86febaa1779e","ref":"refs/heads/main","pushedAt":"2023-12-04T13:50:32.000Z","pushType":"push","commitsCount":2,"pusher":{"login":"lefou","name":"Tobias Roeser","path":"/lefou","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1321393?s=80&v=4"},"commit":{"message":"Better detect Windows Subsystem for Linux environments (#2901)\n\nOn WSL(1), Mill does not support client-server mode.\r\nAs a consequence, we always need to run without a server.\r\n\r\nPull request: https://github.com/com-lihaoyi/mill/pull/2901","shortMessageHtmlLink":"Better detect Windows Subsystem for Linux environments (com-lihaoyi#2901"}},{"before":null,"after":"d84125ff184e7f30a9a11578c82d52efee929fbf","ref":"refs/heads/dependabot/github_actions/actions/setup-java-4","pushedAt":"2023-12-04T11:35:35.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"dependabot[bot]","name":null,"path":"/apps/dependabot","primaryAvatarUrl":"https://avatars.githubusercontent.com/in/29110?s=80&v=4"},"commit":{"message":"Bump actions/setup-java from 3 to 4\n\nBumps [actions/setup-java](https://github.com/actions/setup-java) from 3 to 4.\n- [Release notes](https://github.com/actions/setup-java/releases)\n- [Commits](https://github.com/actions/setup-java/compare/v3...v4)\n\n---\nupdated-dependencies:\n- dependency-name: actions/setup-java\n dependency-type: direct:production\n update-type: version-update:semver-major\n...\n\nSigned-off-by: dependabot[bot] ","shortMessageHtmlLink":"Bump actions/setup-java from 3 to 4"}},{"before":"423f174a81867e6bbfea85cbd1ca0a417e508380","after":"223ae8f5304196d8ad76a4493cd2d2e295c60e5a","ref":"refs/heads/main","pushedAt":"2023-11-30T07:09:39.000Z","pushType":"push","commitsCount":8,"pusher":{"login":"lefou","name":"Tobias Roeser","path":"/lefou","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1321393?s=80&v=4"},"commit":{"message":"Update scalafmt to 3.7.15 (#2896)\n\nNewer versions no longer support Java 8.\r\nAdded some Scala Steward in-place configuration.\r\n\r\nPull request: https://github.com/com-lihaoyi/mill/pull/2896","shortMessageHtmlLink":"Update scalafmt to 3.7.15 (com-lihaoyi#2896)"}},{"before":"503547bc4957a84126ae0ec09e36bf26a57cdaff","after":null,"ref":"refs/heads/mima","pushedAt":"2023-11-23T12:42:29.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"lefou","name":"Tobias Roeser","path":"/lefou","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1321393?s=80&v=4"}},{"before":"e93f6bd21daca8dbe1ede6c2236ea8a3fcb08ad9","after":"423f174a81867e6bbfea85cbd1ca0a417e508380","ref":"refs/heads/main","pushedAt":"2023-11-23T12:38:14.000Z","pushType":"push","commitsCount":8,"pusher":{"login":"lefou","name":"Tobias Roeser","path":"/lefou","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1321393?s=80&v=4"},"commit":{"message":"Rebootstrap Mill 0.11.6 (#2887)","shortMessageHtmlLink":"Rebootstrap Mill 0.11.6 (com-lihaoyi#2887)"}},{"before":"80ff26ea53fe99980f5b36f49755862867e66289","after":"e93f6bd21daca8dbe1ede6c2236ea8a3fcb08ad9","ref":"refs/heads/main","pushedAt":"2023-11-15T09:40:47.000Z","pushType":"push","commitsCount":41,"pusher":{"login":"lefou","name":"Tobias Roeser","path":"/lefou","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1321393?s=80&v=4"},"commit":{"message":"Add check for right Tests traits in ScalaJS and Native (#2874)\n\n## Motivation\r\n\r\nPeople get confused when they do things like:\r\n```scala\r\nimport mill._\r\nimport mill.scalalib._\r\nimport mill.scalajslib._\r\n\r\nobject root extends ScalaJSModule {\r\n def scalaVersion: T[String] = \"3.3.1\"\r\n def scalaJSVersion: T[String] = \"1.14.0\"\r\n object test extends ScalaTests with TestModule.Utest\r\n}\r\n```\r\n\r\nwhich doesn't work since we need to extend `ScalaJSTests` instead of\r\n`ScalaTests`.\r\nNow we crash with an exception:\r\n```\r\n[build.sc] [49/53] compile \r\n[info] compiling 1 Scala source to /Users/lorenzo/scala/repro/out/mill-build/compile.dest/classes ...\r\n[info] done compiling\r\n[build.sc] [53/53] methodCodeHashSignatures \r\nmill.api.MillException: root is a `ScalaJSModule`. root.test needs to extend `ScalaJSTests`.\r\n mill.scalalib.ScalaModule$ScalaTests.$init$(ScalaModule.scala:37)\r\n millbuild.build$root$test$.(build.sc:8)\r\n millbuild.build$root$.test$lzycompute$1(build.sc:30)\r\n millbuild.build$root$.test(build.sc:30)\r\n java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\r\n java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\r\n java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n java.base/java.lang.reflect.Method.invoke(Method.java:566)\r\n mill.resolve.ResolveCore$.$anonfun$resolveDirectChildren0$10(ResolveCore.scala:272)\r\n```\r\n\r\nPull Request: https://github.com/com-lihaoyi/mill/pull/2874\r\n\r\n---------\r\n\r\nCo-authored-by: Tobias Roeser ","shortMessageHtmlLink":"Add check for right Tests traits in ScalaJS and Native (com-lihaoyi#2874"}},{"before":"e05b6dd27c98be88e062f4d34696e089eafb5a72","after":null,"ref":"refs/heads/update-0.10.x-semanticdb","pushedAt":"2023-11-11T22:52:44.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"lefou","name":"Tobias Roeser","path":"/lefou","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1321393?s=80&v=4"}},{"before":null,"after":"e05b6dd27c98be88e062f4d34696e089eafb5a72","ref":"refs/heads/update-0.10.x-semanticdb","pushedAt":"2023-11-11T21:02:40.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"lefou","name":"Tobias Roeser","path":"/lefou","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1321393?s=80&v=4"},"commit":{"message":"Update semanticdb-scalac to 4.8.13","shortMessageHtmlLink":"Update semanticdb-scalac to 4.8.13"}},{"before":"ee9b8efbf4c428f64af2bb04418b94d0f1305c67","after":"80ff26ea53fe99980f5b36f49755862867e66289","ref":"refs/heads/main","pushedAt":"2023-09-27T20:49:04.000Z","pushType":"push","commitsCount":27,"pusher":{"login":"lefou","name":"Tobias Roeser","path":"/lefou","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1321393?s=80&v=4"},"commit":{"message":"Cycle detection in `moduleDeps` and `compileModuleDeps` (#2790)\n\nThe problem: We currently don't detect cycles in transitive `moduleDeps`\r\nand `compileModuleDeps`. Cycles in these result in a\r\n`StackOverflowError`.\r\n\r\nTo solve this, we first need to detect cycles. This is the newly added\r\n`ModuleUtils.recursive` method. Instead of `moduleDeps` we use the\r\n`moduleDepsChecked` in some places to ensure we checked for cycles\r\nbefore accessing module dependencies.\r\n\r\nUnfortunately, we can't easily report the detected cycle at evaluation\r\ntime, as `moduleDeps` needs to be evaluated before we can create a task\r\ngraph (via `T.traverse`), hence we need an alternative way to propagate\r\nthe exception. The current approach throws an `BuildScriptException`\r\nwhich itself derives from `mill.api.MillException`.\r\n\r\nIn `MillMain` as well as `MillClientMain`, our central entry points to\r\nMill, we catch the `MillException` and report the error to the user.\r\n\r\nPull request: https://github.com/com-lihaoyi/mill/pull/2790","shortMessageHtmlLink":"Cycle detection in moduleDeps and compileModuleDeps (com-lihaoyi#…"}},{"before":"b074ac92a9a5ed234d1ca21a444edb7e4ec38f6c","after":"ee9b8efbf4c428f64af2bb04418b94d0f1305c67","ref":"refs/heads/main","pushedAt":"2023-09-19T06:00:41.000Z","pushType":"push","commitsCount":6,"pusher":{"login":"lefou","name":"Tobias Roeser","path":"/lefou","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1321393?s=80&v=4"},"commit":{"message":"0.11.4","shortMessageHtmlLink":"0.11.4"}},{"before":"53dabcda58131660d0bd1da1fb5a543eaedfa959","after":null,"ref":"refs/heads/dependabot/github_actions/actions/checkout-4","pushedAt":"2023-09-18T08:04:15.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"dependabot[bot]","name":null,"path":"/apps/dependabot","primaryAvatarUrl":"https://avatars.githubusercontent.com/in/29110?s=80&v=4"}},{"before":"2f4468687237861950e203e37573493d3abb49c2","after":"b074ac92a9a5ed234d1ca21a444edb7e4ec38f6c","ref":"refs/heads/main","pushedAt":"2023-09-18T08:03:39.000Z","pushType":"push","commitsCount":48,"pusher":{"login":"lefou","name":"Tobias Roeser","path":"/lefou","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1321393?s=80&v=4"},"commit":{"message":"Some post-release tasks (#2745)\n\nPull request: https://github.com/com-lihaoyi/mill/pull/2745","shortMessageHtmlLink":"Some post-release tasks (com-lihaoyi#2745)"}},{"before":null,"after":"53dabcda58131660d0bd1da1fb5a543eaedfa959","ref":"refs/heads/dependabot/github_actions/actions/checkout-4","pushedAt":"2023-09-11T11:14:50.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"dependabot[bot]","name":null,"path":"/apps/dependabot","primaryAvatarUrl":"https://avatars.githubusercontent.com/in/29110?s=80&v=4"},"commit":{"message":"Bump actions/checkout from 3 to 4\n\nBumps [actions/checkout](https://github.com/actions/checkout) from 3 to 4.\n- [Release notes](https://github.com/actions/checkout/releases)\n- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)\n- [Commits](https://github.com/actions/checkout/compare/v3...v4)\n\n---\nupdated-dependencies:\n- dependency-name: actions/checkout\n dependency-type: direct:production\n update-type: version-update:semver-major\n...\n\nSigned-off-by: dependabot[bot] ","shortMessageHtmlLink":"Bump actions/checkout from 3 to 4"}},{"before":"c9c83cfa0b7ce47284b7b41f13bf41d4686a499a","after":"734d02887b2b57d5205aafe3857a18841d3d112d","ref":"refs/heads/resolveDeps","pushedAt":"2023-09-03T09:04:54.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"lefou","name":"Tobias Roeser","path":"/lefou","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1321393?s=80&v=4"},"commit":{"message":"Use CoursierModule.resolveDeps to resolve dependencies","shortMessageHtmlLink":"Use CoursierModule.resolveDeps to resolve dependencies"}},{"before":"95c2ea2205e9a97c4bf242b529d5d2ceec9c8555","after":"2f4468687237861950e203e37573493d3abb49c2","ref":"refs/heads/main","pushedAt":"2023-08-01T14:44:50.000Z","pushType":"push","commitsCount":10,"pusher":{"login":"lefou","name":"Tobias Roeser","path":"/lefou","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1321393?s=80&v=4"},"commit":{"message":"Bytecode reachability analysis for fine-grained target invalidation (#2417)\n\nThis PR replaces the coarse-grained script-import based\r\ntarget-invalidation-on-code-change (introduced in\r\nhttps://github.com/com-lihaoyi/mill/pull/1663) with fine-grained\r\ninvalidation based on the JVM-level method callgraph ([previous\r\ndiscussion](https://github.com/com-lihaoyi/mill/issues/2348)). This\r\ncallgraph is constructed by using [ASM](https://asm.ow2.io/) to parse\r\nthe bytecode generated by compiling `build.sc`: We compute a code-hash\r\nfor every method based on its bytecode, and propagate that hash\r\nthroughout the call graph Merkle-Tree-style, resulting in a\r\n`methodCodeHashSignatures: Map[String, Int]` dictionary that associates\r\neach method with a hash signature representing both its own code as well\r\nas the code of all methods that it calls (transitively). A more limited\r\nanalysis is applied to upstream library code. This hash signature is\r\nthen used as part of the `inputsHash` for each target, causing a Target\r\nto invalidate after a code change only if the code change affected the\r\nTarget's implementation method or some other method that it calls\r\n(transitively)\r\n\r\nOf the 7,000+ lines in this PR, only about ~1,000 are implementation\r\ncode, the other ~6,000 are test cases. Most of the 100+ files added in\r\nthis PR are also just tiny standalone Java/Scala files serving as test\r\ncases, with the main logic in `main/codesig/` being about a half-dozen\r\nfiles and a handful more for the test utilities\r\n\r\n## Algorithm\r\n\r\nAt a high level, the analysis logic living in the `main/codesig/` folder\r\ndoes the following:\r\n\r\n1. `LocalSummary` parses the compile output of `build.sc`. This extracts\r\nthe class hierarchy, computes method code hashes, and identifies method\r\ncallsites within method bodies.\r\n* Notably, JVM lambdas (InvokeDynamic bytecodes) are treated as if they\r\nare called at-definition-site. Even though that is not strictly precise,\r\nit is conservatively correct, and is the same thing the Scala.js\r\noptimizer does according to @sjrd.\r\n\r\n3. `ExternalSummary` parses any super-classes of locally-defined classes\r\nthat are in the upstream classpath - Java/Scala stdlib, Mill code, or\r\nthings that are `import $ivy`ed - and extracts the class hierarchy and\r\nmethod signatures without analyzing method bodies.\r\n* This is necessary so that calls against externally-defined interfaces,\r\ne.g. `InputStream#read`, can be properly traced to local implementations\r\neven if they do not extend the interface directly (e.g. a `MyInputStream\r\nextends ByteArrayInputStream` class)\r\n\r\n5. `ResolvedCalls` uses the class hierarchy to identify what potential\r\nimplementations each callsite will resolve to.\r\n* For calls to virtual methods, we treat them as potentially calling any\r\nimplementation of any subclasses across the whole program. This is done\r\nwithout regard to whether the sub-class would be instantiated or not\r\n* For calls to external methods for which we did not analyze the\r\nbytecode, we treat them conservatively and assume that they could\r\npotentially call any other external methods defined in the receiver\r\nclass, the function parameter types, or any of their superclasses, and\r\nthus any locally-defined overrides for those external methods. This is\r\nsimilar to the approach taken in [AVERROES: Whole-Program Analysis\r\nwithout the Whole\r\nProgram](https://plg.uwaterloo.ca/~olhotak/pubs/ecoop13.pdf)\r\n\r\n6. `ReachabilityAnalysis` converts the method resolution results into a\r\nheterogenous call graph (comprising `LocalDef`s, `Call`s, and\r\n`ExternalClsCall`s) that we feed into Tarjan's to generate topo-sorted\r\nstrongly-connected-components. We then walk the SCCs in topological\r\norder to do the Merkel-Tree hash propagation in one pass, with every\r\nmethod within a SCC having the same hash (because they all call each\r\nother circularly, and so they must have the same transitive call graph)\r\n\r\n* Here, we do some special casing for Single-Abstract-Method types and\r\nimplementations, to make them work like InvokeDynamic lambdas (i.e.\r\ncount reachability via instantiation, rather than via interface method\r\ninvocation). This is the right thing to do in principle since SAM\r\nmethods and Indy lambdas have similar use cases, and is necessary in\r\npractice because the compiler often chooses between the two styles of\r\ngenerating code somewhat arbitratily\r\n(https://github.com/scala/scala/pull/3616)\r\n\r\n7. For every `Target`, we look up the hash of it's `def` method, as well\r\nas the `` constructor methods for each of its enclosing `Module`\r\nclasses, and combine them to form the final hash of the `Target`\r\n\r\n* Mill `Target`s must live on `Module`s which form a tree of static\r\n`object`s. We walk the object tree from the `Target`'s directly\r\nenclosing module to the root module using a new `enclosingModule`\r\nproperty on the `mill.define.Ctx` value, and look up all their ``\r\nmethods. They should all only have one `` method since they're\r\nstatic objects\r\n\r\nFrom the Call Graph Analysis literature, this is a relatively\r\nstraightforward Class Hierarchy Analysis (CHA), with some tweaks called\r\nout above. Notably we aren't able to generate callgraphs via more\r\nprecise algorithms such as Rapid Type Analysis (RTA), which would\r\ngeneralize our the lambda special-casing for all user-implemented\r\ntraits.\r\n\r\n## Limitations\r\n\r\n1. We do not do any kind of dataflow analysis. \r\n* That means we do not know when a Target calls a method whether it is\r\npassing values in, receiving return values out, causing side effects, or\r\nsome combination of these.\r\n* We just assume that if a Target end up transitively calling a method,\r\nit is likely to affect it *somehow*, and we thus invalidate it to be\r\nre-run\r\n* This also means that `def`s and `val`s in the body of a class are\r\nhandled differently, despite looking superficially similar. `def`s are\r\nfull methods and participate in the call graph, so changes would only\r\ninvalidate downstream Targets that transitively end up calling that\r\nmethod. `val`s are simply fields that are initialized in the class'\r\n`` constructor method or `` class initialization, so\r\nchanging in `val` ends up invalidating any downstream Targets that\r\ncreate a `new` instance of that class: probably a much broader set of\r\nTargets\r\n\r\n2. Both the handling of JVM lambdas and the handling of virtual methods\r\nare approximations\r\n* For JVM lambdas, we assume that they are \"live\" just because they are\r\ninstantiated, without checking if they get called\r\n* This means that a lambda that is created but doesn't end up getting\r\ncalled gets imprecisely marked as a callgraph dependency\r\n* For virtual methods, we assume they are \"live\" just because they get\r\ncalled, without checking if the relevant class gets instantiated\r\n* That means that if a Target makes a virtual method call to e.g.\r\n`InputStream#read`, we will assume that Target depends on all\r\n`InputStream#read`s across the codebase, even if for subclasses of\r\n`InputStream` that never get instantiated\r\n* Both are conservative approximations that would cause some unnecessary\r\ncallgraph edges, but both approximations are well fit to actual usage\r\npatterns (JVM lambdas have one method and tend to have tons of different\r\nimplementations, whereas virtual methods tend to have multiple methods\r\nand a small number of implementations)\r\n* Doing the \"proper\" thing and checking _both_ for calling and\r\ninstantiation is done by Rapid Type Analysis (and other more advanced\r\ncallgraph analyzers). These typically require a separate whole-program\r\nanalysis for every Target we may want to invalidate, which would be\r\nunfeasible in a large build with 100s of Modules and 10,000s of Targets.\r\nThere are some approaches that may mitigate that (e.g. [Overlap-aware\r\nrapid type analysis for constructing one-to-one matched call graphs in\r\nregression test\r\nselection](https://www.semanticscholar.org/paper/Overlap-aware-rapid-type-analysis-for-constructing-Kim-Jeong/68b3dfadb2a67a13a1ec78f90f8d8192f71be0e6?citedSort=relevance),\r\n[Incrementalization of Analyses for Next Generation\r\nIDEs](https://www.semanticscholar.org/paper/Incrementalization-of-analyses-for-next-generation-Kloppenburg/414fa60c65fc1aed05e4f522386de50066641321))\r\nbut for now I'm just leaving it for future work.\r\n\r\n\r\n3. We avoid detailed callgraph analysis of external/upstream libraries\r\nbecause the entire transitive codebase would be very large: e.g. all of\r\nMill, all of `com.lihaoyi`, all of Scala std lib, all of Java std lib.\r\nLimiting our detailed analysis to the relatively-small `build.sc`\r\ncompile output simplifies things and greatly speeds things up, at the\r\nexpense of some precision in the callgraph\r\n\r\n\r\n4. Apart from the question of CHA v.s. RTA, it is probably also possible\r\nto get more precise callgraphs by performing the analysis at the Scala\r\nlevel rather than at the JVM bytecode level ([Constructing Call Graphs\r\nof Scala Programs](https://plg.uwaterloo.ca/~olhotak/pubs/ecoop14.pdf)).\r\nI chose to work at the JVM bytecode level because it's\r\nsimple/stable/familiar to me, but trying to do the analysis at the\r\nScala-level is potential future work.\r\n\r\n* We are able to ignore the Scala-level program information because at\r\nthe end of the day what runs on the JVM is bytecode. Thus the bytecode\r\nfully defines the runtime semantics of the program, and changes in\r\nbytecode are sufficient to tell us if a target needs to be invalidated,\r\neven if it's not as precise as what the Scala type information can give\r\nus\r\n\r\n## Notes\r\n\r\n1. I special-case handling of `new sourcecode.Line(_)` and explicitly\r\nignore the argument (~always a literal number). This is because we\r\nexplicitly do *not* want line number changes to force things to\r\nre-compile: even if they can in theory affect the output of a target,\r\nthe assumption is that it's just there for debugging/inspect purposes\r\nand won't. The way I do this in a bit hacky and relies on the bytecode\r\npattern generated by the Scala compiler, but it seems to work.\r\n\r\n2. This analysis in ignores fields entirely. In general, any change to\r\ninstance or static fields that matters would be accompanied by\r\ncorresponding changes to method bodies, even if just changes to the\r\n`` constructor method.\r\n\r\n3. The approximations we use to handle InvokeDynamic lambdas and virtual\r\nmethod calls is necessary for this analysis to be precise enough to be\r\nuseful.\r\n* If we treat lambdas like any other set of trait/implementations, the\r\ncallgraph would end up including an edge from every `def foo = T{}`\r\nTarget to every other Target in the `build.sc`.\r\n* Conversely, if we treated virtual methods the same way we treat\r\nlambdas, then every `mill.Module` someone instantiates (e.g. by\r\nreferencing the module name, which invokes the class static initializer,\r\nwhich instantiates the class itself) would have all its methods and\r\nTargets depended-on by default by the instantiating method.\r\n* Both of these would cause enough additional callgraph edges to make\r\neverything depend on everything else, rendering callgraph analysis\r\nuseless\r\n\r\n4. We ignore calls to `Target`-returning methods as part of the static\r\ncallgraph analysis: `Target`s, `Input`s, `Worker`s, etc. Those\r\n`NamedTask`s will themselves get invalidated if necessary if the\r\nnon-`NamedTask` code _they_ depend on changes, and invalidation will\r\nalready get propagated through the runtime `Task` graph without needing\r\nto also model it in the static callgraph. This would improve precision\r\nwhile also greatly shrinking the static callgraphs\r\n\r\n* This relies on an additional assumption over the current analysis: it\r\nassumed that methods returning `NamedTask`s are pure and do not generate\r\nside effects. This is probably an OK assumption to make, because proper\r\noperation of the runtime `Task` graph also relies on that assumption\r\n\r\n* This greatly improves precision for our use case. Virtual methods\r\nimplemented in many different places is a big weakness of CHA since it\r\nhas to assume any implementation could be called, especially combined\r\nwith calls to methods on external library traits that are also treated\r\nconservatively and have to assume that any virtual method defined in\r\nthat library (and thus any local implementation) can be called. However,\r\nin a Mill build, the vast majority of \"virtual methods implementations\"\r\nand \"external library method calls\" are the\r\n`mill.define.Target[A]`/`T[A]` methods we override and call. Skipping\r\nstatic callgraph analysis for these and relying on the exact runtime\r\nbuild graph is thus a huge improvement to the precision of target\r\ninvalidation.\r\n\r\n5. I wire up JSON logging (to disk in the `T.dest` folder) to the\r\n`--debug` flag. It's too expensive to generate all the time (~doubles\r\nthe time taken for `methodCodeHashSignatures`), but we definitely want\r\nit to be available when things are misbehaving and we need to debug\r\nthings\r\n\r\n## Automated Testing\r\n\r\n1. `main/codesig/test/cases/callgraph/` contains unit tests that assert\r\nthe (simplified) call graphs are the right shape for a variety of\r\nminimal scenarios. This also includes a few less trivial scenarios in\r\nthe `realistic/` folder: some of my old [Java\r\ngames](https://github.com/lihaoyi/Java-Games), a [parallel merge-sort\r\nalgorithm](https://github.com/handsonscala/handsonscala/blob/ebc0367144513fc181281a024f8071a6153be424/examples/13.7%20-%20ParallelMergeSort/MergeSort.sc)\r\nand some [Castor actor\r\ncode](https://github.com/handsonscala/handsonscala/blob/ebc0367144513fc181281a024f8071a6153be424/examples/16.8%20-%20LoggingRearrangedPipeline2/LoggingPipeline.sc)\r\nfrom my book Hands-on Scala, a Fastparse parser. Both Java and Scala\r\ncode work because we perform the analysis at the bytecode level.\r\n\r\n2. We also exercise the transitive hash-propagation logic in some of the\r\n`callgraph/` tests by replacing the \"hash sum\" logic with \"Set union\",\r\nusing it to compute transitive closures of the graph, and asserting that\r\nthe transitive closures are what we expect. This adds test coverage for\r\nmost of the logic around the hash propagation, without the expense of\r\nperforming runtime code-changes and re-compilation that we need to\r\nperform in the integration tests.\r\n\r\n3. `main/codesig/test/cases/methodhash/` contains tests that assert that\r\nthe method code hash signature has the correct properties: that it\r\nchanges when it should change and is un-changed when it should be\r\nun-changed (e.g. when there are only formatting/line-number/comment\r\nchanges)\r\n\r\n4. `integration/feature/codesig{simple,trait,scalamodule}` contain some\r\nintegration tests in that test the workflow end-to-end including the\r\nintegration into Mill's evaluator. These ensure that when you edit a\r\nfile and re-run Mill, the correct targets are re-evaluated. Test cases\r\ninclude a single Target, Targets inside module `object`s and `trait`s, a\r\nsingle `ScalaModule`, and a bunch of dependent/unrelated `ScalaModule`s\r\n\r\nI had to update a bunch of existing tests - e.g. those in\r\n`integration/feature/invalidation/` - because now random code changes no\r\nlonger invalidate targets within that file unless you actually change\r\nthe target or method bodies that the target calls.\r\n\r\nThis is a binary compatible change. I had to add some MIMA exclusions\r\ndue to MIMA not correctly considering things nested within a `private\r\nobject` as `private` (https://github.com/lightbend/mima/issues/771).\r\n\r\nI added an opt-out flag `--disable-callgraph-invalidation` to swap back\r\nto the previous script-import-graph-based invalidation system, as a\r\nchange-management measure to mitigate the risk of the novel invalidation\r\nalgorithm\r\n\r\n## Manual Testing\r\n\r\nI did some end-to-end tests on this PR, running `__.compile` on a few\r\ndifferent projects:\r\n\r\n1. On the com-lihaoyi/fansi build, the incrementality seems to work as\r\nexpected: e.g. changing the `fansi.JsFansiModule#scalaJSVersion` from\r\n`1.10.1` to `1.10.0` invalidates 87/796 targets which all seem Scala.js\r\nrelated, while changing the `FansiTestModule#ivyDeps` uTest version from\r\n0.8.1 to 0.8.0 invalidates 67/875 targets that are all test suite\r\nrelated. This is working as expected\r\n\r\n2. On the `com-lihaoyi/upickle` build, changing `bench.scalaJSVersion`\r\nfrom `1.13.0` to `1.13.1` invalidates 28/3695 targets, while changing\r\n`CommonJsModule#scalaJSVersion` invalidates 412/3695 targets\r\n\r\n3. On the `com-lihaoyi/mill` codebase:\r\n\r\n1. Adding a whitespace at the top of `build.sc`, to offset everyone's\r\nline numbers without changing their logic, invalidates 16/8154 targets,\r\nall downstream of `millVersion` which is invalidated by the dirty hash\r\nof the repo checkout changing\r\n\r\n2. Changing `scalajslib.worker[1].ivyDeps` by re-ordering them,\r\ninvalidates 17/8154 targets (just the ones above, + the one edited)\r\n\r\n3. Changing `MillScalaModule#scalacOptions` by removing `-feature`,\r\ninvalidates 836/8154 targets\r\n\r\n4. Changing `contrib.playlib.WorkerModule#sources` generating a\r\ntemporary file, invalidates 37/8154 targets, mostly stuff in\r\n`contrib.playlib.worker[_]`\r\n\r\n5. Changing a constant `Deps.scalaVersion` ends up invalidating\r\n~4343/8154 targets. I skimmed through the results and they seem\r\nreasonable: all `.scalaVersion` and `.compile` targets end up\r\ninvalidated, while many external targets aren't invalidated because they\r\ndon't have any local code to be affected by the callgraph analysis and\r\ntheir upstream build graph is not affected by `Deps.scalaVersion` (e.g.\r\n`integration.test.javacOptions`,\r\n`contrib.codeartifact.mandatoryScalacOptions`)\r\n\r\n6. Adding a new `val abc = 123` to the root module invalidates\r\neverything, which is expected since it changes the constructor of\r\n`build` which could potentially affect any module nested within it.\r\n1. One solution for this is to turn them into `lazy val`s. Maybe we can\r\ndo it automatically via a compiler plugin.\r\n2. Dataflow/Purity analysis is another alternative that would let us\r\nkeep them as `val`s while still distinguishing which `val` is used in\r\nwhat method, but that is much more complicated analysis than the\r\ncallgraph analysis this PR implements\r\n\r\nThe Fansi and uPickle results shows that this approach does work for\r\nnon-trivial builds, and the Mill results show that it works even on\r\npretty large builds like Mill's own `build.sc` codebase.\r\n\r\n## Debugging\r\n\r\nThe easiest way to understand what is going on with the callgraph\r\nanalysis is to run Mill with the `--debug` flag. This generates extra\r\noutput JSON files in `out/mill-build/methodCodeHashSignatures.dest/`\r\nwhich give you an insight into what is going on inside the callgraph\r\nanalyzer: the `localSummary.json`, `externalSummary.json`,\r\n`resolvedMethodCalls.json`, etc. These are opt-in with `--debug` due to\r\nthe added overhead it takes to generate them (0.3-0.5s for the Mill\r\ncodebase, would be larger for larger builds)\r\n\r\nOn item of interest is the `spanningInvalidationForest.json` that is\r\ngenerated the second time you run Mill with `--debug` enabled. This file\r\ncontains a tree of nested JSON dictionaries containing every method\r\ndefinition or call that was invalidated, with the roots of the tree\r\nbeing the invalidation \"roots\" (i.e. methods which were invalidated\r\nwithout any parents being invalidated), and the tree structure\r\nindicating an arbitrary path from an invalidation root to each specific\r\ninvalidated method. This is very useful for answering the question \"why\r\nwas this method/target/etc. invalidated in response to code changes\"\r\n\r\n## Performance\r\n\r\nPerformance-wise, ad-hoc benchmarks on `com-lihaoyi/mill`'s own build\r\nshow a ~5% increase in `build.sc` compilation times due to this. Not\r\nnothing, but probably acceptable: the cost is only paid when the\r\n`build.sc` is re-compiled, and it will likely end up saving much more\r\ntime in tasks that we can avoid running (e.g. a single no-op Zinc\r\nincremental compile may be 100s of milliseconds)\r\n\r\n| | methodCodeHashSignatures | compile |\r\n|-|---|---|\r\n| Cold | 685ms | 12,148ms |\r\n| Hot | 253ms | 4,143ms | \r\n\r\nIt's taken some amount of optimizations to reach this point. There are\r\ndefinitely further optimizations that can be done, e.g. replacing the\r\nvarious `Map`s we pass around with `Array` or parallelizing parts of the\r\nanalysis (It's mostly pure functional code and should be easy to\r\nparallelize)\r\n\r\n## What Users can do to improve incrementality\r\n\r\n1. `Module` `val`s should generally be avoided in favor of `def`s or\r\n`lazy val`s. `val`s are all bundled together in the `Module`\r\nconstructor, so any change to any `val` invalidates any downstream code\r\nwho depends on anything in that `Module`\r\n\r\n2. Abstract members of `Module`s should be `Target`s - `T[V]`s -\r\nwhenever possible, rather than plain `V`s. We make a stronger assumption\r\nthat `T[V]` methods are pure that we cannot assume in general for any\r\nmethod returning an un-wrapped `V`, and can rely fully on runtime\r\nbuild-graph analysis which is a lot more precise than the static\r\nclass-hierarchy-analysis we do no non-`Target` abstract methods.\r\n* e.g. a change to the code in `BuildInfo#buildInfoMembers` invalidates\r\nthe code in only that target, because we assume that `Target` method\r\ndependencies and invalidation will be handled by the runtime callgraph\r\n* But a change to `BuildInfo#buildInfoPackageName` invalidates the code\r\nin almost everything on every `JavaModule extends BuildInfo` because we\r\ncannot guarantee that the `` method of those `Module`'s does not\r\ndepend on `buildInfoMembers`.\r\n\r\nThese restrictions are fundamental given the simple\r\nreachability/class-hierarchy analysis that we do in this PR, and lifting\r\nthem would involve a more complicated dataflow analysis or purity\r\nanalysis that would be much less simple to implement\r\n\r\n---------\r\n\r\nCo-authored-by: Tobias Roeser ","shortMessageHtmlLink":"Bytecode reachability analysis for fine-grained target invalidation (c…"}},{"before":"9cdfc96a95220045fed8eb9bb7afb74f9eaff207","after":"95c2ea2205e9a97c4bf242b529d5d2ceec9c8555","ref":"refs/heads/main","pushedAt":"2023-07-20T07:38:32.000Z","pushType":"push","commitsCount":37,"pusher":{"login":"lefou","name":"Tobias Roeser","path":"/lefou","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1321393?s=80&v=4"},"commit":{"message":"Avoid intermediate mapping in ZincWorkerImpl (#2661)\n\nProfiled Mill compiling Mill itself and 0.07% of the time was spent in\r\nmapping the result of `compileMixed0` to `CompilationResult`.\r\nThis changes the `compileInternal` method to already use the right types\r\nso less conversions are needed.\r\n\r\nPull request: https://github.com/com-lihaoyi/mill/pull/2661","shortMessageHtmlLink":"Avoid intermediate mapping in ZincWorkerImpl (com-lihaoyi#2661)"}},{"before":"d732aac1a0f6190c9c46e436587c0092d1073453","after":null,"ref":"refs/heads/backport-2338","pushedAt":"2023-06-30T09:53:08.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"lefou","name":"Tobias Roeser","path":"/lefou","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1321393?s=80&v=4"}},{"before":"16718394bcacdc170605031c015916210682c812","after":"9cdfc96a95220045fed8eb9bb7afb74f9eaff207","ref":"refs/heads/main","pushedAt":"2023-06-09T08:05:58.827Z","pushType":"push","commitsCount":1,"pusher":{"login":"lefou","name":"Tobias Roeser","path":"/lefou","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1321393?s=80&v=4"},"commit":{"message":"CI: Make bin-compat-checks mandatory (#2581)\n\nPull request: https://github.com/com-lihaoyi/mill/pull/2581","shortMessageHtmlLink":"CI: Make bin-compat-checks mandatory (com-lihaoyi#2581)"}},{"before":"0bd4c71cbb0ac9c079cb3805fe63cc6c56feb3ba","after":"16718394bcacdc170605031c015916210682c812","ref":"refs/heads/main","pushedAt":"2023-06-09T06:02:46.747Z","pushType":"push","commitsCount":173,"pusher":{"login":"lefou","name":"Tobias Roeser","path":"/lefou","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1321393?s=80&v=4"},"commit":{"message":"Refined fix for `ScoverageModule` inner module (#2584)\n\nThis is a follow-up on #2583.\r\n\r\nUsing a ModuleRef make accesss to the inner module for the CLI\r\nimpossible.\r\nI added that case to the integration test and changed to use a `val`.\r\n\r\nPull request: https://github.com/com-lihaoyi/mill/pull/2584","shortMessageHtmlLink":"Refined fix for ScoverageModule inner module (com-lihaoyi#2584)"}},{"before":"13156d55a4e522cc56a44a3ec41019447f22c3f9","after":"a6b83d6d2044e15e58c9508f067f9fb680a38419","ref":"refs/heads/detect-cycles","pushedAt":"2023-04-26T19:05:41.000Z","pushType":"push","commitsCount":5,"pusher":{"login":"lefou","name":"Tobias Roeser","path":"/lefou","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1321393?s=80&v=4"},"commit":{"message":"Merge branch 'main' into detect-cycles","shortMessageHtmlLink":"Merge branch 'main' into detect-cycles"}},{"before":"e475e8777cbc20fa9d20b173c89b9d8ed46bc35f","after":"13156d55a4e522cc56a44a3ec41019447f22c3f9","ref":"refs/heads/detect-cycles","pushedAt":"2023-04-24T15:33:44.000Z","pushType":"push","commitsCount":2,"pusher":{"login":"lefou","name":"Tobias Roeser","path":"/lefou","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1321393?s=80&v=4"},"commit":{"message":"Merge branch 'main' into detect-cycles","shortMessageHtmlLink":"Merge branch 'main' into detect-cycles"}},{"before":"7202d6a0efbe5cbb90c716d3460a66e7a61e3643","after":"e475e8777cbc20fa9d20b173c89b9d8ed46bc35f","ref":"refs/heads/detect-cycles","pushedAt":"2023-04-24T12:42:36.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"lefou","name":"Tobias Roeser","path":"/lefou","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1321393?s=80&v=4"},"commit":{"message":"Streams cleanup","shortMessageHtmlLink":"Streams cleanup"}}],"hasNextPage":true,"hasPreviousPage":false,"activityType":"all","actor":null,"timePeriod":"all","sort":"DESC","perPage":30,"cursor":"djE6ks8AAAAEa8692gA","startCursor":null,"endCursor":null}},"title":"Activity · lefou/mill"}