Replace macros with codegen - POC/WIP #632

L-Lavigne · 2019-08-03T00:57:46Z

This PR introduces a work-in-progress compiler plugin (not an sbt plugin) to generate "managed" sources from the @service definitions (here @serviceCodegen, to preserve the originals) without using macros but by reusing most of the existing macro implementation. Note that this requires a 2-subproject setup so that the app's compilation can see the sources generated by the protocol's compilation. This is shown in example/codegen.

Please refer to #631 for the rationale behind this and the pros & cons of this approach.

How it works: Compiling example/codegen/app should trigger compilation of example/codegen/protocol which will generate the object in the protocol module's target/scala-2.12/src_managed for the app module to refer to. The generator is in the common module's ServiceGenerator.scala and basically wraps a modified copy of the internal module's serviceImpl inside a compiler plugin instead of a macro.

Note the following current limitations and remaining TODOs:

The compiler plugin resides in the common project (where the protocol keywords are) instead of depending on it, because of the limitations described in Better support for compiler plugins that have dependencies sbt/sbt#2255. We should move it to a different module and implement a workaround based on sbt-assembly.
The new annotation is temporarily called @serviceCodegen to preserve the existing @service macro's functionality; the goal if we adopt this approach is to replace the @service implementation.
The generated object is simply called ${service}Object and contains everything the macro currently generates in the service companion; we'll want to split this into client and server classes for better readability.
The generated object is in a hardcoded example/codegen package to match the example modules, but we'll change it to the actual service's package.
The generated object doesn't have the imports of the service trait; we'll need to add them to the generated source.
The example module doesn't do anything with the generated object except confirm that it can see it; we'll want to add interesting service methods and call them.

Comments, questions and ideas welcome!

@service

Created a compiler plugin (not an sbt plugin) to generate managed sources from the @service definitions (here @serviceCodegen to preserve the originals) without using macros but by reusing most of the macro's implementations. Note that this requires a 2-subproject setup so that the app's compilation can see the sources generated by the protocol's compilation. This is shown in example/codegen.

juanpedromoreno

Thanks @L-Lavigne , this is definitely a great step forward in the project! I left some minor code formatting suggestions and a couple of questions, one of them related to the two-steps code generation stages that we will here.

Probably, we'll want to provide the compiler plugin automatically when you are including the sbt-mu-idl-gen sbt plugin in the project. Would it make sense?

juanpedromoreno · 2019-08-05T08:00:46Z

build.sbt

+  .settings(
+    scalacOptions ++= Seq(
+      // recompile whenever plugin changes (in IDEA this might require using sbt shell for imports & builds)
+      "-Jdummy" + (`common`/Compile/packageBin).value.lastModified,


Suggested change

"-Jdummy" + (`common`/Compile/packageBin).value.lastModified,

"-Jdummy" + (`common` / Compile / packageBin).value.lastModified,

juanpedromoreno · 2019-08-05T08:01:02Z

build.sbt

+    scalacOptions ++= Seq(
+      // recompile whenever plugin changes (in IDEA this might require using sbt shell for imports & builds)
+      "-Jdummy" + (`common`/Compile/packageBin).value.lastModified,
+      "-P:service-generator:outputDir=" + (Compile/sourceManaged).value


Suggested change

"-P:service-generator:outputDir=" + (Compile/sourceManaged).value

"-P:service-generator:outputDir=" + (Compile / sourceManaged).value

juanpedromoreno · 2019-08-05T08:02:27Z

build.sbt

+  .settings(noPublishSettings)
+  .settings(moduleName := "mu-rpc-example-codegen-app")
+  .settings(
+    Compile/sourceGenerators += Def.task(((`example-codegen-protocol`/Compile/sourceManaged).value ** "*.scala").get)


Suggested change

Compile/sourceGenerators += Def.task(((`example-codegen-protocol`/Compile/sourceManaged).value ** "*.scala").get)

Compile / sourceGenerators += Def.task(

((`example-codegen-protocol` / Compile / sourceManaged).value ** "*.scala").get)

juanpedromoreno · 2019-08-05T08:02:52Z

modules/common/src/main/scala/higherkindness/mu/rpc/codegen/ServiceGenerator.scala

+  private lazy val serviceFilter = new FilterTreeTraverser({
+    case md: MemberDef =>
+      findAnnotation(md.mods, "serviceCodeGen").isDefined // TODO: rename to "service"
+    case x => false


Suggested change

case x => false

case _ => false

juanpedromoreno · 2019-08-05T08:04:21Z

modules/common/src/main/scala/higherkindness/mu/rpc/codegen/ServiceGenerator.scala

+    parsedOptions.getOrElse(
+      optionName,
+      throw new IllegalArgumentException(
+        s"Plugin $name missing option $optionName.\n${optionsHelp.get}"))


Could we avoid this Option.get?

juanpedromoreno · 2019-08-05T08:08:17Z

modules/common/src/main/scala/higherkindness/mu/rpc/codegen/ServiceGenerator.scala

+      }
+    }
+
+    case class HttpOperation(operation: Operation) {


we'll want to split this into client and server classes for better readability.

Would it make sense to split the Http part too?

Absolutely! Currently we have a monolithic straight-up copy of the current macro output but there's lots of improvements we can make so it becomes more user-friendly.

Also moving off macros might help resolve the issue that led us to put the HTTP stuff in there in the first place, where we couldn't get two processors for the same macro applied from different classes.

juanpedromoreno · 2019-08-05T08:13:21Z

modules/examples/codegen/protocol/src/main/scala/example/codegen/ExampleService.scala

+
+@message case class Hello(words: String)
+
+@serviceCodeGen(Protobuf) trait ExampleService[F[_]] {


I wonder how the code generation would behave when we would be generating code in two different stages:

From IDL to Scala code (scala files with @serviceCodeGen) (sbt-mu-idl-gen)

@serviceCodeGen to the expanded version of the Scala Code (scala compiler plugin)

I haven't tried it yet but I suppose the SBT plugin generates the @service-annotated sources from IDL, and then the compiler plugin generates the service infrastructure from those sources afterwards.

We could explore combining these two steps into one, however I don't think that would work since the developer needs the intermediate @service-annotated traits in order to implement their methods.

L-Lavigne · 2019-08-06T02:37:08Z

Probably, we'll want to provide the compiler plugin automatically when you are including the sbt-mu-idl-gen sbt plugin in the project. Would it make sense?

The compiler plugin processes the @service annotation and as such would be part of the core functionality of Mu just as the macro is today, so its module should always be imported by developers whether or not they're using IDL generation.

L-Lavigne · 2019-08-06T02:43:34Z

Note that the CI build is failing because the generated sources are either not created by the example protocol or not seen by the example app. It Works on my Machine (™) so I'll investigate what happened there.

rafaparadela · 2019-08-12T18:58:45Z

@L-Lavigne CI is failing because you need to project example-codegen-app and compile.

L-Lavigne · 2019-08-12T19:57:45Z

@L-Lavigne CI is failing because you need to project example-codegen-app and compile.

That should be triggered by the root project's compile though, and it should trigger compilation of example-codegen-protocol which uses the plugin to generate the service object. I do see this project compiling here: https://travis-ci.org/higherkindness/mu/jobs/567183204#L968, and it seems like it's either not using the plugin, or the example-codegen-app compilation isn't adding the protocol's generated sources.

Still not sure why this is the case. Locally it works although I do recall needing a full project re-import in IDEA at some point. I don't see previous build state being much of an issue for CI though.

ivanov-ia · 2019-12-07T09:43:15Z

This doesn't compile for me when using sbt console and running example-codegen-app/compile for the first time or after clean. For the second time it compiles without errors.

The reason is that the task at https://github.com/higherkindness/mu/pull/632/files#diff-fdc3abdfd754eeb24090dbd90aeec2ceR496 is evaluated before the plugin generates ExampleServiceObject.scala. You can see this, if you use:

Compile/sourceGenerators += Def.task {
      val files = ((`example-codegen-protocol` / Compile / sourceManaged).value ** "*.scala").get
      println("FILES: " + files.mkString(", "))
      files
}

The list of files is empty and is printed before the compilation of example-codegen-protocol.

The example works fine this way:

Compile/sourceGenerators += Def.sequential(
    `example-codegen-protocol` / Compile / compile,
    Def.task(((`example-codegen-protocol` / Compile / sourceManaged).value ** "*.scala").get)
)

But it doesn't seem to be a good solution.

rfan-debug · 2021-08-27T02:40:15Z

Just a follow-up, is this PR still in progress? It would be nice if the macro can be replaced by the codegen because it is very hard to intelliJ to resolve those macros.

cb372 · 2022-05-04T08:38:59Z

Closing. We did end up going with srcgen, but not quite like this.

L-Lavigne requested a review from juanpedromoreno August 3, 2019 00:57

L-Lavigne mentioned this pull request Aug 3, 2019

Replace macros with code generation #631

Open

L-Lavigne requested review from fedefernandez and rafaparadela August 3, 2019 00:58

juanpedromoreno reviewed Aug 5, 2019

View reviewed changes

cb372 mentioned this pull request Apr 27, 2020

Start thinking about Scala 3 #894

Open

cb372 closed this May 4, 2022

fedefernandez deleted the feature/mu-631-replace-macros-with-codegen branch November 15, 2022 08:39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replace macros with codegen - POC/WIP #632

Replace macros with codegen - POC/WIP #632

L-Lavigne commented Aug 3, 2019

juanpedromoreno left a comment

juanpedromoreno Aug 5, 2019

juanpedromoreno Aug 5, 2019

juanpedromoreno Aug 5, 2019

juanpedromoreno Aug 5, 2019

juanpedromoreno Aug 5, 2019

juanpedromoreno Aug 5, 2019

L-Lavigne Aug 6, 2019

juanpedromoreno Aug 5, 2019

L-Lavigne Aug 6, 2019

L-Lavigne commented Aug 6, 2019

L-Lavigne commented Aug 6, 2019 •

edited

Loading

rafaparadela commented Aug 12, 2019

L-Lavigne commented Aug 12, 2019

ivanov-ia commented Dec 7, 2019

rfan-debug commented Aug 27, 2021

cb372 commented May 4, 2022

	"-Jdummy" + (`common`/Compile/packageBin).value.lastModified,
	"-Jdummy" + (`common` / Compile / packageBin).value.lastModified,

	"-P:service-generator:outputDir=" + (Compile/sourceManaged).value
	"-P:service-generator:outputDir=" + (Compile / sourceManaged).value

	Compile/sourceGenerators += Def.task(((`example-codegen-protocol`/Compile/sourceManaged).value ** "*.scala").get)
	Compile / sourceGenerators += Def.task(
	((`example-codegen-protocol` / Compile / sourceManaged).value ** "*.scala").get)


		@message case class Hello(words: String)

		@serviceCodeGen(Protobuf) trait ExampleService[F[_]] {

Replace macros with codegen - POC/WIP #632

Replace macros with codegen - POC/WIP #632

Conversation

L-Lavigne commented Aug 3, 2019

juanpedromoreno left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

L-Lavigne commented Aug 6, 2019

L-Lavigne commented Aug 6, 2019 • edited Loading

rafaparadela commented Aug 12, 2019

L-Lavigne commented Aug 12, 2019

ivanov-ia commented Dec 7, 2019

rfan-debug commented Aug 27, 2021

cb372 commented May 4, 2022

L-Lavigne commented Aug 6, 2019 •

edited

Loading