New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Shared dependency-free reverse router across multiple projects #1390

Closed
andrewconner opened this Issue Jul 26, 2013 · 43 comments

Comments

Projects
None yet
@andrewconner
Contributor

andrewconner commented Jul 26, 2013

We use multi-projects with split routes files. We would like to use the Play reverse router in all of the projects. i.e., from ServiceA, call serviceB.routes.SomeController.someAction(id, name) (where ServiceA and ServiceB are not dependent on each other). Specifically, we only want to share the reverse routes, not the controllers or other classes.

Relevant group discussions:
https://groups.google.com/forum/#!topic/play-framework/h9Iab7SvOIY (this is the most thourough explaining the current issue, and the use case)
https://groups.google.com/forum/#!topic/play-framework/VwwM3K9mGDo
and eventually, a "solution"
https://groups.google.com/forum/#!topic/play-framework/RXzAXNLd_Jc

The solution, however, just builds it's own reverse router, instead of using Play's. That works, but means developers have to maintain their own routes parser, and doesn't work immediately with custom types. It would be really nice to be able to export the reverse routes, by themselves, into a common project.

This is most definitely possible (since it's simply a mapping from a method call with parameters to URL), but is not compatible with the reverse routing classes Play generates (which are dependent on the controllers).

Along with dependency-free reverse route classes, the ability to easily export them into a project would make multi-projects a lot more accessible for more developers.

@gmethvin

This comment has been minimized.

Member

gmethvin commented Aug 7, 2013

+1

It'd be awesome to be able to export an API and use it in another Play project. @godenji's https://github.com/godenji/play-reverse-router does basically what I want, but it'd be nice for this to be supported natively by Play.

@diwa-zz

This comment has been minimized.

diwa-zz commented Aug 7, 2013

+100 for this. I am trying to sell play to a large dev team (coming from .NET) evaluating node.js and modularity was an important requirement. The sub-project routing issue was a sticking point in the type-safety story and we would love to have a native solution from the framework. In addition, sub-projects do magic to the compilation times.

Looking at the mailing list, it seems only a matter of time before most people using sub-projects encounter this limitation.

@seantbrady

This comment has been minimized.

Contributor

seantbrady commented Aug 7, 2013

+1 We've invented a hacky work-around that requires keeping two different routes files in sync (one local to the controller and one local to the caller). We haven't explored the "solution" described at the start of this thread.

@godenji

This comment has been minimized.

Contributor

godenji commented Aug 7, 2013

Yes, would be great if this was a built-in feature of the framework. Play team stance seems to be, if A does not depend on B, then A should not know how to (or even need to) generate a reverse route to B.

Generally this is true, but when you start breaking your application up into bite size sub project chunks (to avoid routes compilation crucifixion), then you need a global reverse router to fallback on, there's no other choice, nothing type safe at any rate.

@2is10

This comment has been minimized.

2is10 commented Aug 23, 2013

+1. A service communication dependency should not imply or require a compile-time module dependency.

@bln

This comment has been minimized.

bln commented Aug 26, 2013

+1 +1 +1 for this.
Please offer either a blessed solution or document a best practice/sample project for this case.
For new adopters like us, such a guidance is essential.

@mikesname

This comment has been minimized.

Contributor

mikesname commented Sep 13, 2013

+1 Add me to the list of folks who desperately need this.

@adis-me

This comment has been minimized.

Contributor

adis-me commented Sep 13, 2013

+1

@benmccann

This comment has been minimized.

Contributor

benmccann commented Sep 14, 2013

+1 I think that this would also greatly improve compilation times by breaking the dependency between the reverse routers and controllers

@godenji

This comment has been minimized.

Contributor

godenji commented Sep 14, 2013

@benmccann sub projects already drastically improve compilation times, for incremental builds at any rate; pretty much indispensable for sane Play development, IMO.

The reverse router I created simply collects specified sub project routes and bundles them together for use application wide. This is currently not possible in Play, but hopefully will be in the not too distant future (fairly crucial bit of functionality if one has any interest in type safety) if enough Play-ers speak up on the issue.

@benmccann

This comment has been minimized.

Contributor

benmccann commented Sep 29, 2013

I love the dependency free aspect of this proposal. It seems like the current solution exports routes across all subprojects into one common shared project? I think a better solution might be to have each compiled project output a myproject.jar (same as today's compiled jar) and a myproject-routes.jar that it depends on. Then you could depend on the routes jars of the projects you want to link to. The benefit of this is that you could share routes between projects without having to own them all. Imagine you made an open source blog platform in Play. With this solution, anyone who wanted to integrate with it and create links to various pages in the blog platform could import the routes.jar. Otherwise, everyone would have to download the blog platform source code and have the routes exported into their own global routes project, which seems much harder than just adding a dependency on the appropriate routes jar.

@andrewconner

This comment has been minimized.

Contributor

andrewconner commented Sep 29, 2013

@benmccann 👍 Big fan of putting reverse routes into their own jar dependency based on project.

edit: Reverse routes. Regular (forward) routes do have a dependency on the controllers. I would like an easy way to produce a URL call given a reverse router-like syntax, ideally generated from the routes files themselves.

@benmccann

This comment has been minimized.

Contributor

benmccann commented Sep 29, 2013

I've filed an issue which I think would make this much easier. Would love to hear what you guys think: #1768

@huntc

This comment has been minimized.

Collaborator

huntc commented Oct 3, 2013

This issue "feels" like it is mainly driven by slow compilation times - is that a fair statement?

@benmccann

This comment has been minimized.

Contributor

benmccann commented Oct 3, 2013

@huntc this issue is talking more about the ability to share routes between projects. i think that the best way to do that also has the very nice side effect of likely helping compilation times quite a bit (see #1768), but this issue at it's core is really talking about the ability to export your routes so that they can be used by other projects. as our usage of play grows, it becomes more and more frequent that we have different web apps we're developing and occasionally we'd like to place a link from one of them to another. today there's no compile safe way to do that and we end up just typing out the links in the html, which could potentially break if we ever change the URL of a page

@huntc

This comment has been minimized.

Collaborator

huntc commented Oct 3, 2013

Thanks @benmccann I've assigned this to the 2.3 milestone

@diwa-zz

This comment has been minimized.

diwa-zz commented Oct 3, 2013

Our main driver for this is Modularity. We would like to have sub-projects to reflect the various modules of the application and also will have different teams working on each of them. To call out routes across projects, we would need this ability (also with the javascript router)

Our UI is in angular and with sbt 0.13, scala 2.10.2, play 2.2, fine-granular imports & one item per file, we are fairly ok with the compilation times.

@andrewconner

This comment has been minimized.

Contributor

andrewconner commented Oct 3, 2013

Our main motivation is actually because we deploy services separately. During development, we're able to run all or a subset of projects (most typically, one or all) — this is a big win for Play, because it means we're able to run a decently sized system on a regular development machine in one JVM.

In production, however, we deploy projects separately to several different clusters. These clusters have both public and internal only endpoints. i.e., the server that responds to / isn't necessarily the server the responds to /user, and services talk to each other over HTTP using private internal routes.

As-is, the three options for one service knowing the URL of a remote controller call are:

  • Hardcode URL paths, defeating the great typed routing system and breaks easily
  • Hand build a custom reverse router, keeping typing, but losing automation (which has no guarantees the routes actually exist)
  • Create a custom routes parser which generates dependency-free routes, which is hacky and can break if the routes format changes.

The third is probably the best right now, and we do the second, but none are ideal. The perfect solution is a separate projectName-reverseRoutes.jar that exports to either a specific filesystem location or subproject.

Thanks @huntc for taking a look at this.

@godenji

This comment has been minimized.

Contributor

godenji commented Oct 4, 2013

@huntc let's not forget to give the javascript reverse router some love as well -- generating the js routes on every single request in production is not terribly efficient, and even then it's restricted to the same sub project isolation issue as the scala reverse router.

I generate a global routes.js file at compile time since my app is ajax-heavy, need the routes cached and need sub projects to be able to "talk" to each other.

Hopefully Play 2.3 can bring the goods in the routing department as current built-in solutions are less than ideal for sub project builds.

Of course, overall Play is an incredible framework so interim workarounds are fine here until something better comes along.

@seantbrady

This comment has been minimized.

Contributor

seantbrady commented Oct 5, 2013

One use case for us is that we reference web routes in email templates. Email templates are used in the services-tier which does not depend on the web-tier where the routes are compiled. For now, we maintain an email routes file and have Global run a check onStart that ensures all route strings in the email routes exist in one of the web routes files.

@ejstrobel

This comment has been minimized.

ejstrobel commented Oct 8, 2013

+1 I need this too.

@magro

This comment has been minimized.

Contributor

magro commented Oct 16, 2013

+1

@magro

This comment has been minimized.

Contributor

magro commented Mar 10, 2014

@huntc Is this still on the list for 2.3?

@huntc

This comment has been minimized.

Collaborator

huntc commented Mar 10, 2014

It is still on the list presently

@jroper

This comment has been minimized.

Member

jroper commented Mar 11, 2014

I don't think we (the Play core devs) will have the time to implement this for 2.3. However, that doesn't mean to say that if someone in the community implements it, it won't happen. This is what I think will need to be done:

  • Add a configuration option to SBT to say whether it should generate the router or not. Currently, you can turn on/off generation of the reverse router, but we need something to turn on/off generation of the router.
  • Make the list of source directories for routes files configurable. Currently its hard coded to pass in the confDirectory setting, but a setting, maybe routesSourceDirectories, should be created that defaults to confDirectory, should be made. Additionally, a task, routesFiles, should be created that actually locates the files, and this would by default depend on routesSourceDirectories, so the routes compiler isn't responsible for locating routes files, it just gets passed a list of files to compile.
  • Make a convenience method for configuring the most common configuration. This can probably be an implicit conversion on sbt.Project, adding a generateReverseRoutesFor method that accepts a sequence of modules to generate reverse routes for, and configures the settings as necessary.
  • Document how to use the router in this way.
  • Implement an sbt scripted test that tests projects in this configuration.
  • (Optional) refactor the routes compiler to extract the parsing logic into its own class, the router generation into its own class, and each reverse routers generation into their own classes. The routes compiler is possibly the ugliest code in Play, this would be a good opportunity to fix that up.

Having done that, this is (very roughly) what a project would look like if it used it:

lazy val common = project.in(file("common"))
  .settings(playScalaSettings:_*)
  .generateReverseRoutesFor(Seq(moduleA, moduleB, main))

lazy val moduleA = project.in(file("modulea"))
  .settings(playScalaSettings:_*)
  .settings(
    generateReverseRouter := false
  )
  .dependsOn(common)

lazy val moduleB = project.in(file("moduleb"))
  .settings(playScalaSettings:_*)
  .settings(
    generateReverseRouter := false
  )
  .dependsOn(common)

lazy val main = project.in(file("."))
  .settings(playScalaSettings:_*)
  .settings(
    generateReverseRouter := false
  )
  .dependsOn(moduleA, moduleB)

So given the above, each module declares its own routes files, and the main project also declares its own routes files, but the common project compiles all routes files into reverse routers, but not into the main router, while each module and the main project compiles their own routes files into a router but not a reverse router.

As for the implementation details, the generateReverseRoutesFor method would look something like this:

implicit class ReverseRoutesHelper(project: sbt.project) {
  def generateReverseRoutesFor(modules: => Seq[ProjectRef]) = {
    project.settings(
      generateRouter := false,
      generateReverseRouter := true,
      generateRefReverseRouter := false,
      routesFiles <++= modules.map(module => routesFiles in module).join.map(_.flatten)
    )
  }
}

Might take some SBT foo to get it right. And the by name parameter is important, otherwise you'll get an infinite recursion (that's why it has to be Seq[ProjectRef], and not ProjectRef*).

@jroper jroper added the community label Mar 11, 2014

@jroper jroper removed this from the 2.3.0 milestone Mar 27, 2014

@huntc

This comment has been minimized.

Collaborator

huntc commented Apr 20, 2014

Having now had a good read of all of this here are my thoughts:

  • http is the interface with the routes file used to declare it - this should be all of the specification an external entity such as another system requires - any re-factoring within the routes file cannot be automatically applied externally
  • reverse routes are a convenience mechanism for safe re-factoring within the project and its modules
  • a per-module declaration of routes makes sense

I'm away from my laptop so I can't look right now, but I'm a bit mystified as to why a reverse router should have a dependency on a controller.

@benmccann

This comment has been minimized.

Contributor

benmccann commented Apr 20, 2014

@huntc i completely agree that the reverse router should not have a dependency on a controller. i'd really love to fix that historical quirk and eventually see play remove that dependency. sent an email to the developers list suggesting fixing that https://groups.google.com/forum/#!topic/play-framework-dev/-a3tqyNfaKs

@godenji

This comment has been minimized.

Contributor

godenji commented Apr 20, 2014

@jroper has defined a game plan for attacking the problem but it doesn't seem to address the reverse router/controller dependency problem.

I'd like to see the reverse router decoupled from Java legacy baggage as @benmccann explains in his linked play dev thread above.

Otherwise, while having a framework supported global reverse router would be great, dealing with slow-ish builds is a noop, will stick with a custom reverse router.

Finally, the javascript reverse router is not suitable for production, IMO, since the client has to fetch the server generated javascript on every request (vs. fetching a static cachable routes.js file from the server).

Routing is arguably the primary weak spot in an otherwise fantastic framework, hope a solution starts to evolve much sooner than later.

@andrewconner

This comment has been minimized.

Contributor

andrewconner commented Apr 20, 2014

@huntc "this should be all of the specification an external entity such as another system requires" I'd agree in the general case, but when working with multi-projects, it's a unique scenario. Having type safety across projects is incredibly valuable, which HTTP does not give. Since both ends are inside the JVM eco-system (and share a common parent project), it makes sense and would be very helpful to keep the patterns of the reverse router.

In a larger SOA system, being able to have type safe and IDE support when forming remote requests is an asset. We currently hand write our own common reverse router — it's a mess (just from the size), but gives us the above benefits. Others (I believe @godenji) have a custom reverse router generator (replacing Play's), which also works but is unfortunate.

@jroper

This comment has been minimized.

Member

jroper commented Apr 24, 2014

The dependency on the controllers is already solved, the dependency comes from the ref router, and the ref router generation can be turned off - @nraychaudhuri implemented this.

Also, it's not Java legacy baggage - Java is and will continue to be a first class citizen in Play, there's nothing legacy about the need to provide Java tests with a mechanism to reference actions.

@benmccann

This comment has been minimized.

Contributor

benmccann commented Apr 24, 2014

Totally agree Java should be well supported. Not 100% sure that we need the ref router to support Java well necessarily. I can see arguments on either side. I like the idea of just using native Java syntax in place of the ref router:

Result result = callAction(() -> { controllers.Application.index("Kiki"); });

amertum added a commit to amertum/securesocial that referenced this issue Sep 20, 2014

Disable generateRefReverseRouter to make routes exportable
As stated in SBTSubProjects documentation page :
> To export compiled routes to other projects disable reverse ref routing generation using generateRefReverseRouter := false sbt settings.
See also playframework/playframework#1390
@cjskahn

This comment has been minimized.

cjskahn commented Oct 15, 2014

I this actually being worked on or on a list to be done?

+1 from me as well. I have (yet another) project that I want to split into several modules, but those modules need to access each others' routes and I can't find any way to do it without having circular dependencies or a massive hack. It would be genius if the "common" module all my modules depend on could collect and provide reverse-routes globally.

@ghost

This comment has been minimized.

ghost commented Oct 16, 2014

+1

It's a definite annoyance in taking advantage of a multi-module setup. Sibling projects should have an easy way to route to each other without hardcoding URLs or implementing our own reverse routers.

@jroper

This comment has been minimized.

Member

jroper commented Oct 21, 2014

We've made major changes to the routes compiler, and this requirement has been important in our priorities, so while we haven't directly worked on this, the changes we've done have now made implementing this possible. What's left to do is to either document how to configure it as is, or provide helpers to help configure it (and document the helpers).

@andrewconner

This comment has been minimized.

Contributor

andrewconner commented Oct 21, 2014

Thanks James.

@dmitriy-yefremov

This comment has been minimized.

dmitriy-yefremov commented Nov 3, 2014

I've found another workaround for this issue. It's not perfect, but may work while we are waiting for a proper solution.
Description: http://yefremov.net/blog/play-shared-routes/
Sample app: https://github.com/dmitriy-yefremov/play-shared-routes
Please let me know what do you think.

@chych

This comment has been minimized.

chych commented Jan 13, 2015

+1
jroper says : I don't think we (the Play core devs) will have the time to implement this for 2.3
please create an article to explain how to to do a custom implementation in scala and Java.

I'm a newbie on JAVA i don't know very well yet the framework, so help us.

@benmccann

This comment has been minimized.

Contributor

benmccann commented Mar 2, 2015

There have been lots of good changes to the routing lately that should help with this. I've just submitted another PR that will help quite a bit with making progress towards implementing this. #4006

@jroper jroper added this to the 2.4.0 milestone Mar 2, 2015

@jroper jroper self-assigned this Mar 24, 2015

jroper added a commit to jroper/playframework that referenced this issue Mar 25, 2015

Reverse router aggregation
Fixes playframework#1390.

* Implemented reverse router aggregation
* Replaced routesFiles task with sources in routes task
* Added scripted test for testing router aggregation
* Added support for compiling/including .sbt code snippets in the
  documentation
@jroper

This comment has been minimized.

Member

jroper commented Mar 25, 2015

See attached pull request #4131 that implements this.

@jroper

This comment has been minimized.

Member

jroper commented Mar 25, 2015

For those interested in what the final result looks like, it's really simple:

lazy val common: Project = (project in file("common"))
  .enablePlugins(PlayScala)
  .settings(
    aggregateReverseRoutes := Seq(api, web)
  )

lazy val api = (project in file("api"))
  .enablePlugins(PlayScala)
  .dependsOn(common)

lazy val web = (project in file("web"))
  .enablePlugins(PlayScala)
  .dependsOn(common)

That's all you need in your build.sbt to get this to work.

@andrewconner

This comment has been minimized.

Contributor

andrewconner commented Mar 25, 2015

Wonderful, thanks a ton James.

jroper added a commit to jroper/playframework that referenced this issue Mar 25, 2015

Reverse router aggregation
Fixes playframework#1390.

* Implemented reverse router aggregation
* Replaced routesFiles task with sources in routes task
* Added scripted test for testing router aggregation
* Added support for compiling/including .sbt code snippets in the
  documentation
@godenji

This comment has been minimized.

Contributor

godenji commented Mar 25, 2015

Awesome, @jroper thanks!

Question: can this functionality be extended into javascript reverse routes? One thing I dislike about the current 2.3.x js router is the fact that it generates the routes at runtime (i.e. on every single request).

I currently print the routes to static file(s) at compile time, but it would be nice to leave behind my hacks and have a framework supported version, all the more so given the rise of Scala.js ;-)

jroper added a commit to jroper/playframework that referenced this issue Mar 25, 2015

Reverse router aggregation
Fixes playframework#1390.

* Implemented reverse router aggregation
* Replaced routesFiles task with sources in routes task
* Added scripted test for testing router aggregation
* Added support for compiling/including .sbt code snippets in the
  documentation
@jroper

This comment has been minimized.

Member

jroper commented Mar 25, 2015

@godenji this functionality isn't really related to javascript reverse routes, generating static javascript reverse routers would be completely unrelated.

There are two things that you may want for javascript reverse routes:

  1. A new routes generator, that delegates to the existing routes generators, but also generates a javascript file containing all the Javascript reverse routes.
  2. Something that aggregated all routes files in a project into one compilation unit, so that you can get the paths right, and output only one javascript router, instead of outputting one per routes file. Currently all routes files are compiled separately, but a refactor of the routes plugin could mean they could be compiled together.

ClaraAllende pushed a commit to ClaraAllende/playframework that referenced this issue Aug 28, 2015

Reverse router aggregation
Fixes playframework#1390.

* Implemented reverse router aggregation
* Replaced routesFiles task with sources in routes task
* Added scripted test for testing router aggregation
* Added support for compiling/including .sbt code snippets in the
  documentation
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment