use fs2 to rewrite SearchService #1351

Closed
fommil opened this Issue Mar 13, 2016 · 25 comments

Comments

Projects
None yet
7 participants
@fommil
Contributor

fommil commented Mar 13, 2016

UPDATE: (the original ticket was to replace all of akka with something else, but reducing the scope to use a non-akka solution for the indexer seems a much more achieveable goal)

Depending on akka brings in a huge dependency tree and restricts our ability to work on bleeding edge scala releases.

Also, I've become increasingly unhappy with akka as a framework and we're in Typelevel so we should be using more a functional paradigm, e.g. when cats / fs2 becomes viable. @non may have thoughts on this.

This ticket is specifically about replacing our akka actors with a different framework / paradigm. Minimalism and backpressure (especially for the horrid SearchService) are important, but we also have to bare in mind that we are fundamentally wrapping a black box API around the compiler.

I'm pretty sure core.async in clojure could do what we want.

@fommil fommil added this to the TNG 2.0 milestone Mar 13, 2016

@fommil

This comment has been minimized.

Show comment
Hide comment
@dwijnand

This comment has been minimized.

Show comment
Hide comment
@dwijnand

dwijnand Jun 11, 2016

Member

Perhaps Monix has the primitives that ensime-server requires?

@alexandru do you think that ensime-server could switch to use Monix instead of Akka?

Member

dwijnand commented Jun 11, 2016

Perhaps Monix has the primitives that ensime-server requires?

@alexandru do you think that ensime-server could switch to use Monix instead of Akka?

@alexandru

This comment has been minimized.

Show comment
Hide comment
@alexandru

alexandru Jun 15, 2016

Hello, just noticed this ticket :-)

A Monix Observable might be the right abstraction, having these characteristics:

Back-pressure and everything associated (e.g. overflow policies in case of buffering) is for free, versus Akka actors for which you have to implement your own back-pressure protocol (I can also give hints for Akka btw).

Monix works best if it models a unidirectional flow of messages. For a dialog (e.g. ping-pong), you can think of a client-server communication through WebSocket if that helps, where both can listen to receiving messages and send replies. In such a case, actors are easier to use.

I mean, you can model bidirectional communications with Monix, but in order to replace a bidirectional actor you'll work with an input channel (where you push messages) and an output channel (the Observable, on which you can subscribe for listening) and it gets awkward, because having a dialog means keeping shared state between these two channels. You've got basically the same problem with say, Iteratees in Play, where for websocket you receive an input/output pair and you can compare it with usage of actors, also in Play, which is actually much simpler.

That said, if you don't have dialogs, or lets say difficult dialogs, then Monix's Observable certainly leads to a cleaner model, as it's more decoupled and it gives you stuff for free. You guys think of Clojure's core.async and Monix has many of the same advantages.

Currently I do not have time to look at Ensime's implementation. I'm very busy with releasing a stable 2.0 release for Monix. But afterwards I could take a look if you want. Not sure what the time table is like. I hope you're not in a hurry :-P

alexandru commented Jun 15, 2016

Hello, just noticed this ticket :-)

A Monix Observable might be the right abstraction, having these characteristics:

Back-pressure and everything associated (e.g. overflow policies in case of buffering) is for free, versus Akka actors for which you have to implement your own back-pressure protocol (I can also give hints for Akka btw).

Monix works best if it models a unidirectional flow of messages. For a dialog (e.g. ping-pong), you can think of a client-server communication through WebSocket if that helps, where both can listen to receiving messages and send replies. In such a case, actors are easier to use.

I mean, you can model bidirectional communications with Monix, but in order to replace a bidirectional actor you'll work with an input channel (where you push messages) and an output channel (the Observable, on which you can subscribe for listening) and it gets awkward, because having a dialog means keeping shared state between these two channels. You've got basically the same problem with say, Iteratees in Play, where for websocket you receive an input/output pair and you can compare it with usage of actors, also in Play, which is actually much simpler.

That said, if you don't have dialogs, or lets say difficult dialogs, then Monix's Observable certainly leads to a cleaner model, as it's more decoupled and it gives you stuff for free. You guys think of Clojure's core.async and Monix has many of the same advantages.

Currently I do not have time to look at Ensime's implementation. I'm very busy with releasing a stable 2.0 release for Monix. But afterwards I could take a look if you want. Not sure what the time table is like. I hope you're not in a hurry :-P

@fommil

This comment has been minimized.

Show comment
Hide comment
@fommil

fommil Jun 15, 2016

Contributor

that'd be cool. We're just starting looking at the 2.0 milestone and I'm taking some time off, so this isn't urgent. I'd be happy if we at least knew what the best alternative to akka is.

Both myself and Rory have been using akka for years, it's the battle wounds that's put us off... not lack of understanding.

Contributor

fommil commented Jun 15, 2016

that'd be cool. We're just starting looking at the 2.0 milestone and I'm taking some time off, so this isn't urgent. I'd be happy if we at least knew what the best alternative to akka is.

Both myself and Rory have been using akka for years, it's the battle wounds that's put us off... not lack of understanding.

@fommil

This comment has been minimized.

Show comment
Hide comment
@fommil

fommil Jun 15, 2016

Contributor

the only thing that might be a problem is the number of dependencies you're using in monix. We'd like to remain as scala-free as possible so that we can quickly support new versions of scala. We're in a little bit of a special position that way: consider that cats / whatever might be waiting for ensime to be available before they can work on upgrading their library and also we tend to be a lot more conservative about JDK requirements (akka has jumped JDK7 and gone straight to JDK8 for example, whereas we really care about JDK7).

Contributor

fommil commented Jun 15, 2016

the only thing that might be a problem is the number of dependencies you're using in monix. We'd like to remain as scala-free as possible so that we can quickly support new versions of scala. We're in a little bit of a special position that way: consider that cats / whatever might be waiting for ensime to be available before they can work on upgrading their library and also we tend to be a lot more conservative about JDK requirements (akka has jumped JDK7 and gone straight to JDK8 for example, whereas we really care about JDK7).

@fommil

This comment has been minimized.

Show comment
Hide comment
@fommil

fommil Jun 15, 2016

Contributor

I see your wiki is marked as old, let me know when it's ready to read. A single HTML or PDF file works best as I like to read things offline (if it's only one set of links deep, I can write a script to generate a single PDF)

Contributor

fommil commented Jun 15, 2016

I see your wiki is marked as old, let me know when it's ready to read. A single HTML or PDF file works best as I like to read things offline (if it's only one set of links deep, I can write a script to generate a single PDF)

@alexandru

This comment has been minimized.

Show comment
Hide comment
@alexandru

alexandru Jun 15, 2016

the only thing that might be a problem is the number of dependencies you're using in monix. We'd like to remain as scala-free as possible so that we can quickly support new versions of scala

If you'll take a look at build.sbt, the only external dependency is cats-core (and cats-laws for testing) and these dependencies are at the top of the stack, not at the bottom, meaning that the monix-cats sub-project is completely optional. All the other dependencies are part of Monix, including the testing framework ;-)

I would build versions for Scala 2.12.0-M4 right now, if it weren't for the monix-cats sub-project being part of the build.sbt. At least for now, I don't know how to exclude a sub-project if a certain Scala version is selected. The other problem has been sbt-scoverage. I actually considered dropping it from the project at some point.

alexandru commented Jun 15, 2016

the only thing that might be a problem is the number of dependencies you're using in monix. We'd like to remain as scala-free as possible so that we can quickly support new versions of scala

If you'll take a look at build.sbt, the only external dependency is cats-core (and cats-laws for testing) and these dependencies are at the top of the stack, not at the bottom, meaning that the monix-cats sub-project is completely optional. All the other dependencies are part of Monix, including the testing framework ;-)

I would build versions for Scala 2.12.0-M4 right now, if it weren't for the monix-cats sub-project being part of the build.sbt. At least for now, I don't know how to exclude a sub-project if a certain Scala version is selected. The other problem has been sbt-scoverage. I actually considered dropping it from the project at some point.

@fommil

This comment has been minimized.

Show comment
Hide comment
@fommil

fommil Jun 15, 2016

Contributor

you have the same problem as us then 😄

We are stuck on 2.11 until we can get rid of akka (at least the reactive / http parts). They are the slowest to move.

Contributor

fommil commented Jun 15, 2016

you have the same problem as us then 😄

We are stuck on 2.11 until we can get rid of akka (at least the reactive / http parts). They are the slowest to move.

@timcharper

This comment has been minimized.

Show comment
Hide comment
@timcharper

timcharper Jun 15, 2016

Contributor

JDK7 is for chumps. Move to 8 already. :P

WRT to comparison of Monix and Akka Actors... a better comparison would be Monix and Akka Streams.

I'm a bit surprised to hear about Akka battle wounds. I don't mean to one-up anyone or diminish anybody's pain, and, I don't share that experience. However, I could see heavy actor usage would lead to it. I see Actors as a powerful technology that is often too powerful for the job. Sometimes they are absolutely the right abstraction. Often, they're not and they're overused. I use akka streams and futures unless I have a good reason to use an actor.

And, I should mention, that Monix and Akka Streams are better comparisons. Both projects implement the reactive streams protocol.

Monix looks like a lighter option. Functional stream processing libraries have a tendency to grow large in the number of operations they support (for good reasons).

Contributor

timcharper commented Jun 15, 2016

JDK7 is for chumps. Move to 8 already. :P

WRT to comparison of Monix and Akka Actors... a better comparison would be Monix and Akka Streams.

I'm a bit surprised to hear about Akka battle wounds. I don't mean to one-up anyone or diminish anybody's pain, and, I don't share that experience. However, I could see heavy actor usage would lead to it. I see Actors as a powerful technology that is often too powerful for the job. Sometimes they are absolutely the right abstraction. Often, they're not and they're overused. I use akka streams and futures unless I have a good reason to use an actor.

And, I should mention, that Monix and Akka Streams are better comparisons. Both projects implement the reactive streams protocol.

Monix looks like a lighter option. Functional stream processing libraries have a tendency to grow large in the number of operations they support (for good reasons).

@fommil

This comment has been minimized.

Show comment
Hide comment
@fommil

fommil Jun 18, 2016

Contributor
Contributor

fommil commented Jun 18, 2016

@fommil

This comment has been minimized.

Show comment
Hide comment
@fommil

fommil Jun 18, 2016

Contributor
#!/bin/bash

COUNT=0
for A in execution/scheduler \
             execution/cancelable \
             execution/future-utils \
             eval/task \
             eval/callback \
             eval/coeval \
             best-practices/blocking \
         ; do
    COUNT=$((COUNT + 1))
    wkhtmltopdf --page-width 20.25cm \
                --page-height 27cm \
                --margin-top 15mm \
                --margin-bottom 15mm \
                --user-style-sheet style.css \
                https://monix.io/docs/2x/$A.html monix-${COUNT}.pdf
done

gs -dBATCH -dNOPAUSE -q \
   -dPDFSETTINGS=/ebook \
   -dCompressFonts=true \
   -dDetectDuplicateImages=true \
   -sDEVICE=pdfwrite \
   -r150 \
   -g1600x1200 \
   -dPDFFitPage \
   -sOutputFile=monix.pdf \
   monix-*.pdf

with (mostly ignored)

@font-face {
    font-family: Hack;
    src: url(/home/fommil/.fonts/Hack-Regular.ttf) format('truetype');
}

html {
    font-size: 14pt;
    margin: 0;
    padding: 0;
    /*margin: 0 2em 0 2em;*/
}

pre { font-family: Hack; }
code { font-family: Hack; }

p { text-align: justify; }

/* annoyingly ignored */
.content { margin: 0; padding: 0; }
.container { margin: 0; padding: 0; }
section { margin: 0; padding: 0; }
article { margin: 0; padding: 0; }

.sidebar { visibility: hidden; }

h1 {
    font-size: 18pt;
}
Contributor

fommil commented Jun 18, 2016

#!/bin/bash

COUNT=0
for A in execution/scheduler \
             execution/cancelable \
             execution/future-utils \
             eval/task \
             eval/callback \
             eval/coeval \
             best-practices/blocking \
         ; do
    COUNT=$((COUNT + 1))
    wkhtmltopdf --page-width 20.25cm \
                --page-height 27cm \
                --margin-top 15mm \
                --margin-bottom 15mm \
                --user-style-sheet style.css \
                https://monix.io/docs/2x/$A.html monix-${COUNT}.pdf
done

gs -dBATCH -dNOPAUSE -q \
   -dPDFSETTINGS=/ebook \
   -dCompressFonts=true \
   -dDetectDuplicateImages=true \
   -sDEVICE=pdfwrite \
   -r150 \
   -g1600x1200 \
   -dPDFFitPage \
   -sOutputFile=monix.pdf \
   monix-*.pdf

with (mostly ignored)

@font-face {
    font-family: Hack;
    src: url(/home/fommil/.fonts/Hack-Regular.ttf) format('truetype');
}

html {
    font-size: 14pt;
    margin: 0;
    padding: 0;
    /*margin: 0 2em 0 2em;*/
}

pre { font-family: Hack; }
code { font-family: Hack; }

p { text-align: justify; }

/* annoyingly ignored */
.content { margin: 0; padding: 0; }
.container { margin: 0; padding: 0; }
section { margin: 0; padding: 0; }
article { margin: 0; padding: 0; }

.sidebar { visibility: hidden; }

h1 {
    font-size: 18pt;
}
@fommil

This comment has been minimized.

Show comment
Hide comment
@fommil

fommil Jun 18, 2016

Contributor

@alexandru any chance you could use gitbook for your docs? It makes it a lot easier to convert the docs into whatever format and means you don't need to do any devops like you currently do.

Contributor

fommil commented Jun 18, 2016

@alexandru any chance you could use gitbook for your docs? It makes it a lot easier to convert the docs into whatever format and means you don't need to do any devops like you currently do.

@mwielocha

This comment has been minimized.

Show comment
Hide comment
@mwielocha

mwielocha Jun 25, 2016

Contributor

What about swave? https://github.com/sirthias/swave Its basically depends only on shapeless...?

Contributor

mwielocha commented Jun 25, 2016

What about swave? https://github.com/sirthias/swave Its basically depends only on shapeless...?

@fommil

This comment has been minimized.

Show comment
Hide comment
@fommil

fommil Jun 25, 2016

Contributor

We already depend on loads of Mathias' stuff, if we go with this we'll have to offer to host his builds! 😁

Contributor

fommil commented Jun 25, 2016

We already depend on loads of Mathias' stuff, if we go with this we'll have to offer to host his builds! 😁

@fommil

This comment has been minimized.

Show comment
Hide comment
@fommil

fommil Jul 24, 2016

Contributor

@mwielocha I just watched the scaladays talk, very impressive.

Contributor

fommil commented Jul 24, 2016

@mwielocha I just watched the scaladays talk, very impressive.

@fommil fommil changed the title from replace akka with something else to use a lightweight streaming toolkit to rewrite SearchService Jul 24, 2016

@sirthias

This comment has been minimized.

Show comment
Hide comment
@sirthias

sirthias Jul 25, 2016

@fommil What's your time-line on this ticket?
It'll be a few more weeks before the first stable release of swave, which will come with proper documentation. However, swave is currently somewhat high up on my prio list (because we need it for out own stuff) and probably will be for quite some time, so I'd be able to react rather quickly to support/feature requests.

@fommil What's your time-line on this ticket?
It'll be a few more weeks before the first stable release of swave, which will come with proper documentation. However, swave is currently somewhat high up on my prio list (because we need it for out own stuff) and probably will be for quite some time, so I'd be able to react rather quickly to support/feature requests.

@fommil

This comment has been minimized.

Show comment
Hide comment
@fommil

fommil Jul 25, 2016

Contributor

@sirthias no solid timelines but certainly not within the next few weeks. We just cut ensime 1.0 so 2.0 is probably going to take us a year, realistically (I'll be delighted if its faster than that). I don't expect volunteers to come forward until the weather gets worse 😄 the number of people attending the monthly hack days has noticeably declined as the sun gets shinier...

Contributor

fommil commented Jul 25, 2016

@sirthias no solid timelines but certainly not within the next few weeks. We just cut ensime 1.0 so 2.0 is probably going to take us a year, realistically (I'll be delighted if its faster than that). I don't expect volunteers to come forward until the weather gets worse 😄 the number of people attending the monthly hack days has noticeably declined as the sun gets shinier...

@sirthias

This comment has been minimized.

Show comment
Hide comment
@sirthias

sirthias Jul 25, 2016

@fommil I see. So there is chance that swave might present another alternative to the existing stack (next to monix). I'll keep you posted on the progress.

@fommil I see. So there is chance that swave might present another alternative to the existing stack (next to monix). I'll keep you posted on the progress.

@fommil

This comment has been minimized.

Show comment
Hide comment
@fommil

fommil Jul 25, 2016

Contributor

yup. Currently it's swave vs monix vs fs2 vs scala-csp vs our own home-cooked minimal thing. We'd like to strive for zero scala dependencies (to make it easier to support new versions of scala) but we'll settle for lightweight scala frameworks.

Contributor

fommil commented Jul 25, 2016

yup. Currently it's swave vs monix vs fs2 vs scala-csp vs our own home-cooked minimal thing. We'd like to strive for zero scala dependencies (to make it easier to support new versions of scala) but we'll settle for lightweight scala frameworks.

@alexandru

This comment has been minimized.

Show comment
Hide comment
@alexandru

alexandru Aug 31, 2016

Note, I've just released Monix 2.0.0, now also cross-compiled for Scala 2.12.0-M5:
https://monix.io/blog/2016/08/31/monix-v2.0.0.html

Note, I've just released Monix 2.0.0, now also cross-compiled for Scala 2.12.0-M5:
https://monix.io/blog/2016/08/31/monix-v2.0.0.html

@pascr

This comment has been minimized.

Show comment
Hide comment
@pascr

pascr Sep 4, 2016

Contributor

@fommil to avoid scala dependencies ReactiveX/RxJava (Observable) could also be an option

Contributor

pascr commented Sep 4, 2016

@fommil to avoid scala dependencies ReactiveX/RxJava (Observable) could also be an option

@fommil

This comment has been minimized.

Show comment
Hide comment
@fommil

fommil Sep 4, 2016

Contributor

that's a good point, although at that point the code isn't going to look very pretty so I'm not sure it would be an improvement over what we currently have. I'd be willing to tolerate a lib for this, only if the author was committed to making active releases.

Contributor

fommil commented Sep 4, 2016

that's a good point, although at that point the code isn't going to look very pretty so I'm not sure it would be an improvement over what we currently have. I'd be willing to tolerate a lib for this, only if the author was committed to making active releases.

@fommil fommil referenced this issue Oct 30, 2016

Closed

Remove akka #1634

@fommil

This comment has been minimized.

Show comment
Hide comment
@fommil

fommil Oct 30, 2016

Contributor

Big problem with swave is lack of 2.10 support and limited docs sirthias/swave#18

Contributor

fommil commented Oct 30, 2016

Big problem with swave is lack of 2.10 support and limited docs sirthias/swave#18

@fommil

This comment has been minimized.

Show comment
Hide comment
@fommil

fommil Oct 30, 2016

Contributor

Whatever we choose for searchservice should scale to replace akka

Contributor

fommil commented Oct 30, 2016

Whatever we choose for searchservice should scale to replace akka

@sirthias

This comment has been minimized.

Show comment
Hide comment
@sirthias

sirthias Oct 31, 2016

Docs will completed very soon now (like: end of November).
However, I don't know whether it makes sense for swave to invest into 2.10 support.

Docs will completed very soon now (like: end of November).
However, I don't know whether it makes sense for swave to invest into 2.10 support.

@fommil fommil removed this from Backlog in Sponsorship Apr 22, 2017

@fommil fommil modified the milestones: Big Crunch 2.0, Purely Functional 3.0 Aug 10, 2017

@fommil fommil changed the title from use a lightweight streaming toolkit to rewrite SearchService to use fs2 to rewrite SearchService Aug 10, 2017

@fommil fommil removed the Blocked label Aug 15, 2017

@fommil fommil removed this from the Purely Functional 3.0 milestone Sep 24, 2017

@fommil fommil closed this Mar 26, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment