Scalaz 8 Roadmap

[jdegoes]

Checklist

• Core
  • Type
    • Leibniz
    • Liskov
    • AList
    • ...
  • Data
    • IList — Current version not functional
    • Maybe
    • Iso
    • Disjunction
    • Heap
    • Map
    • These
    • NonEmpty — Replaces OneAnd and NonEmptyList (type Nel[A] = NonEmpty[List, A])
    • Array — This has questionable value outside ST
    • Fingertree
    • EphemeralStream
    • ...
  • Category Theory
    • Functor Hierarchy
      • ...
  • Effects
    • ApplicativeAsk
    • ApplicativeError
    • ApplicativeLocal
    • FunctorListen
    • FunctorTell
    • MonadState
    • ...
  • Abstract Algebra
    • Magma
    • Quasigroup
    • Monoid
    • Semigroup
    • Group
    • AbelianGroup
    • Ring
    • QuotientRing
    • Field
    • Module
    • VectorSpace
    • RingAlgebra
    • FieldAlgebra
    • AssociativeAlgebra
    • LieAlgebra
    • LieGroup
    • Lattice
    • BoundedLattice
    • Semilattice
    • JoinSemilattice
    • MeetSemilattice
    • HeytingAlgebra
    • BooleanAlgebra
    • LieGroup
    • ...
  • Other CT
    • ...
• Syntax
  • Zero-cost, by-name syntax macros
  • ...
• IO
  • IO
  • IORef
  • RTS (JVM)
  • RTS (JS)
  • Fallible
  • Terminatable
  • Foreign
  • Effect
  • ...
• Documentation
  • Examples
    • scalaz.effect
  • Scaladoc
    • scalaz.effect
  • README.md
    • scalaz.effect
• Test
  • ScalaProps
    • Port to Scalaz 8
    • Remove Arbitrary
    • Add Shrinking
    • Add Exhaustivity
  • Algebra
  • Category Theory
  • Mutable
  • IO
  • Effects
• Benchmark
  • scalaz.effect.IO
  • scalaz.effect.IORef
  • scalaz.typeclass.Equal
  • scalaz.typeclass.Ord
  • ...
• Design Guide

---

• Third-Party
  • scalaz-deriving
  • scalaz-nio

Overview

Core

Data

Needed: Someone to spearhead this work.

Functor Hierarchy

The Scalaz 8 functor hierarchy needs to be generalized to minimize the number of new abstractions needed to capture distinctions between different types of functors.

A more generic encoding of functor, more faithfully following category theory, will allow custom arrows and refined objects.

While the scope and nature of these changes is TBD, one useful suggestion by @ekmett is to factor out Productive and Coproductive classes:

trait Productive[F[_]] {
  def product[A, B](fa: F[A], fb: F[B]): F[(A, B)]
  def unit: F[Unit]
}

trait Coproductive[F[_]] {
  def coproduct[A, B](fa: F[A], fb: F[B]): F[A \/ B]
  def counit: F[Void]
}

These abstractions permit more sharing between, for example, an invariant applicative and a covariant applicative, reducing the number of type class variations that would otherwise have to exist.

Similarly, modeling Bind as F[F[A]] => F[A] (rather than (F[A], A => F[B]) => F[B])) allows many more types of monads to exist.

Lead: @alexknvl

MTL

Type classes to describe all classes of effects. Unlike MTL, constraints should be minimized (e.g. Functor or Apply / Applicative when possible, Monad only when necessary), and as a result, names will differ substantially from Haskell.

While the type classes are essential, the dirty secret in Scala is that actual monad transformers do not perform well and are not a realistic solution for most code. An alternative to stacks of monad transformers that is better suited to Scala must be developed.

Needed: Someone to spearhead this work.

Algebra

Needed: Someone to spearhead this work.

Other CT

Syntax

Effect

Lead: @jdegoes

Mutable

High-performance ST-based mutable data structures and constructors allowing importing impure Scala code to directly manipulate mutable data structures. Instances for MTL type classes.

Needed: Someone to spearhead this work. @jdegoes ?

Documentation

Comprehensive documentation, readmes, and examples for all data types and abstractions in Scalaz 8.

Test

Benchmark

The benchmark package should help ensure Scalaz 8 encoding techniques, data structures, and effect systems are competitive with other techniques, up to the limits of the design goals.

Design Guide

Scalaz 8 needs a design guide that can ensure uniform standards are followed.

An initial version of the Design Guide can be found here.

• Policies for laws — algebraic and operational
• Policies for eager / lazy
• Policies on overloading
• Policies for variance (Invariance?)
• Policies for polymorphic values (Pascal)
• Policies for naming conventions
  • Instances, classes, etc.
• Policies for impure code
  • OK to accept if RT (e.g. lift into IO), not OK to return
  • OK to hide fast impure code beneath pure interface
• Policies for PartialFunction
• Detailed guide on type class encoding
• Policies on native methods vs. extension methods
• Detailed guide on syntax encoding

Needed: Someone to spearhead this work.

Third-Party

Scalaz-Deriving

Machinery to derive type classes automatically given instances supplied by the type class author.

Lead: @fommil

Scalaz-Nio

Type safe, minimal interface to Java NIO, including cost-free newtype wrappers for mutable data types.

[jdegoes]

TODO: Port over these notes from @aloiscochard

[fommil]

Deriving will need iotaz if it is to be part of core, so perhaps best left as an addon module?

[TomasMikula]

Can you expand on this?

Similarly, modeling Bind as F[F[A]] => F[A] (rather than (F[A], A => F[B]) => F[B])) allows the question of arrow type to be deferred and isolated to map in the corresponding type of functor.

[jdegoes]

@fommil Sure, it can live anywhere, although there are other reasons to consider including iotaz (composable error sets, for example). I view some of the other major areas as potentially existing elsewhere, in different repos.

@TomasMikula The current definition of Bind marries it to covariant functors. But if you modify the definition as suggested, then if you want a covariant monad, you combine Functor with Bind, and if you want an invariant monad, you combine Invariant with Bind.

[jdegoes]

Btw, to all Scalaz contributors: Please feel free to add to, edit, amend, and fill in!

[emilypi]

Have we enumerated what's going to be included in data yet, or is it, or do we just have List and Map outstanding currently

[Jacoby6000]

There's definitely a lot missing from that list. I know for sure it is missing OneAnd, NonEmptyList, These/Ior, and some sort of Array backed collection.

[TomasMikula]

There was a proposal to have 2 versions of some data types: a direct, fast, but stack-unsafe encoding, and a slower but stack-safe encoding. Applies to e.g. State monad.

[jdegoes]

@TomasMikula We have discussed something even more radical, which is to not have the State monad at all, but only the type class MonadState or rather ApplicativeState (I suppose). Let's discuss sometime w/ @edmundnoble on the private http://gitter.im/scalaz channel.

[NeQuissimus]

Should we maybe convert the example project to use tut, so that Markdown can be used to explain certain pieces of code, thereby providing rich documentation?

[TomasMikula]

👍 for tut. Is there a way to automate the process of publishing tuts with every build?

[NeQuissimus]

I have only ever done this for an internal project (on GitHub Enterprise) and we have our build server build the documentation, then commit it into a separate repo, whose contents we expose via GitHub Pages. The separate repo was mostly to keep the tut commits out of our code history.

I am unaware of a better way but surely there must be one :D

[jdegoes]

@NeQuissimus Sounds like a great idea. 👍

[caente]

Could you elaborate a bit more on the Algebra entry?

[jdegoes]

@caente I think this is a catch all for abstractions from abstract algebra, including things like Semigroup, Lattice, Field, and Ring. I have sketched some detail out above, and I think we can take inspiration from PureScript's hierarchy (and some work has been done in Scala, as well, in Spire and other libraries). Any interest in contributing to this one? 😁

[caente]

@jdegoes I was hopping you would say that. I'm definitely interested in contributing. From what you say it seems like there isn't even a desired roadmap? I don't think my abstract algebra kung fu is enough for planing an strategy/hierarchy.
Although perhaps it could be enough to "copy" PureScript as you said?
Do you have experience with Spire's algebra? Any reason for not follow a very similar path to theirs?

[vmarquez]

@caente would be awesome to have someone tackle this in depth! One thing to keep in mind though is that it may make sense to decide how we want to do laws first. Algebraic structures such as Group, Semigroup, Monoid, SemiLattice, etc. all look pretty similar with different laws... hence why when I added SemiLattice to 7.3.x, I didn't add to Scalaz8. Food for thought. :)

[puffnfresh]

@caente copy PureScript

[dchenbecker]

Would improving scaladoc be considered part of "Examples"?

[jdegoes]

@caente I can definitely help out with that (as can others). As @puffnfresh says, starting with the PureScript hierarchy would be a good first step. It's not quite fine-grained enough but it has the right general shape. There is overlap with Spire but Spire contains "non-things" like Trig which would definitely not be desired in an abstract algebra package.

@dchenbecker I rewrote that section to be Documentation, which now contains Scaladoc, examples, and README.md. Any contributions to any of these would be enormously beneficial.

Also: hi! 😄

[caente]

@jdegoes @puffnfresh I've been looking into purescript docs, and their algebra typeclasses seem to be distributed among several modules:

• https://pursuit.purescript.org/packages/purescript-prelude
• https://pursuit.purescript.org/packages/purescript-group
• https://pursuit.purescript.org/packages/purescript-monoid
• etc

I find strange that Ring and Semigroup are in prelude, but Group and Monoid and in different libs... wouldn't we want that Ring to be "composed" with abelian group and monoid, for example?

what would be the advantage of such non-hierarchical hierarchy?

[puffnfresh]

@caente Ring and Semigroup are there because that's where (+) and (-) come from. Prelude without (+) and (-) would be a bit weird! You'll also see Semiring and EuclideanRing for (*) and (/)

[puffnfresh]

@caente yeah, I think you're right, don't copy this part!

[caente]

thanks @puffnfresh, on the other hand this seems like a nice guide http://a-guide-to-the-purescript-numeric-hierarchy.readthedocs.io/en/latest/

I see, there are Monoid and Semigroup, so it seems to make sense to implement groups and abelian group, and then rings and fields?
but also, I don't see any Monoid or Semigroup instances in scalaz8, nor tests nor laws... am I missing something? why are they checked off?

Any thoughts on how to enforce laws? I see that in the current version of scalaz they are "built-in" on the typeclass itself, e.g. https://github.com/scalaz/scalaz/blob/series/7.3.x/core/src/main/scala/scalaz/Monoid.scala#L69-L82
I don't see any tests of that kind on scalaz8, is there a consensus about it?

btw I'd love to hear of any use cases for these typeclasses...

[jdegoes]

@caente For now, I wouldn't worry about enforcing laws. The laws can be described in comments. Ultimately we want to add them as methods on the type classes, like Scalaz 7, but possibly closer to what Cats is currently doing, which is representing laws as data (case class IsEq[A](lhs: A, rhs: A), with <-> syntax for construction).

You are also right that PureScript makes some tradeoffs when it comes to the Prelude and also how fine-grained the hierarchy is. That's why we don't want to just copy wholesale, although it's a good place to start. See also the abstractions I posted above, Algebra (possibly the Gold Standard?), Spire, Cats Algebra, and of course Wikipedia and other free resources.

[jdegoes]

@caente I found Edward Kmett sketched out a hierarchy here. It uses a someone non-idiomatic style of Scala but looks clean and well put together (Kmett also wrote Algebra for Haskell).

[caente]

@jdegoes last night I began implementing Group in terms of monoid, then I saw Kmett's code and I was intimidated and confused: The code seems bottomless. Hence I repeat the question I asked before: are there use cases we want to prioritize? I imagine we don't exactly want to use scalaz to proof theorems, but rather to automate certain operations?
I'm afraid I'm not the target audience for these typeclasses, but still wanna help.

[jdegoes]

@caente Well, let's start with something simple then:

• Abelian
• Commutative
• Group
• Loop
• Magma
• Module
• Monoid
• Quasigroup
• Rig
• Ring
• Ringoid
• Semigroup
• Semiring

This is a decent foundation and we can tweak or add other things (Lattice and friends) later. I can also sketch out the relationships between these, if you like (though it's all available above in the stated references). For example, Semigroup is a Magma, and so forth.

[vmarquez]

@jdegoes I'm most interested in SemiLattice, does that not belong on the list? They're very useful for distributed systems. (Idempotency + commutativity)

[TomasMikula]

I'm also more interested in lattice-like structures than ring-like structures.

[jdegoes]

@vmarquez @TomasMikula I think we need them all, but we need to start somewhere. 😄

[caente]

@jdegoes I'll make a first stab at this tomorrow, it will probably be lacking, but I don't think I'll need anything else for now, between Kmett's code and Pinter's book I could be fine.

[garyb]

@vmarquez @jdegoes If you add Idempotent you can get Band (Semigroup + Idempotent) and then Semilattice (Commutative + Band).

[jdegoes]

@caente Awesome, let me know if you run into any trouble!

@garyb Hehe. Thanks. Good to see you around these parts! 😄

[fommil]

What about a decent test library? It's still an open gap in FP... imagine a suite of matchers like scalatest that uses Equal and Order, and maybe even a "choose your Monad" runner so you could write Id tests, Either tests, or IO tests. With structural diffs of things that don't match the expectations. Maybe even macros to easily create "mock" versions of a finally tagless algebra backed by an mvar.

[tonymorris]

If you are going to do a decent test library, look at hedgehog.

https://hackage.haskell.org/package/hedgehog

Don't copy quickcheck/scalacheck. You'll be way ahead of anything else that exists for Scala.

IMO.

[caente]

@jdegoes @vmarquez @garyb @TomasMikula @puffnfresh I oppened this PR #1551 to move the conversation there, if possible I rather add small changes at a time.
Do let me know if you think we should start a "lower level".

[NeQuissimus]

@tonymorris @fommil I would be very interested in this actually. I just figured it would be beyond the Scalaz 8 scope. There is scala-dedgehog but I have never used it. In #1540 @jdegoes mentioned that ScalaProps is meant to become part of Scalaz 8, we should definitely keep that in mind.

[jdegoes]

@NeQuissimus @tonymorris Indeed, Hedgehog is my favorite too. ScalaProps is closer than ScalaCheck and can be made closer still, so it's a good base.

[jdegoes]

@fommil It's needed. All the test libraries are non-functional and do not understand functional concepts (e.g. Monad), which makes them difficult to use. Something small but fully integrated into Scalaz can be used for internal tests, but also as a dependency scalaz-test in other projects.

[edmundnoble]

https://github.com/scalaz/testz is now up. It needs Hedgehog-style property-based testing, but for now it should at least let us write laws.

[Atry]

While the type classes are essential, the dirty secret in Scala is that actual monad transformers do not perform well and are not a realistic solution for most code. An alternative to stacks of monad transformers that is better suited to Scala must be developed.

In fact, Codensity performs pretty good in Scala, and because continuation is the mother of all monads, it is easy to implement MTL MonadXxx type class instances with specialized Codensitys.