Modularization

[denisrosset]

Spire currently has good support for:

• concrete data types (Rational, SafeLong, etc...)
• abstract typeclasses that mimic abstract algebraic structures (rings, fields, etc...)

We want to reduce Spire's JAR size and improve code coverage in the modules where Spire is robust.

Roadmap:

• Deprecate code by moving it to a spire-legacy package. That includes lawless or redundant typeclasses, and data types whose definition is not clear (ex: Jet).
• Algorithms such as sorting, ..., should be separated from data types.

[kschwarz1116]

This could be a holding ground for #689, #708, and #709.

[armanbilge]

@kschwarz1116 since you've taken an interest in this project (which is wonderful :) here seems a good a place as any to share some thoughts. In my personal perspective, these are the next big steps ahead for Spire:

1. Modularization, in the style of Cats and Cats Effect. This generally means a kernel module for typeclasses, and a core module with datatypes, and possibly some auxiliary modules. Although, this structure is debatable: cats-core includes both typeclasses and datatypes, and not everyone agrees this was worth it in Cats Effect.
2. Full adoption of Scala 3. Spire is full of Scala 2 specific machinery for syntax extensions with minimal runtime overhead. I don't think it's worth poking at that code anymore, because in Scala 3 this becomes a first class feature: syntax can be defined directly on the typeclass without syntax imports and inline makes it easy to vanish overhead.
3. Figure out how to operate in a world without @specialized. Scala 3 has not implemented primitive specialization and may never do so, which was critical for Spire's performance on Scala 2. So for Spire to truly make the jump to Scala 3 it will need a very clever redesign to workaround this, or simply redefine itself (plenty of stuff in Spire is awesome without specialization).

Personally, I'm interested in Spire wrt how it can be applied to probabilistic programming and statistical inference.

[kschwarz1116]

Thanks for all the info! This is really helpful to know.
I kind of jumped into the issue backlog, hoping to get a better sense of what the current state of the project is, as well as what the goals of the project are (both for maintainers as well as users). Spire piqued my interest because I wanted to contribute to a Typelevel project and I was a math major in college, so it seemed like a reasonable intersection.

Still trying to get my feet wet, but definitely agree with your vision. It makes sense to me to take advantage of scala 3 features where possible. I have eyes on the benchmarks too -- I'm all for performance, but I think it's reasonable to question if that comes at the cost of expressive types.

[armanbilge]

Spire piqued my interest because I wanted to contribute to a Typelevel project and I was a math major in college, so it seemed like a reasonable intersection.

That's fantastic, really glad to have you! Spire was the first Typelevel library I used when getting started with Scala :)

hoping to get a better sense of what the current state of the project is, as well as what the goals of the project are (both for maintainers as well as users)

Yes, this has been helpful for me too. I haven't seen the founding maintainers around for a while, so it's on us to figure out the way forward 😅

That's a great question about user goals and you reminded me to check the download stats. The vast majority of downloads are on Scala 2.12, which was dropped in #978.

scala	downloads last month
2.11	20,064
2.12	652,551
2.13	34,095
3	252

Even more interesting, 357,425 of those 2.12 downloads are for v0.17.0-M1 and another 236,504 are for 0.14.1! 0.17.0 final clocks in at 56,532. So, yeah 🙃

It makes sense to me to take advantage of scala 3 features where possible. I have eyes on the benchmarks too -- I'm all for performance, but I think it's reasonable to question if that comes at the cost of expressive types.

For the record, Scala 2 will be around for a while, so it's definitely valuable to continue developing Spire primarily targeting 2.13 for performance while also cross-compiling to Scala 3, if that's of interest to you. But IMHO any larger efforts to modularize or reorganize should think hard about how this will all work on Scala 3. I'm excited for this, but it's daunting :)

Thanks again for all your work so far!

[armanbilge]

@typelevel/steering can we add @kschwarz1116 to the spire team? Thanks!
https://github.com/orgs/typelevel/teams/spire/members

[rossabaker]

Invite sent.

[benhutchison]

Hi @armanbilge the roadmap makes sense to me

Modularization, in the style of Cats and Cats Effect. This generally means a kernel module for typeclasses, and a core module with datatypes, and possibly some auxiliary modules. Although, this structure is debatable: cats-core includes both typeclasses and datatypes, and not everyone agrees this was worth it in Cats Effect.

Yes please 🙏 Personal taste is more to the fine-grained end of the spectrum, ie more modules. In an ideal world, I'd read / broadly understand most of the code in my classpath, so limiting how much unneeded stuff I pull in is desirable to me.

Full adoption of Scala 3. Spire is full of Scala 2 specific machinery for syntax extensions with minimal runtime overhead. I don't think it's worth poking at that code anymore, because in Scala 3 this becomes a first class feature: syntax can be defined directly on the typeclass without syntax imports and inline makes it easy to vanish overhead.

I mainly care about Spire on 3 at this point, since at minimum it looks like I'll be using it as a dependency of Coulomb,

Figure out how to operate in a world without @specialized. Scala 3 has not implemented primitive specialization and may never do so, which was critical for Spire's performance on Scala 2. So for Spire to truly make the jump to Scala 3 it will need a very clever redesign to workaround this, or simply redefine itself (plenty of stuff in Spire is awesome without specialization).

Hard 😓 IIRC, benchmarks I saw suggested that Graal Enterprise edition (specifically) seems to do something akin to specialization at the JVM JIT level.

[erikerlandson]

I can imagine a library factorization like

• spire-kernel: any core typeclasses for algebras or numeric value conversion
• spire-core: numeric types (or more generally algebras, including complex and quaternion)
• numeric computing, like Jet and prime sieving, as separate libs

IIRC a lot of the algebraic typeclasses are now actually aliases of algebraic typeclasses in cats-algebra. It might be nice to just cut out the aliases, if that isn't too hard. I spent some time code-diving to work out where it was all coming from at some point.

[erikerlandson]

In scala 3, unboxing is intended (IIUC) to be accomplished via inline or transparent inline although my experience playing with these ideas in coulomb is that it's difficult to reason about, or predict, what efficiencies you will actually get, and the only way to really know is the examine the bytecode, see for example: scala/scala3#12009

It's also difficult to see how one takes real advantage of scala-3 without making the leap to leaving scala-2 behind, as I ultimately did with coulomb

On the bright side, one can also do some very interesting things with scala-3 metaprogramming, which is definitely superior to what was available in scala-2.

Oh, also explicit type specialization:
scala/scala3#12011