Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Use Spire's typeclasses. #99

Open
avibryant opened this Issue · 22 comments

5 participants

@avibryant
Collaborator

Not trolling. The definitions of Semigroup, Monoid, VectorSpace etc in https://github.com/non/spire are not obviously incompatible with us, and it might be nicer for the broader Scala community if we could unify.

@non
non commented

Hey, thanks for thinking of Spire! If you have features you need from those type classes please let me know.

@tixxit

One difference is that we (Spire) don't parameterize vectors on the scalar type in Module, VectorSpace, etc. Is this a problem?

@avibryant
Collaborator

Spire seems to require 2.10, which probably makes it a non-starter for us for a while. Is extracting a 2.9-compatible version of the alegbra typeclasses that we could both depend on infeasible?

@non
non commented

It is definitely possible to extract the algebra part. I was thinking of backporting all of Spire to 2.9.x but if that fails (or is prohibitive) we could definitely extract spire.algebra.

@non
non commented

If I was working on a 2.9 version, do I need to target 2.9.0? Could I target something more recent instead? (I'm not sure if there is any significant drift, maybe 2.9.0 will be just as easy. I'm just curious.)

@avibryant
Collaborator

AFAIK everyone using algebird is on either 2.9.1 or 2.9.2.

@non
non commented

Great, thanks!

@non
non commented

OK, so https://github.com/non/spire/tree/2.9-backport is building and passing tests. I need to take some more time to make sure specialization is working, and do some documentation clean up. But it should be in good enough shape for some initial integration work.

I'll comment as soon as I have a reasonable 2.9.x milestone published.

@johnynek
Collaborator

Thanks for doing this. When the 2.9.2 milestone is available, let us know.

@johnynek
Collaborator

@non one question: we have a lot of monoids/groups etc.. On built in collections. We resolve these implicitly by putting them in the Monoids object. Will you accept the same in spire so we can get easy implicit resolution for built in java and scala types?

@non

Sure. As long as it has a well-defined identity/operator (which pretty much all collections do) there should be no problem. I'll take a look at what Algebird has and go about adding them.

@johnynek
Collaborator

@non thanks. PS: be careful with the map-(semigroup/ring). The means of joining the keys matters a lot for performance.

Also take a look at: https://github.com/twitter/algebird/blob/develop/src/main/scala/com/twitter/algebird/GeneratedAbstractAlgebra.scala

Those are generated by a script.

We use these pretty heavily in scalding.

@tixxit

I saw that and will make sure we have the necessary generated tuple-versions. Thanks!

@tixxit

I'll add benchmarks in Spire for Map semigroup op too and make sure there are no perf. regressions.

@tixxit

Hey, so we've implemented many more instances for Scala collections, including Map and tuples (the tuples are autogenerated on compile). We've made a tentative 0.3.0-RC1 release for 2.10 and will backport the changes shortly. Before we do the backport, would you mind taking a look and seeing if there is anything you feel we should change?

One thing that is different, is that we only do vector addition with Modules/VectorSpaces (so Vector(1, 2) + Vector(3, 4) === Vector(4, 6)). Default Monoid for all Seq-likes is concatenation (so Vector(1, 2) |+| Vector(3, 4) === Vector(1, 2, 3, 4)). You can use Module[List[Int], Int].additive to get a Monoid[List[Int]] that does vector addition. If you'd like to be able to do vector-style addition when the scalar is only in, eg, a Monoid, then we can implement an optional import for this.

The map implementation is like yours though, where, eg, Monoid[Map[K, V]] uses a Semigroup[V] to combine values.

https://github.com/non/spire/blob/2.10.0/src/main/scala/spire/algebra/map.scala
https://github.com/non/spire/blob/2.10.0/src/main/scala/spire/algebra/seq.scala
https://github.com/non/spire/blob/2.10.0/project/GenProductTypes.scala

@johnynek
Collaborator

I get 404 when I click the map/seq links above, and I'll take your word about the tuple generated ones (actually looking at the generated code is probably more fun).

About the Vector difference, I guess that's a fine approach too. The rule that we do element-wise operations on Vector but not on List was always a bit ad-hoc.

About the 2.9 backport... Honestly, my guess is that the internal cost of porting our code from algebird to spire will be significant enough that we probably won't get to it before we upgrade to 2.10. So, realistically, it might be better for you to save the effort of backporting if you are doing it on our behalf.

What do you think of some of the approximation structures? CountMinSketch, etc...?

Also, what is your policy about contributions? It looks like only 5 people have ever committed to the spire project. Do you welcome pull requests?

@non

Now that I've (almost) completed the backport I will probably try to keep it up. There are places where Spire on 2.9 would be nice for me personally. But it's true that Spire is a lot nicer on 2.10.

I'll try to look at CountMinSketch soon, so I can say more about it. Maybe Tom has opinions?

About contributions: Spire is released under an MIT license. As long as the submitter is fine with that then we welcome pull requests from anyone.

P.S. The link for those source files actually changed to:

https://github.com/non/spire/blob/2.10.0/core/src/main/scala/spire/algebra/map.scala
https://github.com/non/spire/blob/2.10.0/core/src/main/scala/spire/algebra/seq.scala

@avibryant
Collaborator

To the extent that it makes sense to keep these projects separate (ie, assuming we aren't just trying to merge them), I think the approximation structures are definitely on the algebird side of the line... @tixxit and @non can comment, but it feels like Spire is more about setting up these typeclasses for existing Scala types, and Algebird is more about developing new, more exotic data structures that can also be tied into the same framework.

@tixxit

Yes, I agree with @avibryant -- Spire's focus is mostly on the type classes themselves and also new numeric types (ie. Complex, Rational, Real, FPFilter, Number, etc.), so more exotic/fun data structures may be a bit out of scope (though I am looking forward to Avi's talk at NEScala).

@tixxit

And, as Erik said, we definitely welcome PRs, as long as you are fine with the fairly liberal license.

@mhlr

So what happened? Everybody seemed in agreement, the Spire prep work was done but almost 2 years later the issue is stil open and there are no PRs.

@non
non commented

So it's basically my fault. The algebra project is almost there but I never got everyone's complete agreement, and I also didn't do the Spire integration I needed.

It's on my radar and I'm hoping to get it working. Thanks for the nudge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.