ClojureScript support #81

Open
podviaznikov opened this Issue Nov 20, 2013 · 41 comments

Projects

None yet
@podviaznikov

Is it possible to implement this library in ClojureScript?
Or at least some parts of it.

@mikera
Owner
mikera commented Nov 21, 2013

ClojureScript support should certainly be possible in core.matrix.

Some general points:

  • The API should work for both Clojure and ClojureScript, with virtually no modification.
  • Likewise the protocols that support the API should also work for both ClojureScript
  • The implementations will generally need to be different, as they will depend on host-specific interop and/or libraries

Getting this done will need someone reasonably proficient in ClojureScript to take it on as a project. If anyone is interesting in doing this, let me know (on this issue or by email) and I'm happy to help give some guidance.

@zcaudate

I'm happy to work on this. Would wrangling the protocols with http://sylvester.jcoglan.com/ work?

@mikera
Owner
mikera commented Nov 25, 2013

Possibly - I'm not too familiar with Sylvester. Other options that it might make sense to implement the protocols for:

@zcaudate

There is a js implementation of ndarray here: https://github.com/mikolalysenko/ndarray

I think the goog.matrix and Sylvester are strictly 2d. Are the protocols for n-dimensional arrays?

There is http://www.numericjs.com/ which looks pretty nice and it is for native js arrays. The benchmarks look interesting http://www.numericjs.com/benchmark.html

@mikera
Owner
mikera commented Nov 26, 2013

Looks like there are some good implementation options! My suggestion would be to try and get two or three of them working, it's fairly easy to implement just the mandatory protocols.

The protocols are designed to handle N-dimensional arrays, but you can implement them for 2D arrays and have the implementations fall back to other implementations for different dimensionalities. For example, Clatrix currently only supports 2D matrices natively (although it used a trick of wrapping a 2D matrix to implement 1D vectors as well).

@zcaudate

Okay. that's great. I'm going to go with js-ndarray then. Its the most well thought out of all of them:

http://0fps.wordpress.com/2013/05/22/implementing-multidimensional-arrays-in-javascript/

is there a project namespace/name that I should just? Else I'll just name it as cljs-ndarray

On 27/11/2013, at 3:58 AM, Mike Anderson notifications@github.com wrote:

Looks like there are some good implementation options! My suggestion would be to try and get two or three of them working, it's fairly easy to implement just the mandatory protocols.

The protocols are designed to handle N-dimensional arrays, but you can implement them for 2D arrays and have the implementations fall back to other implementations for different dimensionalities. For example, Clatrix currently only supports 2D matrices natively (although it used a trick of wrapping a 2D matrix to implement 1D vectors as well).


Reply to this email directly or view it on GitHub.

@zcaudate

Hmmm... looking at the source of js-ndarray, it is written for node.js. Node has its own packaging system which is really quite nice.

The ndarray implementation uses some sort of compilation process
https://github.com/mikolalysenko/ndarray/blob/master/ndarray.js

as well and the operations build on top of them:
https://github.com/mikolalysenko/cwise/blob/master/cwise.js

There is a way to turn node code into browser code through browserify - http://browserify.org/. I'm not too concerned as I'm targeting node but it may be for other people

@mikera
Owner
mikera commented Nov 27, 2013

If you can, it would be good to minimise the number of dependencies if possible (e.g. on Node), to make it easier for people to run the implementation on different platforms / clients.

It probably makes sense for the ClojureScript implementations themselves to have their own repo. This gives us more flexibility around releases, naming conventions etc, and means that they can have separate dependencies as appropriate (Clatrix and vectorz-clj have their own repos for similar reasons)

In the main core.matrix repo itself, the main thing to figure out is how to define the API / protocols in a way that works for both Clojure and ClojureScript. I think this will need some kind of proof of concept.

@zcaudate

hmmm... in that case, I'll port the code over. It'll take a bit longer but it should be worth it.

@si14
Collaborator
si14 commented Nov 27, 2013

I think it should also be possible to extend current NDArray implementation to support CLJS. I've tried to implement NDArray in a way that will make translation to CLJS easier: most of the code uses very primitive stuff like aset and aget, and backend-specific details like type hints are mostly wrapped in macros (here clojure.core.matrix.impl.ndarray-macro). There are few Java calls like java.utils.Array/copy, but they can be replaced with macros.

The problem here is that, being core.* lib, core.matrix can't depend on cljx. However, it's still possible to copy an entire namespace into .cljs using macro.

@zcaudate

That could work. We can copy it manually too =) Or another option is lein-dalap to generate the .cljs file

There are also a couple of java-specific things hanging about in protocols.clj. Can that be refactored somehow?

@nyx
nyx commented Mar 13, 2014

I'm interested in using core.matrix functionality in the browser and I'm curious if there's been movement on this front not reflected in this ticket. I'd like to help, but I'll be slow since I'm not very familiar with core.matrix internals or ClojureScript yet.

@zcaudate

I've been slack and haven't looked at it for a while, but i do remember thinking that the protocols file may have to be refactored for cljs in order to get started. But this was half a year ago.

On 14/03/2014, at 1:13, Andrew Andkjar notifications@github.com wrote:

I'm interested in using core.matrix functionality in the browser and I'm curious if there's been movement on this front not reflected in this ticket. I'd like to help, but I'll be slow since I'm not very familiar with core.matrix internals or ClojureScript yet.


Reply to this email directly or view it on GitHub.

@mikera
Owner
mikera commented Mar 22, 2014

I'm not aware of anyone having a serious go at this yet (i.e. I don't think there are any working core.matrix implementations on ClojureScript yet, unless someone is hiding something?)

My view is that it all should be possible, but will need quite good knowledge of the internal workings of both Clojure and ClojureScript to make it work smoothly.

Also it needs a really good design to make sure it coexists nicely with core.matrix JVM version. My suggested approach would be:

  • Refactor the API and protocols to make them ClojureScript friendly.
  • Create a core.matrix.cljs library that contains the ClojureScript specific implementations, and depends on the standard core.matrix for the API and protocol definitions

I'm happy to support this effort / provide advice, but someone else needs to drive it forward as it isn't really a priority for me right now.

@mschuene
Collaborator

would it be possible to share the main code (ie the protocol definitions,
etc) between clojure and clojurescript with tools like
cljx https://github.com/lynaghk/cljx ?

On Sat, Mar 22, 2014 at 5:28 AM, Mike Anderson notifications@github.comwrote:

I'm not aware of anyone having a serious go at this yet.

My view is that it should be possible, but will need quite good knowledge
of the internal workings of both Clojure and ClojureScript to make it work
smoothly.

Also it needs a really good design to make sure it coexists nicely with
core.matrix JVM version. My suggested approach would be:

  • Refactor the API and protocols to make them ClojureScript friendly.
  • Create a core.matrix.cljs library that contains the ClojureScript
    specific implementations, and depends on the standard core.matrix for
    the API and protocol definitions

I'm happy to support this effort / provide advice, but someone else needs
to drive it forward as it isn't really a priority for me right now.

Reply to this email directly or view it on GitHubhttps://github.com/mikera/core.matrix/issues/81#issuecomment-38342733
.

@mikera
Owner
mikera commented Mar 22, 2014

Possibly cljx would work. I haven't experimented with it. It would need to work with Maven builds (that's what core.matrix uses, and it is a clojure contrib requirement too...) - any idea if it does this?

@xcthulhu

This is almost certainly too beta right now, but it would be amazing if core.matrix could eventually wrap the new WebCL standard to make use of the GPU for matrix computations...

https://www.khronos.org/news/press/khronos-releases-webcl-1.0-specification

@skrat
skrat commented Aug 20, 2014

We could just have separate .cljs sources, optimized for JS. core.async does it too afaik.

@mikera
Owner
mikera commented Aug 21, 2014

@xcthulhu this looks possible. It should just be another core.matrix implementation

@mikera
Owner
mikera commented Aug 21, 2014

@skrat that could make sense. Although I'd quite like the public API to be shared.... it makes things like maintaining consistent docstrings much easier!

@skrat
skrat commented Aug 21, 2014

Unfortunately it seems that core.async duplicates a lot of code, while in core.matrix a lot can be reused. That makes me incline toward using cljx.

@nybbles
nybbles commented Oct 4, 2014

@skrat, @si14 says above that cljx cannot be used, because this is a core.* library. I wonder if this constraint is still true.

@skrat
skrat commented Oct 5, 2014

I don't think it is, I'm not aware of any law mandating that cljx cannot be used. It would be used during the publish phase anyway, and thus there wouldn't be any dependency in the published jar.

@gleenn
gleenn commented Aug 16, 2015

Has there been any progress on this?

@mikera
Owner
mikera commented Aug 17, 2015

Nobody has created any patches that I'm aware of. Anyone want to take a shot at it?

I think I'm happy with either cljx or separate cljs files. This mostly applies to just the clojure.core.matrix API namespace itself, since the implementations will generally need to be separate.

My only firm requirement is that it mustn't mess up the standard build/ release process, i.e.:

  • I still need to be able to do mvn release:prepare and mvn release:perform to create a release.
  • Tests need to test both Clojure and ClojureScript side
@skrat
skrat commented Aug 24, 2015

cljx was superseded by reader conditionals in Clojure 1.7. I don't see any reason why should it change the build/release process. The conversion should be fairly simple. Fast matrix storage and operations should be implemented with Javascript's typed arrays.

@mikera
Owner
mikera commented Aug 24, 2015

Currently we still need to support Clojure 1.6, so I think that rules out reader conditionals right now.

@skrat
skrat commented Aug 24, 2015

I have a .cljc port in progress. Trying to find the core issues with a port to JS runtime. There are more issues than I previously thought. To have this done would require a lot of willpower from the maintainers. I don't expect this to happen before the Clojure 1.6 requirement is out of the way.

@mikera
Owner
mikera commented Aug 24, 2015

@skrat Can you share a link to your repo?

@mikera
Owner
mikera commented Oct 12, 2015

Hi all I see an interesting opportunity to get ClojureScipt support using the NDArray implementation here: https://github.com/thi-ng/ndarray

The author @postspectacular would be happy with a PR that implements core.matrix support (although we would need to get the ClojureScript-side protocols working first...)

See this issue for context and discussion::

thi-ng/ndarray#1

@rosejn
Collaborator
rosejn commented Nov 6, 2015

Hey guys,
I've been lurking on this thread without contributing, but I'm very interested in getting this off the ground so I put some effort into it today. Here's a branch with core.matrix converted to cljc with the modifications needed to get it to compile and load in Clojurescript. There is no doubt lots more to do in order to get it working cleanly, but this is a big step as we can at least load it in figwheel and iterate.

https://github.com/thinktopic/core.matrix/tree/cljs

I'm starting on an implementation using thi.ng.ndarray now, which I'm already using on the client, and that branch will go here once I get something working.

https://github.com/thinktopic/ndarray/tree/matrix

Mike let me know if this looks like a reasonable path.

(I've also got core.matrix arrays sent as fressian binary data over a websocket and then deserialized into thi.ng.ndarray arrays on the client... could be a nice protocol addition to core.matrix to support serialization.)

@mikera
Owner
mikera commented Nov 7, 2015

Hi Jeff,

This looks great, thanks for taking a shot at this! I think I'm it is OK to drop the 1.6 support now that 1.8 is nearly out of the door so happy to take a PR on this.

The only potential blocker I see is whether we can do proper integrated tests that cover both the ClojureScript and Clojure side (ideally with Maven, since that is a clojure.core requirement). did you take a look at how to solve this?

@rosejn
Collaborator
rosejn commented Nov 7, 2015

OK, good to hear. I'll keep working on it then. Not sure about testing and I'm not familiar with Maven, but once I get the basic functionality working I'll look into it.

@nilern
nilern commented Nov 17, 2015

Hi,

I am using goog.math.Matrix on a project (mostly for solving groups of equations) and started implementing core.matrix for that using rosejn's branch. Mostly doing this to learn about the protocols and see what the state of cljs support is at the moment. Eventually I'd prefer to do the https://github.com/scijs/ndarray implementation.

@mikera
Owner
mikera commented Nov 18, 2015

Hi @nilern, sounds like a good idea! though it might prove tricky until @rosejn's branch is fully opeartive :-)

Jeff, you may want to sync up with the latest develop, I'm doing some work on the dataset code at the moment and it would probably be best to stay as close to master as possible.

@rosejn
Collaborator
rosejn commented Dec 12, 2015

Hey everyone, a working cljc branch has been merged.

https://github.com/mikera/core.matrix/tree/cljc

There's also a basic core.matrix implementation for ndarrays here:

https://github.com/thinktopic/ndarray/tree/matrix

If people would like to pitch in it would be great to add support for javascript arrays and typed arrays to act like vectors, like they do in Java. This is done by extending some protocols like in defaults.cljc if you want to look at some examples. To get basic support it only takes a few very simple protocol implementations to get the shape, read and write values.

@mikera
Owner
mikera commented Dec 12, 2015

Fantastic work @rosejn !

Would be great if people could pile in and test this out, I'm not such a big expert in ClojureScript so hard for me to review effectively. If all goes well I hope to be able to merge this into the main core.matrix develop branch before too long.

@bmabey
Contributor
bmabey commented Jul 1, 2016

Hey @rosejn, how far did you get with your cljs wrapper? Would you recommend using it or just using the ndarray library API for now?

@mikera
Owner
mikera commented Jul 2, 2016

@bmabey as far as I am aware there is decent core.matrix support in ClojureScript now (since 0.50.0)... but not sure how much testing it has had yet... please try it out and let us know how you get on

@bmabey
Contributor
bmabey commented Jul 2, 2016 edited

Ah, thanks @mikera. I did a quick google search for support and this was the first link. :) I'll try it out and let you know if I run into any issues.

@rosejn
Collaborator
rosejn commented Jul 5, 2016

Hi Ben,  The Clojurescript support is complete in that all of the unit tests pass; however, no effort has gone into making it fast yet.  The design of core.matrix lets us bootstrap a new implementation with fairly little effort now, but there are no doubt many operations that will fall back to slow paths.  If you need something to go faster feel free to ping me and I can help.
Cheers,Jeff

    _____________________________

From: Ben Mabey notifications@github.com
Sent: Saturday, July 2, 2016 3:42 PM
Subject: Re: [mikera/core.matrix] ClojureScript support (#81)
To: mikera/core.matrix core.matrix@noreply.github.com
Cc: Jeff Rose rosejn@gmail.com, Mention mention@noreply.github.com

Ah, thanks @mikera. I did a quite google search for support and this was the first link. :) I'll try it out and let you know if I run into any issues.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment