Clojure support?

[bsima]

Are there any plans or interest in using Javelin from Clojure? I was thinking earlier today that it would be nice to have cells on the backend too.

Yeah, core.async works for most of the same problems as Javelin, but the cell abstraction is often easier and simpler to use than channels.

[alandipert]

There is a Clojure port kicking around: https://github.com/hoplon/javelin/blob/master/src/javelin/core_clj.clj

It's prototype-quality and we've never used it for anything, but if you have an idea about using it and want to contribute, we're happy to accept PRs.

[alandipert]

There is a Clojure port kicking around: https://github.com/hoplon/javelin/blob/master/src/javelin/core_clj.clj

It's prototype-quality and we've never used it for anything, but if you have an idea about using it and want to contribute, we're happy to accept PRs.

[bsima]

I saw that and had a feeling that's what that was, but wasn't 100% sure. I'll take a look at it

[bsima]

I saw that and had a feeling that's what that was, but wasn't 100% sure. I'll take a look at it

[onetom]

When this question comes up, @micha and @alandipert usually conclude the discussion by saying they haven't seen a really compelling case yet for using cells on the backend.

I would be very interested to see a really characteristic use case, so @bsima if you have one, please share!

[onetom]

When this question comes up, @micha and @alandipert usually conclude the discussion by saying they haven't seen a really compelling case yet for using cells on the backend.

I would be very interested to see a really characteristic use case, so @bsima if you have one, please share!

[bsima]

@onetom, well I'm currently working on a streaming data analysis service, and I thought having the reactive cell graph would be useful, especially since I'll be working with a lot of graph data structures. A few discussions on SO about Haskell lenses seem to corroborate my assumption (1, 2). I have yet to get to the implementation of the service, still reading and designing the system, so we'll see where my research leads me.

[bsima]

@onetom, well I'm currently working on a streaming data analysis service, and I thought having the reactive cell graph would be useful, especially since I'll be working with a lot of graph data structures. A few discussions on SO about Haskell lenses seem to corroborate my assumption (1, 2). I have yet to get to the implementation of the service, still reading and designing the system, so we'll see where my research leads me.

[jconti]

Seems like anyone doing pub-sub to long-polling streams would like this kind of thing on the server. Basically anyone implementing event-push to web socket or long-polling web clients needs to distribute events and record who has received what. All that state-full book-keeping is a bother, and easy to write bugs into. Mix in an async implementation (which is the only solution to having any kind of scale with long-polling) and the bugs tend to multiply.

[jconti]

Seems like anyone doing pub-sub to long-polling streams would like this kind of thing on the server. Basically anyone implementing event-push to web socket or long-polling web clients needs to distribute events and record who has received what. All that state-full book-keeping is a bother, and easy to write bugs into. Mix in an async implementation (which is the only solution to having any kind of scale with long-polling) and the bugs tend to multiply.

[bsima]

just dumping some notes here for later...

• The Cell Manifesto Kenny Triton's Tilton's original implementation

• Python implementation, Trellis

  • in particular, this section is a good read.

• Another one from Tilton, Cells secret transcript. Great quote:

  I like to decribe the role for Cells being in any situation where one has an unpredictable stream of data and one is keeping a model with a sufficiently large amount of internal state consistent with that stream of inputs.

  This sounds just like what I want to use Javelin in Clojure for. Another good quote:

  A fun note is that I have in the past applied Cells to a database, specifically the old AllegroStore persistent CLOS database. This works two ways. One is that a user can be looking at a screen and as the underlying data changes the screen changes. That may sound like old news but with Cells one doe not have to write any code to make it happen. One just says "this view shows this users overdue books" and when the date changes the overdue status on every book gets updated and a new book appears in the list on the screen if someone happens to be looking, simply by someone having written code to list overdue books on the screen as if it were an unchanging value.

  So the idea is, you encapsulate a database read in a cell on the server, and whenever the database changes, the new value will propagate automatically to your cell. Now imagine hooking this up to a Castra endpoint, which connects on the client to another cell. If the database changes, wouldn't this value propagate all the way to the client? Unless I'm missing something, this sounds a lot like remote sync in other frameworks sans fancy caching, but with way less overhead and complication. Super cool.

It looks like this kind of cell abstraction, in some form or another, was used primarily for UI programming. E.g. Garnet, which predated even Tilton's Cells. However, the case of using cells on the backend is getting stronger the more I read.

[bsima]

just dumping some notes here for later...

• The Cell Manifesto Kenny Triton's Tilton's original implementation

• Python implementation, Trellis

  • in particular, this section is a good read.

• Another one from Tilton, Cells secret transcript. Great quote:

  I like to decribe the role for Cells being in any situation where one has an unpredictable stream of data and one is keeping a model with a sufficiently large amount of internal state consistent with that stream of inputs.

  This sounds just like what I want to use Javelin in Clojure for. Another good quote:

  A fun note is that I have in the past applied Cells to a database, specifically the old AllegroStore persistent CLOS database. This works two ways. One is that a user can be looking at a screen and as the underlying data changes the screen changes. That may sound like old news but with Cells one doe not have to write any code to make it happen. One just says "this view shows this users overdue books" and when the date changes the overdue status on every book gets updated and a new book appears in the list on the screen if someone happens to be looking, simply by someone having written code to list overdue books on the screen as if it were an unchanging value.

  So the idea is, you encapsulate a database read in a cell on the server, and whenever the database changes, the new value will propagate automatically to your cell. Now imagine hooking this up to a Castra endpoint, which connects on the client to another cell. If the database changes, wouldn't this value propagate all the way to the client? Unless I'm missing something, this sounds a lot like remote sync in other frameworks sans fancy caching, but with way less overhead and complication. Super cool.

It looks like this kind of cell abstraction, in some form or another, was used primarily for UI programming. E.g. Garnet, which predated even Tilton's Cells. However, the case of using cells on the backend is getting stronger the more I read.

[bsima]

More notes...

• an old (2008) email thread about cells in Clojure here
• another email thread

A big problem that Rich and Stuart bring up is transactional order. From the first link, a message from Rich:

if agent A notifies B and C, and B notifies C, then
a change to A will send actions to B and C, and B will subsequently
send to C, but A's message to C and B's message to C can arrive in any
order. Once you are ok with that, you can do multithreaded reactive
programming.

I guess core.async was created to get around this problem. I don't know if Javelin has this problem or not, but if it does, a cell abstraction built on top of core.async would be cool.

Alternatively, there is the FrTime language, which is mentioned in the second email thread.

1. Embedding Dynamic Dataflow in a Call-by-Value Language
2. FrTime: Functional Reactive Programming in PLT Scheme
   • ftp://ftp.cs.brown.edu/pub/techreports/03/cs03-20.pdf

A first scan of the papers seems to indicate that FrTime influenced Javelin, since paper 1 talks about a "lift" function and here we see a lift implemented in Javelin. I'll read FrTime more closely this weekend.

There is also an old dataflow api and source, and an SO thread.

Interesting "cells.clj" gist by Rich from 6 years ago.

[bsima]

More notes...

• an old (2008) email thread about cells in Clojure here
• another email thread

A big problem that Rich and Stuart bring up is transactional order. From the first link, a message from Rich:

if agent A notifies B and C, and B notifies C, then
a change to A will send actions to B and C, and B will subsequently
send to C, but A's message to C and B's message to C can arrive in any
order. Once you are ok with that, you can do multithreaded reactive
programming.

I guess core.async was created to get around this problem. I don't know if Javelin has this problem or not, but if it does, a cell abstraction built on top of core.async would be cool.

Alternatively, there is the FrTime language, which is mentioned in the second email thread.

1. Embedding Dynamic Dataflow in a Call-by-Value Language
2. FrTime: Functional Reactive Programming in PLT Scheme
   • ftp://ftp.cs.brown.edu/pub/techreports/03/cs03-20.pdf

A first scan of the papers seems to indicate that FrTime influenced Javelin, since paper 1 talks about a "lift" function and here we see a lift implemented in Javelin. I'll read FrTime more closely this weekend.

There is also an old dataflow api and source, and an SO thread.

Interesting "cells.clj" gist by Rich from 6 years ago.

[alandipert]

@bsima Excellent stuff! I'll throw this in:

This is the paper and FRP library that most directly influenced Javelin: http://cs.brown.edu/~sk/Publications/Papers/Published/mgbcgbk-flapjax/paper.pdf lift comes from Haskell and is the operation in several FRP frameworks that converts "plain" functions to functions over Behaviors/Signals. Behaviors and Signals in these frameworks correspond directly to cells. From the FRP perspective, Javelin is an FRP library with only Behaviors/Signals, and no event streams.

[alandipert]

@bsima Excellent stuff! I'll throw this in:

This is the paper and FRP library that most directly influenced Javelin: http://cs.brown.edu/~sk/Publications/Papers/Published/mgbcgbk-flapjax/paper.pdf lift comes from Haskell and is the operation in several FRP frameworks that converts "plain" functions to functions over Behaviors/Signals. Behaviors and Signals in these frameworks correspond directly to cells. From the FRP perspective, Javelin is an FRP library with only Behaviors/Signals, and no event streams.

[bsima]

Yeah, the FlapJax paper cites the FrTime papers, and the FrTime author Gregory Cooper mentions it on his website and is an author on the FlapJax paper. So that's the basic historic path from Conal Elliot's invention circa 1997, all the way up to the current Javelin implementation.

But there are really two (or three?) historical lines; one for FRP (and/or "dataflow" which is a rather ambiguous term), and one for cells. They aren't really compatible, because cells update synchronously, whereas "dataflow" is typically concerned with managing concurrency, and FRP is concerned with managing asynchronous dataflow.

So we have 3 concepts that are being tangled together and confused. This really helps me understand why the old dataflow API was dropped, and the cell discussions killed, they all mention the problems of concurrency/asynchronicity without unpacking (decomplecting) the problem. Looking at these implementations from 6 years ago, they're rather naive, basically just simplified ports of Tilton's cells. Trying to build cells on top of agents is interesting as it attempts to make synchronous cells work with concurrent processing, but ultimately fails because there is no time constraint, so values could arrive at cells in any order -- cells are meant for sychronicity after all. So, the two relevant solutions to this concurrency problem are:

1. CSP/core.async, time constraints emerge out of the (rather imperative) "channel" abstraction.
2. FrTime & FlapJax impose a time constraint in the evaluation strategy of "signals"

Of course CSP isn't actually reactive, it's just passing messages. If we want reactivity then we need FrTime/FlapJax. (Or it might be possible to implement FrTime-like semanatics using CSP, but that's an experiment for another time.)

The brilliance of Javelin is that it takes the FrTime semantics and wraps it in the cell abstraction, which is like painting a simple interface over the academic lingo of FRP. I think micha mentioned in his 20-minute talk a few weeks ago that Javelin cells are kinda alternative to core.async, but that's not strictly true. Maybe it is in ClojureScript where you don't have true concurrency, but really cells and CSP are complementary and solving two slightly different problems.

Okay, after all this long-winded thinking-out-loud, I'm concluding that Javelin would absolutely be useful in Clojure. I'm gonna finish reading the FrTime papers closely and also read the Javelin code again more closely, and then I'll work on experimenting with Javelin on the JVM next week or so.

[bsima]

Yeah, the FlapJax paper cites the FrTime papers, and the FrTime author Gregory Cooper mentions it on his website and is an author on the FlapJax paper. So that's the basic historic path from Conal Elliot's invention circa 1997, all the way up to the current Javelin implementation.

But there are really two (or three?) historical lines; one for FRP (and/or "dataflow" which is a rather ambiguous term), and one for cells. They aren't really compatible, because cells update synchronously, whereas "dataflow" is typically concerned with managing concurrency, and FRP is concerned with managing asynchronous dataflow.

So we have 3 concepts that are being tangled together and confused. This really helps me understand why the old dataflow API was dropped, and the cell discussions killed, they all mention the problems of concurrency/asynchronicity without unpacking (decomplecting) the problem. Looking at these implementations from 6 years ago, they're rather naive, basically just simplified ports of Tilton's cells. Trying to build cells on top of agents is interesting as it attempts to make synchronous cells work with concurrent processing, but ultimately fails because there is no time constraint, so values could arrive at cells in any order -- cells are meant for sychronicity after all. So, the two relevant solutions to this concurrency problem are:

1. CSP/core.async, time constraints emerge out of the (rather imperative) "channel" abstraction.
2. FrTime & FlapJax impose a time constraint in the evaluation strategy of "signals"

Of course CSP isn't actually reactive, it's just passing messages. If we want reactivity then we need FrTime/FlapJax. (Or it might be possible to implement FrTime-like semanatics using CSP, but that's an experiment for another time.)

The brilliance of Javelin is that it takes the FrTime semantics and wraps it in the cell abstraction, which is like painting a simple interface over the academic lingo of FRP. I think micha mentioned in his 20-minute talk a few weeks ago that Javelin cells are kinda alternative to core.async, but that's not strictly true. Maybe it is in ClojureScript where you don't have true concurrency, but really cells and CSP are complementary and solving two slightly different problems.

Okay, after all this long-winded thinking-out-loud, I'm concluding that Javelin would absolutely be useful in Clojure. I'm gonna finish reading the FrTime papers closely and also read the Javelin code again more closely, and then I'll work on experimenting with Javelin on the JVM next week or so.

[bsima]

I've got an in-progress branch here: bsima/javelin/jvm

I haven't worked much with custom Clojure data structures so this took quite a bit of research, but so far it's going well. I can construct a cell, but set-formula! doesn't work and I'm not sure why yet. All I get is the extremely unhelpful java.lang.ClassCastException: clojure.lang.PersistentArrayMap cannot be cast to clojure.lang.IRef when calling add-watch on a Cell object.

The CLJS code is really dense and undocumented so it takes me a while to figure out exactly what it's doing. But a lot of the code is shared between CLJ and CLJS, so once I get the CLJ working we could consolidate using .cljc

My goal is to get this working by the end of the month. Thinking toward the future: I like the idea of specifying protocols for the cells to implement, because (as I said in the above analyses) the cell abstraction is useful outside of FRP, so perhaps an ICell (or similar) protocol could be used to implement cells with different backends - cells with core.async, or threaded cells, for example.

[bsima]

I've got an in-progress branch here: bsima/javelin/jvm

I haven't worked much with custom Clojure data structures so this took quite a bit of research, but so far it's going well. I can construct a cell, but set-formula! doesn't work and I'm not sure why yet. All I get is the extremely unhelpful java.lang.ClassCastException: clojure.lang.PersistentArrayMap cannot be cast to clojure.lang.IRef when calling add-watch on a Cell object.

The CLJS code is really dense and undocumented so it takes me a while to figure out exactly what it's doing. But a lot of the code is shared between CLJ and CLJS, so once I get the CLJ working we could consolidate using .cljc

My goal is to get this working by the end of the month. Thinking toward the future: I like the idea of specifying protocols for the cells to implement, because (as I said in the above analyses) the cell abstraction is useful outside of FRP, so perhaps an ICell (or similar) protocol could be used to implement cells with different backends - cells with core.async, or threaded cells, for example.

[sjmackenzie]

Really looking forward to this!

[sjmackenzie]

Really looking forward to this!

[alandipert]

@bsima @sjmackenzie would you be interested in a code walkthrough via Hangout or similar? Javelin/cells come from a handful of ideas that code mashes together and obfuscates. I'll have some time this weekend, can probably enlist @micha also.

[alandipert]

@bsima @sjmackenzie would you be interested in a code walkthrough via Hangout or similar? Javelin/cells come from a handful of ideas that code mashes together and obfuscates. I'll have some time this weekend, can probably enlist @micha also.

[cddr]

Ah the cells manifesto. That was my introduction to reactive programming. BTW @bsima the name is Tilton not Triton.

Also, I'd be interested in this hangout too. If you decide to go ahead with it, will you put a link in here?

[cddr]

Ah the cells manifesto. That was my introduction to reactive programming. BTW @bsima the name is Tilton not Triton.

Also, I'd be interested in this hangout too. If you decide to go ahead with it, will you put a link in here?

[alandipert]

@cddr will do, and will mention anyone else who expressed interest.

[alandipert]

@cddr will do, and will mention anyone else who expressed interest.

[sjmackenzie]

Yes indeed it would be quite cool to listen in to that talk.

[sjmackenzie]

Yes indeed it would be quite cool to listen in to that talk.

[bsima]

@alandipert yes please! The Thanksgiving holiday and general work distracted me from working on this, but a code walkthrough would be super helpful and just generally interesting.

Saturday morning or early afternoon would be perfect, I'm on East Coast time.

@cddr oops! Not sure how I made that typo.. thanks :)

[bsima]

@alandipert yes please! The Thanksgiving holiday and general work distracted me from working on this, but a code walkthrough would be super helpful and just generally interesting.

Saturday morning or early afternoon would be perfect, I'm on East Coast time.

@cddr oops! Not sure how I made that typo.. thanks :)

[xificurC]

I'd love to join as well if you choose a good time.
Edit: Oh it's weekend you're thinking of, that won't work for me. Have fun!

[xificurC]

I'd love to join as well if you choose a good time.
Edit: Oh it's weekend you're thinking of, that won't work for me. Have fun!

[jconti]

I'll likely be on a plane, but would love to review a recording of the walk through if possible.

Sent from my iPhone

On Dec 1, 2015, at 12:43 PM, Peter Nagy notifications@github.com wrote:

I'd love to join as well if you choose a good time.

—
Reply to this email directly or view it on GitHub.

[jconti]

I'll likely be on a plane, but would love to review a recording of the walk through if possible.

Sent from my iPhone

On Dec 1, 2015, at 12:43 PM, Peter Nagy notifications@github.com wrote:

I'd love to join as well if you choose a good time.

—
Reply to this email directly or view it on GitHub.

[alandipert]

@xificurC @sjmackenzie @cddr @bsima I made a doodle, please select the time that works best for you: http://doodle.com/poll/z55yyt2cx6s5nuia

I'll try to set up a Hangout on Air and drop the link here when it's ready so we can save the video.

[alandipert]

@xificurC @sjmackenzie @cddr @bsima I made a doodle, please select the time that works best for you: http://doodle.com/poll/z55yyt2cx6s5nuia

I'll try to set up a Hangout on Air and drop the link here when it's ready so we can save the video.

[sjmackenzie]

I have no idea what timezone the doodle times are using. Once decided please put the time + tz here and I'll convert.

[sjmackenzie]

I have no idea what timezone the doodle times are using. Once decided please put the time + tz here and I'll convert.

[alandipert]

@sjmackenzie timezone is EDT

[alandipert]

@sjmackenzie timezone is EDT

[alandipert]

For those joining us this weekend for the code walkthrough, http://cs.brown.edu/~sk/Publications/Papers/Published/mgbcgbk-flapjax/paper.pdf Section 3.4 is a concise description of what we'll be looking at. The rest of the paper also has a lot of useful context. Worth a read, and don't worry, you won't be tested :-)

[alandipert]

For those joining us this weekend for the code walkthrough, http://cs.brown.edu/~sk/Publications/Papers/Published/mgbcgbk-flapjax/paper.pdf Section 3.4 is a concise description of what we'll be looking at. The rest of the paper also has a lot of useful context. Worth a read, and don't worry, you won't be tested :-)

[onetom]

I remember it was also mentioned that cells and data-flow in general is the most useful when it backs UI.
Well, here is an example when it powers a JavaFX game UI which is written in Clojure:
Game Development Development - Michael Nygard & Ragnar Svensson

[onetom]

I remember it was also mentioned that cells and data-flow in general is the most useful when it backs UI.
Well, here is an example when it powers a JavaFX game UI which is written in Clojure:
Game Development Development - Michael Nygard & Ragnar Svensson

[alandipert]

OK the time is set, Sat 12/5/15 3:00 PM EDT. Here is the Hangout on Air link: https://plus.google.com/events/ccnkeqnkgfhv8933s3ff1l2obd0

[alandipert]

OK the time is set, Sat 12/5/15 3:00 PM EDT. Here is the Hangout on Air link: https://plus.google.com/events/ccnkeqnkgfhv8933s3ff1l2obd0

[cddr]

Hey I'm here. Watching. But there's nothing I can type in the google video

:-)

[cddr]

Hey I'm here. Watching. But there's nothing I can type in the google video

:-)

[alandipert]

@cddr change of plans, we're here now: https://unhangout.media.mit.edu/h/TurboComputingFortress

[alandipert]

@cddr change of plans, we're here now: https://unhangout.media.mit.edu/h/TurboComputingFortress

[bsima]

Just a heads up, I'm still working on this. Today reading about Elm's concurrent FRP ideas for possible use in training a neural network.

As for the Javelin code, I think it would be prudent to start with a few minor refactorings:

• Break out the inline fns into more readable code, add docstrings.
• delete some dead and/or duplicate code (notice catch? is in that highlighted code block twice)
• extract the cell API into a protocol. This could eventually be useful for adding core.async channels or maybe manifold streams or some Java lib as a backend. Basically, we could wrap the cell abstraction over whatever we want

I could do all these refactorings over the next few weeks, and it would help me understand the codebase besides. Let me know what you think

[bsima]

Just a heads up, I'm still working on this. Today reading about Elm's concurrent FRP ideas for possible use in training a neural network.

As for the Javelin code, I think it would be prudent to start with a few minor refactorings:

• Break out the inline fns into more readable code, add docstrings.
• delete some dead and/or duplicate code (notice catch? is in that highlighted code block twice)
• extract the cell API into a protocol. This could eventually be useful for adding core.async channels or maybe manifold streams or some Java lib as a backend. Basically, we could wrap the cell abstraction over whatever we want

I could do all these refactorings over the next few weeks, and it would help me understand the codebase besides. Let me know what you think

[actsasgeek]

I'm very heartened to see this discussion. I've been wondering if something like this might not be very useful for data analysis. When doing ETL, it's a bit of a bother in the REPL to go back and fix mistakes or assumptions and then re-execute everything (and error prone at that). I actually though of this the other day and couldn't remember if Javelin was ClojureScript only...and found this.

[actsasgeek]

I'm very heartened to see this discussion. I've been wondering if something like this might not be very useful for data analysis. When doing ETL, it's a bit of a bother in the REPL to go back and fix mistakes or assumptions and then re-execute everything (and error prone at that). I actually though of this the other day and couldn't remember if Javelin was ClojureScript only...and found this.

[cddr]

Hopefully you'll forgive the tangential nature of this comment but one thing I found really useful about Common Lisp when I was doing lots of ETL stuff was the debugger. It is beyond anything in any other language I've used. You could be near the end of an hour long ETL job, and some exception is thrown due to unexpected data. The debugger would allow you to navigate the stack trace to find the offending data, redefine the function that processes it to include a fix for the bug, then re-commence the job where it left off rather than having to start again.

[cddr]

Hopefully you'll forgive the tangential nature of this comment but one thing I found really useful about Common Lisp when I was doing lots of ETL stuff was the debugger. It is beyond anything in any other language I've used. You could be near the end of an hour long ETL job, and some exception is thrown due to unexpected data. The debugger would allow you to navigate the stack trace to find the offending data, redefine the function that processes it to include a fix for the bug, then re-commence the job where it left off rather than having to start again.

[bsima]

Wow, this was a riveting read. Basically explains how to construct a "functional database" out of the same types of cells that Javelin uses

https://en.wikipedia.org/wiki/Functional_Database_Model

[bsima]

Wow, this was a riveting read. Basically explains how to construct a "functional database" out of the same types of cells that Javelin uses

https://en.wikipedia.org/wiki/Functional_Database_Model

[hierophantos]

Has anyone here looked at potentially using propagators as proposed in the very-riveting talk by Gerry Sussman (of Scheme and SICP fame) titled "We Really Don't Know How to Compute!".

There's also a Clojure library implementation called Propaganda.

[hierophantos]

Has anyone here looked at potentially using propagators as proposed in the very-riveting talk by Gerry Sussman (of Scheme and SICP fame) titled "We Really Don't Know How to Compute!".

There's also a Clojure library implementation called Propaganda.

[bsima]

Looks like this aspect of the pulsar library is basically Javelin http://docs.paralleluniverse.co/pulsar/#dataflow-reactive-programming

[bsima]

Looks like this aspect of the pulsar library is basically Javelin http://docs.paralleluniverse.co/pulsar/#dataflow-reactive-programming

[metasoarous]

Has there been any more progress on this?

[metasoarous]

Has there been any more progress on this?

[bsima]

I haven't been working on it unfortunately :(

[bsima]

I haven't been working on it unfortunately :(

[spoerri]

Here's a good academic paper to hang our hat on: https://arxiv.org/abs/1406.2063v1

[spoerri]

Here's a good academic paper to hang our hat on: https://arxiv.org/abs/1406.2063v1

[stathissideris]

I'd like to use javelin for a JavaFX project, could anyone point me to the work in progress so that I can study it and possibly pick it up from where it was left? @bsima ?

[stathissideris]

I'd like to use javelin for a JavaFX project, could anyone point me to the work in progress so that I can study it and possibly pick it up from where it was left? @bsima ?

[bsima]

@stathissideris I haven't been working on it, I'm using Kafka on the backend for dataflow-like programming. You might also want to look at pulsar as it has support for dataflow programming http://docs.paralleluniverse.co/pulsar/#dataflow-reactive-programming

[bsima]

@stathissideris I haven't been working on it, I'm using Kafka on the backend for dataflow-like programming. You might also want to look at pulsar as it has support for dataflow programming http://docs.paralleluniverse.co/pulsar/#dataflow-reactive-programming

[metasoarous]

@stathissideris You also might want to look at Onyx; You can use it for dataflow style programming, it has a very data-driven API, and you can execute your workflows in the browser, on the backend, or even distributed over a zookeeper cluster. The fact that it's built primarily for the latter most of these cases means that the setup is a little more involved than simple cell computing usage in the ballpark of javelin, but I'd bet one could wrap the functionality up in some macros that make it more buttery for this purpose.

[metasoarous]

@stathissideris You also might want to look at Onyx; You can use it for dataflow style programming, it has a very data-driven API, and you can execute your workflows in the browser, on the backend, or even distributed over a zookeeper cluster. The fact that it's built primarily for the latter most of these cases means that the setup is a little more involved than simple cell computing usage in the ballpark of javelin, but I'd bet one could wrap the functionality up in some macros that make it more buttery for this purpose.

[stathissideris]

@bsima @metasoarous Thanks for the suggestions! I like javelin's small footprint, so I think as a first step I'll do a spike to see how easy it would be to port properly to clojure. If I fail, I will investigate pulsar and onyx (which I thought couldn't run without zookeeper!)

[stathissideris]

@bsima @metasoarous Thanks for the suggestions! I like javelin's small footprint, so I think as a first step I'll do a spike to see how easy it would be to port properly to clojure. If I fail, I will investigate pulsar and onyx (which I thought couldn't run without zookeeper!)

[stathissideris]

Wow, hitting a bit of a wall because it of Cljs/Clj differences when it comes to deftype. It seems that Javelin relies on the fact that you can set! fields of existing instances of a deftype, but in Clojure all the fields are immutable by default (there is a a way to override this):

(deftype foobar [field])
(def foo (foobar. 5))
(set! (.-field foo) 100)

Will throw:

IllegalAccessException Can not set final java.lang.Object field datacore.cells.foobar.field to java.lang.Long sun.reflect.UnsafeFieldAccessorImpl.throwFinalFieldIllegalAccessException (UnsafeFieldAccessorImpl.java:76)

This is a way to override this, but it's starting to show that I may need to take this to a slightly different direction for Clojure.

[stathissideris]

Wow, hitting a bit of a wall because it of Cljs/Clj differences when it comes to deftype. It seems that Javelin relies on the fact that you can set! fields of existing instances of a deftype, but in Clojure all the fields are immutable by default (there is a a way to override this):

(deftype foobar [field])
(def foo (foobar. 5))
(set! (.-field foo) 100)

Will throw:

IllegalAccessException Can not set final java.lang.Object field datacore.cells.foobar.field to java.lang.Long sun.reflect.UnsafeFieldAccessorImpl.throwFinalFieldIllegalAccessException (UnsafeFieldAccessorImpl.java:76)

This is a way to override this, but it's starting to show that I may need to take this to a slightly different direction for Clojure.

[bsima]

@stathissideris When I was working on it, I had to set ^volatileMutable (or something) for each field. More problems came up when I was trying to get the tests to pass, some weird errors I could never figure out. My work if you're interested: https://github.com/bsima/javelin/tree/jvm

[bsima]

@stathissideris When I was working on it, I had to set ^volatileMutable (or something) for each field. More problems came up when I was trying to get the tests to pass, some weird errors I could never figure out. My work if you're interested: https://github.com/bsima/javelin/tree/jvm

[stathissideris]

@bsima thanks for the link, I'll have a look. volatileMutable makes the fields of the type mutable but also removes any thread safety that you get with Clojure's "regular" data structures. It's very likely that the weird errors you were getting were due to race conditions or other similar problems.

[stathissideris]

@bsima thanks for the link, I'll have a look. volatileMutable makes the fields of the type mutable but also removes any thread safety that you get with Clojure's "regular" data structures. It's very likely that the weird errors you were getting were due to race conditions or other similar problems.

[alandipert]

6 months ago I made a lot of progress on the cljc branch.

I can't remember exactly how far I got, but I think... kinda far? I used refs for the mutable fields.

I ended up pretty disgusted by the way it came out. I think it's much better to have separate Clojure and ClojureScript namespaces, or perhaps even different Clojars artifacts.

So, the cljc work wouldn't be helpful as-is, but if you can figure out a way to spit out .clj from the .cljc, you'd have a decent starting point that's more up-to-date than the existing javelin_clj.clj.

[alandipert]

6 months ago I made a lot of progress on the cljc branch.

I can't remember exactly how far I got, but I think... kinda far? I used refs for the mutable fields.

I ended up pretty disgusted by the way it came out. I think it's much better to have separate Clojure and ClojureScript namespaces, or perhaps even different Clojars artifacts.

So, the cljc work wouldn't be helpful as-is, but if you can figure out a way to spit out .clj from the .cljc, you'd have a decent starting point that's more up-to-date than the existing javelin_clj.clj.

[stathissideris]

@alandipert thanks, I had a look at that branch and started uncommenting some of the tests to see how far you got (most of them did pass!). I have a question about dosync*: it relies on binding which modifies the bindings per-thread. This is not relevant to cljs (because it has one thread), but could it prove problematic for the JVM?

[stathissideris]

@alandipert thanks, I had a look at that branch and started uncommenting some of the tests to see how far you got (most of them did pass!). I have a question about dosync*: it relies on binding which modifies the bindings per-thread. This is not relevant to cljs (because it has one thread), but could it prove problematic for the JVM?

[flyboarder]

Self hosted javelin would be really awesome and also affects the cljc thing

[flyboarder]

Self hosted javelin would be really awesome and also affects the cljc thing

[alandipert]

@stathissideris oh yeah: that's one of the things I handwaved.

Reflecting on it now, I'm not sure Javelin needs to have its own dosync in Clojure if refs are used.

Reasoning chain:

On ClojureScript, binding is the way to establish a global dynamic scope. In Clojure the binding is scoped to the thread, as you point out.

ClojureScript doesn't have ref or dosync. There, binding is thin veneer over setting a global variable and then setting it back, which is what we needed to make sure dosync nested appropriately.

But in Clojure, dosync is thread-global: even refs in different threads participate in the same transaction. I think it makes sense to apply to cells the same semantics that already apply to refs. At least, I can't think of a good reason for them to behave differently.

[alandipert]

@stathissideris oh yeah: that's one of the things I handwaved.

Reflecting on it now, I'm not sure Javelin needs to have its own dosync in Clojure if refs are used.

Reasoning chain:

On ClojureScript, binding is the way to establish a global dynamic scope. In Clojure the binding is scoped to the thread, as you point out.

ClojureScript doesn't have ref or dosync. There, binding is thin veneer over setting a global variable and then setting it back, which is what we needed to make sure dosync nested appropriately.

But in Clojure, dosync is thread-global: even refs in different threads participate in the same transaction. I think it makes sense to apply to cells the same semantics that already apply to refs. At least, I can't think of a good reason for them to behave differently.

[stathissideris]

So I've been doing some reading and decided to have a go at implementing cells in Clojure from scratch. Here is my effort so far (be warned, this is work in progress!):

https://github.com/stathissideris/datacore/blob/518abe53729661e7c9d6a217fa1dd64b2ce88466/src/datacore/cells.clj

And here are some tests to show how it behaves:

https://github.com/stathissideris/datacore/blob/518abe53729661e7c9d6a217fa1dd64b2ce88466/test/datacore/cells_test.clj

This implementation is closer to how I think Kenny Tilton's rube works (although I've yet to read his actual code). Tilton is the original developer of cells in CL.

The main differences of my implementation to Javelin are:

1. No static code analysis. Links between cells are established the first time a formula cell is calculated, and the library "becomes aware" that in that context other cells are deref-ed. This is similar to rube.
2. There is a central registry of cells, their values, and dependencies between them. This could be used to visualize state and the links.
3. No lenses.
4. Formula cells need to refer to other cells using deref. In my opinion this makes the code look more like normal Clojure code (which it is!). See tests for examples.

Check the propagate function, it's very similar to Javelin's!

I'm planning to use this as part of my java-fx project and see if it's enough for my needs. For now, I'm glad that it's a simpler implementation than Javelin.

Any feedback welcome!

[stathissideris]

So I've been doing some reading and decided to have a go at implementing cells in Clojure from scratch. Here is my effort so far (be warned, this is work in progress!):

https://github.com/stathissideris/datacore/blob/518abe53729661e7c9d6a217fa1dd64b2ce88466/src/datacore/cells.clj

And here are some tests to show how it behaves:

https://github.com/stathissideris/datacore/blob/518abe53729661e7c9d6a217fa1dd64b2ce88466/test/datacore/cells_test.clj

This implementation is closer to how I think Kenny Tilton's rube works (although I've yet to read his actual code). Tilton is the original developer of cells in CL.

The main differences of my implementation to Javelin are:

1. No static code analysis. Links between cells are established the first time a formula cell is calculated, and the library "becomes aware" that in that context other cells are deref-ed. This is similar to rube.
2. There is a central registry of cells, their values, and dependencies between them. This could be used to visualize state and the links.
3. No lenses.
4. Formula cells need to refer to other cells using deref. In my opinion this makes the code look more like normal Clojure code (which it is!). See tests for examples.

Check the propagate function, it's very similar to Javelin's!

I'm planning to use this as part of my java-fx project and see if it's enough for my needs. For now, I'm glad that it's a simpler implementation than Javelin.

Any feedback welcome!

[alandipert]

@stathissideris cool!

Re: 1, when you say that Javelin does static code analysis, are you talking about the cell= macro? If so, it's worth noting that the cell= macro doesn't know anything about the relationships between cells. It just generates code that will build the cell graph at runtime. It does leverage a few heuristics to minimize the number of cells that are created when the code does run.

Basically, it transforms expressions like (cell= (+ a b)) into ((formula +) a b), where formula does all the work, like recognizing if a or b are cells, and linking them up if so.

If that differs from your perception about what cell= does it would be helpful for me to know, because you won't have been the first to be misled. It's on my short list to clean up the Javelin README and de-emphasize cell=. Actually, thank you in advance for any feedback you might have about the README. It's kind of... overparticular.

Re: 4, interesting choice! I can see the appeal of visually demarcating references to cells in formulas. Javelin gained a macro for doing something similar recently, formula-of

Anyway you're doing really cool work and I enjoy keeping up with it. Keep us posted 😄

[alandipert]

@stathissideris cool!

Re: 1, when you say that Javelin does static code analysis, are you talking about the cell= macro? If so, it's worth noting that the cell= macro doesn't know anything about the relationships between cells. It just generates code that will build the cell graph at runtime. It does leverage a few heuristics to minimize the number of cells that are created when the code does run.

Basically, it transforms expressions like (cell= (+ a b)) into ((formula +) a b), where formula does all the work, like recognizing if a or b are cells, and linking them up if so.

If that differs from your perception about what cell= does it would be helpful for me to know, because you won't have been the first to be misled. It's on my short list to clean up the Javelin README and de-emphasize cell=. Actually, thank you in advance for any feedback you might have about the README. It's kind of... overparticular.

Re: 4, interesting choice! I can see the appeal of visually demarcating references to cells in formulas. Javelin gained a macro for doing something similar recently, formula-of

Anyway you're doing really cool work and I enjoy keeping up with it. Keep us posted 😄

[alandipert]

@stathissideris forgot to mention, clojure has an IAtom interface now, which you could extend instead of having your own swap!

[alandipert]

@stathissideris forgot to mention, clojure has an IAtom interface now, which you could extend instead of having your own swap!

[stathissideris]

@alandipert

Maybe static analysis is too grandiose a term, I meant the fancy code walking necessary for the hoisting of the args for cell=. My implementation does not do that and as a consequence, the linking happens when a sink is first dereferenced, instead of when the cell is constructed (which is what happens in Javelin). I tried using my cells in practice and I was caught off guard a couple of times by this limitation, so it may be worth having some code walking after all.

About dosync: A Clojure version of Javelin could use Clojure's dosync but because transactions within dosync can be retried, that would potentially mean that cell= could no longer used reliably for side effects, despite the propagate function being smart enough to only visit each formula cell once, because if the transaction is tried more than once you'd get the same side effect more than once. I was thinking of a way around this, and I'm leaning towards having a 3rd type of cell (maybe called effect=) which would be run after all the formula cell changes have been propagated, outside a transaction, and would guarantee at most one execution per swap!.

I'm aware of IAtom but I'm not sure yet whether I want my cells to look like atoms completely - still deciding on the interface! I implemented IRef to get the convenience of typing @.

Did a bit more work today for cycle detection. :)

[stathissideris]

@alandipert

Maybe static analysis is too grandiose a term, I meant the fancy code walking necessary for the hoisting of the args for cell=. My implementation does not do that and as a consequence, the linking happens when a sink is first dereferenced, instead of when the cell is constructed (which is what happens in Javelin). I tried using my cells in practice and I was caught off guard a couple of times by this limitation, so it may be worth having some code walking after all.

About dosync: A Clojure version of Javelin could use Clojure's dosync but because transactions within dosync can be retried, that would potentially mean that cell= could no longer used reliably for side effects, despite the propagate function being smart enough to only visit each formula cell once, because if the transaction is tried more than once you'd get the same side effect more than once. I was thinking of a way around this, and I'm leaning towards having a 3rd type of cell (maybe called effect=) which would be run after all the formula cell changes have been propagated, outside a transaction, and would guarantee at most one execution per swap!.

I'm aware of IAtom but I'm not sure yet whether I want my cells to look like atoms completely - still deciding on the interface! I implemented IRef to get the convenience of typing @.

Did a bit more work today for cycle detection. :)

[alandipert]

@stathissideris Hm, I don't see what you do about the relationship between code-walking and cell attachment. In the expression ((formula +) a b),which evaluates to a formula cell, formula is a function, not a macro.

Re: dosync, it's for coordinating changes to input cells, not formula cells -- the values of which can't be set (ignoring lenses). So, assuming add-watch is what's used to detect changes in refs and potentially trigger more propagation if any ref's values have changed, the watches would all be fired after the transaction and its possible retries are completed.

The other required aspect for dosync to work the way I imagine is that formula cells need to be "strict". That is, they need to acquire values as soon as they're created, not when first derefed. Otherwise, propagation would need to happen in the transaction, which does indeed seem bad.

Anyway, it seems like it should work... but you've inspired me to experiment a little because I'm not sure. I'll post whatever I find 👍

[alandipert]

@stathissideris Hm, I don't see what you do about the relationship between code-walking and cell attachment. In the expression ((formula +) a b),which evaluates to a formula cell, formula is a function, not a macro.

Re: dosync, it's for coordinating changes to input cells, not formula cells -- the values of which can't be set (ignoring lenses). So, assuming add-watch is what's used to detect changes in refs and potentially trigger more propagation if any ref's values have changed, the watches would all be fired after the transaction and its possible retries are completed.

The other required aspect for dosync to work the way I imagine is that formula cells need to be "strict". That is, they need to acquire values as soon as they're created, not when first derefed. Otherwise, propagation would need to happen in the transaction, which does indeed seem bad.

Anyway, it seems like it should work... but you've inspired me to experiment a little because I'm not sure. I'll post whatever I find 👍

[stathissideris]

@alandipert I realize that formula is a function. I was just trying to implement a syntax similar to your cell= in my implementation, but because I ended up doing it with a function rather than a macro (to avoid code walking), I also have the tradeoff of resolving links at first deref instead of at the time of cell creation. Maybe it's better to accept that the slightly less convenient syntax of formula is preferable to having links be resolved "late".

About propagation: I'm confused, it seems that add-watch is used, but it seems that propagate! is called when doing reset! and it looks like propagate! does take care of formula cells ((if-let [f (x/get (.-thunk next))] (f) (x/get (.-state next)))!?

About dosync: in a multithreaded environment, I think it's an advantage to have swap! and the subsequent propagation of values happen atomically, within the same transaction. Imagine what would happen if you are in the middle of a propagation and another swap! happened on a different input cell. You'd get interleaving propagations happening at the same time, leaving your graph in an inconsistent state. Multiple threads change everything! So I think swap!+propagation should be in a transaction (like in my implementation), and side effects could be in special "output" or "effect" cells.

I'm glad you're feeling inspired! :-)

[stathissideris]

@alandipert I realize that formula is a function. I was just trying to implement a syntax similar to your cell= in my implementation, but because I ended up doing it with a function rather than a macro (to avoid code walking), I also have the tradeoff of resolving links at first deref instead of at the time of cell creation. Maybe it's better to accept that the slightly less convenient syntax of formula is preferable to having links be resolved "late".

About propagation: I'm confused, it seems that add-watch is used, but it seems that propagate! is called when doing reset! and it looks like propagate! does take care of formula cells ((if-let [f (x/get (.-thunk next))] (f) (x/get (.-state next)))!?

About dosync: in a multithreaded environment, I think it's an advantage to have swap! and the subsequent propagation of values happen atomically, within the same transaction. Imagine what would happen if you are in the middle of a propagation and another swap! happened on a different input cell. You'd get interleaving propagations happening at the same time, leaving your graph in an inconsistent state. Multiple threads change everything! So I think swap!+propagation should be in a transaction (like in my implementation), and side effects could be in special "output" or "effect" cells.

I'm glad you're feeling inspired! :-)

[stathissideris]

@alandipert Just thought to give you a little demo of the debugging facilities:

(def a (cell 100))
(def b (cell 2))
(def c (formula (partial * 2) a b))
(def d (formula (partial * 10) c))

(print-cells (all-cells))

Prints:

| :id | :label | :formula? | :value | :error | :code | :sinks | :sources |
|-----+--------+-----------+--------+--------+-------+--------+----------|
|   0 |        |     false |    100 |        |       |   #{2} |          |
|   1 |        |     false |      2 |        |       |   #{2} |          |
|   2 |        |      true |    400 |        |       |   #{3} |   #{0 1} |
|   3 |        |      true |   4000 |        |       |        |     #{2} |

[stathissideris]

@alandipert Just thought to give you a little demo of the debugging facilities:

(def a (cell 100))
(def b (cell 2))
(def c (formula (partial * 2) a b))
(def d (formula (partial * 10) c))

(print-cells (all-cells))

Prints:

| :id | :label | :formula? | :value | :error | :code | :sinks | :sources |
|-----+--------+-----------+--------+--------+-------+--------+----------|
|   0 |        |     false |    100 |        |       |   #{2} |          |
|   1 |        |     false |      2 |        |       |   #{2} |          |
|   2 |        |      true |    400 |        |       |   #{3} |   #{0 1} |
|   3 |        |      true |   4000 |        |       |        |     #{2} |

[alandipert]

Re: synchronization, I think I understand now, thanks for clarifying. Threads: they change it all!

Re "effect cells", if I understand them as you do, they seem very much like watches, but with the added semantic that they're only called when the dependent cell changes value. As opposed to cells, and more like watches, they wouldn't contain a value and couldn't be depended on directly. If you wanted to retain a value from the effect, you'd swap!/reset! a real cell with it.

So, when these watches are called, the old/new value arguments would always be different. That way you'd have cell-like value dedupe, but all the other properties would be those of watches as they exist already.

Is all of that in-line with what you are thinking?

Re: cell debug - love the table! I can't think of a downside to keeping a list of the cells. An even cooler thing you could do on the JVM is pop a real data grid in a frame, with the ability to edit cell values.

I'm working in a similar area for a work project, based on something I made years ago: http://tailrecursion.com/~alan/ttt/basic.html

We have thousands of cells in the project and it's getting hard to know what's going on, so I'm going to revive this visualization, but include cell names & source file locations when known. I can imagine those would be useful to see in your table also.

[alandipert]

Re: synchronization, I think I understand now, thanks for clarifying. Threads: they change it all!

Re "effect cells", if I understand them as you do, they seem very much like watches, but with the added semantic that they're only called when the dependent cell changes value. As opposed to cells, and more like watches, they wouldn't contain a value and couldn't be depended on directly. If you wanted to retain a value from the effect, you'd swap!/reset! a real cell with it.

So, when these watches are called, the old/new value arguments would always be different. That way you'd have cell-like value dedupe, but all the other properties would be those of watches as they exist already.

Is all of that in-line with what you are thinking?

Re: cell debug - love the table! I can't think of a downside to keeping a list of the cells. An even cooler thing you could do on the JVM is pop a real data grid in a frame, with the ability to edit cell values.

I'm working in a similar area for a work project, based on something I made years ago: http://tailrecursion.com/~alan/ttt/basic.html

We have thousands of cells in the project and it's getting hard to know what's going on, so I'm going to revive this visualization, but include cell names & source file locations when known. I can imagine those would be useful to see in your table also.

[alandipert]

Oops, one difference between cells and watches I overlooked is an obvious one, that cells can depend on many other cells, but watches can depend on only one reference type. So the other feature an "effect cell" would need that watches don't do, is the ability to depend on multiple cells.

I suppose one way to do this would be just to make a formula cell, and then add a watch to it, but maybe there are limitations to that I'm missing. It's definitely less convenient.

[alandipert]

Oops, one difference between cells and watches I overlooked is an obvious one, that cells can depend on many other cells, but watches can depend on only one reference type. So the other feature an "effect cell" would need that watches don't do, is the ability to depend on multiple cells.

I suppose one way to do this would be just to make a formula cell, and then add a watch to it, but maybe there are limitations to that I'm missing. It's definitely less convenient.

[stathissideris]

@alandipert Sorry for the delay, didn't get an email notification and I thought that the thread had died off.

About effect cells: I think you're right in how you've described them.

About the debug table: my interest in cells is because I'd like to make a small ETL/Excel-like JavaFX app, and anything Excel-like needs cells by definition! So I'm thinking of using cells as the overarching metaphor for data processing and the UI as well. Because of this uniformity, I should be able to use the UI both for processing data and for debugging/developing the app itself, so the interactive table that you described is one of the early targets.

Very cool graph visualisation, the fact that it's interactive caught me off guard :-)

I got my cells to the point where all the operations are done in an immutable way (so you can have a graph of cells that's backed by an immutable data structure) and then there are a few thin convenience functions that use the immutable API to mutate the global-cells atom. I hope that having an immutable core will allow me to be more adventurous with the operations that I will support, such as "muting" certain cells (making them "passthrough"). Latest code and tests here:

https://github.com/stathissideris/datacore/blob/62eda26a4e51c3f4c935b02e5d7106831905f292/src/datacore/cells.clj

https://github.com/stathissideris/datacore/blob/62eda26a4e51c3f4c935b02e5d7106831905f292/test/datacore/cells_test.clj

Immutable API shown here:

https://github.com/stathissideris/datacore/blob/62eda26a4e51c3f4c935b02e5d7106831905f292/test/datacore/cells_test.clj#L56-L63

I'm starting to think I should extract this from the app and release it as a separate library.

[stathissideris]

@alandipert Sorry for the delay, didn't get an email notification and I thought that the thread had died off.

About effect cells: I think you're right in how you've described them.

About the debug table: my interest in cells is because I'd like to make a small ETL/Excel-like JavaFX app, and anything Excel-like needs cells by definition! So I'm thinking of using cells as the overarching metaphor for data processing and the UI as well. Because of this uniformity, I should be able to use the UI both for processing data and for debugging/developing the app itself, so the interactive table that you described is one of the early targets.

Very cool graph visualisation, the fact that it's interactive caught me off guard :-)

I got my cells to the point where all the operations are done in an immutable way (so you can have a graph of cells that's backed by an immutable data structure) and then there are a few thin convenience functions that use the immutable API to mutate the global-cells atom. I hope that having an immutable core will allow me to be more adventurous with the operations that I will support, such as "muting" certain cells (making them "passthrough"). Latest code and tests here:

https://github.com/stathissideris/datacore/blob/62eda26a4e51c3f4c935b02e5d7106831905f292/src/datacore/cells.clj

https://github.com/stathissideris/datacore/blob/62eda26a4e51c3f4c935b02e5d7106831905f292/test/datacore/cells_test.clj

Immutable API shown here:

https://github.com/stathissideris/datacore/blob/62eda26a4e51c3f4c935b02e5d7106831905f292/test/datacore/cells_test.clj#L56-L63

I'm starting to think I should extract this from the app and release it as a separate library.

[dm3]

hey, I've done some work on https://github.com/dm3/javelin/tree/macros-split-clj. It now has a Clojure implementation based loosely on Plumbata. The implementation is not concurrency-safe. All the cell's fields are stored in refs, so on conflicting transactions any watches set on cells might be retried. However, when run from a single thread the implementation passes all of the Javelin tests.

The tests also pass in self-host Clojurescript mode. To try that install Planck or Lumo and run:

planck -c src/:test/:~/.m2/repository/net/cgrand/macrovich/0.2.0/macrovich-0.2.0.jar
> (require '[javelin.core-test :as ct])

The big problem is I can’t figure out how to make the codebase work under normal JVM Clojurescript again!

The only significant change I made was splitting macros out into a separate namespace - javelin.macros. I think the macros can be returned into javelin.core but I’d like to figure out JVM Clojurescript first… Any help greatly appreciated!

[dm3]

hey, I've done some work on https://github.com/dm3/javelin/tree/macros-split-clj. It now has a Clojure implementation based loosely on Plumbata. The implementation is not concurrency-safe. All the cell's fields are stored in refs, so on conflicting transactions any watches set on cells might be retried. However, when run from a single thread the implementation passes all of the Javelin tests.

The tests also pass in self-host Clojurescript mode. To try that install Planck or Lumo and run:

planck -c src/:test/:~/.m2/repository/net/cgrand/macrovich/0.2.0/macrovich-0.2.0.jar
> (require '[javelin.core-test :as ct])

The big problem is I can’t figure out how to make the codebase work under normal JVM Clojurescript again!

The only significant change I made was splitting macros out into a separate namespace - javelin.macros. I think the macros can be returned into javelin.core but I’d like to figure out JVM Clojurescript first… Any help greatly appreciated!

[flyboarder]

@dm3 you probably want something like this for the cljs macros

(ns javelin.core (:require [javelin.core :include-macros true]))

[flyboarder]

@dm3 you probably want something like this for the cljs macros

(ns javelin.core (:require [javelin.core :include-macros true]))

[onetom]

A bit more info on this "macros from cljs" topic: https://github.com/clojure/clojurescript/wiki/Differences-from-Clojure#namespaces In latest versions of the CLJS compiler, you can :require-macros within a cljs NS and when that NS is required by another cljs NS, macros are implicitly available for :referring.

…

-- tom

[onetom]

A bit more info on this "macros from cljs" topic: https://github.com/clojure/clojurescript/wiki/Differences-from-Clojure#namespaces In latest versions of the CLJS compiler, you can :require-macros within a cljs NS and when that NS is required by another cljs NS, macros are implicitly available for :referring.

…

-- tom

[dm3]

@flyboarder @onetom thanks, I'm well aware of the basics. The problems I was struggling with are a bit more nuanced.

I think it's the compilation of macros injavelin.macros which chooses a wrong code walking namespace when in JVM Clojurescript as discussed on Slack with flyboarder. However, I haven't inspected the generated javascript, so the assumption may be wrong.

[dm3]

@flyboarder @onetom thanks, I'm well aware of the basics. The problems I was struggling with are a bit more nuanced.

I think it's the compilation of macros injavelin.macros which chooses a wrong code walking namespace when in JVM Clojurescript as discussed on Slack with flyboarder. However, I haven't inspected the generated javascript, so the assumption may be wrong.