Concurrency in Lux

[LuxLang]

I've been thinking for a while on what the right concurrency primitives should be for Lux.
There are a few considerations the design should take into account in order to satisfy all that's needed.

1. The library/model should work on all of the supported platforms for Lux in the same way.
2. Lux should ideally provide the most primitive, but usable, concurrency model, so that more advanced infrastructure can easily be built on top.

There were 5 models I considered while doing research and design for concurrency.

1. Software Transactional Memory (STM)
2. Promises/Futures
3. Communicating Sequential Processes (CSP)
4. Functional-Reactive Programming (FRP)
5. Actor model

Thinking about those models, I was first considering which models could be implemented on top of other ones, hoping that many ways of doing concurrency could be supported while doing minimal base work on Lux.

Actor model on top of of CSP

This one seems to me to be the easiest one to do.
All that would be needed to do that is to have some form of ID assigned to each CSP process and to assign channels for the exclusive use of processes to serve as mailboxes.
Finally, some error-handling & monitoring mechanisms would need to be added to handle the failure/crash of actors.

Actors, however, seem a bit too heavyweight to be a concurrency mechanism for programs.
That being said, I do think they would be an excellent choice for doing distributed programming, as having identity for processes means that you could have actors on one machine communicating with actors on another machine.

Because of that, I think it would be best to leave actors outside of the Lux standard library in order to implement them in a more heavyweight distributed programming framework.

Simplifying Promises and Futures

From what I've seen, languages that have both seem to show a somewhat confusing picture.
The story goes like this:
A future points to a value that will eventually be made available.
Futures are read-only.
A promise also points to a value that will eventually be made available.
Promises can be written to.
You set the value of a future by writing to its associated promise.

For those of you familiar with Clojure's promises and futures, that story might sound a bit weird.
You only know that promises are little black boxes you can write once to, while futures run code whose return value is the value of the future.
On the inside, the story is the same as the one above, even if it looks a bit simpler.

I think this whole divide between futures and promises sounds a bit ridiculous, so I'd rather simplify things a bit.

I'm thinking of having a type (Async a). It would be implemented natively on every platform Lux runs. You will only be able to set it once.
If you try to get it's value and it's already set, you'll just get it.
Otherwise, you'll give it a callback that will be run once the value has been set.

Async will have both Functor and Monad implementations, with the idea of making asynchronous computation feel synchronous (having the same effect that you get from using Clojure's core.async).

Lux will also provide a thread-pool that will be used to run the callbacks every time a promise is set.
Functions such as timeouts/delays, and I/O would return Async values in order to async-io.
Processes could communicate with each other by messaging through Async.

A function called future could transform (IO a) into (Async a), to make interfacing with synchronous code easy,

CSP on top of Async

Traditionally, CSP relies on channels to do inter-process communication.
As flexible and cool as that might be, channels are actually unnecessary. You can do all of your communication and synchronization just using Async values.

With that said, channels could be implemented on top of Async:

(deftype (Chan a)
  (Async (Maybe (, a (Chan a)))))

The use of Maybe means channels can be closed by just setting an Async to #None.
Channels implemented this way would be unbounded, unlike channels in other languages; though I don't see that causing much of a problem.

Finally, it will be possible to access old/previous values in a channel by just holding on to previous nodes.

Any CSP process that needed to constantly receive inputs could just process a channel step-by-step, dropping older nodes and only using the latest one if it doesn't need to remember the past.

FRP on top of Async

FRP is much easier to do on top of Async (by using channels), as it just needs to piggy-back on the Functor implementation for Chan, or on the Monad implementation, if fancier work needs to be done.

All that would be necessary is adding the functions and combinators commonly expected for FRP.

STM ???

Actors can be done on top of CSP.
CSP and FRP can be done on top of Async (our little promise/future hybrid).

What happens now with STM?

The thing with STM is that it's fundamentally different from all the other ones.
Promises/futures, CSP and Actors can all be seen as ways to get processes to communicate with each other.
FRP can be interpreted as the same thing, assuming that transformations on channels will be done concurrently by processes hosted by the thread-pool.

STM is not really about communication.
STM is about how do you get multiple processes to bang on shared state without breaking anything.

For that reason, I get the feeling implementing STM on top of promises is probably not feasible (at least with anything resembling good performance).

Now... I'll be honest. I've never used Clojure's STM in any significant way.
I just used atoms for everything until core.async arrived, and then CSP became my hammer of choice.

For that reason, I'm somewhat biased to just dismiss STM and go on the Async (promisses) route.

This is still an open issue

I'm pretty confident that going in the Async route and forgetting STM is a good way to go, but there's nothing set on stone right now.

I made this repo issue to see if there were any ideas anyone else might have regarding concurrency in Lux.

Is it a bad idea to implement CSP & FRP on top of promises?

Is STM more useful and necessary than I think?

Feel free to give your 2 cents.

I plan on implementing promises, CSP and FRP for version 0.3.2, provided that I stay on the path described on this issue.

Here's your chance to alter the course of history.

[wavebeem]

It's extra confusing when you think about JS-land where "promise" is the readable portion of a future-like thing, and the constructor function is the writable portion.

When you get around to making Lux compile to JS, there will need to be at least support for promises to deal with native APIs returning them.

Can the other approaches be implemented in a single-threaded environment like JS?

[LuxLang]

Can the other approaches be implemented in a single-threaded environment like JS?

For that, I'd just need to do the same tricks ClojureScript uses for its core.async implementations.
It's definitely do-able.

When you get around to making Lux compile to JS, there will need to be at least support for promises to deal with native APIs returning them.

That could be done by having a function for translating JS's promises to Lux's.
Another alternative would be to just use a JS-native promise implementation (though I don't know how feasible that would be, as I don't know if there are JS-promises in all modern browsers...)

[wavebeem]

Promise support is decent but not perfect http://caniuse.com/#search=promise

Using native has the benefit of better console debugging, so it might be worth including an optional shim and otherwise using native.