Highland.js Integration

[svozza]

We had a bit of a discussion here about possibly integrating Highland streams into Ramda so I said I'd create an issue here explore the issue more fully. @caolan has been very busy recently so I'm not sure if he has time to get involved but after him @vqvu probably knows the code better than anyone else - not to mention @jeromew, @LewisJEllis and @quarterto - and I'm sure we can all help in some way. What do you have in mind and what would you need from our side?

[caolan]

Ramda looks fun, I'll have to take it for a spin :) Seems a good match for Highland. Perhaps it would be a good opportunity to produce a more 'skinny' Highland core for you to integrate too...

[buzzdecafe]

some background: we already can dynamically dispatch to non-list objects in list position. That's how we support transducers, and how we are implementing the fantasy-land spec. so in some (many?) ways highland may just drop right in with ramda and work, e.g., map(add(1), stream) should be the same as stream.map(add(1)). But I am not familiar enough with highland api to know how well integrated we are. also, we will probably need to expand the number of functions that dispatch in this fashion.

So I'd like to firm this up s.t. you can just drop it in and start using it point-free.

full disclosure: I'm working on a relational programming thing that i want to recast to use streams. so this is personally useful to me 😸

[svozza]

Yeah, map works fine but unfortunately reduce doesn't because Highland's version has the arguments in a different order: _.reduce(memo, iterator). Really don't know how we can get around that one.

[vqvu]

Interesting...so we'll need to figure out a list of Highland <-> Ramda operator equivalencies. We'd also have to make sure their semantics match.

One possible issue is that sometimes the names don't match between the two libraries. For instance, R#forEach === hl#each. Perhaps we can add aliases on our side for some of these inconsistencies, and Ramda can do the _dispatchable magic.

Plus, due to the nature of the two libraries, Ramda implements more operators than Highland does (e.g., findFirst, findLast). Maybe we can dispatch to the native HL transform if it exists and use transducers if it doesn't? Or just use transducers everywhere (the native transform is probably slightly faster in the sync case).

[kevinbeaty]

but unfortunately reduce doesn't because Highland's version has the arguments in a different order:

The Ramda implementation of reduce (and by extension transduce and into) also delegates to a reduce(rf, init) method internally which would almost work directly except the signature of Highland's reduce is reversed. What are the chances of changing the argument order in Highland's reduce? Too late?

One thing I proposed when discussing the transformer protocol was to use @@transducer/reduce as a marker for a reduce function with known arguments for use in eduction. It was implemented in transduce, but, unfortunately, was not (yet) formalized in the protocol. If it were formalized, Ramda could potentially prefer dispatch to @@transducer/reduce over reduce in the implementation and Highland could add a @@transducer/reduce implementation that has the correct order. Then into should "just work".

Ping @tgriesser as he may have thoughts on this as well.

[svozza]

Yeah, if we were still in the pre-1.0.0 phase we could probably make the case for changing the parameter order of reduce but I don't think it's really an option now. I like your transducer approach though, could be an elegant solution.

I've compiled a list of Highland transform functions and their equivalents in Ramda excluding a few of the more stream specific ones though:

hl#append === R#append
hl#batch === R#???
hl#collect === R#of
hl#compact === R#???
hl#drop === R#drop
hl#filter === R#filter
hl#find === R#find
hl#findWhere === R#???
hl#flatten === R#flatten
hl#group === R#groupBy
hl#head === R#head
hl#intersperse === R#???
hl#invoke === R#invoke
hl#keys === R#keys
hl#last === R#last
hl#map === R#map
hl#pairs === hl#toPairs
hl#pick === R#pick
hl#pickBy === R#pickBy || R#filterObjIndexed
hl#pluck === R#pluck
hl#reduce === R#reduce
hl#reduce1 === R#???
hl#reject === R#reject
hl#scan === R#???
hl#scan1 === R#???
hl#sortBy === R#sortBy
hl#split === R#???
hl#splitBy === R#split
hl#take === R#take
hl#transduce === R#transduce
hl#uniq === R#uniq
hl#uniqBy === R#uniqWith
hl#values === R#values
hl#where === R#where
hl#zip === R#zip
hl#zipAll === R#???

• Initially I though hl#batch was the same as R#aperture but it's not:

var xs = [1, 2, 3, 4, 5];
hl.batch(2, hl(xs));  // => [1, 2], [3, 4], [5]
R.aperture(2, xs); //=> [[1, 2], [2, 3], [3, 4], [4, 5]]

• I'm guessing hl#compact will probably never have an equivalent Ramda implementation.
• hl#findWhere is a convenient form of hl#where that takes a spec object rather than a function and returns the first matching item. Again, I can't see this making it into Ramda.
• hl#group could be problematic because it accepts either a string or function, it should probably have been split into hl#group and hl#groupBy.
• hl#intersperse is effectively the inverse of hl#splitBy.

var xs = hl(['ba', 'a', 'a']);
hl.intersperse('n', xs);  // => ba, n, a, n, a

• pickBy || filterObjIndexed could have issues because the function passed to it in Highland is f(key, value) whereas Ramda is f(value, key).
• hl#reduce suffers from the argument order problem: hl#reduce(memo, iterator) vs R#reduce(iterator, memo)
• hl#reduce1 is basically just reduce with the memo set to the first value of the stream, I vaguely remember Ramda having a similar function but I'm guessing it was axed.
• hl#scan is a handy version of reduce when used with streams, it emits each intermediate value of the reduction as it is calculated.
• hl#scan1 is scan with the memo set to the first value of the stream.
• hl#split splits on new line characters.
• hl#transduce only takes two parameters - the transducer and the stream - compared to the four parameters in R#transduce.
• hl#zipAll is like zip but takes an arbitrary number of lists.

var xs = hl([1,2,3])
hl.zipAll([[4, 5, 6], [7, 8, 9], [10, 11, 12]], xs)
// => [ [ 1, 4, 7, 10 ], [ 2, 5, 8, 11 ], [ 3, 6, 9, 12 ] ]

• hl#where only takes a spec object but this should be fine once separate R.where into two simpler functions: R.where and R.whereEq #1036 is merged.

[buzzdecafe]

• batch would make a good addition to ramda. i'll do a pr for that.
• compact "I'm guessing hl#compact will probably never have an equivalent Ramda implementation" maybe so, since we like to pretend that we're dealing with lists.
• intersperse maybe worth another PR
• reduce1 is tricky for lists; what happens when you give it an empty list? (boom)
• we have R.scan but not scan1 (see reduce1 above)
• since zipAll takes a list of lists and is not variadic, i think that could work too. another PR for me.

hl.pickBy: this gives me some extra ammo to keep pickBy as the name, rather than the prosaic at best filterObjIndexed. The param order is problematic, but if we find a way to solve reduce, maybe we can do the same for pickBy

[svozza]

I'd be more than happy to help out with some of those PRs, if you want. Although, that said, you'd probably get the three done in the time it took me to do one because no doubt I'd make a bunch of n00b mistakes with the build. Its a bit of an intimidating process to say the least.

[buzzdecafe]

thanks @svozza i can probably bang 'em out pretty quick. i am interested to hear if you have ideas about making contributing easier though. but that is yet another discussion ....

[svozza]

Hehe. Yeah, let's stick to one thing at a time. Anyway, the offer is there if you get sidetracked.

[bergus]

@buzzdecafe Notice that scan1 does not suffer from BOOM with empty lists, it just returns another empty list. I often found it more natural to use than scan, which always increases the length of the list by one.

[buzzdecafe]

fair enough @bergus i will add that to my PR-TODO list

[CrossEye]

Something like batch is being discussed in #1028

Is there any appetite for some sort of user-defined aliasing capability that would allow us to dispatch on different names? This might possibly include a lodash-style rearg. It would make integrations like this quite nice, I think.

[buzzdecafe]

couple more:

• hl.flatMap <=> R.chain
• hl.??? <=> R.ap

would be nice to have these integrated, and treat streams as fantasy-style monads

[apaleslimghost]

We're looking at Fantasy Land compatibility after 3.0. Discussion and WIP implementation is at caolan/highland#114. Part of this is aliasing flatMap to chain and a fantasy-compatibe ap.

[vqvu]

hl#transduce only takes two parameters - the transducer and the stream - compared to the four parameters in R#transduce.

hl#transduce is R#into with an implicit empty stream as the accumulator argument (more or less).

[vqvu]

We also have

• hl#zipAll0, to be added in latest release. It operates an a list of list directly. Not awesome name cause we weren't prescient enough. Ideally hl#zipAll -> hl#zipEach and hl#zipAll0 -> hl#zipAll.
• hl#slice === R#slice

Highland folks, do we want to consider fixing some of the naming & arg order issues in 3.0? Or would that complicate things even more?

[svozza]

I'd definitely be in favour of changing the names and argument orders for 3.0. That said, @CrossEye 's suggestion about the user defined aliases could be very nice too. The 'glue' to get the two libraries to work together could live in a separate module so it would reduce the impact on the core of both APIs.

[buzzdecafe]

The 'glue' to get the two libraries to work together could live in a separate module so it would reduce the impact on the core of both APIs.

FWIW i think in the near term at least that is going to wind up being the solution.

[davidchambers]

Perhaps we should create ramda/ramda-highland and add the Highland contributors. Alternatively, @caolan could create caolan/highland-ramda and we could collaborate there.

[svozza]

Yeah, that sounds like a good plan. Probably makes sense to go for ramda/ramda-highland as we're still trying to figure out what way to go for extensions to the Highland.

[buzzdecafe]

👍 @davidchambers

[davidchambers]

I've created an empty repository at ramda/ramda-highland. We can discuss specifics over there.

[CrossEye]

Ok, I guess. But from Ramda's point of view, I would hope that Highland is just one possible integration, and that there is no specific Ramda internal code to make it work (except for fundamental infrastructure usable by any integration.)

[davidchambers]

I agree that Ramda should have no specific knowledge of Highland. A compatibility layer seems to me a reasonable way to resolve API differences between the two projects.

[vqvu]

I wonder how much of the Ramda API can be implemented via the Fantasy Land Monad interface + transducers + reduce.

Lots of common operators can be implemented as transducers (e.g., map, filter, take, drop), and they can all be plugged directly into hl#transduce (or some other standardized name). Benefit here being Ramda implements things like filterIndexed that Highland doesn't, but it can still be used with Highland via transducers.

Fantasy Land can help this along even more. I implemented partial Fantasy Land support + reduce and got commute to work(*): https://gist.github.com/vqvu/07bebdab23fed81c2325. Any other Ramda function that can be implemented via the Fantasy Land API will also just work.

Getting integration this way could actually go pretty far, and would reduce the size of the compatibility layer.

(*)The caveat here is that Highland can't actually implement reduce due to possible asynchrony. It actually implements a a close analogue.

-- C for collection
-- C can be [] or HL or whatever type implements reduceC
reduceC :: ((acc, x) -> acc) -> acc -> C x -> C acc

This means commute actually returned a thing of the type HL (HL [a]) instead of HL [a].

[svozza]

A compatibility layer seems to me a reasonable way to resolve API differences between the two projects.

Yes, and it's a much less invasive way for us than having to add several aliases to the Highland core.

[CrossEye]

A compatibility layer seems to me a reasonable way to resolve API differences between the two projects.

I've been considering ways for Ramda to easily support such compatibility layers, and it strikes me that what we want is to offer configuration at the dispatch level. Something like

R.alias('toPairs', 'pairs')

that possibly also takes some rearg configuration.

And then the dispatching mechanism would know to look for either 'pairs' or 'toPairs' on the object in question.

[CrossEye]

There's a huge downside to my suggestion, though, perhaps insurmountable: it would introduce state
into Ramda itself. So... other options?

[kevinbeaty]

Perhaps it would be a good opportunity to produce a more 'skinny' Highland core for you to integrate too...
I wonder how much of the Ramda API can be implemented via the Fantasy Land Monad interface + transducers + reduce.

As Ramda supports transducers in more list functions, I'm also wondering how much can be done directly with transducers + Fantasy Land. Probably quite a bit. The "compatibility layer" of transducers + hl.transduce will go a long way... maybe even making an external layer or user defined aliases unnecessary.

[wayneseymour]

@CrossEye When you say state, I immediately thought you are referring to the idea that ramda would have to remember the alias. Is that the case? If not, what do you mean?

[CrossEye]

@wayneseymour Yes, that's what I mean. Right now. all Ramda functions will return the same results for the same parameters. But if we added the ability to dynamically change the aliases for dispatching, then this would no longer be true:

    var hand = new PokerHand(['4d', 'Jh', '2c', '4h', '7s']);
    R.toPairs(hand); 
    //=> [
    //   ['cards', ['4d', 'Jh', '2c', '4h', '7s']],
    //   ['pairs', [['4d', '4h']]],
    //   ['threesOfKind', []],
    //   ['straight', null],
    //   /* ... */
    // ]
    R.alias('toPairs', 'pairs'); //=> ??
    R.toPairs(hand); //=> [['4d', '4h']]

We've lost our referential integrity. R.toPairs(hand) has different behavior before and after we made that alias call, because Ramda has updated some internal state that it's using to do its dispatching. That's something that we've worked very hard to avoid.

[vqvu]

Is it possible to have alias return a different instance of the ramda object? That would keep referential transparency at the cost of pushing management of the different objects to the user.

var R = require('ramda'),
    _ = require('highland');

// Ramda-for-Highland
var HighlandR = R.alias([
    ['chain', 'flatMap']
    ...
];

console.log(R === highlandR); // => false

R.split(R.add(1), [1, 2, 3, 4]); // => [2, 3, 4, 5]
R.split(R.add(1), _([1, 2, 3, 4])); // => some errror.

HighlandR.split(HighlandR.add(1), _([1, 2, 3, 4])).toArray(_.log); // => [2, 3, 4 ,5]
HighlandR.split(HighlandR.add(1), [1, 2, 3, 4]); // => some error

[apaleslimghost]

@CrossEye could we consider something like Bilby's environments?

var hand = new PokerHand(['4d', 'Jh', '2c', '4h', '7s']);
R.toPairs(hand); 
//=> [
//   ['cards', ['4d', 'Jh', '2c', '4h', '7s']],
//   ['pairs', [['4d', '4h']]],
//   ['threeOfKinds', []],
//   ['straight', null],
//   /* ... */
// ]
var R_ = R.withAlias('toPairs', 'pairs');
R_.toPairs(hand); //=> [['4d', '4h']]
R.toPairs(hand); 
//=> [
//   ['cards', ['4d', 'Jh', '2c', '4h', '7s']],
//   ['pairs', [['4d', '4h']]],
//   ['threeOfKinds', []],
//   ['straight', null],
//   /* ... */
// ]

edit: @vqvu got there before me :)

[CrossEye]

Yes, that could work. It would take some interesting reworking of Ramda's internals. And a user who doesn't mind could always assign this right back to R. An intriguing possibility...

[wayneseymour]

@CrossEye
Thanks for explanation. Makes sense.

[buzzdecafe]

closed for inactivity