Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cycle Diversity: generalize for any stream library #196

Closed
staltz opened this issue Jan 9, 2016 · 74 comments
Closed

Cycle Diversity: generalize for any stream library #196

staltz opened this issue Jan 9, 2016 · 74 comments

Comments

@staltz
Copy link
Member

@staltz staltz commented Jan 9, 2016

@TylorS and I started discussing how to migrate to RxJS 5, and/or support most.js streams without the need for a fork (Motorcycle.js) which mostly has copy-pasted code from some Cycle.js libraries.

We arrived at a proposal: Build a "base" run() function which is generic and assumes no stream library. It uses "stream adapters" to agree on an interface for each stream library. We need to build an adapter for each stream library. We also need to build "rx run()", "most run()" which basically utilizes the common "base run()" but specify which stream library adapter is used. Drivers are given the stream adapter object from run(), and can written using whatever stream library.

@cycle/rx-run

import baseRun from '@cycle/base'
import streamAdapter from '@cycle/rx-adapter'

const Cycle = {
  run: (main, drivers) => baseRun(main, drivers, {streamAdapter})
}

export default Cycle

@cycle/dom

function domDriver(sink$) {...}

domDriver.streamAdapter = rxAdapter // mostAdapter 

Tasks to do

  • Create @cycle/base repo (essentially this PR #198 there) https://github.com/cyclejs/base
  • Write test for @cycle/base ensuring one-liner drivers work properly with default adapter.
  • Build https://github.com/cyclejs/rx5-adapter with tests, using StreamAdapter interface from base
  • Build https://github.com/cyclejs/rx4-adapter with tests, using StreamAdapter interface from base
  • PR on Cycle Core to implement it as @cycle/base + @cycle/rx-adapter, with #200 in mind
  • Make canary version for core
  • Make @cycle/rx-run npm package alias for @cycle/core
  • Rewrite Cycle DOM Driver in TypeScript with transposition off by default
  • Update base to export CycleSetup interface
  • Fix Cycle DOM diversity to adapt the out-facing streams for the app's stream library
  • Convert some cycle-examples to use the canary version of dom with RxJS 5
  • Cycle HTTP driver in TypeScript and using driverFn.streamAdapter = rxAdapter
  • Make canary versions for http, test it around
  • Update power-ui to Diversity
  • xstream-adapter
  • xstream-run
  • Fix Cycle DOM last bugs with DOMSource.events() and isolate
  • Soft release core, dom, *-run, http, jsonp RC versions
  • Build https://github.com/cyclejs/most-adapter in JS with tests, using StreamAdapter interface from base
  • create @cycle/most-run https://github.com/cyclejs/most-run with #200 in mind
  • Unnecessary: Update motorcycle drivers to use driverFn.streamAdapter = mostAdapter
  • Update examples and TodoMVC to Diversity & xstream
  • Make bmi-typescript example, and fix Cycle DOM TypeScript problems, issue #307
  • Write http-random-user in TypeScript to make sure HTTP Driver is also working well
  • Change HoldSubject to Subject in stream adapters. It's not needed.
  • Rework makeHTMLDriver(callback)
  • mockDOMSource should return a stream of the app's own stream library.
  • DOMSource.select().elements should return WhateverStream interface, not type any
  • foo-adapter and foo-run: remove dispose(), add remember()
  • Update cycle history to use stream adapter remember()
  • Update TodoMVC using cycle history
  • Fix Cycle DOM documentation generation
  • Update cycle.js.org
  • cycle.js.org documentation page
  • Write Release notes and migration guide
  • Merge branches diversity to master, and publish non-rc versions
    • Cycle Core / Rx Run
    • DOM
    • HTTP
    • JSONP
    • examples
    • cyclejs.github.io
    • others
  • npm deprecate @cycle/core, use rx-run
  • Cycle DOM rxjs-typings, rx-typings, most-typings
  • Cycle HTTP rxjs-typings, rx-typings, most-typings

Breaking changes so far

  • DOMSource.select().observable => DOMSource.select().elements()
  • new Cycle.run API for those cases where sources and sinks are wanted: #200
  • Snabbdom instead of virtual-dom
    • breaking changes with hyperscript way of using attrs
  • top-level <svg> element is created with hyperscript-helpers but children elements created with hyperscript
  • mockDOMSource(sa, config) now takes a Stream Adapter as the first argument
  • makeHTMLDriver() => makeHTMLDriver(effectFn, options)
  • No more onError callback as an option to makeDOMDriver()
  • No more sinks.dispose and sources.dispose
  • HTML Driver returns and HTMLSource mimicking DOMSource. HTMLSource.element$ is the stream of HTML strings
  • HTTP Source is not response$$ but contains it. It's an object with select(category: string): Observable<Observable<Response>>, filter(predicate: (res$: ResponseStream) => boolean): HTTPSource and response$$ property.
@staltz staltz added the feature label Jan 9, 2016
@erykpiast
Copy link
Contributor

@erykpiast erykpiast commented Jan 9, 2016

I'm not sure if I understand correctly, so please tell me, which statement is the closest to your idea.

  1. you wish to allow people to create Cycle drivers with whatever library they want, not only Rx, and make Cycle-core transparent (library agnostic), so if all the drivers are written with, ex., most.js, application may be written using most too and there is no dependency on Rx?
  2. or you propose to make inside Cycle kind of translation from any-streams-library.js to other-stream-library.js, to allow different drivers be written with different libraries and application in any other supported library?
@TylorS
Copy link
Member

@TylorS TylorS commented Jan 9, 2016

I have already begun experimenting with this here at https://github.com/TylorS/cycle-core/tree/rxjs-most-interoperability. It requires no breaking changes to the current cycle-* drivers. Most.js drivers are supported using (this is not a agreed upon or final API)

function driverFn() {}
driverFn.type = 'most'

There are some issues still that need to be resolved. Cycle-DOM for example has 2 failing tests, but I haven't had the time just yet to investigate the issues.

I'm also interested in simply creating a set of streamLibrary -> streamLibrary conversion functions to experiment with drivers being in control of where to convert from one library to another.

@erykpiast
Copy link
Contributor

@erykpiast erykpiast commented Jan 9, 2016

"streamLibrary -> streamLibrary" approach would make Cycle's interoperability and interchangeability outstanding! Especially if it's done inside core (or somewhere else, but to make it transparent to the framework user and driver creator).

@staltz
Copy link
Member Author

@staltz staltz commented Jan 9, 2016

@erykpiast The purpose of this issue/initiative is to:

  • Allow main() to be written in any of these stream libraries: RxJS 4, RxJS 5, most, Bacon.js, Kefir.
  • Allow using most-based drivers, such as Motorcycle's outstanding most + snabbdom driver.
  • Keep the simple API for drivers: just functions that input and/or output streams. In particular, it's important to allow people to create one-liner drivers without a lot of boilerplate. We need a solution that assumes drivers, unless specified, are using the same stream library used by run() when specified in options.streamLibrary in Cycle.run(main, drivers, options).
@TylorS
Copy link
Member

@TylorS TylorS commented Jan 9, 2016

I'm working on an external library to convert between different stream implementations. I believe it will likely be required for a driver to supply a place to be able to convert it's streams when there are more complex streams such as the DOM Driver.

@staltz
Copy link
Member Author

@staltz staltz commented Jan 9, 2016

Great 👍
Modularity for the win

@erykpiast
Copy link
Contributor

@erykpiast erykpiast commented Jan 9, 2016

What about interoperability of nested cycles? There is plan to make possible connections between cycle applications built with different libraries? To be honest I don't really believe people would like to use many stream libraries in production-ready applications (at the end of the day size matters), but if we make it possible for drivers, it will be consistent to provide the same abilities to nested cycles.

@TylorS do you think this conversion utility should be used in drivers internally? I see another possible approach: for drivers like Cycle-DOM, thin compatibility layers may be built, one for each library. If driver creator defines some interface to build ex. multi-level streams, implementation (from separated package like cycle-dom-kefir) may be passed to the driver.

import { run } from '@cycle/core';
import makeDomDriver from '@cycle/dom';
import kefirAdapter from '@cycle/dom-kefir-adapter';

Cycle.run(main, {
  DOM: makeDomDriver('#app-container', kefirAdapter)
});

The adapter will implement functions like createEventsStream etc. in Kefir, using stream conversion library internally. Driver itself will stay small and library agnostic, hovewer its configuration will become more complicated.

Just an idea.

@staltz
Copy link
Member Author

@staltz staltz commented Jan 9, 2016

We can always endlessly abstract everything away, but I don't think that's the purpose. It's enough if we allow people to build main() in their preferred stream library. For nested components (if I understood that's what you meant by "nested cycles") I think a better idea will be a utility library to convert a Cycle main() to a Web Component (or a Web standard custom element).

@TylorS
Copy link
Member

@TylorS TylorS commented Jan 9, 2016

@erykpiast Your approach is bascially what I have in mind, but converting from one to another is actually pretty easy and general so it would probably be handled as an option to run(). This is just an idea too.

function driverFn(sink$, conversionFn) {
  return conversionFn(sink$.map(x => x * x))
}

driverFn.streamLibrary = 'rx' // Tells run that it's written in rx

run(main, {other: driverFn}, {streamLibrary: 'rx'}) // means write your main() with Rx
run(main, {other: driverFn}, {streamLibrary: 'most'}) // write your main() with most

So anywhere that a stream is produced and returned back to main, conversionFn() would just have to be called to convert the stream from x -> y.

@HighOnDrive
Copy link

@HighOnDrive HighOnDrive commented Jan 9, 2016

Very exciting, Cycle as a "Web Component (or a Web standard custom element)" sounds great too. Embedding anywhere everywhere sounds like a famous-ly awesome idea!

@TylorS
Copy link
Member

@TylorS TylorS commented Jan 9, 2016

Here is the start of this stream conversion library
https://github.com/TylorS/stream-conversions

@staltz
Copy link
Member Author

@staltz staltz commented Jan 9, 2016

👍 I like the API!

@Cmdv
Copy link
Contributor

@Cmdv Cmdv commented Jan 9, 2016

I like the idea giving devs the option to chose stream library of preference.
But I worry a little with giving people support. Most & Rx are quite similar and all the examples on Docs assume Rx but how do we support people using Bacon.js?
Just wonder if it will add a level of confusion as to what the right way to do things would be.
Superb for people with previous Cycle experience but a little daunting for new comers?
Just a devils advocate thought process but I always like to think of new comers :)

@staltz
Copy link
Member Author

@staltz staltz commented Jan 9, 2016

Good concerns @Cmdv. The point here is to give choice. That's different from forcing people to choose. I'll still advocate in cycle.js.org docs one stream library (most likely RxJS 5), but I'm happy to support people who want (or need) to use most or Bacon.js or anything else. For people who know what they need, I want to make it easy. For people who don't know what they need, I also want to make it easy.

I'm still planning to write that "Real world apps" chapter for cycle.js.org, where there will be a suggested stack of tools.

@TylorS
Copy link
Member

@TylorS TylorS commented Jan 9, 2016

@staltz This API for which? stream-conversions or above that for drivers?

@Cmdv I don't know anything about Bacon.js or any of the other libraries, but I think we're just getting started with just Rx and Most. Basically the idea is to have an approach to allow Cycle.js to be a general implementation of a pattern, ultimately paving the way for when we can just use ES standard Observables and the interoperability that will bring.

@staltz
Copy link
Member Author

@staltz staltz commented Jan 9, 2016

stream-conversions

@Cmdv
Copy link
Contributor

@Cmdv Cmdv commented Jan 9, 2016

@staltz @TylorS lovely I just wanted to make sure there was an advocated way and more experienced Devs could chose the stream library of choice.

Didn't want to add lots of confusion to the table, because Cycle and FRP isn't the easiest thing to get your head around if you've never used streaming libraries. So clarity and less abstraction is needed for those new devs. This can be just at a Documentation level or advocated level but wanted to make sure new dev's weren't coming to cycle thinking:

What are these stream libraries which one do I use? I just want to have a little play and already I have to make important choices about the core of what I'm building.

Then ultimately scaring them off :)

@TylorS
Copy link
Member

@TylorS TylorS commented Jan 9, 2016

@Cmdv The default for the foreseeable future at least will be rx 4 and rx 5 when that's ready. I don't think there is any current plan or need for breaking changes application side. This should all be handled in run() and inside of the drivers transparently.

@staltz
Copy link
Member Author

@staltz staltz commented Jan 9, 2016

Actually one of the biggest things is being able to use most.js in a DOM Driver. In principle it should have better performance.

@TylorS
Copy link
Member

@TylorS TylorS commented Jan 9, 2016

@staltz Yes, that's another good reason.

Most.js in theory can be anywhere from ~200x-400x* faster than Rx 4 and still roughly 30x-50x* faster than Rx 5 so even with these conversions there should still be a performance boost where there are expensive operations like diffing and patching the DOM.

(* - https://gist.github.com/TylorS/55573b29df19c064c6b1)

@Cmdv
Copy link
Contributor

@Cmdv Cmdv commented Jan 9, 2016

Agreed, I'm really liking the idea to let run() handling that behind the scene and perf improvement is always a good route to take.

Also if there is a core Rx library that could fit in really nicely rather than the whole of Rx.

Would you be able to mix say Cycle-dom use most.js and run use Rx etc? I assume thats what @TylorS stream-conversions is about?

@TylorS
Copy link
Member

@TylorS TylorS commented Jan 9, 2016

@Cmdv Yes, thats the idea. The drivers can be written in x and your application will be written in whatever library you prefer.

run(main, driver, {streamLibary: 'rx'}) // write your application using RxJS
run(main, driver, {streamLibary: 'most'}) //write your application using Most.js
@Cmdv
Copy link
Contributor

@Cmdv Cmdv commented Jan 9, 2016

Great this is a nice way to have everyone working together too 👍

@HighOnDrive
Copy link

@HighOnDrive HighOnDrive commented Jan 9, 2016

Very nice indeed 👍

@TylorS
Copy link
Member

@TylorS TylorS commented Jan 9, 2016

Yes! Motorcycle will essentially will be rolled back into Cycle.js 👍

@TylorS
Copy link
Member

@TylorS TylorS commented Jan 10, 2016

My stream conversion library has been updated to make use of an interface to allow for easier support of additional libraries and quite a bit less code overall. Tests are now partially generated, in a way, too! I'll try to get started on a version of cycle-core that takes advantage of this, and add support for some of the common drivers. e.g. cycle-dom and cycle-http

@TylorS
Copy link
Member

@TylorS TylorS commented Jan 11, 2016

I opened PR #197 for my work on stream conversions being done in run(). Drivers would now passed a function as the second parameter for use of converting between streams between different libraries.

@Frikki
Copy link
Member

@Frikki Frikki commented Jan 11, 2016

Very interesting. An idea we discussed months ago. Hopefully, I’ll soon be back working with you lot.

@staltz
Copy link
Member Author

@staltz staltz commented Jan 11, 2016

@Frikki 💙

@wclr
Copy link
Contributor

@wclr wclr commented Apr 26, 2016

Web-streams will be shipping natively in browsers soon

yes, like observers, ES7, http2.. soon

@SteveALee
Copy link
Contributor

@SteveALee SteveALee commented Apr 26, 2016

heh, and don't forget web components.

@tusharmath
Copy link

@tusharmath tusharmath commented May 14, 2016

Here is another thought, how about the API being exposed via simple callbacks or event emitters. This would make the API much simpler to consume and convert to whichever stream people want to use. This would also encourage people to write truly stream agnostic drivers which is a must have.

@wclr
Copy link
Contributor

@wclr wclr commented May 14, 2016

@tusharmath what API? you actually may use adapter's API to create lib agnostic drivers. It is probably not perfect for this task, but basic cases can be implemented quite easily.

@tusharmath
Copy link

@tusharmath tusharmath commented May 15, 2016

@whitecolor

which api

The Cycle.run method could actually get rid of any dependency on a stream lib completely. The same applies to @cycle/dom

Here are my concerns in detail —

  • I am using most.js for my application but I still need to have a copy of the humungous RxJS lib simply because Cycle depends on it. IMHO letting go of Rx or any other streaming lib will make cycle easier to adopt and also not be limited by the performance of that streaming lib. Every now and then there is going to be something faster and its not realistic to expect Cycle and its drivers to immediately upgrade.
  • Cycle is about following a MVI pattern but should not be enforcing FRP as a way to achieve it. FRP makes literally everything easier, but that should not become a criteria for how Cycle's interface is designed.
@staltz
Copy link
Member Author

@staltz staltz commented May 15, 2016

@tusharmath There will be a Cycle.run for each stream library, e.g. @cycle/rx-run, @cycle/most-run etc. Cycle Diversity is about making these libraries with each other so that if you have a driver written in most, you can use it even though your app is in Rx. But the endgoal with all this is to gradually and softly migrate to xstream. If we had started from the beginning with xstream, Cycle Diversity and the interop between these stream libraries would not be necessary.

@Cmdv
Copy link
Contributor

@Cmdv Cmdv commented May 15, 2016

@staltz is the long term goal, to have xstream be the only streaming library involved?

@staltz
Copy link
Member Author

@staltz staltz commented May 15, 2016

I don't mean we will cut support for other libraries. That's not the point. The point is that xstream is meant for Cycle and Cycle for xstream.

@wclr
Copy link
Contributor

@wclr wclr commented May 15, 2016

But the endgoal with all this is to gradually and softly migrate to xstream. If we had started from the beginning with xstream, Cycle Diversity and the interop between these stream libraries would not be necessary.

I can not completely agree with this. I believe cycle has potential of staying/becoming really stream library agnostic framework, and no need to set this restricting "endgoal". I believe @cycle/core will become eventually xstream-run instead of rx-run, but time will show if Cycle for xstream (sounds kinda nazy) is better vision than real diversity.

Though I can not disagree that dealing with one universal stream library is simpler and eventually better approach - but this should be very natural way of coming to this point.

@tusharmath for using components that were developed with different stream libs I've created #305

@alkhe
Copy link

@alkhe alkhe commented May 15, 2016

@whitecolor Diversity is cool and all, but if we want to have minimal bundle sizes and maximum performance, then sticking to one stream implementation is the way to go.

The last thing we want to do is to send the client a >1 MB distribution including xstream, rx4, rx5, most, bacon, and kefir just because we flaunted the ability to choose whatever stream library suited your fancy.

@staltz
Copy link
Member Author

@staltz staltz commented May 15, 2016

@edge Not just that, but having more choices leads to more choice paralysis for beginner programmers, but it also means more maintenance for the core developers. I am building Cycle as a framework with smart defaults, but with some degree of choice/freedom, not as a generic layer for everything anyone wants.

@wclr
Copy link
Contributor

@wclr wclr commented May 15, 2016

@edge nothing restricts you from this with diversity approach. Use drivers and components written with your library of choice.

And I agree that xstream can become a good standard for shared components. But i'm also convinced that harmonic approach to diversity will only make framework stronger.

@staltz

but it also means more maintenance for the core developers

Not always, it depends on chosen approach - if generalized approach is chosen correctly, it will make framework and tools even less prone to errors for specific cases eventually.

@download13
Copy link

@download13 download13 commented Jun 9, 2016

I'm curious why xstream was chosen over something that could be more interoperable like es-observable`.

If the spec is added to a future version of JavaScript we'll likely have a built-in implementation of Observable that all libraries can use (much like Promise). At that point libraries will only need to depend on utility functions, which can be imported as bundles (a la rx) if they don't mind being bulky or individually (something like @cycle/flatMapLatest).

@staltz
Copy link
Member Author

@staltz staltz commented Jun 9, 2016

I am closely following es-observable closely and I am involved in discussions. For now, it is a moving target. We are building Cycle Diversity with tools from today, not from tomorrow.

@staltz
Copy link
Member Author

@staltz staltz commented Jun 29, 2016

@mightyiam
Copy link

@mightyiam mightyiam commented Jun 29, 2016

Yay! Good job!

staltz pushed a commit that referenced this issue Jul 18, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet