Build on top of node.js streams

[mitar]

Probably for maximum compatibility it would be great if data layer would be build on top of node.js streams. Then one can simply use any node.js stream as a transport layer. This could build into many community efforts to port node.js on top of websockets, webrtc, IoT protocols and so on.

[arunoda]

I think this leads to a proposal for a common GraphQL transport. Currently, most popular one it the HTTP transport (and a kind of protocol) comes with express-graphql.

We can do more to build a better graphql transport with query caching, streaming and so on.

Basically, this is more of a question we should answer together as a graphql community, not just for Apollo.

[stubailo]

I totally agree, we should find something that will work for all GraphQL. Are there any proposals for this?

Also, would it instead be better to have a non-streaming option so that you could build on any transport that can do request/response?

[arunoda]

I think there is no such project. Lee of one one from FB said once their version of GraphQL has caching and so on. He said, they are planning to bring it to the open source version as well.
(There version is with HACK I guess)

So, I think there is a open opportunity to build a common spec where anyone could use.

We could work with Graphene, Sangria and Reindex for this. (of course with FB :) )

[stubailo]

Yeah, great idea. We are actually having the author of Graphene over for lunch tomorrow, would be cool to talk to some of the others as well. I heard Sangria in particular has some really cool stuff going on.

[venturecommunism]

There's a Sangria-Ion for Ion marshalling
https://github.com/sangria-graphql/sangria-ion.

Maybe Ion would solve a lot of GraphQL's protobuf, caching, static -> dynamic typing, universal schema, storage and message routing issues in a more p2p context.

http://tutorials.jenkov.com/iap/ion-vs-other-formats.html
http://amznlabs.github.io/ion-docs/
https://news.ycombinator.com/item?id=11546098

Maybe maybe deal with graphs more general than trees by using JSON-LD http://json-ld.org when a client has not opted in to some of Ion's more circular dependency hell-ish features.

Just a thought.

[stubailo]

Yeah exploring alternate transports for normalized data is definitely something we want to do once we build out the basic functionality. The Relay/GraphQL people seem to be interested in that as well.

[venturecommunism]

Might solve some of Sequelize's issues with GraphQL to do it this way graphql/graphql-js#290

[MasterJames]

Hi just learning by ozzmosis these days (from my fellow comrades and heroes visible in the community). I'm a little lost here, being still formulizing alpha, with limited docs, time limits and distractions of other topics/projects etc. and looking for Simple Solutions to complex problems too. For me "one great way" is all I need, and not multi datasource integration to legacy methods, so not sure this wont be overkill but hoping its more like rather good templates and guildlines. I know its all idealistic and not so realistic, more futuristic and visionary as an indy dev thinking ahead seems the only way to not be left behind. Also I'm using 3D with THREE.js and find the DOM irrelevant. Alright that's just a quick hello with an orientation bio, to help understand my position, direction and goals.
I also feel like "I'll display whatever my way thanks just give me the data." No Web page needed just use a schema that's accepted by the global community and I'll display my way for me. "So your selling products. Well what schema where's the DDP IP:port? Maybe I'll display your logo maybe not." Okay I think you get the direction I have for my future.

Non-pooling bidirectional WebSockets sounds like progress to the future, as is formulating thoughts on DDP integrated with schema.org. I found the ion "symbol tables" interesting too, thanks. Diffing and signing, are decentralized points in general as well (with deeper possibilities to contimplate).
Long ago with NeXT we used an Object's toString method to serialize/archive/create objects, people said "Oh XML is the way now" and I thought but I'm already doing a more direct approach where the dictionary is the central key value way to identify the constituents. Then along came JSON and the meteor varient with extra types like dates in BSON, which seemed a terrific compromise (aka obvious improvement). [Note new improvements of optimized run time compiling means worring about types is important for speed opimizations where needed but not a critical concern. Flixiblity is very important in my world.] Now I think "but I'm using Objects in Objects with Javascript (akin to dictionaries) sometimes JS passed there too (delivered through synced DB).", so kind of back to a NeXT way of thinking, if i can call it that. Seems to me no one got that. Those smart people from NeXT back then did not make it to Apple and saving/integrating OS9 as it was now going off target and adding unnecessary complications with what seemed already the right direction back then, but thats another sad story like how the scientific tool, the 3DKit turned into Pixar aka movies for children. A sad time for the world in my opinion. I guess we weren't ready then will we ever be? Is questionable.
Anyway I see the Shared File System as the preexisting quorum so no need for all the layers doing the same thing. AWS' (still in preview) EFS is a cheap scalable Shared File System (really the starting point) is perported to be slow and this is unfortunate as it seems like it shouldn't be if it's massively scalable. So one maybe puts a booster like redis in where needed.
I've looked at ETCD as a way to store the data as JS object incapsulations but only conceptually as an example of alternatives to Mongo.
I thought if Meteor with Mongo (or whatever JS object store) instances are spun up as needed from an EFS would you need a master slave logic in the server setup aka Meteor/Mongo? or if they are all "citizens" (as I've coined it also "death to the slaves and masters", which is not true because the EFS quorum works that way already) that all are both writing and reading the oplog tail then the EFS quorum would handle the locking and syncing in a time based way etc.
Further any global user could spin up a system and mount with granted permissions the EFS directly but at that point we're getting ahead of ourselves but seems to be not an impossible future if EFS is fast enough.
Anyway this seemed like a good enough place to introduce my vision and maybe the healthier seeds of thought will more likely grow when introduced to the right interpretive minds, so thanks for listening.

So really keeping things simple and more to the point "pure" seems like the direction I hope this project will help give me guidence to achieve.

In example, I hope to sync a 3D objects attributes by passing back JavaScript objects within objects that can inject and overwrite code changes directly writing and reading with the local cache Diffing ideas that transport over the more direct non-polling WebSockets and some efficiencies the transport layer handles like binary compression symbol tables etc. that are hidden from the developer.
Schema that are known and agreed apon would also be accessible directly through a kind of API so I can display them as needed or more to the point as I personally prefer (which could be defaulted to the data providers style if you want) and so everyone's visualizations could be easily different.

This then extends to visualizing the code and being editable in an interactive 3D way. Signed Diffs and usage tracking provide Etherium type contracts to pay out those that do the real work. Alas in a global quorum decentral concepts like Etherium also become irrelevant thanks to the larger quorum, which would take a larger manifesto to explain better. All you need is the digital signing authority in the block chain and that can be rolled over as edits and uplates aré continuous. Also I see git hub iteration and 3D code visualization parsing of code as critical components.
Basically I ponder and pose the question is there a minimum amount of code you need to write to not need to write code any more? but rather have a virtual interactive environment to do it in, a place where all code exists in the database as signed diffs that boil down to precompiled hotloaded byte codes. A new landscape were automation validates improvements and usage dictates value.
I also see the possibility for code evolution based on random known routines for optimization etc., which with automated testing will exceed human comprehension in finding better distiled code when we finally extrapolate programming logic into a conceptual form.
One more point. If the client is waiting for data changes does it even need a callback? No when the P2P processing system (oh right that's another important concept) or what-have-you finishes the job/method I'll likely get the data update before the callback can be called.
Example I need a video frame rendered, it's broken into jobs, the global communities shared resources work in randomized tandem to valid each pixel or block. Statistically two executers per job will always return the result faster on average and the second validates it or 4 more (nodes via random obsifucated uri) check the same job again. The client can check the job whenever they want later as needed or be notified if waiting but they really just get sent a diff of the pixel or block they're subscribed to or in general "the data change" as a result without the need of promises and callbacks. In a sense the GUI is monitoring the result data directly so in a way there's a promise one could say.
Anyway if it's food for thought in this quickly posted format it's surely swiss cheese but with a little filler these ideas as idealistic and futuristic as they still seem to me was what we pioneers saw for the future like way back in 1969 at the dawn of the computer age when the clock was 0 or at least when VR was announced and we still had no limits in what would be possible. It's been 40+ years but with the advent of WebGL and VR/AR on the mobile platform the waiting is over and the future is something we must as an opensource community "will to be." today, because corporate goals are too short term.

"In the transition time, a hybrid will exist." - Master James

[stubailo]

@MasterJames to be honest, I'm having a hard time reading your comment due to limited formatting. Could you split it up into sections with headings and paragraphs? My eyes get lost because of the large volume of text, but it looks like you have some interesting points.

[MasterJames]

Yes sorry my psychotic Manifesto really does need a separate thread or whatever sorry to hijack.
It was just for the record to have some thouhts seeded. One has to take the time to read between the lInes to extrapolate the missing details. Did it on mobile too.
My attempts in the past at this I never posted in the end either. Its lIke a compression of a lifetime of knowledge or tcpdump. To figure out what's going on in there amounts to debugging. LOL. I'll try to subjugate things with a master thread and separate out subsections somehow. If you can spare guidance from time to time a propotype might present itself.

In these times our problems have become more complicated. It takes complicated solutions to solve complicated problems and to explain them is well... complicated bother to write and read.

Consider it a trial run, or first draft etc. maybe with time you'll find and interpret something that can be integrated out of a more clarified working draft of my lofty wishful thinking.

[MasterJames]

After another month of searching for alternatives to the overly tardy GA of AWS' EFS. I found someone inline with my thinking.
IPFS.io
This video helps explain where to begin
https://www.youtube.com/watch?v=HUVmypx9HGI

I fear the main original goal of Meteor, "simplicity", might be getting subverted by the recent attentions it's acquired due to popularity and the new desire to spend %90 percent of effort to satisfy the last %10 of everybody else's needs that possibly one could say Apollo represents.
It seems like TypeScript? is for the Big companies and ex-Java "Type-heads" coming on board, that didn't really understand JavaScripts advantages (that have alternative solutions in intelligent run time compiler systems now too).
I've been pondering "const" with some dismay and a lot of rejection (coming from a hot code push from code injection via database storage mentality).
Ironically the Merkle Tree route is of course "immutable" and so I'm not sure there's a clear route to reactivity that's clear to me [or that I love the idea of adapting yet]. Nor do I see clearly the 'diff' logic in versioning with this solution although Git uses it (so it must be plausible), [but it has many good qualities that I will be contemplating over the next few months (along with socket.io)]. There is a Garbage Collection concept that hints it can achieve what's needed for a reactive model.

Anyway a Distributed Shared File System functioning with duality as the transport & reactive datastore is this way forward in my opinion. One to also help try to dissolve the undesirably dirty word "complexity". So I'm just saying don't loss track of that notion in the headstrong march towards the singularity.
Still Lemmings are a reality and a system that also appears to work, despite the tragedy of mother nature and evolution it somehow makes sense? At least to cleanse away a myriad of possibilities and resolve to a true pure path of simplistic logic.
These idea actually goes back to the original visionaries views of what the Inter(Galactic)Net is suppose to be, and so superior and luckily not entirely lost wisdom in my opinion.
The other irony is the notion that such a system is possibly not as monitizable as we've been breed to believe is critically important & sustaining. I believe that too, in time, is finally shifting.

[mitar]

@MasterJames: I could not agree with you more.

[mquandalle]

my psychotic Manifesto really does need a separate thread

I think so :-) Because despite how much I like IPFS I don’t see how it helps the original issue of using node.js streams in Apollo.

[MasterJames]

Sorry mquandalle didn't know you were listening to 17 with such focused intent. It's more of a new starting point for my admitantly blurted out rant maybe as an explanation or appology. Peace and good will to you too. It's actually great for steams because all files are broken into separate cat-able pieces/blocks. A brilliant solution really.

[MasterJames]

In regards to streaming in ipfs i can't say this makes things any clearer for me but maybe someone with the time and foresight will find it useful.
ipfs/notes#129

[venturecommunism]

@MasterJames Would you be willing to start a repository for us to discuss some of these things? Just point me to it and we can talk.

[MasterJames]

Hi @venturecommunism I'm glad you have some interest to follow this up. Thanks for making an effort.
#53
You can reach me at masterjames@master-domain.net too bad they got ride of private messages here on GitHub.