Simplify angular demo

[victornoel]

I wasn't really finished last time when you merged #365 :)
Let's keep this open until we finished discussing what can be improved!

This simplify some things:

• Use ngx-pipes to transform object to array of pairs (instead of my ugly code)
• Extract GunDb in its own file (just for clarity)
• Introduce deletion (by clicking on an element).

It works well, but still, as I explained in my previous PR, the problem is that I think that gun wasn't meant to be really used like this…
From what I read in your examples and demos, when you want to iterate over the elements of the todo object, you use gun.get('todos').map().on(...), i.e., you execute a function for EACH of the properties which will mutate the view.

With angular and reactive programming, a common way of writing this would be more declarative: you specify how the view is expressed in function of the set of todos (see the html code). In my case, I use something that would end up calling gun.get('todos').on(....) (it is gun-edge and its $rx function that does this by making the new state of the object, i.e. the object passed to the on(...), into an Observable event).
And unfortunately, this object will contain not only the elements that would be present in the map, but also some special properties such as _ which I then have to manually remove.

So, I'm not sure how this kind of situation is meant to be handled with gun… and maybe gun should evolve w.r.t. to this kind of need? Or is there already some utility libraries that ease this kind of us?

[victornoel]

@amark some quick questions:

1. apart from on(...) is there other types of event?
2. is there some way to cancel a subscription to changes done with on(...)?
3. how do you remove a value previously added with put(...)?
4. are you active on a gitter or something like that if I have more questions?

I plan to drop gun-edge for now because I think it relies on old API (and also to workaround #322 as you noted in #365) and reimplement what I need from it in this demo, and maybe then make some PR to gun-edge or water-gun :)

Thanks!

edit: added a 4rd question :)

[amark]

Oh, sorry for merging early! I was just excited to get angular in. :)

Yeah, I'm most active on the gitter channel :) you should join it.

I think we might be misunderstanding each others API familiarities? The api on gun is FRP on the items, maybe this is slightly off topic but hopefully it'll give you some context to then correct me in my misunderstanding:

In gun list.map(cb).val() the callback acts as a map filter, anything other than returning undefined gets sent down the chain (as in, in JS, if you return undefined you get a list full of undefineds, which is annoying, gun doesn't do this, it ignores/filters them out).

While list.map().val() if no callback is specified, rather than defaulting to "return undefined for everything" (which is kinda silly), it uses a default callback which just return data which causes each data point to get passed down the chain. Thus acting like a forEach.

Further, because gun has a chaining API, you can combine map in some cool ways:

list.map().on(cb) get items in the list (and as they are added in the future), and updates to the items themselves.
list.map().val(cb) get items in the list (and as they are added in the future) only once, no updates to the item itself.
list.val().map().val(cb) get the list once (no future updates), and each item only once.
list.val().map().on(cb) get the list once (no future updates), but get updates to just those items over time.

As a result, live updates to DOM elements is easy with gun's core FRP/observable API gun.get('todos').map().on((item, id) => { $("#"+id).reuseOrAppend().text(item) }) (pardon the pseudo jquery code, it is what I'm familiar with).

So I'm guessing similar for Angular, right? in the on(cb) callback, you'll get data and then bind it to an Angular scope, and then it should auto-update (or maybe you have to call setState or dirty or something). Yes / no?

1. on covers all changes. It should be easy to case detect null === data // remove etc.
2. Yes, .val(cb) gets called once and then cancels the subscription, so you'll probably want to use that. on(cb) the cb gets called with extra parameters: gun, data, key, at, ev where ev = event which you can ev.off() at anytime.
3. With put(null) this deletes/tombstones the data. It looks like you are doing this correctly already.
4. Yes :)

So what am I missing? Please let me know!

[victornoel]

Generally

I don't think you are missing anything, but I see things a bit different.
One point I think is important is that with angular, you do want to work in a reactive way, but usually, you don't want the callback to be triggered once for every key/val of an object, but once for the whole object.
For example compare:

1. https://github.com/amark/gun/blob/master/examples/todo/index.html#L66
2. https://github.com/amark/gun/blob/master/examples/angular/src/app/app.component.html#L9

In the first case, you call the callback for each of the value to mutate the DOM, in the second case, you express the DOM as a function of the whole object.
This gives you the possibility to clearly separate how you build the data and how you define your DOM.

But that's not a problem with gun, since you can do get('myObject').on(...).
The only problem is that we are missing some utility function to easily extract, from a given node, only its key/val and not the gun metadata such as the special property _.

On the questions

About 2., I meant cancelling an on, I don't want just one value but I want all values until I decide I don't want them anymore.
I tried to use the following:

// subscribe to some changes
var g = gun.get('todos').on(cb);
// later on, cancel the subscription
g.off();

But it didn't work (I got an exception, I don't have it next to me, but if it is the way it should be done, I will reproduce it).

About 3, that's what I thought, but as explained above I am using gun.get('object').on(...) and the object I retrieve via the callback contains the key with a null value… is it expected of the user to check himself for the nullness of the value? Again I think some utility functions could be of help here maybe…

[victornoel]

Oh and I wanted to add: if you are advocating for a FRP experience, it could be interesting to have a full rxjs compliant wrapper for gun or even rewrite gun with rxjs directly ;)

[victornoel]

And question 5: is there any documentation on what exactly the callback of on() gets as arguments? The source code is too hard to read for me :)

[amark]

Ah, yes, I see now, it is indeed easier to just give Angular the data and let IT handle the iteration. Stupid question though: Doesn't that require re-rendering the entire DOM list (versus just re-rendering the one item that changed?), which would be bad performance for large lists, and thus bad practice?

To hand Angular the full object (I think you mentioned this already), do:

gun.get('list').on(function(data){
  delete data._ // this should now be safe in 0.7.3+
  scope.bind = data; // or whatever Angular is
});

If you want to remove any nulls (these are still important for "tombstones" in gun, especially for UI libraries that render individual elements versus a whole list), change the line to:

scope.bind = Gun.obj.map(data, function(item, key, filter){
  if(null === i){ return }
  filter(key, item);
})

Or use any of your favorite lodash tools instead. Finally, we can wrap this in a new gun extension to make yours (and everybody else's) life easier:

(note, there are several extensions to gun like this that tweak gun's API to fit whatever exact need you want, that you can reuse, rather than having to use gun's universal API)

Gun.chain.bindToAngular = function(scope, cb){
  return this.on(function(data, key){
    delete data._ // this should now be safe in 0.7.3+
    scope.bind = Gun.obj.map(data, function(item, key, filter){
      if(null === i){ return }
      filter(key, item);
    });
    cb(scope.bind, key);
  });
}

Off

Yes, this is what I meant by:

gun.get('foo').on(function(data, key, at, ev){
  if(true === someCondition){
    ev.off(); // unsubscribe
  }
});

In the future gun.get('foo').off() is suppose to work too (actually, it'll purge the data from memory too, as to not waste space between views). However right now it is broken, thus why it is crashing on you :( sorry about that.

The docs should be updated:

data is the raw data (one level deep)

key is the property name of the value.

at is the context for the update.

ev is the event listener.

RxJS

There are some wrappers here:

https://github.com/kristianmandrup/water-gun
https://github.com/JuniperChicago/cycle-gun
https://github.com/ctrlplusb/gun-most

Rewriting gun to use RxJS would have some seriously damaging repercussions: gun's tens of millions of API ops/sec wouldn't be possible, and gun would have dependencies (currently gun has 0 dependencies, other than server-side plugins). Not that RxJS isn't great, it is, but there would be a severe performance penalty and dependency requirement.

---

Did I answer everything? Let me know if I missed something.

I especially think you'll find extensions to be very powerful and solve most of your itches.

[amark]

oh, and is this ready to pull yet? :D :D

[victornoel]

oh, and is this ready to pull yet? :D :D

haha, let me get back to you on that, I need to process your answers (great by the way!).
I think all is good except for the off() thing, I would really love to have a way to stop listening a on() (and only that one, not all of the listeners bound to a given node) without doing it from inside the callback, but I still need to see how the existing rxjs wrappers handle that (I hacked something together for this PR but I want to do it clean.).

I will do that in the next few days :)

Doesn't that require re-rendering the entire DOM list

Yes, I guess (actually I'm not even sure, maybe it's even optimized not to recreate the list element themselves, which can be angular components for example, if the elements of the node do not change) it will need to rewrite the DOM for the loop on the node, not less, not more. Not our concern I think, at this scale this is nothing, angular handles that at best as it can.
As always, there exists a trade-off between imperative code and declarative one...

[amark]

the ev.off() is for just-that-one-listener. When gun.off() is working, it unfortunately won't be able to detect which of many listeners it is associated with (although if you only add 1 listener per object, then that would make it simple). There used to be a way to get the that-one-listener by passing in an options object (I don't think this works any more) it was something ugly like var opt = {}; gun.on(cb, opt); opt.on.off(); Any recommendations on how I could make this easy with the API? I've struggled with this myself for a while, but can't think of any elegant solution, so your input would be valuable.

Yeah. When I was working on the performance version of gun, I discovered all sorts of evil/terrible things about V8 and JS. Since then, I've lost any/all hope that it is possible for a framework to do this for you, because simple things like re-rendering the entire list for you (which is easy/nice) versus idempotently updating items, wind up being such different performance profiles (O(N) vs O(1)). Even supposed "magical" things like React's DOM diffing still have to do a lot of work (aka: performance loss) underneath to figure this out for you (somebody I know at a big SVG shop said React was too slow updating thousands of items in a dataviz).

At this point, I'm starting to turn my eyes towards things like PixiJS and pure WebGL GPU shaders. A bit overkill, but I'm excited for the day when rendering a web page on modern hardware will be less intensive than running Crysis on my decade old desktop. :P

[amark]

Oh, and you might like this extension: https://github.com/IMGNRY/load (it recursively loads the full depth of any object you ask for, and then returns it to you once it is done. Having a live updating version of this might be nifty too)

[victornoel]

about performances

Maybe we are not talking about the same kind of use cases anyway :)
For me, I prefer choosing a framework like angular where I can express things declaratively (it is easy to write, easy to maintain and easy to reason about), and anyway I'm not trying to render in the browser a page with 10000000 items, but for this kind of use case, maybe angular is not the best.

I know also that you can implement some kind of interfaces on your iterable so that the angular loop does not repaint everything but detect where are the added or removed elements (https://angular.io/docs/js/latest/api/core/index/IterableDiffer-interface.html).
This could be another improvement that would take advantage of get().map().on() (and not get()on() directly as I currently do) I think… I will keep that in mind :)

about off()

I understand that since you have chaining, you can't simply return an object via on() that corresonds to this current listener… or you could but you would need to change the API and on() would return a different context tied to THIS listener (but exactly similar to the context of the get(), only with information about the registration on top of it).

Something like the option would work too, even if it's not the most elegant…

With the current behaviour, it is impossible to off() an on() until there was ONE change, right? So in any case, you will have to tackle this issue at one point or another :)

About rxjs

Yes, I understand your arguments, again, I think it is about the use cases, and mine are maybe not the main ones of gun.js :)

I checked the libraries, only water-gun is doing rxjs bindings, but it seems it is based on old APIs and does not take care of unsubscriptions (calling off()) so that's why I was writing my own even though I would prefer reuse something else!

Conclusion

To complete this PR, I would prefer to have a complete rxjs wrapper, and for this, I would need a way to off() a current on() without doing it from inside the callback.
If you think it will take time, I can hack something around, for this :)

In the longer term, I would like to see if we can do something with IterableDiffer from angular to improve performances :)

[victornoel]

For the record, delete data._ seems to break things, so I use underscore's omit instead, even though it's more costly.

Also filtering of the null values gives me this kind of error:

Error: Uncaught (in promise): TypeError: Cannot read property 'obj' of undefined
TypeError: Cannot read property 'obj' of undefined
    at Gun.<anonymous> (http://localhost:8081/main.bundle.js:181:69)
    at Object.ok [as use] (http://localhost:8081/vendor.bundle.js:28399:12)
    at Object.use [as next] (http://localhost:8081/vendor.bundle.js:27967:7)
    at Object.input [as next] (http://localhost:8081/vendor.bundle.js:27779:11)
    at Object.onto [as on] (http://localhost:8081/vendor.bundle.js:26751:19)
    at Object.map (http://localhost:8081/vendor.bundle.js:27496:16)
    at Type.obj.map (http://localhost:8081/vendor.bundle.js:26690:19)
    at Object.next (http://localhost:8081/vendor.bundle.js:27456:5)
    at Function.onto [as on] (http://localhost:8081/vendor.bundle.js:26751:19)
    at Object.Gun.HAM.synth [as next] (http://localhost:8081/vendor.bundle.js:28247:9)
    at Object.onto [as on] (http://localhost:8081/vendor.bundle.js:26751:19)
    at Object.Gun.on.ack (http://localhost:8081/vendor.bundle.js:27541:10)
    at Object.root [as next] (http://localhost:8081/vendor.bundle.js:27427:13)
    at Object.onto [as on] (http://localhost:8081/vendor.bundle.js:26751:19)
    at Gun.chain.on (http://localhost:8081/vendor.bundle.js:28355:14)
    at Gun.<anonymous> (http://localhost:8081/main.bundle.js:181:69)
    at Object.ok [as use] (http://localhost:8081/vendor.bundle.js:28399:12)
    at Object.use [as next] (http://localhost:8081/vendor.bundle.js:27967:7)
    at Object.input [as next] (http://localhost:8081/vendor.bundle.js:27779:11)
    at Object.onto [as on] (http://localhost:8081/vendor.bundle.js:26751:19)
    at Object.map (http://localhost:8081/vendor.bundle.js:27496:16)
    at Type.obj.map (http://localhost:8081/vendor.bundle.js:26690:19)
    at Object.next (http://localhost:8081/vendor.bundle.js:27456:5)
    at Function.onto [as on] (http://localhost:8081/vendor.bundle.js:26751:19)
    at Object.Gun.HAM.synth [as next] (http://localhost:8081/vendor.bundle.js:28247:9)
    at Object.onto [as on] (http://localhost:8081/vendor.bundle.js:26751:19)
    at Object.Gun.on.ack (http://localhost:8081/vendor.bundle.js:27541:10)
    at Object.root [as next] (http://localhost:8081/vendor.bundle.js:27427:13)
    at Object.onto [as on] (http://localhost:8081/vendor.bundle.js:26751:19)
    at Gun.chain.on (http://localhost:8081/vendor.bundle.js:28355:14)

[victornoel]

So I hacked sometimes for the unsubscription, so if you want you can merge.

Are you using some kind of obfuscating tool before pushing code to github? The code in src/ is really HARD to read, it's impossible to understand how things work :/ I tried to find the source of my problem with Gun.obj.map but it's impossible, I ended using underscore's pick instead.

[victornoel]

note also that the code in gun.helper.ts could be made a library similar to water-gun but using simpler code and supporting unsubscription.
@kristianmandrup feel free to reuse it to reimplement the rx bindings in water-gun :)

[amark]

Sorry been gone the last week, my wife was graduating with her PhD. I'll check back in on this soon :).

[victornoel]

@amark no problem, congrats to her, big moment, I also had the pleasure of going through a PhD :)

[amark]

Sorry this took way too long!

@victornoel what was your PhD in?

Pulling finally! Thanks so much. :)

[victornoel]

thanks!

It was on software engineering for self-organising and multi-agent systems :)

[amark]

@victornoel woah, that sounds incredible. Did you implement any prototypes? I'm definitely trying to build in that direction ;) would love your ideas/thoughts.

[victornoel]

The research team I was in did only that, building prototype and real applications of self-organising and emergent systems, mostly for problem solving, optimization and other kind of complex IA stuffs.
I was more on the tooling/programming/modelling part (I made a Tool/DSL for describing/implementing architectures for programming multi-agent systems during my thesis), but I did work on some systems: as a researcher, mainly one prototype for (ASCENS) to illustrate methodological guidelines on the engineering of self-organising systems, the code is on github actually: victornoel/ascens-robots-amas and the publication quite easy to read :)

If you are interested I also highly recommend reading about the AMAS approach promoted in my previous research team, it's not easy to understand at first, but it's impressive while you get the way of thinking of it :)
It's an approach to emergent system design focused on programming at the local level while having the system exhibiting a desired global behaviour, one of the paper I wrote for ASCENS (above) explains it in simple terms I think.
Nowadays I don't work on this anymore, but I still do a bit of distributed systems, but for oldschool SOA architectures, not the same :)

Thank you for your interest, I rambled a lot there, I must miss this part of my professional life ^^

[amark]

I particularly appreciated this page: http://ascens-ist.eu/casestudies/cloud.html which I got to through trying to find the "search and rescue" case study.

A lot of the terms are way over my head :( but it looks super fascinating. What was the overall conclusion, that you drew personally, from it? Do you think they will work? Do you think most systems will have those sorts of things as their foundation?

No no, I'm definitely curious. For instance, I gave this talk ( https://vimeo.com/208899228/b9bc9eaaa4 skip to 13:30 ) a few months ago, about how I think the future of programming will be based on emergent behavior from databases and codebases understanding their own Turing Complete structures. It is the future I'm wanting/trying to actively build.

[victornoel]

Sorry, took me some time to answer :)

Basically, my personal conclusion is that these are hard questions and most people don't realise it, and don't have any good method/methodology to approach the design of self-organising systems (it's normal I have this opinion, since this was what my work was about ; )
In the ASCENS project, for example for people working on the cloud example, nothing great was really produced. They produced results in their field, made some link to the example, and that was, in a way, the objective of this kind of huge project with 10+ partners, so it's ok (usually these huge European projects are mainly a way to get money more than to really solve the problems of the project. But it's ok, research gets done, and advances are made on many subjects. Also, I personally think smaller projects are better, because it is easier for the partners to actually collaborate).
But the use case were not solved, and I think existing engineered solutions (from the opensource community for example) to cloud management works better to whatever could have been done in this project :)

Engineering this kind of system is complex, and that's why you need completely new design paradigms to approach them. You need to forget about the global functionality and look at local functions and how they could interact so that globally things still work (without trying to understand what's happening globally).
For example, the AMAS approach to design I was talking about says: you have local functionalities of a bigger system (so the designer must decide how to decompose the system in local functionalities, it's mainly what my work in ASCENS was about), and if all these local functionalities only "cooperate" with each others (the AMAS approach has a clear definition of what cooperation means, and there has been a lots of experimentation on how this can be implemented), then the global behaviour will be adequate for whatever it is meant to be. It seems strange like that, but it actually works :)

I don't really think approaches were there is some kind of self-aware software, understanding its own functioning, and stuffs like that have a real future. If you look at a complex system working, after something happens, you can say "ah, it did this then that, because of that", but you couldn't have known before hand, and the system itself couldn't either. Emergence is exactly because the complexity resulting from the interactions is too much complex :)

I haven't watched the video in details, but if I understood right, you talk about code self-elvolving because of their execution context, something like that, right?