Our federation model produces dangling data #6806
I wasn't sure of the best place to put it. The question I want to raise is quite fundamental for the federation, but too technical to put it on loomio, so I think github fits better. And it's actually an issue. I'm also not aware of any previous discussions on that topic, but it is well-known for sure for those who work on the federation.
So, we use push model for the most of our interactions. However we use pull model for some. This includes profiles fetching and posts fetching now and will include more in future, like comments fetching and if we ever implement the message editing feature it'll be affected as well.
Our push model implementation now supposes that users are exchanging messages only when they are sharing. It doesn't matter if the sharing is mutual or not. But it must be there. Updates, including post comments, retractions, profile updates and profile deletions are delivered only to those pods where a user have contacts.
On the other hand, we use pull model that allows us to fetch profile or post without having a sharing relashionship set up. This leads to the problem, that if there is no one on our pod who shares with some person whose content we have fetched, then we can't trust this content. We will definetely miss any updates on this content (user deletion #6791, retraction #6436, comments). This also happens if there was sharing but it was removed then.
I guess this applies to the public posts only, since we don't use the pull model for private posts. The issue wasn't noticed too much by users because most of the time we have some kind of sharing set up, and also most of the users use big pods so there is at least someone on your pod who shares with the other person, so you get updates through his subscribtion anyway. But if the federation grow, it'll become worse. Also if #4491 is fixed it'll make the thing worse. Also this will make the account migration less reliable.
We shouldn't allow this to happen.
Here is a solution I think of. First, if there ever was sharing between Alice from podA and Bob from podB, and they stop sharing, both pods must still send updates on user profiles, even if there are no one left who share between the pods. Second, the same stands for posts. If a post was federated, all the updates must be delivered anyway, including retractions, comments, in future - editions, again even if Alice and Bob don't share anymore and there is no one between podA and podB who shares. Third, if there was no user sharing between podA and podB, and podA fetches some profiles and posts from podB, then podA must also send some feedback address to podB, so podB knows that it must send update information somewhere. Thus it'll be podA's responsibility to set a correct address to receive updates. For podB podA then becomes just another subscriber along with ordinary users.
@SuperTux88, since you are the main federation contributor at the moment, I hope to get feedback from you. Maybe you already have idea on solution for that or maybe you're going to fix it? You don't want me to participate to your federation work, but if it is the part you aren't working on, then maybe I can work on it?
I'm really interested to get this fixed, because it also affects in some ways the issues I promised to fix as a part of my campaign including the account migration itself. However I won't start any work on this before getting any feedback from the core.
So for clarification do pods only fetch a particular post once and not the updates? And if so, how does it decide which posts to fetch? Does pod A try to dig through all public posts in pod B that are tagged with something that a user on pod A follows and then check whether they already have a local copy?
@theworldbright, well, I'm not sure about the present state, posts fetch was switched off due to a security issue, but it's going to be enabled again when the new fetch procedure is integrated (see https://github.com/SuperTux88/diaspora_federation/issues/31). So the main cases for that issue now are different:
referenced this issue
Jul 5, 2016
Not much changed here. You can subscribe to a post now (by clicking the little bell icon), and then you will receive all updates to this post (the post-author-pod needs to have the new federation), independent of the contact-state. This didn't work properly before (AFAIK), and works now. But that's everything that changed now.