Support for DecSync

[Hund]

Quoted from their repository:

DecSync (Decentralized Synchronization) synchronizes key-value mappings using the file system in a conflict-free way. It uses a synchronized directory to synchronize the mappings. This can be done without a server by using for example Syncthing.

https://github.com/39aldo39/DecSync

It would be a neat thing to have. Things like TT-RSS is nice, but it eats resources llike crazy.

[Minoru]

I wouldn't compare DecSync with TT-RSS and other such services, since DecSync only stores a part of the metadata (see their RSS doc). TT-RSS stores everything, and it also fetches things even if Newsboat is offline.

Given users will have to synchronize DecSync's directory anyway, I don't see why they won't synch their cache.db and urls files directly. This gives them the same benefits that DecSync does, plus they don't have to fetch articles multiple times (and thus don't risk losing some from the high-volume feeds).

@Hund, can you please list some specific problems of yours that would be solved if Newsboat started supporting DecSync?

[Hund]

I meant it was an alternative to TT-RSS. I know that TT-RSS outperforms DecSync in terms of functionality several times over. :)

So you can sync the files cache.dband urls between two running isntances of Newsboat without any issues? That could be an alternative since both my clients have acess to my the same disk.

If I can use the same database across different instances of Newsboat that's running and writing to them at the same time, well, then there's no need fo DecSync. :)

[Hund]

But If I share the same database, they'll still update the feeds twice as much as with TT-RSS or DecSync. Which will cause a lot of unnecessary read and writes.

[Minoru]

Sorry, why did you close the issue? I'll reply anyway, and if you want to continue the discussion, please re-open this. I don't this it's resolved yet.

So you can sync the files cache.dband urls between two running isntances of Newsboat without any issues?

Ah, no, that's not possible. Theoretically we could implement multiple read-only access. Writes are much harder, since we'll need conflict resolution and a way to notify other Newsboats of the change (which is a big change architecture-wise, because Newsboat doesn't expect the cache to change under it).

But If I share the same database, they'll still update the feeds twice as much as with TT-RSS or DecSync. Which will cause a lot of unnecessary read and writes.

That can be fixed: we can add a "last updated" field to the database. Newsboat will check it ocasionally, and if it's older than the reload-time, set a "reloading" flag in the DB and reload feeds. But that opens us to interesting user configuration errors; e.g. if one client uses ignore-mode "download", some of the ostensibly-filtered articles might still get through if they're downloaded by another client with a different ignore-mode setting or a different set of ignore-article commands.

I'd really like to see a list of problems that can be solved with multiple clients running simultaneously. Can't assess the importance without it.

[Hund]

You may open it up if you want to. But I decided to stick with TT-RSS. It worked okay when I was just using Newsboat on two computers, but I missed reading the news on the phone. :)

[ghost]

I support this! DecSync is surprisingly powerful. I've got a setup on two computers + my phone that works seamlessly. I'd love to switch back to using newsboat but I'm hung up on DecSync

[Minoru]

DecSync doesn't look like any of the other APIs we support, so I'm not quite sure where to fit it in the current scheme of things.

[39aldo39]

I'm trying to understand how Newsboat handles syncing. My current understanding:

Newsboat gets the subscribed URLs from the syncing service at startup, but it doesn't support changing this list. However, it looks like you can subscribe locally to additional URLs by adding them to a file.

Every time an article or feed is marked as (un)read, Newsboat sends this to the syncing service. However, I don't understand how changes are sent back. Some services have a special fetch_feed function (Tiny Tiny RSS, Newsblur and Nextcloud), which could have this information attached. But how does this work for the other services?

[Minoru]

Newsboat gets the subscribed URLs from the syncing service at startup, but it doesn't support changing this list.

Yup, that's correct. There is #106 about the editing part.

However, it looks like you can subscribe locally to additional URLs by adding them to a file.

I believe only query feeds can be added this way. Query feeds aggregate items from "normal" feeds; for example, one can have a query feed of all unread items, or a query feed of items from sports-themed feeds. This is a local thing, Newsboat doesn't synchronize the state of query feeds with the remote feeds (because that's unnecessary).

Every time an article or feed is marked as (un)read, Newsboat sends this to the syncing service.

Yes.

However, I don't understand how changes are sent back. Some services have a special fetch_feedfunction (Tiny Tiny RSS, Newsblur and Nextcloud), which could have this information attached. But how does this work for the other services?

Indeed, these three return an array of items, each item having an unread flag. These special methods are used in src/rssparser.cpp. When Newsboat tries to fetch a feed, RssParser either uses a special method of the current remote API, or just fetches and parses the feed directly. For other services, this only works one way: from Newsboat to the service. Newsboat won't know if an item was already read somewhere else. Now that I think about it more, it appears that DecSync does fit with the second category, and we can even improve the sync there if we hack RssParser further: for these services, it can fetch the feed directly and then ask the service for metadata of each item. Does that make sense to you?

…

-- Regards, Alexander Batischev

[39aldo39]

That makes sense to me! Indeed, syncing from Newsboat to DecSync should be straightforward (regarding the APIs).

For the other way around, it is a bit different. Currently the DecSync API assumes that every stored entry corresponds to a 'global' action in the application. For example, if an article is read, you give a function that marks the article as read in the database. However, in Newsboat the 'plugin' doesn't have directly access to the database. Maybe this can be made possible, or I could add the feature to get the stored entries directly.

[Minoru]

Currently the DecSync API assumes that every stored entry corresponds to a 'global' action in the application.

I'm not sure I understand this correctly. Judging by libdecsync-android, I just create a single Decsync object, and turn each user action (e.g reading an article) into a call to Decsync's method. Isn't that how it works? That can certainly be done in Newsboat, because objects of TtrssApi, NewsblurApi etc live for the whole duration of the program. (Sorry, don't have time yet to dive into any of the apps that already use DecSync).

[Minoru]

Also, with discussion revived, it's high time to re-open the issue.

[39aldo39]

Sorry, I didn't explain it very well. I was talking about getting changes from DecSync to Newsboat. However, rethinking it a bit there isn't really a problem, you can ignore that paragraph.

Also, I had some time and motivation, so I actually started with an implementation in #462! It doesn't fully work yet and there are a few workarounds, but it's mostly done.

[Minoru]

I was talking about getting changes from DecSync to Newsboat.

I'm still curious what you meant :) Reading libdecsync-vala right now, it appears that it can call back into the application upon different events (e.g. a sync ends or a new entry appears). That's indeed in conflict with how Newsboat operates. However, it seems like the library can be used as, well, a library rather than a framework, so there isn't a problem here, really. Is that understanding correct, @39aldo39?

[39aldo39]

Ok, I will elaborate a bit more :)

Let's say we also want to get the read statuses from DecSync to Newsboat. The way to do this with libdecsync-vala, is to call executeAllNewEntries or executeStoredEntries (or initMonitor) and specify the entries which have to be executed. It will then call the corresponding OnEntryUpdateListener. However, this listener is given in the constructor of DecSync, which has no access to the database or the feed object. This is why I thought it would not be possible.

However, there is an Extra parameter which allows you to pass data to the listener in a call of executeAllNewEntries or executeStoredEntries. If we update a feed, we can just pass the feed object around. Since this feed object isn't relevant for all actions, I didn't thought of it. But it can just be an optional type.

[vlamacko]

Is this feature currently planned to be added at some point? I hope this question will not come across as prodding or entitled. Rather I am curious and I would also like to express my support for the request. Thanks!

[Minoru]

Not planned, but it isn't ruled out either.

The comments above and in #462 list some of the problems. I'd be happy to advise if someone were to work on it, but I don't intend to work on it anytime soon—I'm mostly busy porting existing functionality to Rust (#286).

[Minoru]

#668 implements reading half of the functionality via a shared libdecsync library. Write support (marking articles read etc.) is still lacking because Newsboat doesn't have a rich enough API. I'm putting this feature on hold for now because I don't want to add a new dependency just yet; see #668 (comment) for a bit more info on this.

[sergei-mironov]

Hi. My 2c: I miss a feature to synchronize read status between machines. I think that Syncthing-powered cache folder synchronization would be a big step forward because it is very generic, a kind of 'just works' technology.

[joelchrono12]

There are already a couple apps on android that support DecSync thanks to the creator of it who got them up and running!
Because of that I hope we get to see some support from newsboat too!

I just comment to show there is still some interest :P