Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Backfill statuses from remote accounts when first subscribed #34

Open
Gargron opened this Issue Sep 10, 2016 · 26 comments

Comments

Projects
None yet
10 participants
@Gargron
Copy link
Member

Gargron commented Sep 10, 2016

No description provided.

@mjankowski

This comment has been minimized.

Copy link
Collaborator

mjankowski commented Apr 15, 2017

I'd like to take a crack at this, but am still trying to wrap my head around the world of services/workers and which things are triggered when.

My guess here is that somewhere in the collection of remote follow services and workers we want to add a background worker task which pulls the feed of the remote account which was just followed, and import the last 5 or 10 or so items.

The other thing I'm struggling to locate is any code which is effectively "go grab this account's atom feed and import some entries" ... it strikes me as possible that this doesn't exist, because all statuses are pushed onto the server and never (yet) pulled?

Any pointers here are appreciated!

@Gargron

This comment has been minimized.

Copy link
Member Author

Gargron commented Apr 15, 2017

@mjankowski The main roadblock to this is actually #1059, because if you went back and imported 5 statuses from an account that last posted 6 months ago, you would get 5 6mo-old posts on the top of the public timeline (and potentially home timelines as well)

So this should probably stay a wontfix until something is done about IDs and their sorting.

@mjankowski

This comment has been minimized.

Copy link
Collaborator

mjankowski commented Apr 16, 2017

Thanks, I read that thread and looked at more code, and I agree that backfilling is blocked on improving the ordering approach to either get away from ID-sorting, or make ID-sorting reliable for backfilled records. I'll pause on this, and leave a comment over there.

alpaca-tc pushed a commit to alpaca-tc/mastodon that referenced this issue Apr 16, 2017

53ningen pushed a commit to 53ningen/mastodon that referenced this issue Apr 26, 2017

akihikodaki pushed a commit to kagucho/mastodon that referenced this issue May 3, 2017

lnanase pushed a commit to lnanase/mastodon that referenced this issue Aug 9, 2017

walfie pushed a commit to walfie/mastodon that referenced this issue Aug 13, 2017

Merge pull request tootsuite#34 from MasterDalK/default-fixes
Font updates, fix columns past 1080p, settings page, privacy warning

MitarashiDango pushed a commit to MitarashiDango/mastodon that referenced this issue Sep 24, 2017

NCLS
Merge pull request tootsuite#34 from yuzulabo/fix-gauge-icon
knzk-gauge-icon.pngが表示されない問題を修正
@deutrino

This comment has been minimized.

Copy link

deutrino commented Dec 10, 2017

#3307 is about accurate follower counts, but in the comments I discuss some implementation ideas which could theoretically encompass this.

@kit-ty-kate

This comment has been minimized.

Copy link

kit-ty-kate commented Feb 25, 2018

Any updates on this ?

@Gargron

This comment has been minimized.

Copy link
Member Author

Gargron commented Feb 25, 2018

It's now possible to implement this, but it's not clear how many items to fetch (and if you say "all", think about accounts with 100,000 statuses...)

@kit-ty-kate

This comment has been minimized.

Copy link

kit-ty-kate commented Feb 25, 2018

Mmh maybe it can be done by displaying a button where the missing toots are, to fetch the 5 or 10 next toots ? So it would be done manually by users little by little.

@deutrino

This comment has been minimized.

Copy link

deutrino commented Feb 25, 2018

What about just fetching blocks of 20 toots, triggered by scrollbar position?

@kit-ty-kate

This comment has been minimized.

Copy link

kit-ty-kate commented Feb 25, 2018

@deutrino Generally this seems good, but some toots might be already fetched manually by users and so there would be holes to fill here and there.
For example if there is a profile with no visible toots from one instance and you have access to the urls of some toots and you insert them into the search bar they are going to be fetched and added to the profile of the user in question (from your instance point of view).

@kit-ty-kate

This comment has been minimized.

Copy link

kit-ty-kate commented Feb 27, 2018

@deutrino ok, I was thinking back at what you were suggesting and it seems to be the good thing to do considering that mastodon already serve 20 toots at a time when looking at a profile. So everything can probably be done smoothly on the server side with field saying "I know I'm up-to-date up to this toot/date" and contacting the other instance if we are trying to get toots that are in the "probably not up-to-date" range.

@kit-ty-kate

This comment has been minimized.

Copy link

kit-ty-kate commented Mar 12, 2018

Maybe fixing this would also partly fix #6137, as looking for a deleted account toots would give hints to the server to remove the account from its local database ?

imbsky referenced this issue in kibousoft/mastodon Apr 26, 2018

NCLS
Merge pull request #34 from yuzulabo/fix-gauge-icon
knzk-gauge-icon.pngが表示されない問題を修正

imbsky referenced this issue in kibousoft/mastodon Apr 29, 2018

imbsky
Merge pull request #34 from yuzulabo/fix-gauge-icon
knzk-gauge-icon.pngが表示されない問題を修正
@kit-ty-kate

This comment has been minimized.

Copy link

kit-ty-kate commented May 21, 2018

Now that #7459 has been merged, I guess it seems simpler to finally implement this. @ThibG ?

@kit-ty-kate

This comment has been minimized.

Copy link

kit-ty-kate commented May 21, 2018

Mmh, thinking again, maybe not. Sorry. They seemed closely related in my head but seems like I'm too tired today :/

@kit-ty-kate

This comment has been minimized.

Copy link

kit-ty-kate commented May 29, 2018

any chance for this issue to get fixed or partially fixed by the next release (2.5) ?

@ThibG

This comment has been minimized.

Copy link
Collaborator

ThibG commented May 29, 2018

@kit-ty-kate with some logic similar to #7459 we could probably fetch private toots from a remote user, yeah.
Fetching “up to the N last toots” on first follow is easy enough, but:

  • it would not be applied to users already followed before the update
  • it would not be applied to users not followed
  • “up to the last N toots” is pretty arbitrary

The suggestion of having a “gap” that users could click sounds very nice, but it is a lot harder to implement. Indeed, the protocol mandates toots to be strictly ordered, but I don't think there is a mechanism to make sure you're not missing toots, nor to request a certain range of items, so efficiently filling a gap seems pretty hard. Add to that that some items may only be displayed to some authenticated users, and things can get quite complicated…

walfie added a commit to walfie/mastodon that referenced this issue Jun 17, 2018

@deutrino

This comment has been minimized.

Copy link

deutrino commented Jul 21, 2018

I think a good partial solution to the problem of empty or stale profiles of remote users would be to prefetch n last toots whenever there is any interaction between a local user and a toot from remote user, and when particular criteria are met.

For example: if our local user favs, boosts, or even reads a toot from a remote user, we check if there are any toots from that remote user in the db which are less than (for example) 1 week old; if not, we dispatch a job to fetch the last n (and maybe touch a timestamp somewhere that then prevents this type of fetch for a few hours, to prevent thundering herd problems if a toot from an inactive account suddenly becomes popular).

This is a very simple "heat" heuristic. If a local user interacts with a remote user in any way - faving, reading a boost, etc - they are more likely to open that user's profile than that of a random user. With good criteria for when to prefetch, and with the implementation of database compaction in #1554, I think this would be a considerable usability improvement, particularly for small instances / those not using relays.

@deutrino deutrino referenced this issue Jul 21, 2018

Closed

Sweep up old federated toots from local DB #1554

1 of 1 task complete
@trwnh

This comment has been minimized.

Copy link
Contributor

trwnh commented Aug 6, 2018

I'm not sure if this deserves a separate issue, but if you follow a locked account and you're the first person to do so on your server, then you can't fetch old followers-only posts. I followed a locked account that had a pinned public post with followers-only replies, where the public post says only "please read:" and the followers-only replies... were not loaded, and thus unable to be read.

I assume this is because the older statuses weren't backfilled, but I'm not sure if there's a different or better way for your server to discover followers-only replies besides backfilling.

@nightpool

This comment has been minimized.

Copy link
Collaborator

nightpool commented Aug 6, 2018

@trwnh

This comment has been minimized.

Copy link
Contributor

trwnh commented Aug 7, 2018

@nightpool you'd have to know the URL of every single post though, wouldn't you?

@ThibG

This comment has been minimized.

Copy link
Collaborator

ThibG commented Aug 7, 2018

Yes, you would need to know the URLs, which you typically don't know when those are replies to a toot you can see.
It's a bit of an issue, as those replies are likely to be old posts, and thus not fetched by the “fetch last n toots” proposal.

@nightpool

This comment has been minimized.

Copy link
Collaborator

nightpool commented Aug 7, 2018

@trwnh

This comment has been minimized.

Copy link
Contributor

trwnh commented Aug 8, 2018

Yeah, I think that's grounds to maybe reconsider a pure numerical approach. This might also overlap with other issues about loading missing toots in a chain due to privacy settings.

@thebaer

This comment has been minimized.

Copy link

thebaer commented Dec 14, 2018

Anyone else interested in working on this? Or any advice on where to get started? Would really like to get this feature in (as mentioned in #9525).

@sikkdays

This comment has been minimized.

Copy link

sikkdays commented Jan 9, 2019

If this issue is too complex to solve or we cannot agree on how to make this work, what about a button or text message telling users to look at the full profile?

It's just my opinion, but discovery is always an issue in social networks. Thus, I find this issue paramount to growing the decentralized and federated world. New people I invite to my Mastodon instance have no idea that my server has never connected to another and that's why the "toots" section of the profile is empty. Therefore, a simple message in the toots section of a profile that said, "There are no toots here, trying going to the user's page by clicking on their icon above" might be helpful. Something more elegant, I'm just rattling this idea off before I lose it.

@trwnh

This comment has been minimized.

Copy link
Contributor

trwnh commented Jan 10, 2019

@trwnh trwnh referenced this issue Feb 6, 2019

Open

import / export of user data #814

0 of 12 tasks complete
@ealgase

This comment has been minimized.

Copy link

ealgase commented Feb 24, 2019

This is probably my biggest gripe with the platform, and I think it would be a roadblock in trying to get my friends to switch over. It may be hard to implement but it would be VERY useful.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.