[bug] Empty timeline pages can't be properly handled by some clients

[benjaminbhollon]

Describe the bug with a clear and concise description of what the bug is.

While the recent release candidate fixes some inconsistent timeline paging by providing a next link in the header even if the current page is empty, some clients aren't handling this correctly. Specifically, tut and Semaphore so far are not allowing you to scroll past an empty timeline page.

In contrast, someone using Tusky is able to scroll indefinitely.

I'm expecting this is due to clients ignoring the link header and using the ID of the most recent successfully fetched post as the maxid field, leading to repeatedly requesting the same empty page instead of continuing past it.

What's your GoToSocial Version?

v0.12.0-rc1

GoToSocial Arch

arm64 binary

What happened?

On scrolling through the local timeline on alpha.polymaths.social with some clients, the timeline ends unexpectedly.

What you expected to happen?

Being able to continue to scroll through the local timeline.

How to reproduce it?

Use Semaphore and try to scroll through the local timeline; if necessary, I'm happy to provide an account on my instance since I'm aware most GoToSocial instances are single-user and don't really use the local timeline.

Anything else we need to know?

I'm happy to provide any diagnostics you need. Hopefully we can get this fixed before the full release of Spaghetti Sloth comes out? Would really be frustrating to have to wait another whole release cycle. I get that development pace may or may not fit that, though. If necessary, we can find clients that work.

[benjaminbhollon]

Tested some more clients:

• phanpy.social and elk.zone let you keep scrolling, but feed you federated posts, not local posts? Pretty odd behavior imo, not sure what's causing it.
• Ice Cubes stops at the blank page, like tut and semaphore do.
• the toot CLI works perfectlye letting you scroll as far as you like and only showing local posts.

[tsmethurst]

This feels to me like a client issue, since we are returning the link headers now per the mastodon API. We could try to resolve this in GtS by not returning until we've paged down a few times more to see if there's more posts ahead, but this is always going to be imprecise since we'll never know how far we need to page to find posts that aren't filtered out.

That said, since semaphore and pinafore and the like aren't really maintained anymore, perhaps we ought to see if we can do something from GtS's side..

feed you federated posts, not local posts

What on earth :')

[tsmethurst]

Oh, the phanpy issue is this: #2253

So, our fault! Will fix that

[daenney]

Does this issue reproduce with Masto proper? If it does, then I'm inclined to say we shouldn't implement specific workarounds for them in GtS. I get its annoying and the clients are unmaintained so a fix is unlikely. But that doesn't feel like the kind of tech debt that should then get externalised to servers instead.

If it doesn't, it would be interesting to understand why and see if we can figure out where our implementation diverges. We don't have a goal of emulating the Masto API bug-for-bug, but for something like timeline paging I think it's probably worth the effort.

[benjaminbhollon]

I've never seen the issue in Mastodon using the same clients.

My guess is that Mastodon has some sort of way that makes sure they never send empty pages, so clients aren't developed with the expectation that's possible? I don't know what that mechanism would be, I haven't looked at the Mastodon codebase much.

[tsmethurst]

Mm okay, probably something we ought to make a fix for then :) Thanks for looking into it @benjaminbhollon

[benjaminbhollon]

Happy to. Again, I'm glad to share any information you'd like to help with diagnosis.

[VyrCossont]

I should note a closely related issue here: search result paging also uses an empty result set as an indicator that there are no more results, and unlike the timeline paging cases, search doesn't use Link headers at all (in either Mastodon or GtS) so that's the only way to indicate that a client has reached the end of the search results and should stop fetching pages.

I think this can actually manifest if a page of search results is entirely removed by visibility filters: even if there are visible results in the next page, the API will return an empty page, and clients will stop loading past that. Will open a new bug if I can confirm this.

There's no reason GtS couldn't use Link headers on search results, but clients won't be looking for them.

[daenney]

Mmmm. Oh dear. I suppose what we could do is check the state of it after the visibility filtering and if it's empty, fetch the next page internally. We probably don't want to keep paging indefinitely, but we could advance it 1-2 times to significantly decrease the changes of ending up with an empty page.

I'm not sure how easy it would be to wire that up though.

[VyrCossont]

It's not that bad for either search results or timeline entries. Just keep fetching and filtering results in a loop until you fill your requested size, hit the max fetches cutoff, or hit the actual end of results. The other option would be to push visibility filtering into the DB, but this would be a ton of work and not interact well with our existing visibility logic, for a case that is probably not suuuuper common.

[benjaminbhollon]

Even just checking the next couple pages would catch the vast majority of cases on my instance. Having the fetch cutoff be configurable might be the best compromise.

[benjaminbhollon]

In case anyone else is looking for a quick workaround in the meantime, I coded a script to try and predict when the local timeline is about to clear and post a placeholder to prevent any pages from being empty.

https://codeberg.org/benjaminhollon/.dotfiles/src/branch/main/.local/bin/gts-timeline-placeholder

I've been running it for a while day and we have nine days of history now. https://alpha.polymaths.social/@placeholder_bot

Definitely a very hacky workaround, but it works! 😆