Bug: out of order & missing items from user profile

[hacdias]

Hey 👋

I imported quite a lot of bookmarks and annotations to Margin. Since then, I can't see any of the annotations and bookmarks I previously made using Margin on my profile (or on /bookmarks or /annotations). The records still exist, so it's probably some issue with sorting/ordering/indexing somewhere.

Update

Okay. I just did some inspection at the code on my phone while in the train and I have findings to share. There's two issues. Both issues occur only when (1) the user is seeing their own feed; and (2) the user is logged in. This is because the user's own feed is provided by serveUserFeedFromPDS when logged in.

(a) Wrong ordering of items

• serveUserFeedFromPDS relies on listRecords, which sorts by TID.
• When the user imports older bookmarks, chances are the TIDs are still based on newer dates, meaning they will show up first.

I see two possible solutions:

1. Live with it and don't do anything
2. Just provide the user's own feed from the database.

I prefer 2 for consistency, and that would also fix the next issue.

(b) Missing items

• listRecords uses a cursor for pagination. serveUserFeedFromPDS goes around this by using a larger limit (limit + offset).
• On the frontend, hasMore is defined if items >= items.length.
• That's a problem. The PDS may not always return as many items as in the limit, even if there are more afterwards. That's why it returns a cursor to tell if there's more.
• This also means that every time the user presses "Load more", all previous items are fetched, plus the new ones (on the backend). That can become a strain once there's thousands of items.

Possible solutions:

1. Just use the database normally. The current situation is also probably not scalable once the user has 1000 bookmarks. Are we gonna fetch 1000 bookmarks in one go? This would also solve the previously mentioned issue.
2. When the user is logged in, use cursor based pagination. This would involve quite some refactoring.****

[hacdias]

I updated the comment above with the findings. I'm up for implementing it, but I think it needs to be discussed. In my opinion, I'd rather provide the user's own feed from the database. This also ensures that there's less code that needs to be maintained, and that the output will be the same no matter if the user's logged in or not.

However, I see that providing the user's own feed from the PDS is a way of updating the database. So if something goes wrong while ingesting, it'll still be updated here.

My preferred solution would be:

1. Always use the database for providing the user's profile.
2. Create some mechanism through which the user profile could be "repaired" if the user notices something missing.

If we still prefer to use the PDS (wrong ordering of items seems minor issue to me, because not a lot of people will have this issue), we need to make the pagination cursor based for the PDS in order to make it work properly. But that means sending both an offset and cursor, so the frontend needs to be aware of how to handle that.

Wdyt @scanash00?