Bug: stale feed. Large contact list.

[alltheseas]

Tracking Issus

These are steps needed to fix this issue

• Debug elsat's account #2206
• Add test case for large contact list #2207

What happens
Feed seems stuck. Only hours long posts show.
Universe view is showing new posts. DMs work. Notifications work.

What I expect to happen
I dont expect to see a stale feed.

Link to noteID, npub

reported by elsat
npub1zafcms4xya5ap9zr7xxr0jlrtrattwlesytn2s42030lzu0dwlzqpd26k5

Screenshots/video recording
https://github.com/damus-io/damus/assets/64376233/95ae8255-4cfb-4be0-8cf0-b56124ebe0d4

a	b

** Versions **
Damus version: 1.9 (1) 90c68fe
Operating system version: iOS 17.4.1]
Device: e.g. iPhone 13 mini

Steps To Reproduce
Steps to reproduce the behavior:

Unclear.

[alltheseas]

For brief moments feed becomes real time - for instance after not having used the app for an hour.

After app close, and open feed is stale oncemore.

[alltheseas]

Stale feed problem persists

universe view is real time

[jb55]

On Tue, Apr 30, 2024 at 02:08:53PM GMT, alltheseas wrote: Stale feed problem persists  universe view is real time 

can you try reducing your number of relays and see if that fixes anything. I've heard reports of this and I'm not sure why more relays breaks things.

[alltheseas]

reduced relays - does not change stale feed

[alltheseas]

Turned on damus on old phone - same issue present

[alltheseas]

If yall log in with my npub are you able to recreate?

[jb55]

(this was sent via email but also sent to the github issue as a comment) Re: [damus-io/damus] Bug: stale feed (Issue #2194) Link: #2194

On Sat, May 04, 2024 at 07:09:46PM GMT, alltheseas wrote: reduced relays - does not change stale feed [..] Turned on damus on old phone - same issue present

Your contact list is large and contains lots of hashtags and pubkeys. This may have something to do with it. We should use these to build some test cases for this to dig down and see what is going on. I have some suspicions: ## Potential problems - Your contact list is about 150KB, subscribing to that 9x is about 1.4MB of upload every time your phone reconnects. The sheer weight of this is likely causing stalling. - Relays may reject large subscriptions like this, leading to empty feeds. - There may be some data encoding problem in the list leading to a rejected subscription, but I did not see any issue on your contact list. I did see some lists like this though. ## Options - Outbox model to only send subsets of your list to the sets of relays that your follow list is on. This is probably the best path without writing any new NIPs but may lead to missed messages depending on how damus chooses which relay subsets to read from for each pubkey subset. If most users have the damus relay on their list we may be back to the same problem. Relay reliability is also a concern in this scenario as well as all the other outbox model issues in Damus which still needs to be resolved by the local relay model[1][2] - Only choose a subset of the relays to send the timeline subscription to. This may result in missed messages. ## Workarounds Since your contact list is so large, you may need to either: - Start going through and unfollowing people who are no longer active or - Severely reduce your relay list to 1-4 until we have a fix. This is a huge problem with the protocol at the moment with not many obvious solutions. I try to keep my contact list under 1000 for this reason. We should definitely run some simulations on how much data the outbox model will save us here, with real world data from real relay and contact lists such as yours. ## Next steps - Look into ways for damus to suggest stale accounts to unfollow - Put together potential outbox model subscription selection algorithms and see how much it helps us (may only help with a diverse selection of relays on relay lists) - Use elat's current contact list as a testbed for fixing this issue since it is an example testcase that is not working. - Look into follow caps to reduce the damage of this issue but this may be controversial, perhaps we can find a solution without doing this. ## Ideal solution - Look into a potential new NIP for relays. Relays already have your contact list, what if the query was simple as: { "authors": ["kind3:<pubkey>"] } Which would do the same thing as the 150k query except just tell the relay to use the list it has instead of sending it over and over. Then we could send a few bytes instead of 150k * number of relays I prefer this approach, not sure if anyone has proposed it yet. I have a feeling a lot more users are going to be running into this issue so we should starting thinking of solutions soon. Cheers, Will [1] #2041 [2] https://github.com/damus-io/notedeck/

[alltheseas]

Potential additional test case:

note1v2uzw8f488y29mhuafwjrkn45ss79sx2fydn3sedtxkca46rnjcq6u3g74

npub1vwymuey3u7mf860ndrkw3r7dz30s0srg6tqmhtjzg7umtm6rn5eq2qzugd

[alltheseas]

[alltheseas]

Potential test case:

note18mq7htp6dr8qthtdlppeyru9zesxpldxpdsr7evswaxe2wy3qhysmyqmsk

npub1nje4ghpkjsxe5thcd4gdt3agl2usxyxv3xxyx39ul3xgytl5009q87l02j

https://v.nostr.build/LeOaQ.mp4

[danieldaquino]

Reproduced this issue on my end

Device: iPhone 15 simulator
iOS: 17.4
Damus: 1.9 (3) (0d9954290a674e1520164c08050bcfb9291fdd05)
Steps:

1. Login with Elsat's npub
2. Check the home feed. There should be fresh notes.

Results: No fresh notes, most recent post is from several hours ago (Feed is stale)

---

Already working on investigating the root cause.

[danieldaquino]

After several minutes (between 5–10) fresh notes get loaded. I will inspect network traffic and logs

[danieldaquino]

Your contact list is large and contains lots of hashtags and pubkeys. This may have something to do with it.
(...)
Relays may reject large subscriptions like this, leading to empty feeds.

These items seem to be part of the root cause. I am getting several log messages like these ones:

failed to decode event ["NOTICE","ERROR: bad req: total filter items too large"]
failed to parse '["NOTICE","ERROR: bad req: total filter items too large"]'

Relays are rejecting subscription queries for being too large.

I will look deeper to see if there other problems beyond relay rejection

[danieldaquino]

I will look deeper to see if there other problems beyond relay rejection

As far as I can tell:

1. The relay rejection is there and seems to be there and likely the biggest cause of the issue.
2. The stalling due to the large size of the filter might be there as well, but we will be able to better measure it once we fix problem (1)
3. Damus seems to be formatting the contact list filter well, it includes all 2076 people that Elsat is following
4. I believe the actual sending of this contact list filter data is probably working as well, since the relay actively rejects it mentioning too many items.

I will focus on fixing (1) for now. If there are residual issues left after it, those will likely become evident.

[danieldaquino]

@jb55 @alltheseas iteration 1 of the fix is ready. It's a 100% "clean refined solution", but it does seem to resolve the stale feed issue.

Here is a summary of the fix:

Chunk home filter subscriptions to avoid hitting max filter limits

When a user is following several accounts on their follow list, they may
get a stale feed caused by subscriptions being rejected by relays (due to max filter item limits).

This commit creates a temporary fix that gets around the issue by
creating several subscriptions for the home feed, chunking them into
filters with a max of 500 contacts.

Testing:

Procedure:
1. Login with Elsat's npub (Or some account that follows about 2K people, and connected to about 8 relays)
2. Check the home feed. There should be fresh notes.

Fix test details
Device: iPhone 15 simulator
iOS: 17.4
Damus: This new commit
Results:
- Fresh notes appear, most recent post is from a few seconds ago.
- However, profile images or names seem to take longer to load (I might need to chunk those metadata subscriptions as well

I will try to refine this fix on my second iteration:

• I will try to do the same with metadata subscriptions to get profile pictures and names faster (I will do this to improve the experience)
• I can also try to refine this to do something smarter: (I am not sure if this would be within the scope of this ticket or if we want to refine this later)
  • I can try to pull the contact lists from those friends, see which relays they are connected to, to try distributing the friends more intelligently across different relay subscriptions
  • I can try to randomize the distribution of friends across different relay subscriptions

@jb55 any thoughts or opinions on the first iteration of the fix, and the potential refinements?

Ideal solution

• Look into a potential new NIP for relays. Relays already have your contact list, what if the query was simple as:

{ "authors": ["kind3:"] }

@jb55, I like this too, I think that solution would be nice. Perhaps one way we could generalize this solution further would be to add the ability to define Nostr filters as a Nostr Event, and then reference those filters via event IDs?

[danieldaquino]

Iteration 2 of the fix

• Fixed stale profile metadata through a similar chunking mechanism
• Made the chunking logic more reusable
• Added automated tests around the chunking logic
• Changed chunking of subscriptions into chunking of the filters themselves within the same subscription request
• Did more thorough testing

Patch and all other details sent over email: https://groups.google.com/a/damus.io/g/patches/c/3zVnZPoOXhw

@jb55 please let me know if this needs further improvements or if this is good for pushing to master. Thanks!