New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

twitter: fully support oversized block lists #764

Open
snarfed opened this Issue Aug 9, 2017 · 1 comment

Comments

Projects
None yet
1 participant
@snarfed
Owner

snarfed commented Aug 9, 2017

we recently started fetching and obeying twitter block lists in #473. more implementation details in snarfed/granary#40. we use twitter's blocks/ids endpoint (with count=5000), which is rate limited to 15 calls per 15m per user. this means that we can only fetch 75k members of a user's block list at one time.

there are currently three bridgy twitter users with >75k users on their block lists. one has 149996 total (!); i haven't counted the others. they now hit the rate limit and fail pretty much every poll. we cache block lists, but we don't split and coalesce block list fetches across polls.

those three users' last webmentions from bridgy were 2 mos ago, 3 mos ago, and never, and the solutions to this are awkward at best, so i'm not prioritizing this right now.

@snarfed snarfed added the listen label Aug 9, 2017

@snarfed snarfed changed the title from twitter: handle oversized block lists to twitter: fully support oversized block lists Aug 23, 2017

@snarfed

This comment has been minimized.

Show comment
Hide comment
@snarfed

snarfed Aug 23, 2017

Owner

my stopgap solution to this is to gracefully handle when we get rate limited and use the partial block list contents that we've fetched so far (06f5987). the cap i ended up with is actually 40k, not 75k, since i memcache block lists, and memcache values are limited to 1MB.

Owner

snarfed commented Aug 23, 2017

my stopgap solution to this is to gracefully handle when we get rate limited and use the partial block list contents that we've fetched so far (06f5987). the cap i ended up with is actually 40k, not 75k, since i memcache block lists, and memcache values are limited to 1MB.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment