Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.Sign up
GitHub is where the world builds software
Millions of developers and companies build, ship, and maintain their software on GitHub — the largest and most advanced development platform in the world.
twitter: fully support oversized block lists #764
we recently started fetching and obeying twitter block lists in #473. more implementation details in snarfed/granary#40. we use twitter's
there are currently three bridgy twitter users with >75k users on their block lists. one has 149996 total (!); i haven't counted the others. they now hit the rate limit and fail pretty much every poll. we cache block lists, but we don't split and coalesce block list fetches across polls.
those three users' last webmentions from bridgy were 2 mos ago, 3 mos ago, and never, and the solutions to this are awkward at best, so i'm not prioritizing this right now.
the memcache docs are light on size calculation details, but they do say Any type. If complex, will be pickled. i guess i could pickle the list, measure, and incrementally cut it down until it's under 1MB, but for now i'm just going to drop the cutoff down to 35k.