New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Twitter: do not backfeed from blocked accounts #473

Closed
kylewm opened this Issue Sep 15, 2015 · 17 comments

Comments

Projects
None yet
6 participants
@kylewm
Collaborator

kylewm commented Sep 15, 2015

Discussed yesterday in IRC, we should avoid backfeeding tweets from blocked users. I confirmed that the Twitter search API returns tweets from blocked users with no indication that they are blocked.

The blocks/ids endpoint gives up to 5000 ids at a time, and is rate-limited to 15 calls per user per 15 minutes. (the largest shared blocklist I know of has 10k accounts on it)

  • proactively filter out blocked users when searching for replies
  • on individual tweets, serve 410 tombstone so that mentions from blocked accounts will be deleted
  • periodically check for old responses from accounts that are now blocked, and re-propagate them.

@snarfed snarfed added the listen label Sep 15, 2015

@snarfed

This comment has been minimized.

Show comment
Hide comment
@snarfed

snarfed Sep 15, 2015

Owner

interesting! i did indeed see the IRC discussion. thanks for writing it up!

i have to admit, i'm a bit skeptical of the ROI on this one. i definitely do see the value and @kevinmarks's request, but i'm not quite sure the overall impact will be worth the work and added complexity. i'm open to discussion though!

(i'd also be curious to see what the comparable features in other silos are, and how much work it would take to support them too.)

the IRC discussion also covered the key point that webmention recipients (both CMSes and services like @voxpelli's great https://webmention.herokuapp.com/) also need to support at least manually deleting webmentions too, if not blocklists as well. i expect a common case is that you get a bad wm from a new person, so you then add them to your blocklist and also need to remove the bad wm. yes, bridgy could detect blocklist changes, go back and search for all wms from that user, and resend and 410 them, but i'm definitely skeptical of the ROI on that. (not to mention i suspect 410 support among wm recipients is far from widespread.)

sorry for the negativity. definitely still worth discussing more!

Owner

snarfed commented Sep 15, 2015

interesting! i did indeed see the IRC discussion. thanks for writing it up!

i have to admit, i'm a bit skeptical of the ROI on this one. i definitely do see the value and @kevinmarks's request, but i'm not quite sure the overall impact will be worth the work and added complexity. i'm open to discussion though!

(i'd also be curious to see what the comparable features in other silos are, and how much work it would take to support them too.)

the IRC discussion also covered the key point that webmention recipients (both CMSes and services like @voxpelli's great https://webmention.herokuapp.com/) also need to support at least manually deleting webmentions too, if not blocklists as well. i expect a common case is that you get a bad wm from a new person, so you then add them to your blocklist and also need to remove the bad wm. yes, bridgy could detect blocklist changes, go back and search for all wms from that user, and resend and 410 them, but i'm definitely skeptical of the ROI on that. (not to mention i suspect 410 support among wm recipients is far from widespread.)

sorry for the negativity. definitely still worth discussing more!

@snarfed

This comment has been minimized.

Show comment
Hide comment
@snarfed

snarfed Sep 15, 2015

Owner

btw, i'm painfully aware that i'm privileged and don't get harassed online - and deliberately don't participate in [redacted controversial topic] - so i'm probably biased. i'm trying hard to balance that against my usual judgment process, but that's tough. feedback is obviously welcome.

Owner

snarfed commented Sep 15, 2015

btw, i'm painfully aware that i'm privileged and don't get harassed online - and deliberately don't participate in [redacted controversial topic] - so i'm probably biased. i'm trying hard to balance that against my usual judgment process, but that's tough. feedback is obviously welcome.

@voxpelli

This comment has been minimized.

Show comment
Hide comment
@voxpelli

voxpelli Sep 15, 2015

An alternative solution would perhaps be to translate the Twitter blocklist into an IndieWeb blocklist that could be consumed by eg. my endpoint to auto-block certain content (preferably with Pubsubhubbub-support to ensure it stays up to date in realtime). That way the solution will work even if a user starts to maintain its own Silo-independent blocklist.

True IndieWeb WebMentions can never be hidden from the source – they always have to be hidden through the target – so for Brid.gy to bridge also block-data to the IndieWeb it should probably expose that data in a way that non-silos can do as well.

Closest related ticket at my endpoint is: voxpelli/webpage-webmentions#28

Will add one for consuming block lists.

Related wiki-pages:

An alternative solution would perhaps be to translate the Twitter blocklist into an IndieWeb blocklist that could be consumed by eg. my endpoint to auto-block certain content (preferably with Pubsubhubbub-support to ensure it stays up to date in realtime). That way the solution will work even if a user starts to maintain its own Silo-independent blocklist.

True IndieWeb WebMentions can never be hidden from the source – they always have to be hidden through the target – so for Brid.gy to bridge also block-data to the IndieWeb it should probably expose that data in a way that non-silos can do as well.

Closest related ticket at my endpoint is: voxpelli/webpage-webmentions#28

Will add one for consuming block lists.

Related wiki-pages:

@kevinmarks

This comment has been minimized.

Show comment
Hide comment
@kevinmarks

kevinmarks Sep 15, 2015

I like that idea, voxpelli - also translating those you follow to a whitelist may make sense. the challenge is that these are supposed to be non-public, so we'd have to do a confirmation between the services that it is the end-user requesting it, which is delegated auth etc.

I like that idea, voxpelli - also translating those you follow to a whitelist may make sense. the challenge is that these are supposed to be non-public, so we'd have to do a confirmation between the services that it is the end-user requesting it, which is delegated auth etc.

@voxpelli

This comment has been minimized.

Show comment
Hide comment
@voxpelli

voxpelli Sep 15, 2015

If one wants a block list to be possible to be both public and private (Twitters are currently private so any block lists from twitters should probably be private when exposed through Bridgy) then using the brainstorm documented here would be useful: https://indiewebcamp.com/private_posts#Public_Page_Upgrading

Expose a page that can be upgraded to private access through an IndieAuth token and then possibly allow the client to subscribe to it using Pubsubhubbub (or else expect a client to poll it every now and then).

The exact markup and discoverability of a block list is still a bit of an unknown but I would say that as its a blacklist it's the opposite of the markup of the people one follows, which one typically mark up with XFN, so it would look like XFN, but have other relations.

On the discoverability side I don't know how to really do it – typically XFN-data is discovered in a users profile and one traverses rel-me relations to find more profiles and on them one finds more XFN-data. But as rel-me data and the identity graph can grow rather large such a solution would require a crwaler like relspider to manage – so one would probably have to add some specific relation data do make such a Bridgy-block page discoverable – not sure what that would be though – some property of a user's h-card probably – u-blocks (and perhaps a reverse u-follows if one wants to make a whitelist as discoverable)?.

If one wants a block list to be possible to be both public and private (Twitters are currently private so any block lists from twitters should probably be private when exposed through Bridgy) then using the brainstorm documented here would be useful: https://indiewebcamp.com/private_posts#Public_Page_Upgrading

Expose a page that can be upgraded to private access through an IndieAuth token and then possibly allow the client to subscribe to it using Pubsubhubbub (or else expect a client to poll it every now and then).

The exact markup and discoverability of a block list is still a bit of an unknown but I would say that as its a blacklist it's the opposite of the markup of the people one follows, which one typically mark up with XFN, so it would look like XFN, but have other relations.

On the discoverability side I don't know how to really do it – typically XFN-data is discovered in a users profile and one traverses rel-me relations to find more profiles and on them one finds more XFN-data. But as rel-me data and the identity graph can grow rather large such a solution would require a crwaler like relspider to manage – so one would probably have to add some specific relation data do make such a Bridgy-block page discoverable – not sure what that would be though – some property of a user's h-card probably – u-blocks (and perhaps a reverse u-follows if one wants to make a whitelist as discoverable)?.

@voxpelli

This comment has been minimized.

Show comment
Hide comment
@voxpelli

voxpelli Sep 15, 2015

After some brief discussion with tantek I would say u-block-list and u-follow-list, prefixed with x-, would be the more right way to start experimenting with this: http://indiewebcamp.com/irc/2015-09-15/line/1442345907245

If @kevinmarks could add such a link to his h-card then I could discover that on my side and ask him if he wants to enforce that block list on my endpoint and then start consuming and enforcing it.

After some brief discussion with tantek I would say u-block-list and u-follow-list, prefixed with x-, would be the more right way to start experimenting with this: http://indiewebcamp.com/irc/2015-09-15/line/1442345907245

If @kevinmarks could add such a link to his h-card then I could discover that on my side and ask him if he wants to enforce that block list on my endpoint and then start consuming and enforcing it.

@snarfed

This comment has been minimized.

Show comment
Hide comment
@snarfed

snarfed Sep 24, 2015

Owner

thanks for the brainstorming @voxpelli!

if i understand all this right, the asks for bridgy are:

  1. fetch a twitter user's block list and translate it to mf2
  2. serve that mf2 on a private, access controlled page
  3. grant access based on... indieauth token?
  4. bonus: let consumers subscribe to changes via PuSH

I'd happily consider adding #1 to granary. the rest are unlikely at best, at least for me. if someone else wants to add them to bridgy, I'm open to that.

thanks again!

Owner

snarfed commented Sep 24, 2015

thanks for the brainstorming @voxpelli!

if i understand all this right, the asks for bridgy are:

  1. fetch a twitter user's block list and translate it to mf2
  2. serve that mf2 on a private, access controlled page
  3. grant access based on... indieauth token?
  4. bonus: let consumers subscribe to changes via PuSH

I'd happily consider adding #1 to granary. the rest are unlikely at best, at least for me. if someone else wants to add them to bridgy, I'm open to that.

thanks again!

@voxpelli

This comment has been minimized.

Show comment
Hide comment
@voxpelli

voxpelli Sep 24, 2015

@snarfed: That sums it up pretty well.

As block lists are private on Twitter, exporting them to something public seems like something one would like to avoid. Are you thinking that Granary would expose it privately or publicly?

The IndieAuth-token check should be pretty simple to implement – not sure about the relation between Granary and Bridgy and where one would add it though.

@snarfed: That sums it up pretty well.

As block lists are private on Twitter, exporting them to something public seems like something one would like to avoid. Are you thinking that Granary would expose it privately or publicly?

The IndieAuth-token check should be pretty simple to implement – not sure about the relation between Granary and Bridgy and where one would add it though.

@snarfed

This comment has been minimized.

Show comment
Hide comment
@snarfed

snarfed Sep 24, 2015

Owner

agreed, granary definitely can't expose private block lists (or anything else private) publicly.

the first step is just to add it to the granary library. the next step is exposing it via REST and the UI in exchange for a twitter access token, which is nice but not enough for you, since you don't have twitter credentials for your users.

the problem is that granary doesn't store twitter creds either (or anything at all). we'd need a service that does store twitter creds and check IndieAuth tokens (or whatever). I'd love for that to be something else, but right now the only candidate i know of is bridgy...unless you or webmention.io want to start collecting and storing twitter creds? :P

Owner

snarfed commented Sep 24, 2015

agreed, granary definitely can't expose private block lists (or anything else private) publicly.

the first step is just to add it to the granary library. the next step is exposing it via REST and the UI in exchange for a twitter access token, which is nice but not enough for you, since you don't have twitter credentials for your users.

the problem is that granary doesn't store twitter creds either (or anything at all). we'd need a service that does store twitter creds and check IndieAuth tokens (or whatever). I'd love for that to be something else, but right now the only candidate i know of is bridgy...unless you or webmention.io want to start collecting and storing twitter creds? :P

@snarfed snarfed referenced this issue Sep 27, 2015

Closed

support block lists #40

1 of 1 task complete
@snarfed

This comment has been minimized.

Show comment
Hide comment
@snarfed

snarfed Sep 27, 2015

Owner

i thought about this a bit more and filed snarfed/granary#40. a good first step could be to do just public block lists. we'd add granary support, then serve them from bridgy since it has users' silo credentials. probably no UI (or minimal) in bridgy, just a per user endpoint.

this isn't a priority for me personally right now, but I'd love to see it happen!

Owner

snarfed commented Sep 27, 2015

i thought about this a bit more and filed snarfed/granary#40. a good first step could be to do just public block lists. we'd add granary support, then serve them from bridgy since it has users' silo credentials. probably no UI (or minimal) in bridgy, just a per user endpoint.

this isn't a priority for me personally right now, but I'd love to see it happen!

@snarfed

This comment has been minimized.

Show comment
Hide comment
@snarfed

snarfed Dec 16, 2016

Owner

re the last comment: twitter doesn't actually support public block or mute lists. they're all private. so the MVP use case for this is bigger and has more open questions than we (i) thought.

Owner

snarfed commented Dec 16, 2016

re the last comment: twitter doesn't actually support public block or mute lists. they're all private. so the MVP use case for this is bigger and has more open questions than we (i) thought.

@snarfed

This comment has been minimized.

Show comment
Hide comment
@snarfed

snarfed Dec 16, 2016

Owner

more importantly, if you block someone on twitter, they can't see your tweets at all, so they can't reply, favorite, or retweet them. which leaves just @-mentions that bridgy would still need to block.

(apologies if all this was already obvious. i hadn't paid much attention to blocking before this. privilege check, etc.)

Owner

snarfed commented Dec 16, 2016

more importantly, if you block someone on twitter, they can't see your tweets at all, so they can't reply, favorite, or retweet them. which leaves just @-mentions that bridgy would still need to block.

(apologies if all this was already obvious. i hadn't paid much attention to blocking before this. privilege check, etc.)

@tantek

This comment has been minimized.

Show comment
Hide comment
@tantek

tantek Jul 18, 2017

Contributor

This is currently a blocker (no pun intended) for me implementing showing any kinds of comments (including from Bridgy via Twitter), and likely even reposts/likes (because I'd really like to avoid having a direct link from my site/permalinks, even in a responses display section, to e.g. a hate/abusive profile on Twitter) - and this happens with many (most?) POSSE tweets of mine (sometimes "just" sensitive images rather than outright hate but still :/ )

Upstream I could make a feature request to webmention.io to filter out Twitter responses (for now), or barring that the barrier is raised (for me) even higher to "just" directly handle receiving webmentions myself, and write my own filtering code.

However I'd prefer to help push to make these services usable by more devs, which is going to require blocking support for all but the smallest / most privileged accounts that see little abuse on Twitter.

Contributor

tantek commented Jul 18, 2017

This is currently a blocker (no pun intended) for me implementing showing any kinds of comments (including from Bridgy via Twitter), and likely even reposts/likes (because I'd really like to avoid having a direct link from my site/permalinks, even in a responses display section, to e.g. a hate/abusive profile on Twitter) - and this happens with many (most?) POSSE tweets of mine (sometimes "just" sensitive images rather than outright hate but still :/ )

Upstream I could make a feature request to webmention.io to filter out Twitter responses (for now), or barring that the barrier is raised (for me) even higher to "just" directly handle receiving webmentions myself, and write my own filtering code.

However I'd prefer to help push to make these services usable by more devs, which is going to require blocking support for all but the smallest / most privileged accounts that see little abuse on Twitter.

@aaronpk

This comment has been minimized.

Show comment
Hide comment
@aaronpk

aaronpk Jul 18, 2017

Contributor

I just tested this via bridgy to confirm, tho I guess I should have just taken @kylewm's word for it. Bridgy currently does send webmentions even if I've blocked an account. https://brid.gy/log?start_time=1500407904&key=aglzfmJyaWQtZ3lyNQsSCFJlc3BvbnNlIid0YWc6dHdpdHRlci5jb20sMjAxMzo4ODc0MDA0MTc5MzU1MDMzNjEM

Contributor

aaronpk commented Jul 18, 2017

I just tested this via bridgy to confirm, tho I guess I should have just taken @kylewm's word for it. Bridgy currently does send webmentions even if I've blocked an account. https://brid.gy/log?start_time=1500407904&key=aglzfmJyaWQtZ3lyNQsSCFJlc3BvbnNlIid0YWc6dHdpdHRlci5jb20sMjAxMzo4ODc0MDA0MTc5MzU1MDMzNjEM

@snarfed

This comment has been minimized.

Show comment
Hide comment
@snarfed

snarfed Jul 18, 2017

Owner

ok, so the task here is at least the first two items in @kylewm's original issue. (he was right all along! :P)

  • proactively filter out blocked users when searching for replies [..., links, @-mentions, etc]
  • on individual tweets, serve 410 tombstone so that mentions from blocked accounts will be deleted

we're very unlikely to do the third:

  • periodically check for old responses from accounts that are now blocked, and re-propagate them.

...but fortunately aaronpk/webmention.io#85 should have the same effect, and is maybe more likely to actually happen.

Owner

snarfed commented Jul 18, 2017

ok, so the task here is at least the first two items in @kylewm's original issue. (he was right all along! :P)

  • proactively filter out blocked users when searching for replies [..., links, @-mentions, etc]
  • on individual tweets, serve 410 tombstone so that mentions from blocked accounts will be deleted

we're very unlikely to do the third:

  • periodically check for old responses from accounts that are now blocked, and re-propagate them.

...but fortunately aaronpk/webmention.io#85 should have the same effect, and is maybe more likely to actually happen.

@snarfed snarfed added the now label Jul 18, 2017

@tantek

This comment has been minimized.

Show comment
Hide comment
@tantek

tantek Jul 19, 2017

Contributor

Note: there's the historical aspect also. E.g.:

1 receive an abusive @-reply via Bridgy backfeed
2 block that user on Twitter
3 "refresh" (automatically?) Bridgy to update its cache of my Twitter blocklist
4 Bridgy notices any new blocked users, and 410s all previous (historical) webmentions from them, preferably resending webmentions (webmention deletes) to cause those webmentions to automatically get deleted by the receiver (e.g. a webmention proxy like webmention.io)

If that's different / harder enough to merit a separate issue, ok with splitting that out. From a "UX of the service" perspective, it feels like it should "just work", as this is how for example blocking a user on Flickr works (it removes their past comments and favorites from all your photos).

Contributor

tantek commented Jul 19, 2017

Note: there's the historical aspect also. E.g.:

1 receive an abusive @-reply via Bridgy backfeed
2 block that user on Twitter
3 "refresh" (automatically?) Bridgy to update its cache of my Twitter blocklist
4 Bridgy notices any new blocked users, and 410s all previous (historical) webmentions from them, preferably resending webmentions (webmention deletes) to cause those webmentions to automatically get deleted by the receiver (e.g. a webmention proxy like webmention.io)

If that's different / harder enough to merit a separate issue, ok with splitting that out. From a "UX of the service" perspective, it feels like it should "just work", as this is how for example blocking a user on Flickr works (it removes their past comments and favorites from all your photos).

snarfed added a commit to snarfed/granary that referenced this issue Jul 21, 2017

snarfed added a commit to snarfed/granary that referenced this issue Jul 21, 2017

snarfed added a commit to snarfed/granary that referenced this issue Jul 22, 2017

snarfed added a commit to snarfed/granary that referenced this issue Jul 22, 2017

snarfed added a commit that referenced this issue Jul 23, 2017

handle responses by blocked users
ignore responses in poll, return 410 (deleted) in handlers

for #473

snarfed added a commit that referenced this issue Jul 23, 2017

@snarfed

This comment has been minimized.

Show comment
Hide comment
@snarfed

snarfed Jul 23, 2017

Owner

ok, this is mostly done. bridgy now fetches twitter block lists (up to hourly, cached beyond that), ignores responses from blocked users, and 410s their translated tweets and responses. example: https://brid.gy/post/twitter/schnarfed/510676816084754432

resending all past webmentions from newly blocked users (#473 (comment)) is a worthy goal, but nontrivial, and has open UX questions. i've split it out into #762.

Owner

snarfed commented Jul 23, 2017

ok, this is mostly done. bridgy now fetches twitter block lists (up to hourly, cached beyond that), ignores responses from blocked users, and 410s their translated tweets and responses. example: https://brid.gy/post/twitter/schnarfed/510676816084754432

resending all past webmentions from newly blocked users (#473 (comment)) is a worthy goal, but nontrivial, and has open UX questions. i've split it out into #762.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment