Implement importing domain blocklists via tootctl & mastodon:setup by ThisIsMissEm · Pull Request #25691 · mastodon/mastodon

ThisIsMissEm · 2023-07-03T20:08:23Z

For a while now, there's been a criticism of Mastodon which is "when setting up a new instance, it isn't safe by default", this pull request attempts to address that, by:

Introducing a tootctl domains blocklist import [name] [url] method, which downloads the given blocklist to a temporary CSV and then processes it, creating the necessary blocks
Adding a prompt to rails mastodon:setup to (optionally) choose from a few commonly used basic blocklists, which then uses that new tootctl command to import those lists

The code does do some kinda awkward things (like attempting to reuse Admin::Import, which has an awkward dependency in a CLI context on something that looks like an ActiveDispatch::Parameters value, for performing validation of the CSV, and bit of cludge around executing the tootctl command from the mastodon:setup rake task and handling the command's output nicely.

There's probably (almost certainly) a need for tests — currently I've just tested manually.

I think this should be a good step forwards in "safe by default", you can see it in action at: https://www.dropbox.com/s/6ir6zfq8ulxmnct/mastodon-tootctl-blocklist-import.mov?dl=0

ThisIsMissEm · 2023-07-03T20:16:01Z

+
+        say('Fetched blocklist!')
+
+        import = Admin::Import.new(data: DomainBlocklistData.from(filename, tmpfile.path))


This class has some good validation logic inside of it, but ties directly to the http/routing stack, hence needing this workaround.

An alternative would be to rework Admin::Import to accept a filename and file / filepath values, which could then work with my approach of using a tmpfile, but that'd be a bunch of extra changes besides just adding the feature to see how people like it.

ThisIsMissEm · 2023-07-03T20:19:18Z

+          domain_block.save!
+          DomainBlockWorker.perform_async(domain_block.id)


Note: we're not adding to the AdminLog here, as we may not have a user account yet, so there'd be no account to attribute this import to.

Arguably we only need DomainBlockWorker if the server has already been running / has peers, I think!

ThisIsMissEm · 2023-07-03T20:20:15Z

+          domain_block.save!
+          DomainBlockWorker.perform_async(domain_block.id)
+        rescue ArgumentError => e
+          say("Error importing blocklist, #{e.message}", :red)


I'm not really sure when this error could happen, but the original import controller features similar.

ThisIsMissEm · 2023-07-03T20:20:53Z

+
  class Domains < Base
+    desc 'blocklist SUBCOMMAND ...ARGS', 'Tools for working with domain blocklists'
+    subcommand 'blocklist', DomainBlocklist


I could move this class into it's own separate folder, if desired — for now I've just kept everything together

ThisIsMissEm · 2023-07-03T20:21:53Z

      end
    rescue TTY::Reader::InputInterrupt
-      prompt.ok 'Aborting. Bye!'
+      prompt.ok "\nAborting. Bye!"


This fixes an issue in the output where the abort line would be joined to the preceding question line.

… blocklist import process

ThisIsMissEm · 2023-07-03T23:16:22Z

-        prompt.ok 'All done! You can now power on the Mastodon server 🐘'
+        prompt.ok 'All configured!'


Have made this change as ideally we don't want them to start their server until they've imported a blocklist

ThisIsMissEm · 2023-07-03T23:31:11Z

+
+        prompt.say "\n"
+
+        blocklist = prompt.select("Would you like to import a domain blocklist for your instance?\n") do |menu|


We could change this to a 2 or 4 step process:

Would you like to import a domain blocklist for your instance? Yn # If no: > Are you sure? This may decrease the safety of users on your instance? Yn > Okay, you can always download a blocklist later and import it via the Admin Panel or tootctl. Which domain blocklist would you like to import? 1. Oliphant (website) 2. The Bad Space (website) 3. Other # If `Other` then: What is the URL of blocklist you would like to import? What is the name of the blocklist?

As to allow importing something stricter than the basic ones proposed above in domain_blocklists

I guess I should also add an explainer like

"As Mastodon is connected to a network of other servers and services, there's a small minority of servers that are widely known to have problematic users or behaviour, over time, other administrators & moderators have collated lists of these servers such that new administrators can easily block the worst of the worst before your server even joins the fediverse, keeping you and people using your server safe"

jazmichaelking · 2023-07-05T18:37:23Z

Explainer option:

"Some servers are documented to have problematic users or activity, and are blocked by a large number of administrators to reduce network abuse. Blocklists are created by curators who make these lists available to the public."

ocdtrekkie · 2023-07-05T20:24:37Z

One thing I've discussed in a few places is that it would be really nice, especially for a feature included in an "official Mastodon" release, to have a minimal option that syncs with mastodon.social, an "official server". However to do this, Mastodon gGmbH would need to publish an unobfuscated list on some endpoint somewhere.

jazmichaelking · 2023-07-06T18:38:43Z

Can it be pulled via API and then formed into a CSV?

see also https://github.com/iftas-org/rfcs/discussions/4 (I suggest Mastodon just include their blocklist for all installs, either way, should be a viable baseline list for some/many)

jazmichaelking · 2023-07-06T19:01:01Z

Two additional options to consider

https://gardenfence.github.io/
https://raw.githubusercontent.com/gardenfence/blocklist/main/gardenfence-mastodon.csv

and

https://codeberg.org/oliphant/blocklists/raw/branch/main/blocklists/mastodon/100.percent.csv
as a bare minimum option

I also think that the fact there's a choice, we might need to guide folks in increasing order of severity no matter how many - 2 ,3 or more files are available. Installer might skip due to uncertainty, "do I need all of them" "which one is best for me" etc. Maybe "Minimal", "Moderate", "Maximal" or similar? (We can sort by total number of domains)

ThisIsMissEm · 2023-07-17T01:10:29Z

Two additional options to consider

https://gardenfence.github.io/ https://raw.githubusercontent.com/gardenfence/blocklist/main/gardenfence-mastodon.csv

and

https://codeberg.org/oliphant/blocklists/raw/branch/main/blocklists/mastodon/100.percent.csv as a bare minimum option

I also think that the fact there's a choice, we might need to guide folks in increasing order of severity no matter how many - 2 ,3 or more files are available. Installer might skip due to uncertainty, "do I need all of them" "which one is best for me" etc. Maybe "Minimal", "Moderate", "Maximal" or similar? (We can sort by total number of domains)

@jazmichaelking problem is that I can't know how many blocks are on a given list until I request & process that CSV, so I can only really go by a short textual description of the list / link to the details.

evanp · 2023-07-17T02:23:39Z

I'm really glad to see this PR! Making it easy to include this basic user safety for new installations is a huge step in making the fediverse safe and welcoming for everyone. Thanks @ThisIsMissEm for this good work!

ThisIsMissEm · 2023-10-04T15:04:34Z

@Gargron @ClearlyClaire in it's current state, this doesn't work the nicest, and as Eugen pointed out, it doesn't solve for managed instances. I'm not sure what we have in the way of "instance setup" in the admin web UI, but perhaps we can look at just having in rails mastodon:setup a call to get a denylist ("blocklist") and have a documentation page explaining them?

And then further as a stats thing, show a banner at the top of the instance admin UI if domain_blocks.count == 0 and recommend setting up a domain blocks by importing a denylist?

Would this be workable as a way to move forwards on the matter this PR attempts to address?

(Though I also note it probably should be domain_blocks.count < 140, given that's the number of instances on the FediNuke list)

ThisIsMissEm · 2023-10-04T15:28:47Z

We could also look at having an instance start out by default in limited federation mode, until the admin goes "yup, we're good to start federating" and flips a switch.

github-actions · 2023-10-23T13:22:49Z

This pull request has merge conflicts that must be resolved before it can be merged.

ThisIsMissEm · 2024-03-29T18:21:59Z

Closing this as it's not going to move forwards as far as I can tell.

renchap · 2024-04-19T17:44:51Z

A bit of context for people stumbling into this: better blocklist management and an (opt-in) default "worst of the worse" blocklist were planned for 4.3, but we did not had the resources to work on it.

If we provide this default blocklist, we also want it to be updated periodically, so we need to implement blocklist syncing as well, so the scope here is not small, hence the lack of resources to do it.

But it has not being dropped from our todolist!

Implement importing blocklists via tootctl & mastodon:setup

8080724

ThisIsMissEm changed the title ~~Implement importing blocklists via tootctl & mastodon:setup~~ Implement importing domain blocklists via tootctl & mastodon:setup Jul 3, 2023

ThisIsMissEm commented Jul 3, 2023

View reviewed changes

Comment thread lib/mastodon/cli/domains.rb Outdated

ThisIsMissEm commented Jul 3, 2023

View reviewed changes

Comment thread lib/mastodon/cli/domains.rb Outdated

ThisIsMissEm commented Jul 3, 2023

View reviewed changes

Comment thread lib/tasks/mastodon.rake Outdated

ThisIsMissEm added 2 commits July 4, 2023 01:06

Use tty-command for executing commands, and add a progress bar to the…

7d1c149

… blocklist import process

Minor fixes

a0e6bca

ThisIsMissEm commented Jul 3, 2023

View reviewed changes

github-actions Bot added the rebase needed 🚧 label Oct 23, 2023

ThisIsMissEm closed this Mar 29, 2024

ThisIsMissEm deleted the feat/add-import-of-blocklists-during-setup branch March 29, 2024 18:22


		say('Fetched blocklist!')

		import = Admin::Import.new(data: DomainBlocklistData.from(filename, tmpfile.path))

		domain_block.save!
		DomainBlockWorker.perform_async(domain_block.id)

		prompt.ok 'All done! You can now power on the Mastodon server 🐘'
		prompt.ok 'All configured!'


		prompt.say "\n"

		blocklist = prompt.select("Would you like to import a domain blocklist for your instance?\n") do \|menu\|

Uh oh!

Uh oh!

Conversation

ThisIsMissEm commented Jul 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ThisIsMissEm Jul 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jazmichaelking commented Jul 5, 2023

Uh oh!

ocdtrekkie commented Jul 5, 2023

Uh oh!

jazmichaelking commented Jul 6, 2023

Uh oh!

jazmichaelking commented Jul 6, 2023

Uh oh!

ThisIsMissEm commented Jul 17, 2023

Uh oh!

evanp commented Jul 17, 2023

Uh oh!

ThisIsMissEm commented Oct 4, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ThisIsMissEm commented Oct 4, 2023

Uh oh!

github-actions Bot commented Oct 23, 2023

Uh oh!

ThisIsMissEm commented Mar 29, 2024

Uh oh!

renchap commented Apr 19, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

ThisIsMissEm commented Jul 3, 2023 •

edited

Loading

ThisIsMissEm Jul 3, 2023 •

edited

Loading

ThisIsMissEm commented Oct 4, 2023 •

edited

Loading