This is a set of tools for generating information, including descriptions, on every listed Mastodon instance, in various formats, like sqlite and html.
chmod +x buildall.sh && buildall.sh
Look in the data/ directory for your generated files.
Oh wow, library errors? This is may be why. You need:
- Python 3
pip3 install bs4
BeautifulSoup, for parsing HTMLpip3 install requests
, for downloading websites without silly syntactic junkpip3 install cheetah3
, for generating html
Script | Effect | output file |
---|---|---|
tools/federation2json.py | downloads and generates .json database | data/DescribedInstances.json |
tools/json2markdown.py | generates a markdown file from the .json database, above | data/DescribedInstances.md |
tools/TemplateRunner.py | generates two HTML files from .json file, above | data/DescribedInstances.html |
tools/json2sqlite.py | generates an sqlite database of the .json file, above | data/DescribedInstances.sqlite |
buildall.sh | generates all of the above | data/* |
tools/AboutThisInstance.py | delivers the json descripton for a single instance | none |
These tools are ad hoc kludges to get the job done only. This is a simple one-shot tool that you can run for 30 minutes to consume all the info out there. I may turn this into a more interruption-friendly database updater later.
To generate all files, simply:
chmod +x buildall.sh && ./buildall.sh
To generate each file individually:
cd tools
python3 federation2json.py ../data/DescribedInstances.json
# wait about 30 minutes for stuff to download
python3 json2markdown.py ../data/DescribedInstances.json ../data/DescribedInstances.md
python3 TemplateRunner.py ../data/DescribedInstances.json ../data/DescribedInstances
python3 json2sqlite.py ../data/DescribedInstances.json ../data/DescribedInstances.sqlite
This is pretty much what the buildall.sh
script does.
To see a json file describing an instance, try
python3 AboutThisInstance.py mastodon.social
This might be more useful than the monolithic approach above.
To test a smaller set of random instance data, set the TEST=1
environment variable, e.g.:
TEST=1 ./buildall.sh
or
cd tools/
TEST=1 python3 federation2json my_testdata.json
To update a previous .json file, skipping the instance list download from mastodon.xyz:
JSON_FILE=data/DescribedInstances.json ./buildall.sh
Since you read this README all the way to the end, have a gift:
THREADS=50 ./buildall.sh
Play nice! Cheers.