Collect #weeknotes posts from Mastodon

[JackiePease]

e.g. Mastodon

[johnmckerrell]

Or Instagram, but especially Mastodon

[amcewen]

There was Instagram support, until they started locking down their API. That stalled at #19 but given that in 2019 they further limited their API I'm going to mark #19 as "won't fix" because Instagram are web-hostile.

I've changed the title of this one so we'll know when it's fixed (if there are other sources people would like, we should open separate issues for them).

I think we should make a copy of the images rather than hot-linking to them, so we don't hit the same "images don't show up" bug as we have with Twitter hot-linked images #30. As I said in a comment on that issue, it'd be nice to also get permission to use the images the first time someone includes one in a weeknote. (Initially it can just store the consent locally, and we can add it to Optimism as a similar-system-to-noting-inductions)

The main thing I'm not sure about is where we'd store a copy of the images. Should it be on a subdomain of ours - like assets.doesliverpool.com or similar? Then we can have images/weeknotes/USER/file.jpg for ones with just-weeknotes consent, and images/full/USER/file.jpg for those with full consent.

That could live on the same VPS as organisers.doesliverpool.com does, I guess. Any reasons I shouldn't do that, or better suggestions?

[johnmckerrell]

Your image caching suggestions sound reasonable. How would we get the consent though, automatically or.. otherwise?

[amcewen]

Automatically I think. Have the weeknote generator code keep a list of people it's got consent from (until there's an API for that in the members section of optimism ;-) and if it encounters an image from someone it doesn't know, it sends them a DM to ask for consent, with a "reply with yes / no / just weeknotes".

Might want to run that bit more frequently than once-a-week, so the consent has come through before we need to publish things.

[amcewen]

Made some progress on this last night.

There doesn't seem to be an API to Mastodon to get posts for a given hashtag, but you can get the hashtags as an RSS feed, so that works. I've got some code that fetches that and parses it before trying to fetch the details of each post.

Fetching the details of the post works for Mastodon, but fails on Pleroma (at least), so it'll need to be a bit more intelligent in how t gets details in order to message the user to ask for consent.

I'm thinking that we'd run a script to find new posts on Mastodon frequently (one-a-day or something), which then saves them into a full-consent-awaiting-processing or just-weeknotes-awaiting-processing folder if we've already got consent from the user (and discards them if they refused consent). And messages any new users to ask for consent and saves those posts into a pending-consent folder. I think it'd also check for any consent replies, and then move any posts accordingly.

Then the generate_weeknotes.rb script will process anything from the ...-awaiting-processing folders.

I'm tracking the work on this on a new branch: feature/31-add-mastodon-support

[amcewen]

Found the timelines section of the API, which lets us search better through tags and returns all the relevant details in JSON so it no longer crashes on Pleroma posts. I updated the find_toots.rb script on the branch to use that.

I also added code to send a DM with a poll in it to ask for consent:

It needs to only send one request per person, and needs to do something with the replies, so that'll be next.

[johnmckerrell]

Nice. Small comments, the screenshot shows it asking for a reply when the user needs to fill in the poll, and perhaps it would be good to link to the week notes to explain what that bit means (then again maybe we can assume they know given they used the hashtag).

[amcewen]

Yeah, partly that's because the wording was written when I was last working on it, and the poll was an addition last night 😁 I'm not 100% sure that all bits of the Fediverse support polls though, so maybe a "Vote or reply [all / weeknotes / no]" would work? Including a link to our intro weeknotes post isn't a bad idea either.

I did wonder whether or not it should come from @DoESLiverpool@does.social or some new @Weeknotes@does.social or similar - and also whether it should be a DM (which needs them to follow us) or a "private" message (not quite sure what that is, but it's one of the options for visibility alongside "direct")

[johnmckerrell]

Good question about the account, I guess technically if we're posting from a bot we should mark the account as a bot, but then that might confuse people about who they should follow if this bot was never actually posting anything else. I forget if people had to be following DoES, or vice versa, for the toots to show up?

[amcewen]

I've tweaked the wording.

@johnmckerrell, I think at the moment we'll see any #weeknotes toots that the does.social server sees, but we can check for them following us (I'm sure). I think sending them from @DoESLiverpool makes more sense - maybe we could "sign" them with "^ Weeknotes-Bot" at the end of the message text if we want to make it clearer it's a semi-bot?

This evening I got it to only ask each user once, rather than every time it encountered a toot from them; and it now stores the consent in a YAML file (we can make it a more complex database in the future if need be). I've also worked out the API endpoints we'll need to hit to check for responses to the consent request, and started on a script (check_consent_requests.rb) which will hit those endpoints.

I did discover that there isn't any paper trail for who voted (at least I've not seen it yet), just the number of votes. As you can't vote on your own polls, and it's a DM to a single user, I think that'll be okay, but I'd prefer to have a URL to store in the consent database which points to something on the user's instance. We'll have that for replies. That said, the user could delete a reply in future and the URL we've stored will 404, so maybe a poll is better in that case 🤷

[amcewen]

Got the checking for replies to the consent request done today. Need to save the found toots accordingly now (based on both the consent, and also a pending area for ones where we ask for consent), and then process them to include in the weeknotes.

[amcewen]

The found toots get saved: ones where we've asked for consent are put into one folder (from the settings), inside a sub-folder for the user; ones where we've got consent are stored in a folder ready for publication, based on the consent. When consent is given, any pending toots are copied over ready for publication accordingly (or deleted if consent is refused)

We now ignore toots from accounts that we don't follow too.

Next we can actually process them for inclusion in the weeknotes! Will need to download any media and store it somewhere on a server we can serve it from, and generate the relevant HTML for inclusion in the weeknotes blogpost.

[amcewen]

Started on the processing them for inclusion in the weeknotes bit last night. Need to handle downloading the images, so we can serve them from media.doesliverpool.com and then update the message content to point to the new locations.

And then get it deployed for use...

[amcewen]

Added downloading the images and generating the output for Mastodon posts. Needs to only find toots in the past week, and have the consent handling actually ask everyone it finds, rather than printing out what it would've sent and only really ask me (that my debugging has done).

[amcewen]

This has been deployed (see #34 for details) and so far seems to be working as expected. Actually, the poll when it asks for consent only lasts for an hour, which was good for testing but less useful in production. I've upped it to 14 days.