"Search my Mastodon timeline" as a use case?

[jneubert]

A user wants to retrieve something which she has seen in her timeline on Mastodon.

Would automatically index the posts that pop up in the timeline of a certain Mastodon user with links to their original instances.

• Optionally, include boosted posts.
• Optionally whole threads.
• Optionally, include linked web pages, papers etc.

---

I'm not aware of any way to achieve such a search with Mastodon tools.

[minimalparts]

Hi @jneubert, that's a nice idea! I am only half-familiar with the Mastodon API but that sounds in principle very feasible from a local PeARS install. I guess a few handy functions should be added to the CLI interface. Have you had a go at this?

For what it's worth, I know that Mastodon currently has a project aiming at adding search to the instance software. But the question has, I believed, been controversial. So I'm not sure what their current plans are...

[jneubert]

I'm not sure if improvements of search per instance would help, because a least my instance does not buffer posts from other instances. Plus, it would be great to restrict search to the entries I have seen in my timeline (and not the whole of entries from one or many instances). To me, that is what I hope PeARS could solve nicely.

Reading the timeline information via the Mastodon API looks doable - straightforward for direct posts, complicated for boosts. Paging through the Mastodon timeline and keeping track of already indexed entries would be part of a script. I'd to this in Perl (using module Mastodon::Client). I've tried to set up PeARS to feed the URLs into it, but was not able to set it up correctly (PeARSearch/PeARS-federated#191).

[jneubert]

A first experimental script works nicely on actual posts (no boots, no linked URIs), writing an input file for flask pears index localhost:8080 urls.txt (formatted according to regex).

I have timeline entries in English and German. Pears language recognition seems to work quite well. However, I was not able to get German as a second installed language to work, so I followed the advice to just start with English. Unfortunately, that clutters the indexing log (ERROR language de is not supported. Moving to default language.), but the page is indexed anyway.

A more serious issue is "robotcheck ... disallowed", which in my timeline test set prevents 89 out of 311 entries from being indexed. In some cases the check seems over-restrictive - I will probably follow up on this.

Overall, the first results look encouraging.

[minimalparts]

Thanks for those experiments and your report! Regarding the language install: we had a PR lying around supposed to fix this, which I have just merged. Long story short: the language install got broken when GitHub changed the way raw files are displayed. I would be grateful if you could do a git pull and try again. If you want to install German, you should do:

flask pears install-language de

... and restart your instance. Please do report any issues you may have!

And absolutely, there is a bug somewhere in the robots.txt checking. It's actually been a recorded issue for a while: PeARSearch/PeARS-federated#4. If you feel like investigating, that would be amazing! We never got to it...

[jneubert]

Thanks! Indexing with PEARS_LANGS=en,de works fine now.