Indexing: Improve heuristics for guessing the primary web feed

[m-i-l]

Now I've started building a feed reader into the search engine, as per #54 and #71 , it is often detecting multiple feeds on a site, some of which the site owner might not even be aware exist. For example, it turns out that on my personal site (based on Hugo) there are feeds for each of the /tags/ and /categories/, which I wasn't aware existed, e.g. https://www.michael-lewis.com/tags/artificial-intelligence/index.xml. Now in my case I've configured the search to exclude the /tags/ and /categories/ paths so it doesn't actually find them, but few people are configuring their search in this way.

As it is right now, if it finds multiple feeds, it simply picks the most recently found one to use as a primary feed. If that turns out to be (for example) for a rarely used tag, then that isn't going to be too useful e.g. for #34 or some of the functionality that will be based off #71 .

Easiest choice would be to use the path, and have a list of paths in order of preference. With a bit more work there could be other data that could factor into this too, e.g. location of the link (a link on the home page is more likely to be the main feed).

The list of multiple feeds can be obtained from the logs via:

docker logs src_indexing_1 2>&1 | grep "More than one potential web feed"

At the moment (with only around 1 day of data), a prioritised list could look like this:

1. /posts/index.xml
2. /feed/
3. /feed
4. /feed.xml
5. /feed/rss
6. /feeds/all.xml
7. /feeds/all.atom.xml
8. /blog/atom
9. /blog/atom.xml
10. /blog/index.xml
11. /blog/feed.xml
12. /blog?format=rss
13. /blog/feeds/all.xml
14. /notes/index.xml
15. /index.xml
16. /atom.xml
17. /rss.xml
18. /rss

Note that some sites just have feeds for tags so probably just have to pick one of those, some look pretty site-specific so again just pick one, and some are not for content (e.g. one only has a feed of public keys) which is a bit more tricky but not sure I want to start maintaining an allow list so they might have to slip through. As an aside, in many cases the preferred one is the first one in the list, i.e. the first one found, although there are plenty of exceptions so not sure I'd want to make that a rule.

[m-i-l]

The system now selects the web feed based on the above prioritised list, e.g. if the web_feeds it finds are ['https://michael-lewis.com/index.xml', 'https://michael-lewis.com/posts/index.xml'] then it selects 'https://michael-lewis.com/posts/index.xml' because '/posts/index.xml' is higher on the list than '/index.xml'. If it can't find a match, e.g. because the options end '/en/blog/atom.xml' and '/fr/blog/atom.xml', or because it only has feeds for tags, it'll pick the first it finds.

The idea is to try and prioritise feeds of blog posts over whole-site feeds, or whole-site feeds over tag feeds.

Not perfect, but hopefully good enough. Remember also that site owners with full listings can specify their preferred feeds themselves (see #54 ). Not sure how a preferred feed could be specified for basic listings, given there isn't a way for owners to login, unless it is specified on the initial Add Site page, although the issues with that would be (i) ideally the Add Site should be as quick and easy for basic listings as possible but that would turn 3 questions into 4, and (ii) perhaps more importantly there would be no easy mechanism for the feeds for basic listings to be changed after initial submission (I guess they could switch to the free trial to edit the preferred web feed which would be maintained when the free trial lapses back to the basic listing, but that's not exactly intuitive).