Indexing: Add new field to indicate if a page is included in a web feed

[m-i-l]

This could be useful to e.g.:

• Filter Newest Pages, instead of using the current fq=published_date:* (which lets some non article content through, e.g. the PostgreSQL home page whenever it is updated), to ensure Newest Pages is more of an article feed.
• Boost results that are articles, to lower the score of index/landing pages (which might even do away with the need for Search: Exclude content from results, but don't exclude from indexing #66 ).
• Provide RSS/Atom search and aggregation functionality, in combination with Search: Provide OpenSearch Atom responses for standards-based federation #26 .

It could be implemented via a new in_web_feed boolean in the Solr schema, perhaps being combined with the existing web_feed attribute (currently only set on the home page as per #64 ) to indicate which feed the article is from. Would need to get a list of all the pages in a site's feed first (this would be needed for identifying new pages for #34 and could be implemented alongside #54 ).

[m-i-l]

Implemented:

• The web feed is parsed at the start of the indexing, as per Indexing: Crawl from web feed (RSS/Atom) #54 , and a list of links in the feed held in memory during indexing for a site.
• Whenever a page is indexed, it is checked to see if it is the list, and if it is the in_web_feed field is populated in Solr.

Note that, as per usual, the new field won't be fully populated until all sites are reindexed. Note also that, given the web feed URL is populated by indexing, the first time a site is indexed it won't have it to seed the indexing, so the in_web_feed won't be populated until a new site's second index.

The 3 bullet points listed above for features enabled by this are tracked separately:

• Search: Change Newest Pages to Newest Posts, comprising only pages which appear in a web feed  #74 :
• Search: Boost relevance scores for pages which appear in a web feed #73 :
• Search: Provide OpenSearch Atom responses for standards-based federation #26 :