scraper follow the only link

[qjebbs]

In some cases, what the scraper got is only a landing page, with this PR, user can use scraper rules (e.g.: .post.detail .title a) to locate the link of the landing page and follow it for real content

How to use:

1. For those feeds who give you only a landing page, find the selector of the redirect link (e.g.: .post.detail .title a)
2. Setup scraper rules in feed settings

it also fix the wrong scrape rule apply when the server redirects it to another host

Do you follow the guidelines?

• I have tested my changes
• I read this document: https://miniflux.app/faq.html#pull-request

[fguillot]

Can you provide some examples of website to test this change?

[qjebbs]

Sure, settings below works perfect for that feed.

feed: https://toutiao.io/daily.xml
scrape rule: .post.detail .title a
rewrite rule: add_dynamic_image

[thiagowfx]

Another example for testing: https://www.bloomberg.com/authors/ARbTQlRLRjE/matthew-s-levine.rss

[fguillot]

feed: https://toutiao.io/daily.xml

This website doesn't seem to exist anymore.

[qjebbs]

feed: https://toutiao.io/daily.xml

This website doesn't seem to exist anymore.

It's announced to be in maintenance until 31-10-2022