Add a metadata property to reflect general purpose of the UA

[claytonrothschild]

Hi,

Great library!

I am using this with a React app to return a static site if the UA is a scraper rather than a human viewer. The reason is that many scrapers used by sharing UA like Facebook/Twitter/Slack do not render javascript, so my app content does not preview correctly.

It would be very helpful if we could sort and designate which UAs are used for which general purpose. For example, I thought I had captured all the UAs that are typically used by social sites to render preview links, but a user just sent me a screenshot of a Messenger message where the preview did not render correctly. This is because I had not configured my app to return static for that particular UA.

I think a simple property of general_purpose: would be helpful. With options of browser, scrape: search, scrape: preview_render, etc.

What do you think?

[monperrus]

Excellent idea. We would have to add an optional field in the JSON schema and check that the value is in the set of accepted ones. Do you want to make this contribution?

[jarodium]

I would like to also propose a key called robotstxt ( or similar ) with the user agent that is commonly used to add in robots txt. This value could be comma separated, especially when in the case of:

"pattern": "[wW]get", that could yield two rules.

Then, this list could be used as a base for automated generation of robots txt as well as a base for logfile analysis. xD

[monperrus]

Excellent idea, see #265

[chjlsch]

Maybe using "Open Graph Meta Tags" would help you. The official site is ogp.me (stands for open graph protocol).