just some ideas

[digitalist]

collecting of /ads.txt
and https:// certs info

may be useful

[s0md3v]

Thanks for the suggestions, I really appreciate it but I have two questions to ask :')

1. How do you find out if a javascript file/code runs ads and how is that relevant?
2. Don't you think that falls out of the scope for a crawler?

[digitalist]

nah, just checking for ads.txt file, as in 'robots.txt' - it's an RTB standard (example: https://www.gazeta.ru/ads.txt)
checking for https cert is useful for some activities (antifraud etc.).

It's not hard to implement, maybe I should find time for a pull request

[s0md3v]

It's not about the difficulty of implementation, it's about relevance. A crawler isn't supposed to check SSL certificates.

[digitalist]

but as a crawler it CAN check for their presence/grab them - as an important metadata [intel]

[s0md3v]

Sorry but I am closing this issue, I don't think it's relevant at all.

1. ads.txt isn't that common
2. Grabbing SSL certificates is out of scope

Thanks for the suggestions tho, let me know if you have some more ^_^