Fix icon fetching for some sites

[quexten]

A few sites in my vault (f.e rakuten.de, kraken.com, crunchyroll.com, etc.) do not have an icon in the clients. The current vaultwarden logic just checks a few well known paths (/favicon.ico etc.). These sites do not use these paths, but have the icon linked in the html:
<link rel="apple-touch-icon" sizes="144x144" href="https://www.crunchyroll.com/build/assets/img/favicons/apple-touch-icon-144x144.png">

We could parse the index.html and look for the icon link in order to fetch the items correctly. (I'm not sure if we have to take special protections to prevent ddos from large html sites. Maybe this should be an opt-in feature).

[quexten]

Actually I just noticed my brief skimming of the code was not very thorough and vaultwarden already parses the html. However, the sites mentioned above are still not working either way, so this might be a bug.

[stefan0xC]

@BlackDex recently improved the favicon handling #3751 so it might already work. You can test it with the latest testing image.

Also make sure you have not disabled the icon downloading (e.g. by setting DISABLE_ICON_DOWNLOAD=true or having it disabled in your web-vaults preferences).

[quexten]

A select few more that do not work:

• camelcamelcamel.com
• codepen.io
• coolblue.de
• my.konami.net
• rebuy.de

[BlackDex]

Thanks for the sites. I will take a look at them when i have time.
Some sites have strange html layouts.

I have a bunch of test sites i use and also compare it with Bitwarden, Google and DDG.

I quickly checked kraken.com and crunchyroll.com, they don't seem to work on the Bitwarden, Google and other icon fetching tool either.

Also, some sites have Cloudflare protection enabled, and those block access since they require JavaScript.

The reason you see it tries to fetch /apple-touch-icon.png is because that, together with /favicon.ico are fallback's which will be tried if nothing was able to be extracted from the parsed HTML.

Unfortunately for some sites we are just not able to fetch icons from.
Ill try to see if i can dive a bit deeper into the mentioned sites, but it might just be not possible.

Trying to use manifest.json adds more checks and more requests coming from the server. It might be an option to add this, i did had this in mind already, but it will add extra burden and complexity, and it needs more testing then the changes i made just a few weeks ago.

Also, we can't really change the user-agent. The one currently used helps bypassing some sites via Cloudflare with some protection enabled. Adding a different try with a different UA would also make it take downloading icons very long and complex in my opinion. But maybe it's something to look at.

So, it might be some sites just will not work because of protections or other strange way of html or JavaScript stuff.

[quexten]

Also, some sites have Cloudflare protection enabled, and those block access since they require JavaScript.

Just noticed this too while debugging. When accessing the site from the server I host bitwarden on, I get a captcha.

Trying to use manifest.json adds more checks and more requests coming from the server. It might be an option to add this, i did had this in mind already, but it will add extra burden and complexity, and it needs more testing then the changes i made just a few weeks ago.

Yeah... rakuten.de specifically also offers svg directly but this would also require some svg2png library/renderer and probably be too complex. The html does link to the manifest, so at least we would only fetch it if the site actually has a manifest.

Also, we can't really change the user-agent. The one currently used helps bypassing some sites via Cloudflare with some protection enabled. Adding a different try with a different UA would also make it take downloading icons very long and complex in my opinion. But maybe it's something to look at.

Agreed.

[BlackDex]

The html does link to the manifest, so at least we would only fetch it if the site actually has a manifest.

That still adds more complexity though. I have the same issue with sites that use http-equiv to redirect.

We would need to parse it, see if there are no link items at all, if so redirect/fetch again, keep track of the redirect amounts etc...
It is something on my to-do list though. But it is not something which really breaks the UX i think. It will enhance it though.

[BlackDex]

I did some quick testing:

• kraken.com > Needs a security check/Javascript
• rakuten.de > Provides a SVG only, which i deem unsafe to be implemented directly, and converting to a PNG is in my eyes a bit too much.
• www.crunchyroll.com > Works
• camelcamelcamel.com > Works
• codepen.io > Works
• coolblue.de > Cloudflare Block/Javascript
• my.konami.net > Invalid certificate
• rebuy.de > Works

A workaround would be to manually add an icon to the icon cache using the same naming convention as you see in the cache.
Other than that, it would be hard to add these.
The only item which might be interesting would be the manifest part. But, the site you mentioned which has that actually just needs to fix there HTML, since they provide a apple-touch-icon rel, but that points to an invalid, not existing image.