Investigate making *Send nothing as Referer* the default *Rewrite Mode*

[ntninja]

Based on recent user input I've come to believe that many sites that many sites that do referer checking skip their check if the Referer header field is empty / not present. This could be used to simplify the whitelist and reduce the number of issues discovered in the future.

It's hard to properly decide this since we cannot know the total number of websites doing referer checking and which of these properly handle Send URL of target page vs Send nothing as Referer values. However an empty referer value is afaik also sent when loading HTTP content in HTTPS pages and definitely on direct hit. It also makes sense from a programmers perspective to write something like

if(referHeader.present()) {
    checkRefererOrThrow(refererHeader.value(), ["sourceDomain.a", "sourceDomain.b"]);
}

to do the referer checking: The check the presence of the header would likely already be added during development since the header is not present on direct-hit and during development you often have direct-hits while testing; adding the target domain as valid source domain on the other hand, is not something that would be needed during testing or real-life use-cases so it's much less likely. This is of course just a guess, but it's likely what I would do during development as well, simply due to how getting stuff done efficiently when programming works.

Due to lack of a better benchmark, I'll mostly go with the reasoning above if it turns out that at least 30% of the current whitelist entries are unnecessary when changing to direct-hit mode.

• Check existing whitelist entries on whether they are necessary when changing to direct-hit
• Figure out the necessary migration code if results turn out to be promising

[ntninja]

Currently score of tested whitelist entries entries is that 3 of 8 entries seem to be unnecessary. Additionally issue #112 will also likely result in a whitelist entry with a rule that's only necessary in the current default mode. I've asked a few more users for feedback but current results look quite promising (4 / 9 > 40%!).

[neckcen]

Due to lack of a better benchmark, I'll mostly go with the reasoning above if it turns out that at least 30% of the current whitelist entries are unnecessary when changing to direct-hit mode.

May I point out that proceeding this way is biased as most rules were added with Send URL of target page being the default. As such, most rules needed only with Send nothing as Referer would not exist in the current whitelist (for example #34).

[ntninja]

@neckcen: I fully agree with you and I'd wish there was a better way to acquire data on this – I just don't know how. Based on my reasoning above I just believe that almost any page not handling an empty referer correctly would also not handle a self-referer correctly. Getting solid data on this seems very hard through and I could use some input on how to do this.
Any ideas welcome!

[ReporterX]

I remember some sites will be broken if you send empty referrer. I encountered them in the past. I forget which sites though. Definitely there will be other sites added to the whitelist if you switch to another mode. It is hard to know which mode will generate fewer whitelist entries.

[ntninja]

This issue has been moved to GitLab: https://gitlab.com/smart-referer/smart-referer/issues/113