dev.qubes-os.org intermittently unavailable

Qubes OS version

Qubes-OS 4.0.3

Affected component(s) or functionality

Web site: dev.qubes-os.org

Brief summary

The URL https://dev.qubes-os.org/projects/core-admin/en/latest/qubes-backup.html#module-qubes.backup is unavailable. Sometimes the first call works but then all subsequent calls fail. I use TBB as proxy in a qube through which I connect to websites. I have dedicated Firefox profiles for web sites I regular visit.

To Reproduce

Steps to reproduce the behavior:

1. Qube based on Debian 10 with Firefox ESR and TBB
2. Start TBB
3. Point Firefox to TBB as proxy (socks5, proxy DNS)
4. Open https://dev.qubes-os.org/projects/core-admin/en/latest/qubes-backup.html#module-qubes.backup
5. If the site appears, click on one of the source links on the right side, or reload, or open try to open in a new window/tab.
6. I get the error:

Error 1014 …
CNAME Cross-User Banned

Expected behavior

No Error page.

Actual behavior

Error 1014 …
CNAME Cross-User Banned


What happened?

You've requested a page on a website that is part of the Cloudflare network. The host is
configured as a CNAME across accounts on Cloudflare, which is prohibited by security policy.

Screenshots

Additional context

Firefox ESR profiles based on a template profile which contains the plugins noscript and Http Everywhere. NoScript is configured to allow only the web site for which the dedicated profile is meant, e.g. qubes-os.org: allow qubes-os.org and related sites, e.g. the forum or the mail archive, but only https enabled sites.

Solutions you've tried

Nothing I can do. It's Cloudfare which tells me that there is some kind of web site configuration problem.

Relevant documentation you've consulted

Ask in the forum and got advised to open a problem report:
https://github.com/QubesOS/qubes-issues/issues/6080

Related, non-duplicate issues

[marmarek]

Hmm, this is weird. If that indeed be "prohibited by security policy", it shouldn't work at all. Additionally, this is recommended way to configure readthedocs hosting on a custom domain: https://docs.readthedocs.io/en/stable/custom_domains.html#custom-domain-support
Maybe some cloudflare issue? Or a misleading error meaning actually "we don't like Tor"?
Anyway, this seems rather related to readthedocs.org hosting configuration, not something directly controlled by us. Lets keep this issue open here and use as a reference when submitting a report to them.

[andrewdavidwong]

Related/possible duplicate: #4538

Any idea what's causing this, @woju?

[woju]

On Sun, Sep 20, 2020 at 11:16:30AM -0700, Andrew David Wong wrote: Related/possible duplicate: #4538 Any idea what's causing this, @woju?

I believe it's duplicate of #4538.

[andrewdavidwong]

I believe it's duplicate of #4538.

Ok, looks like we decided to close #4538 as not our bug, but since @marmarek asked for this one to remain open, I'll leave this open and #4538 closed.

Any idea about the cause, though, @woju?

[woju]

On Tue, Sep 22, 2020 at 03:26:55PM +0000, Andrew David Wong wrote: Any idea about the cause, though, @woju?

Nothing that wasn't said in both of the bugs. To restate: - We use cloudflare's DNS for our domain `qubes-os.org`. - Read the docs too, for `readthedocs.io`. - RTD has an option to use user's own domain (usually for PR reasons, and the content is also available using standard domains, like https://qubes-os.readthedocs.io/projects/core-admin/en/latest/). - To use the option, we have on our end configure appropriate CNAME from our domain to theirs (and also some config at RTD, which is not relevant here). - CF has policy to disallow CNAME from one customer to another in their DDoS protection products, because historically this was a vector for vulnerabilities. Supposedly this protection should not kick in for very basic DNS setup. - The *target* domain can opt-out of this protection for a specific domain (https://support.cloudflare.com/hc/en-us/articles/360029779472-Error-1014-CNAME-Cross-User-Banned#error1014). Now, as marmarek pointed out, this would mean that our setup should either work always or shouldn't work at all, even without Tor. My wild guess is that there is something wrong in CF DDoS protection logic. Maybe some uninteded edge case between CNAME cross-user detection, then suppressed by manual support action, and some weird interaction with Tor detection. Something is triggered, but the wrong error message is produced. Or, alternatively, this wrong error code is intended, not to make an oracle for attacker. This is something that probably only CF could explain. To start troubleshooting the issue, I'd ask user if the `readthedocs.io` URL works: https://qubes-os.readthedocs.io/projects/core-admin/en/latest/ If no, the problem is that RTD+CF don't like Tor, because operators of CF-protected sites can disable service for entire countries, and Tor counts as one for the purpose of that control (this is grossly oversimplified, and can be wrong). I'd follow up to RTD. If yes, then the problem is likely Tor vs CF alone. I'd follow up to CF. In either case, there is not much we can do about it besides abandoning dev.qubes-os.org and reverting to qubes-os.rtfd.io (if it works over Tor). Again, maybe someone at Cloudflare could tell more.

[marmarek]

CF has policy to disallow CNAME from one customer to another in their DDoS protection products, because historically this was a vector for vulnerabilities. Supposedly this protection should not kick in for very basic DNS setup.

FWIW I do see in our domain configuration that this CNAME is kind of special - CF tells me some of its parameters (like TTL etc) are configured in the target domain's other account. So, it does recognize it is cross-account CNAME, applies some limits, but allows it otherwise (at the domain config level).

[woju]

FWIW I do see in our domain configuration that this CNAME is kind of special - CF tells me some of its parameters (like TTL etc) are configured in the target domain's other account. So, it does recognize it is cross-account CNAME, applies some limits, but allows it otherwise (at the domain config level).

This post: https://community.cloudflare.com/t/error-1014-cname-cross-user-banned/12584/2 mentions something about grey and orange clouds. I don't know what it is, but the OP seems to imply that it does matter.

[marmarek]

Control over "grey and orange clouds" for this entry is also disabled for me.