CORS issues? (Onebox generation and Image fetching broken)

[aschrijver]

Describe the bug

I recently did a fresh Discourse install on Yunohost for the forum https://discuss.coding.social and everything worked fine. When dropping a link in a post the Onebox functionality would kick in and generate a nice link preview from the opengraph data found at the URL.

That all suddenly stopped working. Dropping a link no longer generates the preview, just shows the link text.

This while since initial install I did not do any updates / upgrades of both Yunohost itself or any of the applications I have installed, nor did I tweak anything in the admin interface of Discourse, or even updated the VPS itself. Everything is box-standard Yunohost + Discourse.

Context

• Hardware: VPS bought online
• YunoHost version: 4.3.6.2 (stable)
• I have access to my server: Through SSH as well as through the webadmin
• Are you in a special context or did you perform some particular tweaking on your YunoHost instance?: No
• Using, or trying to install package version/branch: 2.7.13~ynh1

Steps to reproduce

Any URL typed into a discourse post on a single line that would normally trigger onebox generation, just displays the text. The hourglass icon for the generator appears very briefly, then stops. There's no fetching of remote opengraph metadata.

Expected behavior

Dropping a link should yield a onebox similar to the following:

Logs

There are no relevant logs that relate to this issue, neither on Yunohost nor in the admin UI of Discourse. In Yunohost the last log entries date back to when oneboxes were still operational.

The reason I report this issue in this repository is that I operate various Discourse forums that are not hosted on Yunohost, where this never has been a problem. Whereas I know of one other Yunohost + Discourse that has the exact same issue, and it also happened unexpectedly (at the time they blamed themself for trying to install a Discourse plugin, but now it looks like that wasn't the cause)

This forum is https://fediverse.town operated by @paulatothepeople and I create a topic to notify them at: https://fediverse.town/t/inability-to-generate-link-previews-serious-bug/404

I don't know if I can troubleshoot a yuno-hosted Discourse in the same way as a regular installation (via the discourse docker install). Looks like it is needed to dive into SSH and find more information there as to what's causing this.

As a general feedback to the Yunohost project and the Discourse package in particular: It is a pity that the Discourse upgrade facilities cannot be used, as I find them to be pretty solid and stable. In Yunohost I feel that too much information is hidden from me, for the sake of facilitating ease-of-use. This is fine if everything works, but leaves on in the dark when trouble occurs. Maybe the README can be updated with a section on 'Troubleshooting', just enough instruction on where yunohost ends, and discourse sysadmin takes over.

[aschrijver]

First link is to: https://ec.europa.eu/info/law/better-regulation/have-your-say/initiatives/13394-Brain-drain-mitigating-challenges-associated-with-population-decline-communication-_en

[tituspijean]

I might suspect CORS settings, but we will need extracts from your browser console to confirm it.

[aschrijver]

Ah, it is good you mention. I thought there were no logs, but an 'Inspect' in Firefox gives errors relating to Onebox while typing https://fedi.foundation in a new topic. A site that has a whole bunch of opengraph and SEO metadata and should generate a preview.

First a warning message:

Content Security Policy: This site (https://discuss.coding.social) has a Report-Only policy without a report URI. CSP will not block and cannot report violations of this policy.

Followed by a 404 error:

XHRGET https://discuss.coding.social/onebox?url=https://fediverse.codeberg.page/&refresh=false&category_id=5&topic_id=45
[HTTP/1.1 404 Not Found 0ms]

GET https://discuss.coding.social/onebox?url=https://fediverse.codeberg.page/&refresh=false&category_id=5&topic_id=45
Status: 404 Not Found
Version: HTTP/1.1
Transferred: 0 B (0 B size)

Request headers
==============

Referrer Policy: no-referrer-when-downgrade
content-length: 0
content-security-policy: upgrade-insecure-requests
content-security-policy-report-only: default-src https: data: blob: ; object-src https: data: 'unsafe-inline'; style-src https: data: 'unsafe-inline' ; script-src https: data: 'unsafe-inline' 'unsafe-eval'
content-type: text/plain; charset=utf-8
date: Sun, 17 Apr 2022 09:43:01 GMT
permissions-policy: interest-cohort=()
referrer-policy: no-referrer-when-downgrade
server: nginx
strict-transport-security: max-age=63072000; includeSubDomains; preload
vary: Accept
x-content-type-options: nosniff
x-discourse-route: onebox/show
x-discourse-username: myusername
x-download-options: noopen
X-Firefox-Spdy: h2
x-frame-options: SAMEORIGIN
x-permitted-cross-domain-policies: none
x-runtime: 0.072555
x-sso-wat: You've just been SSOed
x-xss-protection: 1; mode=block

Response Headers
================

Accept: text/html, */*; q=0.01
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.5
Connection: keep-alive
Cookie:_t=[redacted]; _forum_session=[redacted]; _bypass_cache=true
Discourse-Logged-In: true
Discourse-Present: true
DNT: 1
Host: discuss.coding.social
Referer: https://discuss.coding.social/t/question-does-having-a-technology-vision-help-adoption/45/4
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-origin
Sec-GPC: 1
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; [redacted]
X-CSRF-Token: [redacted]
X-Requested-With: XMLHttpRequest

[aschrijver]

I just found another issue, and like @tituspijean says it looks like there's CORS problem as the cause.

When you add an image in Markdown and use an external image URL in doing so, then at the time that Discourse will fetch the remote image to store in its own archive locally, a similar error occurs and the image is broken, is no longer displayed.

The warnings in the console is the following:

Content Security Policy: This site (https://discuss.coding.social) has a Report-Only policy without a report URI. CSP will not block and cannot report violations of this policy. 28

Referrer Policy: Less restricted policies, including ‘no-referrer-when-downgrade’, ‘origin-when-cross-origin’ and ‘unsafe-url’, will be ignored soon for the cross-site request: https://socialhub.activitypub.rocks/uploads/default/original/1X/71be05cf3c42e4ae015eb5cdb9f7f956faf8a684.png

Request to access cookie or storage on “https://socialhub.activitypub.rocks/uploads/default/original/1X/71be05cf3c42e4ae015eb5cdb9f7f956faf8a684.png” was blocked because we are blocking all third-party storage access requests and content blocking is enabled.

Here's the result in the UI. The first image was loaded from disk. The second image contains an external image URL and shows a placeholder 'broken image' icon after Discourse tries to fetch it:

There's no error in the log, but the image isn't fetched.

[zamentur]

Note that infobox features could include external trackers compromizing private messages title through referer for example.

[aschrijver]

Somehow:

https://fosstodon.org

Generates a OneBox, but:

https://mastodon.technology

Does not.

They are both running the same version of Mastodon (v3.5.3). I tested this on 2 yunohosted Discourse forums.

In Yunohost Services console I see a logmessage in /var/www/discourse/log/production.log:

Failed to onebox https://mastodon.technology Address family not supported by protocol - getifaddrs (Errno::EAFNOSUPPORT) ["/opt/rbenv/versions/2.7.1/lib/ruby/2.7.0/resolv.rb:410:in `ip_address_list'", "/opt/rbenv/versions/2.7.1/lib/ruby/2.7.0/resolv.rb:410:in `use_ipv6?'", "/opt/rbenv/versions/2.7.1/lib/ruby/2.7.0/resolv.rb:403:in `each_address'", "/opt/rbenv/versions/2.7.1/lib/ruby/2.7.0/resolv.rb:116:in `block in each_address'", "/opt/rbenv/versions/2.7.1/lib/ruby/2.7.0/resolv.rb:115:in `each'", "/opt/rbenv/versions/2.7.1/lib/ruby/2.7.0/resolv.rb:115:in `each_address'", "/opt/rbenv/versions/2.7.1/lib/ruby/2.7.0/resolv.rb:58:in `each_address'", "/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/excon-0.81.0/lib/excon/socket.rb:110:in `connect'", "/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/excon-0.81.0/lib/excon/ssl_socket.rb:170:in `connect'", "/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/excon-0.81.0/lib/excon/socket.rb:49:in `initialize'", "/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/excon-0.81.0/lib/excon/ssl_socket.rb:10:in `initialize'", "/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/excon-0.81.0/lib/excon/connection.rb:461:in `new'", "/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/excon-0.81.0/lib/excon/connection.rb:461:in `socket'", "/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/excon-0.81.0/lib/excon/connection.rb:118:in `request_call'", "/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/excon-0.81.0/lib/excon/middlewares/mock.rb:57:in `request_call'", "/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/excon-0.81.0/lib/excon/middlewares/instrumentor.rb:34:in `request_call'", "/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/excon-0.81.0/lib/excon/middlewares/idempotent.rb:19:in `request_call'", "/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/excon-0.81.0/lib/excon/middlewares/base.rb:22:in `request_call'", "/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/excon-0.81.0/lib/excon/middlewares/base.rb:22:in `request_call'", "/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/excon-0.81.0/lib/excon/connection.rb:273:in `request'", "/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/excon-0.81.0/lib/excon.rb:250:in `head'", "/var/www/discourse/lib/final_destination.rb:196:in `public_send'", "/var/www/discourse/lib/final_destination.rb:196:in `resolve'", "/var/www/discourse/lib/oneboxer.rb:425:in `block in external_onebox'", "/var/www/discourse/lib/cache.rb:94:in `fetch'", "/var/www/discourse/lib/oneboxer.rb:399:in `external_onebox'", "/var/www/discourse/lib/oneboxer.rb:233:in `onebox_raw'", "/var/www/discourse/lib/oneboxer.rb:60:in `onebox'", "/var/www/discourse/lib/cooked_post_processor.rb:544:in `block in post_process_oneboxes'", "/var/www/discourse/lib/oneboxer.rb:135:in `block in apply'", "/var/www/discourse/lib/oneboxer.rb:120:in `block in each_onebox_link'", "/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/nokogiri-1.11.7-x86_64-linux/lib/nokogiri/xml/node_set.rb:239:in `block in each'", "/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/nokogiri-1.11.7-x86_64-linux/lib/nokogiri/xml/node_set.rb:238:in `upto'", "/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/nokogiri-1.11.7-x86_64-linux/lib/nokogiri/xml/node_set.rb:238:in `each'", "/var/www/discourse/lib/oneboxer.rb:119:in `each_onebox_link'", "/var/www/discourse/lib/oneboxer.rb:134:in `apply'", "/var/www/discourse/lib/cooked_post_processor.rb:525:in `post_process_oneboxes'", "/var/www/discourse/lib/cooked_post_processor.rb:37:in `block in post_process'", "/var/www/discourse/lib/distributed_mutex.rb:33:in `block in synchronize'", "/var/www/discourse/lib/distributed_mutex.rb:29:in `synchronize'", "/var/www/discourse/lib/distributed_mutex.rb:29:in `synchronize'", "/var/www/discourse/lib/distributed_mutex.rb:14:in `synchronize'", "/var/www/discourse/lib/cooked_post_processor.rb:34:in `post_process'", "/var/www/discourse/app/jobs/regular/process_post.rb:26:in `block in execute'", "/var/www/discourse/lib/distributed_mutex.rb:33:in `block in synchronize'", "/var/www/discourse/lib/distributed_mutex.rb:29:in `synchronize'", "/var/www/discourse/lib/distributed_mutex.rb:29:in `synchronize'", "/var/www/discourse/lib/distributed_mutex.rb:14:in `synchronize'", "/var/www/discourse/app/jobs/regular/process_post.rb:10:in `execute'", "/var/www/discourse/app/jobs/base.rb:232:in `block (2 levels) in perform'", "/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/rails_multisite-4.0.0/lib/rails_multisite/connection_management.rb:80:in `with_connection'", "/var/www/discourse/app/jobs/base.rb:221:in `block in perform'", "/var/www/discourse/app/jobs/base.rb:217:in `each'", "/var/www/discourse/app/jobs/base.rb:217:in `perform'", "/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.2.1/lib/sidekiq/processor.rb:196:in `execute_job'", "/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.2.1/lib/sidekiq/processor.rb:164:in `block (2 levels) in process'", "/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.2.1/lib/sidekiq/middleware/chain.rb:138:in `block in invoke'", "/var/www/discourse/lib/sidekiq/pausable.rb:138:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.2.1/lib/sidekiq/middleware/chain.rb:140:in `block in invoke'", "/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.2.1/lib/sidekiq/middleware/chain.rb:143:in `invoke'", "/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.2.1/lib/sidekiq/processor.rb:163:in `block in process'", "/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.2.1/lib/sidekiq/processor.rb:136:in `block (6 levels) in dispatch'", "/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.2.1/lib/sidekiq/job_retry.rb:112:in `local'", "/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.2.1/lib/sidekiq/processor.rb:135:in `block (5 levels) in dispatch'", "/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.2.1/lib/sidekiq.rb:38:in `block in <module:Sidekiq>'", "/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.2.1/lib/sidekiq/processor.rb:131:in `block (4 levels) in dispatch'", "/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.2.1/lib/sidekiq/processor.rb:257:in `stats'", "/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.2.1/lib/sidekiq/processor.rb:126:in `block (3 levels) in dispatch'", "/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.2.1/lib/sidekiq/job_logger.rb:13:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.2.1/lib/sidekiq/processor.rb:125:in `block (2 levels) in dispatch'", "/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.2.1/lib/sidekiq/job_retry.rb:79:in `global'", "/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.2.1/lib/sidekiq/processor.rb:124:in `block in dispatch'", "/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.2.1/lib/sidekiq/logger.rb:11:in `with'", "/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.2.1/lib/sidekiq/job_logger.rb:33:in `prepare'", "/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.2.1/lib/sidekiq/processor.rb:123:in `dispatch'", "/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.2.1/lib/sidekiq/processor.rb:162:in `process'", "/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.2.1/lib/sidekiq/processor.rb:78:in `process_one'", "/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.2.1/lib/sidekiq/processor.rb:68:in `run'", "/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.2.1/lib/sidekiq/util.rb:43:in `watchdog'", "/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.2.1/lib/sidekiq/util.rb:52:in `block in safe_thread'"]

[aschrijver]

The latest upgrade to 2.8.8 fixed this issue.