Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Link preview should respect robots.txt #3242

Open
26000 opened this issue May 18, 2018 · 2 comments
Open

Link preview should respect robots.txt #3242

26000 opened this issue May 18, 2018 · 2 comments
Labels
O-Occasional Affects or can be seen by some users regularly or most users rarely S-Minor Blocks non-critical functionality, workarounds exist. T-Defect Bugs, crashes, hangs, security vulnerabilities, or other reported issues.

Comments

@26000
Copy link

26000 commented May 18, 2018

Description

When we send a link in Matrix, the client can use the Synapse's integrated prefetcher to fetch the link preview. It though doesn't check if the website allows bots there and crawls the page regardlessly.

We need to have a User-agent for Synapse and to parse robots.txt at the root of the domain user wanted to preview, if access for Synapse is denied, do not visit the URL.
@turt2live turt2live mentioned this issue May 18, 2018
Open
@neilisfragile neilisfragile added z-p2 (Deprecated Label) z-minor (Deprecated Label) labels May 22, 2018
@richvdh
Copy link
Member

richvdh commented May 29, 2018

I really thought we had this issue already, but I can't find it...

@ptman
Copy link
Contributor

ptman commented Jun 25, 2018

Probably just this TODO:

synapse/synapse/rest/media/v1/preview_url_resource.py

Line 281 in aafb0f6

# TODO: we should probably honour robots.txt... except in practice

@H-Shay H-Shay added T-Defect Bugs, crashes, hangs, security vulnerabilities, or other reported issues. O-Occasional Affects or can be seen by some users regularly or most users rarely S-Minor Blocks non-critical functionality, workarounds exist. and removed z-p2 (Deprecated Label) z-minor (Deprecated Label) labels Mar 6, 2023
@matrixbot matrixbot mentioned this issue Dec 21, 2023
Open
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
O-Occasional Affects or can be seen by some users regularly or most users rarely S-Minor Blocks non-critical functionality, workarounds exist. T-Defect Bugs, crashes, hangs, security vulnerabilities, or other reported issues.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@ptman @neilisfragile @richvdh @H-Shay @26000