Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Disallow crawling of data export links #568

Closed
hancush opened this issue Nov 16, 2021 · 3 comments · Fixed by #573
Closed

Disallow crawling of data export links #568

hancush opened this issue Nov 16, 2021 · 3 comments · Fixed by #573
Assignees

Comments

@hancush
Copy link
Member

hancush commented Nov 16, 2021

We do not want bots to crawl the export links. Add nofollow, like we did for source files.

Because the standardized data download is relatively heavy and not cached, we may also want to restrict downloads to users who have given us an email address. Cc: @fgregg

@fgregg
Copy link
Member

fgregg commented Nov 16, 2021

thanks, @hancush. @smcalilly can add the nofollow.

do you have suggestions on how to add the restriction? @hancush ?

@hancush
Copy link
Member Author

hancush commented Nov 16, 2021

iirc, "authenticated" users will have some cookie set. you could check for that 1) when rendering a page w/ the download button, to determine whether to show the download link and 2) in the download view?

@fgregg
Copy link
Member

fgregg commented Nov 16, 2021

we would want this on rendering the download button.

we probably want a more complicated interaction here, where the download button is there, but if the cookie is not present, then clicking prompts the user to login, and if the cookie is there it is a normal link.

I'll open an issue for that behavior. To close this issue, @smcalilly, just add the nofollow link.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants