Deploying on shinyapps.io when crawlers are blocked via robots.txt

[gadenbuie]

shinyapps.io uses a root-level, non-configurable robots.txt file to block all web crawlers. Any web crawlers that respect the robots.txt specification will not show a preview for your app. This is why preview cards won't appear as expected on Twitter.

There are good reasons for this default behavior. Primarily, blocking web crawlers keeps ensures that your app isn't started up (incurring costs and hours against your quota) just to render the preview card. On the other hand, most social media sites implement some sort of caching mechanism.

At the moment, I am not aware of any way to configure the settings in the robots.txt file. It is served globally at the root level of <user_name>.shinyapps.io, so an app-specific robots.txt stored at www/robots.txt won't have an effect or change the setting.

I'll update this issue if these settings become configurable, or if anyone reports a work-around.