robots.txt is a text file that tells web robots what to crawl and what not to crawl.
robots.txt files can give attackers valuable information on potential targets by giving them clues about directories their >owners are trying to protect. robots.txt files tell search engines which directories on a web server they can and cannot >read. they offer clues about where system administrators store sensitive assets because the mention of a directory in a >robots.txt file screams out that the owner has something they want to hide. In the simplest cases, it (robots.txt) will >reveal restricted paths and the technology used by your servers source:https://www.theregister.co.uk/2015/05/19/robotstxt/
Demo:
