Fix log cleanup

• Fixed log cleanup not functioning properly.
  • Additionally made the log cleanup to happen on server start-up
• Added robots.txt to disallow crawlers since they wouldn't have an api key anyway.