remove /old/ pages from search engines

[SethTisue]

e.g. Googling "Scala FAQ" takes you to http://www.scala-lang.org/old/faq which (as @soc recently pointed out at https://groups.google.com/d/msg/scala-internals/nkh7_L4EpHA/9z1DM5QQBQAJ) is embarrassingly outdated.

I think it's fine to leave the old pages up so that links to them don't just go dead, but we really don't want this stuff turning up when newbies google us.

[DarkDimius]

There are a lot of knowledge hidden discussions in old part of forums that would be great to find through google. Such as http://www.scala-lang.org/old/node/9126.html
Could we mark everything but old forums as non-indexed by search engines?

[fsalvi]

Forums are just the archives of google groups mailing list.
Eg, for Inlining Problem:
https://groups.google.com/forum/#!msg/scala-internals/a6Py8kpeiQY/VPDKvJV8SEEJ

[fsalvi]

I thought it would be interesting to know which are the old pages often requested on the scala website.

Here is the list (based on our webserver statistics):

• Code Examples
  http://www.scala-lang.org/node/219.html
• How can I catch exceptions?
  http://www.scala-lang.org/node/255.html
• Scala Actors: A Short Tutorial
  http://www.scala-lang.org/node/242
• Maven For Beginners
  http://www.scala-lang.org/node/345
• Scala Snippets
  http://www.scala-lang.org/node/220.html
• Scala Examples - Advanced
  http://www.scala-lang.org/node/44.html
• Scala in the Enterprise
  http://www.scala-lang.org/node/1658
• Scala levels: beginner to expert
  http://www.scala-lang.org/old/8610
• Scala REPL
  http://www.scala-lang.org/node/2097
• DSLs - A powerful Scala feature
  http://www.scala-lang.org/node/1403
• Scala .net
  http://www.scala-lang.org/node/10299
  http://www.scala-lang.org/node/168
• Scala Manuals
  http://www.scala-lang.org/old/node/198

Searching for these keywords on google always put the old website in top results...

There were other old web pages often requested, but I already added some redirects to point them to new pages which have the same content (eg tour of scala).

[soc]

Thanks, good to know!

[raboof]

I know the README is quite adamant that "Subdirectory scala-lang.org/old is a static copy of the old website. It was generated once and copied there, and it stays like that".

Still, it'd be real neat to put it up on github (even in its current 'raw' form), so we can at least create PR's to add some links to more recent material.

[jarrodu]

It sounds like we could benefit from updating the robots.txt file in the root directory.

[adamvoss]

What @fsalvi did with redirects sounds good. I know I still have recently pulled useful information out of the old website because it appeared in the search results. I did not investigate everything to see if the the current site had the info. I did note that other, similarly ranked, results did not look like that would have been as concise in answering the programming/language question I had.

I guess what I am trying to say is that I would be worried about delisting web pages if we are not sure there are good alternatives as you may be making learning harder.

[fsalvi]

Here's an updated list of old pages still requested (ordered by most watched)

• Scala Code Examples
  http://www.scala-lang.org/old/node/219

• How can I catch exceptions?
  http://www.scala-lang.org/old/node/255.html

• Getting Started with Scala
  http://www.scala-lang.org/old/node/166.html

• Scala Snippets
  http://www.scala-lang.org/old/node/220.html

• How to test String is null or empty?
  http://www.scala-lang.org/old/node/11722.html

• Scala Examples - Advanced
  http://www.scala-lang.org/old/node/44.html

• Maven For Beginners
  http://www.scala-lang.org/old/node/345

• Scala Actors: A Short Tutorial
  http://www.scala-lang.org/old/node/242

• A Tour of Scala: Pattern Matching
  http://www.scala-lang.org/old/node/120

• Scala levels: beginner to expert, application programmer to library designer
  http://www.scala-lang.org/old/node/8610

• A Tour of Scala: Case Classes
  http://www.scala-lang.org/old/node/107.html

• A Tour of Scala: Subclassing
  http://www.scala-lang.org/old/node/125

• A Tour of Scala: Packages
  http://www.scala-lang.org/old/node/119.html

• How to define a string literal bounded by double quotes as a raw string?
  http://www.scala-lang.org/old/node/5514

• A Tour of Scala: Sequence Comprehensions
  http://www.scala-lang.org/old/node/111

• Frequently Asked Questions - Java Interoperability
  http://www.scala-lang.org/old/faq/4

• DSLs - A powerful Scala feature
  http://www.scala-lang.org/old/node/1403

• Scala in the Enterprise
  http://www.scala-lang.org/old/node/1658

[SethTisue]

Thanks, Fabien, for compiling this list.

I think nearly all, perhaps 100%, of these results, are egregiously outdated and actually quite harmful for beginners to be landing on. So I still support my original proposal:

I think it's fine to leave the old pages up so that links to them don't just go dead, but we really don't want this stuff turning up when newbies google us.

[fsalvi]

I updated robots.txt to stop the index of /old directory.

[SethTisue]

Thank you, Fabien!