GetCrawlYear to accompany GetCrawlMonth

[ianmilligan1]

I'm doing some research into a web archive right now, and it's actually long enough that year-over-year data is preferable to more granular day-by-day or month-by-month. We should have GetCrawlYear built into aut.

[ianmilligan1]

Testing in the wild with:

import io.archivesunleashed.spark.matchbox.{ExtractDomain, ExtractLinks, RecordLoader}
import io.archivesunleashed.spark.rdd.RecordRDD._

RecordLoader.loadArchives("/mnt/vol1/data_sets/labour/*.gz", sc)
  .keepValidPages()
  .map(r => (r.getCrawlYear, ExtractLinks(r.getUrl, r.getContentString)))
  .flatMap(r => r._2.map(f => (r._1, ExtractDomain(f._1).replaceAll("^\\s*www\\.", ""), ExtractDomain(f._2).replaceAll("^\\s*www\\.", ""))))
  .filter(r => r._2 != "" && r._3 != "")
  .countItems()
  .filter(r => r._2 > 5)
  .saveAsTextFile("/mnt/vol1/derivative_data/labour/sitelinks-year")

[ruebot]

Resolved with: 010fe24