Add language detection column to webpages.

[ruebot]

GitHub issue(s): #402

What does this Pull Request do?

Add language detection column to webpages.

How should this be tested?

Something like:

import io.archivesunleashed.df._
import io.archivesunleashed.df._

RecordLoader.loadArchives("/home/nruest/Projects/au/sample-data/geocites/1", sc).webpages().show()

Should yield:

+----------+--------------------+--------------------+--------------------+--------+--------------------+
|crawl_date|                 url|mime_type_web_server|      mime_type_tika|language|             content|
+----------+--------------------+--------------------+--------------------+--------+--------------------+
|  20091027|http://geocities....|           text/html|           text/html|      en|HTTP/1.1 200 OK
...|
|  20091027|http://geocities....|           text/html|           text/html|      en|HTTP/1.1 200 OK
...|
|  20091027|http://www.geocit...|           text/html|           text/html|      en|HTTP/1.1 200 OK
...|
|  20091027|http://www.geocit...|           text/html|           text/html|      en|HTTP/1.1 200 OK
...|
|  20091027|http://geocities....|           text/html|           text/html|      en|HTTP/1.1 200 OK
...|
|  20091027|http://geocities....|           text/html|           text/html|      en|HTTP/1.1 200 OK
...|
|  20091027|http://www.talent...|           text/html|application/xhtml...|      en|HTTP/1.1 200 OK
...|
|  20091027|http://geocities....|           text/html|           text/html|      ms|HTTP/1.1 200 OK
...|
|  20091027|http://geocities....|           text/html|           text/html|      en|HTTP/1.1 200 OK
...|
|  20091027|http://www.geocit...|           text/html|           text/html|      en|HTTP/1.1 200 OK
...|
|  20091027|http://www.geocit...|           text/html|           text/html|      en|HTTP/1.1 200 OK
...|
|  20091027|http://geocities....|           text/html|           text/html|      en|HTTP/1.1 200 OK
...|
|  20091027|http://geocities....|           text/html|           text/html|      en|HTTP/1.1 200 OK
...|
|  20091027|http://www.geocit...|           text/html|           text/html|      en|HTTP/1.1 200 OK
...|
|  20091027|http://geocities....|           text/html|application/xhtml...|      en|HTTP/1.1 200 OK
...|
|  20091027|http://geocities....|           text/html|           text/html|      en|HTTP/1.1 200 OK
...|
|  20091027|http://geocities....|           text/html|           text/html|      en|HTTP/1.1 200 OK
...|
|  20091027|http://geocities....|           text/html|           text/html|      en|HTTP/1.1 200 OK
...|
|  20091027|http://www.infoca...|           text/html|           text/html|      en|HTTP/1.1 200 OK
...|
|  20091027|http://geocities....|           text/html|           text/html|      en|HTTP/1.1 200 OK
...|
+----------+--------------------+--------------------+--------------------+--------+--------------------+

Additional Notes:

If it comes up, we don't add Remote HTML and Headers on content because we leave that all in there for ExtractHyperlinks, so we can do the domains derivative.

[codecov]

Codecov Report

Merging #403 into master will increase coverage by 0.04%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master     #403      +/-   ##
==========================================
+ Coverage   77.48%   77.53%   +0.04%     
==========================================
  Files          40       40              
  Lines        1537     1540       +3     
  Branches      292      292              
==========================================
+ Hits         1191     1194       +3     
  Misses        218      218              
  Partials      128      128

[ianmilligan1]

Good idea - one of those important filters we have. I’ll try to review later today or first thing tomorrow.

[ianmilligan1]

Works nicely. For doc purposes, the example script you had was missing an import, but this checked out:

import io.archivesunleashed._
import io.archivesunleashed.df._

RecordLoader.loadArchives("example.arc.gz", sc).webpages().show()