Extract Image Links DF API + Test

[jwli229]

---

GitHub issue(s):

• DataFrames for image analysis #220

What does this Pull Request do?

• Add DataFrame API to extract (source page, image url) pairs in WARRecord
• Add an entry point from DataframeLoader
• Add a test for the API

How should this be tested?

• mvn clean install
• mvn -Dtest=ExtractImageLinksTest test

Additional Notes:

• mvn clean install builds successfully and all tests pass
• I could not get the local snapshot build to run in the spark-shell. When I run spark-shell --packages "io.archivesunleashed:aut:0.16.1-SNAPSHOT", I get

Not sure if I'm starting the shell correctly for a snapshot build.

Interested parties

@lintool @ruebot

[codecov]

Codecov Report

Merging #221 into master will increase coverage by 0.5%.
The diff coverage is 85.71%.

@@           Coverage Diff            @@
##           master    #221     +/-   ##
========================================
+ Coverage    61.2%   61.7%   +0.5%     
========================================
  Files          34      34             
  Lines         665     679     +14     
  Branches      124     124             
========================================
+ Hits          407     419     +12     
- Misses        217     219      +2     
  Partials       41      41

Impacted Files	Coverage Δ
...n/scala/io/archivesunleashed/DataFrameLoader.scala	0% <0%> (ø)	⬆️
src/main/scala/io/archivesunleashed/package.scala	69.56% <100%> (+4.56%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fc8f4bf...3392feb. Read the comment docs.

[ruebot]

@JWZ2018 we don't have a SNAPSHOT repo, so unless you have everything setup perfectly, using --packages locally isn't going to work. Easiest way to move forward is to use --jars /path/to/aut/target/aut-0.16.1-SNAPSHOT-fatjar.jar after you build your branch locally. If you need a hand, just tag me in #aut on Slack.

All that said, I might have to add that commons library back in after #219 and #217 were merged. I'll sort that all out once we get closer to a new release 😄

[ruebot]

Add a doc comment here.

[ruebot]

Need a doc comment.

[lintool]

lg so far.

Anything obvious that we should add to this DF? Maybe the alt text?

Let's get the DF for the images themselves, then we can do interesting analyses like, what's the most common images that are linked to. With de-dup'ing on MD5, we can look at cases where the same exact images are being copied around and renamed.

[jwli229]

I will add the DF for the actual images in a separate PR to not clutter this one.

[lintool]

Okay, make the edits by @ruebot and I'll give a +1.

[ruebot]

@lintool you good for me to merge?

[lintool]

lgtm, go ahead and merge please.