Allowing AWS, updating links

[ianmilligan1]

Just a pull request that does the following:

• some tweaks to the vagrantfile to make it AWS compatible, changing username to ubuntu;
• updating the commands given warcbase refactoring;
• new download points for spark notebooks;
• tiny tweaks and tests.

This lets us spin them up in the cloud.

I don't think we have any workshops ahead, so we could repurpose this as a general purpose warcbase repo. Alternatively, happy to migrate this over to a new repo if we want a cloud specific one.

[ianmilligan1]

Let me know what you think, @ruebot – keep it as one repo, or spin off as a separate one. I've tested and it works well on AWS.

[ruebot]

Let me make a side PR before I merge this to update local to Ubuntu 16.04, and when that happens the documentation will be consistent w/r/t username changing from vagrant to ubuntu, and making sure the code works. Hopefully I can get this done by end of day tomorrow.

...also, nice work! We're gonna make a DevOps engineer out of you 😄

---

Actually, you might be able to do this yourself with one tiny change.

[ianmilligan1]

Hah thanks @ruebot – have been learning a lot the past few weeks on this end. I can try to look into 16.04 myself, actually, could be a good chance to learn a bit more.

[ruebot]

@ianmilligan1 can you edit Vagrant like this? Then we should be consistent across the board. Once you have that, we can get this merged.

...and if you want to go a step further, you can do DigitalOcean like this. (We yanked our aws config a while back, otherwise I would have just told you to steal from there:smile:.)

[ianmilligan1]

ubuntu/xenial64 wasn't supported by aws, but lattice/ubuntu-trusty-64 was – seemed to be the closest box to build off of in the HashiCorp atlas? Just tested on AWS and it works.

[ruebot]

Warcbase didn't build for me. I'm about to disappear to a few meetings, but I'll troubleshoot in the late afternoon.

[ianmilligan1]

OK cool. I will try to build locally and see what's up in the meantime. It works on AWS but must have broken something locally..

[ianmilligan1]

Just ran via vagrant up and built it locally. Some of the messages are around skipping components of warcbase that people aren't going to use.

Just ran and tested via Spark Shell and Notebook, both execute sample scripts well. This ran on both.

import org.warcbase.spark.matchbox._ 
import org.warcbase.spark.rdd.RecordRDD._ 
val r = RecordLoader.loadArchives("/home/ubuntu/project/warcbase-resources/Sample-Data/ARCHIVEIT-227-UOFTORONTO-CANPOLPINT-20060622205612-00009-crawling025.archive.org.arc.gz", sc)
  .keepValidPages()
  .map(r => ExtractDomain(r.getUrl))
  .countItems()
  .take(10)

[ruebot]

Cool. We're good to go. I'll create a separate ticket to update the lessonplan.