Make webpages() consistent across aut and ARCH.

[ruebot]

GitHub issue(s):

• Remove http headers, and html on webpages() #538

What does this Pull Request do?

Make webpages() consistent across aut and ARCH.

• Filter HTTP headers, and HTML from content on webpages so that it is consistent with the app implementation, and the ARCH implementation
• Update PlainTextExtractor to use .all() since HTML is removed from content
• Add domain to all()
• Update csv exports on app so that they are rfc4180 compliant
• Apply GitHub workflows to main branch
• Consistent formating on DataFrameLoader.scala
• Update tests as needed
• Resolves Remove http headers, and html on webpages() #538

How should this be tested?

• GitHub actions build
• Testing using documentation updates: Documentation updates for https://github.com/archivesunleashed/aut/pu… aut-docs#117

[codecov]

Codecov Report

❗ No coverage uploaded for pull request base (main@988f70f). Click here to learn what that means.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #539   +/-   ##
=======================================
  Coverage        ?   92.93%           
  Complexity      ?       42           
=======================================
  Files           ?       39           
  Lines           ?      835           
  Branches        ?       52           
=======================================
  Hits            ?      776           
  Misses          ?       35           
  Partials        ?       24           

[ruebot]

I think we should consider renaming the content column produced by .all() to raw_content or something along those lines. So, it is clearly differentiated from the content column produced by .webpages(). Otherwise, it's easy to presume you should be able to apply particular matchbox utilities or udfs to the column, when in actuality you can't be cause the correct data isn't there. I'm linking of extractBoilerPipe, extractLink, and extractImageLinks, all of which require HTML to still be present in the content column.

Does that make sense? If so, I can get it updated here, and in archivesunleashed/aut-docs#117 as well.

I'm also working on getting the PySpark example notebooks updated as well to reflect this. In the process there, I realized we never made an equivalent for keepValidPagesDF in the Python implementation. I'm struggling with how to reimplement that other than verbosely doing in the documentation examples like I here 🤷‍♂️

[ianmilligan1]

Does that make sense?

Makes a lot of sense – I like raw_content. I can test the PR when it's updated and ready to merge!

[ruebot]

@ianmilligan1 ready for testing when you have time.

Documentation PR: archivesunleashed/aut-docs#117
Updated PySpark notebooks: https://github.com/archivesunleashed/notebooks/tree/main/PySpark%20Examples

[ianmilligan1]

Tested and all works! (Except for the discovery of Spark 3.0.0 having a bug, as discussed on Slack!)