Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ExtractBoilerpipeText to remove headers as well. #253 #256

Merged
merged 3 commits into from Aug 11, 2018
Merged

ExtractBoilerpipeText to remove headers as well. #253 #256

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
e1b0478
ExtractBoilerpipeText now removes headers.
greebie Aug 10, 2018
162e238
Fix scalastyle errors
greebie Aug 10, 2018
a35da9a
Merge branch 'master' into issue-253
greebie Aug 11, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
11 changes: 8 additions & 3 deletions src/main/scala/io/archivesunleashed/matchbox/ExtractBoilerpipeText.scala
View file
Open in desktop
Expand Up @@ -28,12 +28,17 @@ object ExtractBoilerpipeText {
* @param input an html string possibly containing boilerpipe text
* @return text with boilerplate removed or Nil if the text is empty.
*/

def apply(input: String): String = {
val maybeInput = Option(input)
removeBoilerplate(RemoveHttpHeader(input))
}

private def removeBoilerplate(input: String): String = {
val maybeInput = Option(DefaultExtractor.INSTANCE
.getText(input).replaceAll("[\\r\\n]+", " ").trim())
maybeInput match {
case Some(text) =>
DefaultExtractor.INSTANCE
.getText(input).replaceAll("[\\r\\n]+", " ").trim()
text
case None =>
""
}
Expand Down
7 changes: 7 additions & 0 deletions src/test/scala/io/archivesunleashed/matchbox/ExtractBoilerPipeTextTest.scala
View file
Open in desktop
Expand Up @@ -25,6 +25,9 @@ import org.scalatest.junit.JUnitRunner

@RunWith(classOf[JUnitRunner])
class ExtractBoilerPipeTextTest extends FunSuite {
val header = "HTTP/1.0 200 OK Content-Type: text/html;" +
"charset=UTF-8 Expires: Fri, 20 Jul 2018 19:09:28 GMT Date:" +
"Fri, 20 Jul 2018 19:09:28 GMT Cache-Control: private,;\r\n\r\n"
var text = """<p>Text with a boiler plate.<p>
<footer>Copyright 2017</footer>"""
var boiler = """Copyright 2017"""
Expand All @@ -36,4 +39,8 @@ class ExtractBoilerPipeTextTest extends FunSuite {
// scalastyle:on null
assert(ExtractBoilerpipeText("All Rights Reserved.") == "")
}

test("Removes Header information") {
assert(ExtractBoilerpipeText(header + text) == boiler)
}
}