Skip to content

Improved Simplification - #26

Merged
newsch merged 14 commits into
mainfrom
simpler
Aug 15, 2023
Merged

Improved Simplification#26
newsch merged 14 commits into
mainfrom
simpler

Conversation

@newsch

@newsch newsch commented Aug 10, 2023 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Additional work to bring HTML simplification in line with the Wikipedia Extracts API.

Closes #4.

Remaining work:

  • Extracts element denylist
  • Media elements
  • Mediawiki attributes
  • Span attributes
  • Mediawiki ids
  • Comments
  • Whitespace
  • Update section removal
  • Remove empty sections at end of processing
  • Add benchmarks for specific articles
  • Add html pretty printer from Expose HtmlSerializer for customizing output servo/html5ever#359 for comparison
  • Add snapshot tests
  • Get size/speed comparison for planet dump
  • Clean up/refactor simplification code
  • Test random articles

newsch added 2 commits August 10, 2023 11:41
@newsch
Add option to not simplify when extracting
c2d4ded 
Signed-off-by: Evan Lloyd New-Schmidt <evan@new-schmidt.com>
@newsch
Add simplification logging
4b91714 
Signed-off-by: Evan Lloyd New-Schmidt <evan@new-schmidt.com>
@newsch

newsch commented Aug 10, 2023

Copy link
Copy Markdown
Collaborator Author

I've been using the Pomona College article (Q7227384) for comparisons, it's the largest Wikipedia article referenced by OSM in the first 100,000 items in my local dump.

The original file is 1.5mB.
Previously it was reduced to 430.8kB (3.5x reduction)
With the changes so far, the new size is 44.6kB (33.8x reduction).

The output looks like this: pomona_college.html.txt

@biodranik

biodranik commented Aug 10, 2023

Copy link
Copy Markdown
Member

Need to remove whole first paragraph.

Remove section, span.

@newsch

newsch commented Aug 10, 2023

Copy link
Copy Markdown
Collaborator Author

Need to remove whole first paragraph.

Do you mean:

<!DOCTYPE html><html><head><meta charset="utf-8"><title>Pomona College</title><meta http-equiv="content-language" content="en"><meta http-equiv="vary" content="Accept"></head><body lang="en" class="mw-content-ltr sitedir-ltr ltr mw-body-content parsoid-body mediawiki mw-parser-output" dir="ltr"><section>

or:

<p><b>Pomona College</b> (<span><span>(<span><span>listen</span></span>)</span></span> <i title="English pronunciation respelling">pə-<span>MOH</span>-nə</i>) is a private liberal arts college in Claremont, California. It was established in 1887 by a group of Congregationalists who wanted to recreate a "college of the New England type" in Southern California. In 1925, it became the founding member of the Claremont Colleges consortium of adjacent, affiliated institutions.</p>

?

@biodranik

biodranik commented Aug 10, 2023

Copy link
Copy Markdown
Member

Can empty sections like Notes be detected and deleted? Or maybe remove all Notes completely? Is there any value in them?

@biodranik

biodranik commented Aug 10, 2023

Copy link
Copy Markdown
Member

The first one.

@newsch

newsch commented Aug 10, 2023

Copy link
Copy Markdown
Collaborator Author

Can empty sections like Notes be detected and deleted? Or maybe remove all Notes completely? Is there any value in them?

Yes, I'm working on that - I think it should handle most of the galleries too.

@newsch

newsch commented Aug 10, 2023 •
edited
Loading

Copy link
Copy Markdown
Collaborator Author

With empty sections removed, sections expanded, and doctype/html/body/whitespace removed: pomona_college2.html.txt

I think it looks pretty good! Need to try it on more articles.

@newsch

newsch commented Aug 10, 2023

Copy link
Copy Markdown
Collaborator Author

The main issues I've seen, which are present in the old scraper extracts too:

  • The pronunciation guides at the beginning of articles are sometimes shortened to "()".
  • Some sections have a short introduction followed by a table or figure, when those are removed the introduction is still present.

@biodranik

biodranik commented Aug 14, 2023

Copy link
Copy Markdown
Member

Cool, so it works! Let's finish this PR!

biodranik
biodranik approved these changes Aug 15, 2023
View reviewed changes
Comment thread src/html.rs Outdated
Comment thread src/html/pretty.rs
Comment thread tests/data/Q4185820-en/output.html Outdated
Comment thread tests/data/Q4185820-en/output.html
@newsch
newsch marked this pull request as ready for review August 15, 2023 22:07
newsch added 4 commits August 15, 2023 18:26
@newsch
Add snapshot tests for html output
b765c07 
- Article contents are from the 2023-04-01 Wikipedia Enterprise Dump
- Add benchmark for HTML processing

Signed-off-by: Evan Lloyd New-Schmidt <evan@new-schmidt.com>
@newsch
Add denylist from Extracts API
f9fac73 
Signed-off-by: Evan Lloyd New-Schmidt <evan@new-schmidt.com>
@newsch
Remove comments
8396c12 
Signed-off-by: Evan Lloyd New-Schmidt <evan@new-schmidt.com>
@newsch
Expand spans, sections, and body after removing head
2879099 
Signed-off-by: Evan Lloyd New-Schmidt <evan@new-schmidt.com>
newsch added 8 commits August 15, 2023 18:35
@newsch
Remove "(listen)" text
6e73cc1 
Signed-off-by: Evan Lloyd New-Schmidt <evan@new-schmidt.com>
@newsch
Remove empty sections after other removals
4c36435 
Signed-off-by: Evan Lloyd New-Schmidt <evan@new-schmidt.com>
@newsch
Remove doctype and html element
099072a 
Signed-off-by: Evan Lloyd New-Schmidt <evan@new-schmidt.com>
@newsch
Minify whitespace between elements
affe164 
Signed-off-by: Evan Lloyd New-Schmidt <evan@new-schmidt.com>
@newsch
Preserve excerpts
a143036 
Signed-off-by: Evan Lloyd New-Schmidt <evan@new-schmidt.com>
@newsch
Remove coordinates from output
4be39ac 
Signed-off-by: Evan Lloyd New-Schmidt <evan@new-schmidt.com>
@newsch
Refactor simplification
8ae2597 
- Combine expansion steps
- Pull original steps into functions
- Use parent sections for removing specific headers
- Remove head in initial stage

Signed-off-by: Evan Lloyd New-Schmidt <evan@new-schmidt.com>
@newsch
Remove header ids
06f3e63 
Signed-off-by: Evan Lloyd New-Schmidt <evan@new-schmidt.com>
@newsch
newsch merged commit c7fe34f into main Aug 15, 2023
@newsch
newsch deleted the simpler branch August 15, 2023 22:37
@newsch newsch mentioned this pull request Aug 16, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@biodranik biodranik biodranik approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Additional HTML Simplification

2 participants

@newsch @biodranik