Source code for the paper "Web2Text: Deep Structured Boilerplate Removal", full paper @ ECIR'18
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Type Name Latest commit message Commit time
Failed to load latest commit information.


Source code for Web2Text: Deep Structured Boilerplate Removal, full paper at ECIR '18


This repository contains

  • Scala code to parse an (X)HTML document into a DOM tree, convert it to a CDOM tree, interpret tree leaves as a sequence of text blocks and extract features for each of these blocks.

  • Python code to train and evaluate unary and pairwise CNNs on top of these features. Inference on the hidden Markov model based on the CNN output potentials can be executed using the provided implementation of the Viterbi algorithm.

  • The CleanEval dataset under src/main/resources/cleaneval/:

    • orig: raw pages
    • clean: reference clean pages
    • aligned: clean content aligned with the corresponding raw page on a per-character basis using the alignment algorithm described in our paper
  • Output from various other webpage cleaners on CleanEval under other_frameworks/output:


  1. Install Scala and SBT. The code was tested with SBT 0.31.

  2. Install Python 3 with Tensorflow and NumPy.


Recipe: extracting text from a web page

  1. Run ch.ethz.dalab.web2text.ExtractPageFeatures through sbt. The arguments are:
    • input html file
    • the desired output base filename (script produces {filename_base}_edge_feature.csv and {filename_base}_block_features.csv)
  2. Use the python script src/main/ with the 'classify' option. The arguments are:
    • python3 classify {filename_base} {labels_out_filename}
  3. Use ch.ethz.dalab.web2text.ApplyLabelsToPage through sbt to produce clean text. Arguments:
    • input html file
    • {labels_out_filename} from step 2
    • output destination text file path


In Scala:

import ch.ethz.dalab.web2text.cdom.CDOM
val cdom = CDOM.fromHTML("""
        <p>Paragraph with an <i>Italic</i> section.</p>

Feature extraction


import ch.ethz.dalab.web2text.features.{FeatureExtractor, PageFeatures}
import ch.ethz.dalab.web2text.features.extractor._

val unaryExtractor = 
    + LeafBlockExtractor
    + AncestorExtractor(NodeBlockExtractor + TagExtractor(mode="node"), 1)
    + AncestorExtractor(NodeBlockExtractor, 2)
    + RootExtractor(NodeBlockExtractor)
    + TagExtractor(mode="leaf")

val pairwiseExtractor = 
    TreeDistanceExtractor + 
    BlockBreakExtractor + 

val extractor = FeatureExtractor(unaryExtractor, pairwiseExtractor)

val features: PageFeatures = extractor(cdom)


Aligning cleaned text with original source

import ch.ethz.dalab.web2text.alignment.Alignment
val reference = "keep this"
val source = "You should keep this text"
val alignment: String = Alignment.alignment(source, reference) 
println(alignment) // □□□□□□□□□□□keep this□□□□□

Extracting features for CleanEval

import ch.ethz.dalab.web2text.utilities.Util
import ch.ethz.dalab.web2text.cleaneval.CleanEval
import ch.ethz.dalab.web2text.output.CsvDatasetWriter

val data = Util.time{ CleanEval.dataset(fe) }

// Write block_features.csv and edge_features.csv
// Format of a row: page id, groundtruth label (1/0), features ...
CsvDatasetWriter.write(data, "./src/main/python/data")

// Print the names of the exported features in order
println("# Block features")
println("# Edge features")

Training the CNNs

Code related to the CNNs lives in the src/main/python directory.

To train the CNNs:

  1. Set the CHECKPOINT_DIR variable in
  2. Make sure the files block_features.csv and edge_features.csv are in the src/main/python/data directory. Use the example from the previous section for this.
  3. Convert the CSV files to .npy with data/
  4. Train the unary CNN with python3 train_unary.
  5. Train the pairwise CNN with python3 train_edge.

Evaluating the CNN

To evaluate the CNN:

  1. Set the CHECKPOINT_DIR variable in to point to a directory with trained weights. We provide trained weights based on the cleaneval split and a custom web2text split (with more training data.)
  2. Run python3 test_structured to test performance on the CleanEval test set.

The performance of other networks is computed in Scala:

import ch.ethz.dalab.web2text.Main