Accelerate rvest::html_table

[georgevbsantiago]

I currently use the XML package only because of the XML::readHTMLTable function. The xml2 package does not have a function to read tables in the HTML files, correct? I've already tried using the rvest :: html_table function, but readHTMLTable is 10x faster and produces a cleaner data table.

In the example below, the rvest::html_table function creates one more column; included the table footer and handles the lines with NA. Already XML::readTable captures only the header and the body of the table.
However, the biggest "problem" for me is the issue of execution speed.

tab_example.html.gz

library(magrittr)
library(XML)
library(xml2)

# HTML - Table example
html <- "tab_example.html"


# Like XML
get_tab_XML <- html %>% 
        XML::htmlParse(encoding = "UTF-8") %>%
        XML::readHTMLTable(stringsAsFactors = FALSE, which = 2)

# Like xml2
get_tab_xml2 <- html %>% 
        xml2::read_html() %>%
        rvest::html_node("#tabelaResultado") %>% 
        rvest::html_table(fill = TRUE)

Unit: seconds
 expr       min        lq      mean    median        uq       max neval
  XML  3.816312  3.955153  4.173987  4.093994  4.352824  4.611654     3
 xml2 33.720705 34.495118 35.144829 35.269531 35.856891 36.444251     3

[hadley]

Reprex with latest rvest, which is a little faster, but still not great:

path <- "~/Downloads/tab_example.html"
xml1 <- XML::htmlParse(path, encoding = "UTF-8")
xml2 <- xml2::read_html(path, encoding = "UTF-8")

bench::mark(
  XML = XML::readHTMLTable(xml1, stringsAsFactors = FALSE, which = 2),
  rvest = rvest::html_table(rvest::html_node(xml2, "#tabelaResultado")),
  check = FALSE
)
#> Warning: Some expressions had a GC in every iteration; so filtering is disabled.
#> # A tibble: 2 x 6
#>   expression      min   median `itr/sec` mem_alloc `gc/sec`
#>   <bch:expr> <bch:tm> <bch:tm>     <dbl> <bch:byt>    <dbl>
#> 1 XML            1.8s     1.8s    0.555     4.88MB    15.0 
#> 2 rvest           16s      16s    0.0623    8.29MB     2.68

^{Created on 2020-12-19 by the reprex package (v0.3.0.9001)}

[georgevbsantiago]

thank you very much for this improvement in rvest::html_table

[hadley]

Looking at the profile, it looks like only 10% of the time is spent in rvest; the rest is just accessing all the nodes. But it's rather suspicious that so much time is spent in xml_ns().

[hadley]

Ok, that was easy 😄

path <- "~/Downloads/tab_example.html"
xml1 <- XML::htmlParse(path, encoding = "UTF-8")
xml2 <- xml2::read_html(path, encoding = "UTF-8")

bench::mark(
  XML = XML::readHTMLTable(xml1, stringsAsFactors = FALSE, which = 2),
  rvest = rvest::html_table(rvest::html_node(xml2, "#tabelaResultado")),
  check = FALSE
)
#> Warning: Some expressions had a GC in every iteration; so filtering is disabled.
#> # A tibble: 2 x 6
#>   expression      min   median `itr/sec` mem_alloc `gc/sec`
#>   <bch:expr> <bch:tm> <bch:tm>     <dbl> <bch:byt>    <dbl>
#> 1 XML           1.98s    1.98s     0.506    4.88MB     14.7
#> 2 rvest         1.41s    1.41s     0.708    7.88MB     19.8

^{Created on 2020-12-19 by the reprex package (v0.3.0.9001)}

[georgevbsantiago]

Sensational! Amazing!! Unbelievable !!!

You are extraordinary!!!