Support "html5" type to use html5lib parser

[redapple]

Every now and then we get a bug report about some HTML source not being parsed as a browser would.

There was the idea in Scrapy of adding an "html5" type to switch to an HTML5 compliant parser.
One of these is html5lib that can be used with lxml.

[joaquingx]

I'll work on this 💪

[grahamanderson]

Is there any update on this?
In my project, I can scrape items with beautifulsoup that fail with scrapy.
In my case, it happens in 1/20 pages in my scrapy project.
I hate to waste page data--if I don't have to :)
Or, is there an elegant workaround?

Example Code

import requests
from lxml import etree, html
from bs4 import BeautifulSoup
url = 'https://www.homeadvisor.com/rated.CoventryAdditions.60530954.html'
r = requests.get(url)

# Author Fails
from parsel import Selector
sel = parsel.Selector(text = r.text)
print('Page Title is {}'.format(sel.xpath("//title//text()").get())) # Success
print(sel.xpath('//span[contains(@itemprop,"author")]//text()').get()) # []
print(sel.css('span[itemprop="author"]').get()) []

# Author Works
soup = BeautifulSoup(r.content, 'lxml') # also works with html5lib
print('title is: {}'.format(soup.title.text)) #Success
for author in soup.findAll("span", {"itemprop":"author"}):
    print(author.text) # Success

[Gallaecio]

@grahamanderson You can try and review the pull request at #133

Alternatively, you can use the following workaround in a downloader middleware or in the callbacks of your spider:

from bs4 import BeautifulSoup

# …

response = response.replace(body=str(BeautifulSoup(response.body, "html5lib")))

[grahamanderson]

Thank you @Gallaecio !
I used the scrapy-beautifulsoup code...as middleware
Strangely, I did not have to resort to using html5lib. BeautifulSoup's lxml parser seems a bit more robust than Scrapy/Parsel's LXML parser.

class BeautifulSoupMiddleware(object):
    def __init__(self, crawler):
        super(BeautifulSoupMiddleware, self).__init__()

        self.parser = crawler.settings.get('BEAUTIFULSOUP_PARSER', "html.parser")

    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler)

    def process_response(self, request, response, spider):
        """Overridden process_response would "pipe" response.body through BeautifulSoup."""
        return response.replace(body=str(BeautifulSoup(response.body, self.parser)))

[Gallaecio]

From @whalebot-helmsman:

There is html 5 parser implementation for lxml (https://lxml.de/api/lxml.html.html5parser-pysrc.html )

[aryamanpuri]

Hi all,
I am too late to start for GSoC 2020, found this issue interesting and having good knowledge in web-dev with python and javascript.
Can someone help me with how to get started?

[Gallaecio]

Start by reading up http://gsoc2020.scrapinghub.com/participate and the links at the top to Python and Google resources. Mind that student applications have just started and will close in a couple of weeks.

[aryamanpuri]

So should I start contributing to the project or start making a good proposal ?

[Gallaecio]

You can start with whichever you prefer, but you need to do both before the deadline, proposals from student that have not submitted any patch will not be considered.

If you start with your proposal, and you can manage to isolate a small part of the proposal that you can implement in a week or less, you could implement that as your contribution, and that would speak high of your ability to complete the rest of the project.

[aryamanpuri]

Parsel can extract data from the Html and XML but due to some exceptions in the Html like the use of # in the attributes of tag and having the different technique to visualize the tags in HTML, there is need of html5lib parser.
Do I get it right?
Anything more that can help me?

[Gallaecio]

Make sure you have a look at the issues linked from this thread.

Another benefit of supporting a parser like html5lib, for example, is that the HTML tree that it builds in memory is closer to what you see in a web browser when you use the Inspect feature.