[Question]: BeautifulSoup

[twaddington]

Checklist

• I'm asking a question
• I've looked through the README and FAQ for similar questions
• I've searched the bugtracker for similar questions including closed ones

Question

I recently started digging through the youtube-dl code and notice that regular expressions are used all over the place. I'm wondering if the developers have considered migrating to an HTML parser like beautifulsoup instead? This would be a more readable and idiomatic way of extracting data from video pages.

[dstftw]

Regular expressions are used intentionally. Code using HTML parsers will break on any minor change in page layout.

[twaddington]

Regular expressions are used intentionally. — @dstftw

That's great that it's an intentional decision. I'm not advocating a change. Was just wondering if it'd been discussed before.

Code using HTML parsers will break on any minor change in page layout. — @dstftw

I don't mean to be blunt, but this isn't accurate. For example, you can use a parser to query the DOM for all anchor tags with a command like soup.a. The whole idea is that even if the HTML structure changes, the query still works as expected.

[dstftw]

It's all good until it comes to some more realistic and complicated scenarios of extraction, like extraction of something laying arbitrary levels of nesting deep involving several alternative paths and so on. In such cases parser-based code will most likely grow and branch resulting in actually less readable code. Also it won't work for HTML embedded in some JavaScript. Also it may not work in case of malformed HTML. Also it won't work for non-HTML. And lots of other potential corner cases that will require special treatment.
Keeping this in mind I see zero point in switching to HTML parsers.

[twaddington]

I have definitely seen some parsers choke on sites with squirrelly markup. Thanks for offering your opinion. I definitely didn't mean to come in and tell you how to maintain your own project. No reason to change what's already working well for you ;)