URL builder utils

[beardypig]

As part of the solution for the issues raised in #1519 by @amurzeau, I have started adding some URL utils that can be used in plugins to better manipulate URLs.

As well as a useful method for iterating through HTML tags, using regex in a brute force kind of way - like most of the plugins do.

update_qsd and url_concat are the two URL manipulation methods. url_concat will join together URL parts to make a URL, and update_qsd can be used to add, remove or update query string parameters in a URL.

I updated a couple of plugins as examples.

[codecov]

Codecov Report

Merging #1675 into master will increase coverage by 0.09%.
The diff coverage is 85.36%.

@@            Coverage Diff             @@
##           master    #1675      +/-   ##
==========================================
+ Coverage   51.26%   51.36%   +0.09%     
==========================================
  Files         242      243       +1     
  Lines       14358    14383      +25     
==========================================
+ Hits         7361     7388      +27     
+ Misses       6997     6995       -2

[gravyboat]

This fell through and I didn't ever merge it, sorry @beardypig. Do you want to rebase and we can get this in?

[back-to]

well, I tried to rebase it
but the tests won't pass on Python 3.7 / 3.8-dev

=================================== FAILURES ===================================
___________________ TestPluginUtil.test_itertags_multi_attrs ___________________
self = <tests.test_plugin_utils.TestPluginUtil testMethod=test_itertags_multi_attrs>
    def test_itertags_multi_attrs(self):
        metas = list(itertags(self.test_html, "meta"))
        self.assertTrue(len(metas), 3)
        self.assertTrue(all(meta.tag == "meta" for meta in metas))
    
>       self.assertEqual(metas[0].text, None)
E       AssertionError: 'Title' != None
tests/test_plugin_utils.py:49: AssertionError
________________________ TestPluginUtil.test_no_end_tag ________________________
self = <tests.test_plugin_utils.TestPluginUtil testMethod=test_no_end_tag>
    def test_no_end_tag(self):
        links = list(itertags(self.test_html, "link"))
        self.assertTrue(len(links), 1)
        self.assertEqual(links[0].tag, "link")
>       self.assertEqual(links[0].text, None)
E       AssertionError: '' != None
tests/test_plugin_utils.py:68: AssertionError

[beardypig]

I cannot work out why it's failing on 3.7 right now ... When I have some time I will look at it, unless @back-to is able to work it out :)

[back-to]

I know why it fails

re.finditer
Changed in version 3.7: Non-empty matches can now start just after a previous empty match.

https://docs.python.org/3/library/re.html#re.finditer
python/cpython#4471
https://bugs.python.org/issue25054

https://docs.python.org/3/whatsnew/3.7.html#changes-in-the-python-api

---

here is something similar https://bugs.python.org/issue33585
might be the new expected behavior

If you don't want to find an empty string, change you patter so that it will not match an empty string: ".+".

that has maybe something todo with it,
but not sure whats the best way to fix it.

import re
test_html = """
<title>Title</title>
<meta property="og:type" content= "website" />
<meta property="og:url" content="http://test.se/"/>
<meta property="og:site_name" content="Test" />
<script src="https://test.se/test.js"></script>
<link rel="stylesheet" type="text/css" href="https://test.se/test.css">
<script>Tester.ready(function () {
alert("Hello, world!"); });</script>
<a
href="http://test.se/foo">bar</a>
"""
tag_re = re.compile(r'''(?=<(?P<tag>[a-zA-Z]+)(?P<attr>.*?)(?P<end>/)?>(?:(?P<inner>.*?)</\s*(?P=tag)\s*>)?)''', re.MULTILINE | re.DOTALL)
print([m.groups() for m in tag_re.finditer(test_html)])

Python 3.7

[('title', '', None, 'Title'),
('meta', ' property="og:type" content= "website" ', '/', 'Title'),
('meta', ' property="og:url" content="http://test.se/"', '/', None),
('meta', ' property="og:site_name" content="Test" ', '/', None),
('script', ' src="https://test.se/test.js"', '/', ''),
('link', ' rel="stylesheet" type="text/css" href="https://test.se/test.css"', None, ''),
('script', '', None, 'Tester.ready(function () {\nalert("Hello, world!"); });'),
('a', '\nhref="http://test.se/foo"', None, 'bar')]

Python 3.6

[('title', '', None, 'Title'),
('meta', ' property="og:type" content= "website" ', '/', None),
('meta', ' property="og:url" content="http://test.se/"', '/', None),
('meta', ' property="og:site_name" content="Test" ', '/', None),
('script', ' src="https://test.se/test.js"', None, ''),
('link', ' rel="stylesheet" type="text/css" href="https://test.se/test.css"', None, None),
('script', '', None, 'Tester.ready(function () {\nalert("Hello, world!"); });'),
('a', '\nhref="http://test.se/foo"', None, 'bar')]

[beardypig]

I must be missing something - I cannot see why the inner group would be "Title" for the second match. It seems like the inner group is not being reset, and gets the previous value for some reason (same for when it is '').

[back-to]

@beardypig

with removing the (?= ... ) part, the tests will pass on 3.7

diff --git a/src/streamlink/plugin/api/utils.py b/src/streamlink/plugin/api/utils.py
index f6f3f12..00445e2 100644
--- a/src/streamlink/plugin/api/utils.py
+++ b/src/streamlink/plugin/api/utils.py
@@ -7,9 +7,9 @@ from ...utils import parse_qsd as parse_query, parse_json, parse_xml
 __all__ = ["parse_json", "parse_xml", "parse_query"]
 
 
-tag_re = re.compile('''(?=<(?P<tag>[a-zA-Z]+)(?P<attr>.*?)(?P<end>/)?>(?:(?P<inner>.*?)</\s*(?P=tag)\s*>)?)''',
+tag_re = re.compile(r'''<(?P<tag>[a-zA-Z]+)(?P<attr>.*?)(?P<end>/)?>(?:(?P<inner>.*?)</\s*(?P=tag)\s*>)?''',
                     re.MULTILINE | re.DOTALL)
-attr_re = re.compile('''\s*(?P<key>[\w-]+)\s*(?:=\s*(?P<quote>["']?)(?P<value>.*?)(?P=quote)\s*)?''')
+attr_re = re.compile(r'''\s*(?P<key>[\w-]+)\s*(?:=\s*(?P<quote>["']?)(?P<value>.*?)(?P=quote)\s*)?''')
 Tag = namedtuple("Tag", "tag attributes text")
 
 
@@ -26,4 +26,3 @@ def itertags(html, tag):
         if match.group("tag") == tag:
             attrs = dict((a.group("key").lower(), a.group("value")) for a in attr_re.finditer(match.group("attr")))
             yield Tag(match.group("tag"), attrs, match.group("inner"))
-

---

platform linux -- Python 3.7.0, pytest-3.6.3, py-1.5.4, pluggy-0.6.0
rootdir: /run/media/ka/N/github/back-to/streamlink, inifile:
plugins: requests-mock-1.5.2, cov-2.5.1
collected 5 items                                                                                                                                            

tests/test_plugin_utils.py .....

[beardypig]

Not sure that the positive look ahead is even required, might have been left over from a previous iter_tags implementation (I had a few goes) :)

[back-to]

http can be removed, it got added with a rebase.

[beardypig]

True that, forgot to clean up after I rebased. I saw it (using PyCharm to do the rebase), and i was going to tidy it up but I forgot :)

[back-to]

@beardypig

itertags is actually broken now, because the test html code is missing the <html> tag,
which will work for the tests but not for real websites.

it can be tested with the gardenersworld plugin as it is broken with the current changes.

<> must be ignored for inner tags or there will be only one tag html

here is a posible fix

back-to@ab7d6bf
https://travis-ci.org/back-to/streamlink/builds/410256162

[beardypig]

@back-to I updated the test html with your changes, and added a test for inner/outer tags - to maintain the existing behaviour.

I think this might be a bug in Python. I have created a bug report, so we can see what they say.

[back-to]

well, this might take awhile.

Maybe the tests should be ignored on py37 and py38 for now,
so url_concat and update_qsd can be used.

As the Issue was merged in #1693 and it is not really related to this PR,
it was only discovered because of the additional tests.

---

also from streamlink.plugin.api import http got added again with the rebase 🐘

---

https://github.com/beardypig/streamlink/blob/e449aa326a1cece9a8dc4755c413caa179d915f5/tests/test_utils.py#L100-L111

test_url_equal should be removed,
when you moved it to test_utils_url.py it was not removed in test_utils.py.

[beardypig]

🤦‍♂️ The rebase for this one got complex :)

I'll add a skipTest for Python 3.7/3.8.

[beardypig]

I updated the tests so those that were failing allow failure on 3.7+. With a note to monitor bpo-34294.

[beardypig]

@gravyboat, I think this is OK to merge with the exception that it might be wonky on Python 3.7.

[gravyboat]

@beardypig Sounds good, changes look good as well. We can always open another issue depending on what occurs.

[beardypig]

@gravyboat I opened an issue to remind us :)