Support pyparsing v3 (AttributeError downcaseTokens)

[mborsetti]

There appears to be a use of a discontinued feature from the pyparsing library:

Traceback (most recent call last):
[...]
  File "/usr/local/lib/python3.9/dist-packages/httplib2/__init__.py", line 52, in <module>
    from . import auth
  File "/usr/local/lib/python3.9/dist-packages/httplib2/auth.py", line 20, in <module>
    auth_param_name = token.copy().setName("auth-param-name").addParseAction(pp.downcaseTokens)
AttributeError: module 'pyparsing' has no attribute 'downcaseTokens'

[seanlinehan]

This is a breaking issue – the requirements.txt file should be updated to:

pyparsing>=2.4.2,<3.0.0a0 # TODO include v3 after dropping Python2 support

[temoto]

@mborsetti please starting from clean virtualenv, show pip install instructions to get to the problem, show output of pip freeze |fgrep pypa in the end.

@seanlinehan current requirements already excludes all 3.x. pip install -U httplib2 installs pyparsing==2.4.7 for me and it still has downcaseTokens.

[ahopkins]

Anyone here have a winning combo of pyparsing and httplib2 to get past this right now and get CI unstuck?

---

UPDATE: I pinned to httplib2<0.20 and pyparsing==2.4.7. All seems to be working again.

[stoksc]

So, I work on a project that was also depends packaging, which pins >=2.0.2, and I ended up with a line like httplib2 0.20.1 has requirement pyparsing<3,>=2.4.2, but you'll have pyparsing 3.0.1 which is incompatible. (after it pulled in 3.0.1). I manually pinned pyparsing<3.0.0 and my CI is green. I'm by no means an expert in python packaging; I'm sort of surprised this wasn't handled by whatever algorithm resolves the dependencies. But it works 🤷 (edit: after reading, https://pip.pypa.io/en/latest/user_guide/ indicates updating pip to > 20.2 would've also worked for me)

[temoto]

solution

pip install 'httplib2>=0.20.2'

cause

Old pip dependency resolver allowed installation of conflicting requirements. Installing httplib2 (with requirements pyparsing<3) would not downgrade existing pyparsing>=3.

pypa/pip#6969
pypa/pip#988

pip 20.3 has been released, and it has the new resolver by default!

So upgrade pip might also help.

pip install 'pip>=20.3'

[xpxaxsxi]

It seems that pyparsing 2.1.8 (from Aug 2016) had the downcaseToken-funcionality that is currently used in httplib2 (source: CHANGES). Lots of thing have changed since in pyparsing module.

auth.py line 20

[Schroedingberg]

For the long run, could replacing pp.downcaseTokens with pp.pyparsing_common.downcase_tokens be a possible solution? That fixed it in my fresh virtualenv, it seems a bit too easy though, so I might be missing something.

[faddiekh]

The current version of pyparsing doesn't support pp.downcaseTokens methond according to the changelogs of newer versino of pyparsing

. upcaseTokens and downcaseTokens - convert to using pyparsing_common.upcaseTokens and downcaseTokens

[faddiekh]

how can I push that fix?

[mborsetti]

@mborsetti please starting from clean virtualenv, show pip install instructions to get to the problem, show output of pip freeze |fgrep pypa in the end.

Irrelevant because in my virtualenv pyparsing is also needed by matplotlib and packaging.

@seanlinehan current requirements already excludes all 3.x. pip install -U httplib2 installs pyparsing==2.4.7 for me and it still has downcaseTokens.

Also irrelevant, because it turns out that httplib2 is not at the center of the universe, and it too needs to play nice with other more updated libraries (as common sense would dictate).

how can I push that fix?

Thanks @Fahadsaadullahkhan: supporting the current version of the library is the only way to permanently resolve conflicts, as well as ensuring that any eventual security bug present in an obsolete version of a library is not carried into this one.

[temoto]

For the long run, could replacing pp.downcaseTokens with pp.pyparsing_common.downcase_tokens be a possible solution? That fixed it in my fresh virtualenv, it seems a bit too easy though, so I might be missing something.

It fixes import, but parser doesn't work.

Please try following:

git clone https://github.com/httplib2/httplib2 tmp207
cd tmp207
git apply <<END
diff --git a/python3/httplib2/auth.py b/python3/httplib2/auth.py
index 84b5831..9fe69f2 100644
--- a/python3/httplib2/auth.py
+++ b/python3/httplib2/auth.py
@@ -5,6 +5,11 @@ import pyparsing as pp

 from .error import *

+try:
+    downcaseTokens = pp.common.downcaseTokens
+except AttributeError:
+    downcaseTokens = pp.downcaseTokens
+
 UNQUOTE_PAIRS = re.compile(r"\\(.)")
 unquote = lambda s, l, t: UNQUOTE_PAIRS.sub(r"\1", t[0][1:-1])

@@ -17,7 +22,7 @@ token68 = pp.Combine(pp.Word("-._~+/" + pp.nums + pp.alphas) + pp.Optional(pp.Wo
 )

 quoted_string = pp.dblQuotedString.copy().setName("quoted-string").setParseAction(unquote)
-auth_param_name = token.copy().setName("auth-param-name").addParseAction(pp.downcaseTokens)
+auth_param_name = token.copy().setName("auth-param-name").addParseAction(downcaseTokens)
 auth_param = auth_param_name + pp.Suppress("=") + (quoted_string | token)
 params = pp.Dict(pp.delimitedList(pp.Group(auth_param)))
END
python3 -m venv venv
venv/bin/pip install -e . -r requirements-test.txt
venv/bin/pip install 'pyparsing>=3'
venv/bin/pytest -sv tests/test_auth.py

[mborsetti]

For the long run, could replacing pp.downcaseTokens with pp.pyparsing_common.downcase_tokens be a possible solution? That fixed it in my fresh virtualenv, it seems a bit too easy though, so I might be missing something.

It fixes import, but parser doesn't work.

Please try following:

git clone https://github.com/httplib2/httplib2 tmp207
cd tmp207
git apply <<END
diff --git a/python3/httplib2/auth.py b/python3/httplib2/auth.py
index 84b5831..9fe69f2 100644
--- a/python3/httplib2/auth.py
+++ b/python3/httplib2/auth.py
@@ -5,6 +5,11 @@ import pyparsing as pp

 from .error import *

+try:
+    downcaseTokens = pp.common.downcaseTokens
+except AttributeError:
+    downcaseTokens = pp.downcaseTokens
+
 UNQUOTE_PAIRS = re.compile(r"\\(.)")
 unquote = lambda s, l, t: UNQUOTE_PAIRS.sub(r"\1", t[0][1:-1])

@@ -17,7 +22,7 @@ token68 = pp.Combine(pp.Word("-._~+/" + pp.nums + pp.alphas) + pp.Optional(pp.Wo
 )

 quoted_string = pp.dblQuotedString.copy().setName("quoted-string").setParseAction(unquote)
-auth_param_name = token.copy().setName("auth-param-name").addParseAction(pp.downcaseTokens)
+auth_param_name = token.copy().setName("auth-param-name").addParseAction(downcaseTokens)
 auth_param = auth_param_name + pp.Suppress("=") + (quoted_string | token)
 params = pp.Dict(pp.delimitedList(pp.Group(auth_param)))
END
python3 -m venv venv
venv/bin/pip install -e . -r requirements-test.txt
venv/bin/pip install 'pyparsing>=3'
venv/bin/pytest -sv tests/test_auth.py

@temoto Thanks; this seems to work.

[temoto]

It definitely doesn't work. Subscribe to recent pyparsing issue.

…

On Sat, Oct 30, 2021, 12:13 Mike Borsetti ***@***.***> wrote: For the long run, could replacing pp.downcaseTokens with pp.pyparsing_common.downcase_tokens be a possible solution? That fixed it in my fresh virtualenv, it seems a bit too easy though, so I might be missing something. It fixes import, but parser doesn't work. Please try following: git clone https://github.com/httplib2/httplib2 tmp207cd tmp207 git apply <<ENDdiff --git a/python3/httplib2/auth.py b/python3/httplib2/auth.pyindex 84b5831..9fe69f2 100644--- a/python3/httplib2/auth.py+++ b/python3/httplib2/auth.py@@ -5,6 +5,11 @@ import pyparsing as pp from .error import *+try:+ downcaseTokens = pp.common.downcaseTokens+except AttributeError:+ downcaseTokens = pp.downcaseTokens+ UNQUOTE_PAIRS = re.compile(r"\\(.)") unquote = lambda s, l, t: UNQUOTE_PAIRS.sub(r"\1", t[0][1:-1])@@ -17,7 +22,7 @@ token68 = pp.Combine(pp.Word("-._~+/" + pp.nums + pp.alphas) + pp.Optional(pp.Wo ) quoted_string = pp.dblQuotedString.copy().setName("quoted-string").setParseAction(unquote)-auth_param_name = token.copy().setName("auth-param-name").addParseAction(pp.downcaseTokens)+auth_param_name = token.copy().setName("auth-param-name").addParseAction(downcaseTokens) auth_param = auth_param_name + pp.Suppress("=") + (quoted_string | token) params = pp.Dict(pp.delimitedList(pp.Group(auth_param)))END python3 -m venv venv venv/bin/pip install -e . -r requirements-test.txt venv/bin/pip install 'pyparsing>=3' venv/bin/pytest -sv tests/test_auth.py @temoto <https://github.com/temoto> Thanks; this seems to work. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#207 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAGTMJDONSRBFHQKT74U53UJOZM3ANCNFSM5GVGUVCA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[temoto]

Fixed in httplib2>=0.20.2