DeprecationWarning when running pytest on pyramid application

[josherickson]

Hello,

These are warnings that I'm receiving when running pytest on my pyramid application. It appears that they are coming from this package, although it's not clear to me after looking through the source code what it is that's causing the warnings.

Warnings from pytest output:

=============================== warnings summary ===============================
/usr/local/lib/python3.6/site-packages/webob/acceptparse.py:373: DeprecationWarning: invalid escape sequence \`
"""

/usr/local/lib/python3.6/site-packages/webob/acceptparse.py:526: DeprecationWarning: invalid escape sequence \`
"""

/usr/local/lib/python3.6/site-packages/webob/acceptparse.py:577: DeprecationWarning: invalid escape sequence \ 
"""

/usr/local/lib/python3.6/site-packages/webob/acceptparse.py:1488: DeprecationWarning: invalid escape sequence \ 
"""

/usr/local/lib/python3.6/site-packages/webob/acceptparse.py:1594: DeprecationWarning: invalid escape sequence \ 
"""

/usr/local/lib/python3.6/site-packages/webob/acceptparse.py:1703: DeprecationWarning: invalid escape sequence \ 
"""

/usr/local/lib/python3.6/site-packages/webob/acceptparse.py:1855: DeprecationWarning: invalid escape sequence \ 
"""

/usr/local/lib/python3.6/site-packages/webob/acceptparse.py:2456: DeprecationWarning: invalid escape sequence \ 
"""

/usr/local/lib/python3.6/site-packages/webob/acceptparse.py:2560: DeprecationWarning: invalid escape sequence \ 
"""

/usr/local/lib/python3.6/site-packages/webob/acceptparse.py:2664: DeprecationWarning: invalid escape sequence \ 
"""

/usr/local/lib/python3.6/site-packages/webob/acceptparse.py:2833: DeprecationWarning: invalid escape sequence \ 
"""

/usr/local/lib/python3.6/site-packages/webob/acceptparse.py:3034: DeprecationWarning: invalid escape sequence \*
"""

/usr/local/lib/python3.6/site-packages/webob/acceptparse.py:3469: DeprecationWarning: invalid escape sequence \ 
"""

/usr/local/lib/python3.6/site-packages/webob/acceptparse.py:3574: DeprecationWarning: invalid escape sequence \ 
"""

/usr/local/lib/python3.6/site-packages/webob/acceptparse.py:3680: DeprecationWarning: invalid escape sequence \ 
"""

/usr/local/lib/python3.6/site-packages/webob/acceptparse.py:3854: DeprecationWarning: invalid escape sequence \ 
"""

/usr/local/lib/python3.6/site-packages/webob/acceptparse.py:5035: DeprecationWarning: invalid escape sequence \ 
"""

/usr/local/lib/python3.6/site-packages/webob/acceptparse.py:5140: DeprecationWarning: invalid escape sequence \ 
"""

/usr/local/lib/python3.6/site-packages/webob/acceptparse.py:5245: DeprecationWarning: invalid escape sequence \ 
"""

/usr/local/lib/python3.6/site-packages/webob/byterange.py:5: DeprecationWarning: invalid escape sequence \d
_rx_range = re.compile('bytes *= *(\d*) *- *(\d*)', flags=re.I)

/usr/local/lib/python3.6/site-packages/webtest/forms.py:408: DeprecationWarning: invalid escape sequence \s
'''<label\s+(?:[^>]*)for=(?:"|')([a-z0-9_\-]+)(?:"|')(?:[^>]*)>''',

-- Docs: https://docs.pytest.org/en/latest/warnings.html
=============== 1 failed, 1 passed, 21 warnings in 6.84 seconds ================

pytest code:

from unittest import TestCase

from pyramid.paster import get_appsettings
from webtest import TestApp as _TestApp

from app import main


class FunctionalTests(TestCase):
    def setUp(self):
        settings = get_appsettings('pytest.ini', name='main')
        app_ = main({}, **settings)
        self.test_app = _TestApp(app_)

    def tearDown(self):
        self.base_tear_down()

    def test_success(self):
        self.assertTrue(True)

    def test_failure(self):
        self.assertTrue(False)

Is this something that I can fix, or is this an issue with the library?

[mmerickel]

which version of webob?

[josherickson]

which version of webob?

pyramid==1.9.2
pytest==3.91
WebOb==1.8.2
WebTest==2.0.32

By the way, I am not installing and importing WebOb myself. I imagine this occurs through my use of WebTest.

[mmerickel]

Can you try webob 1.8.3 as it was just released this week and report back. I'm not expecting it to fix things but I want to be sure.

[mmerickel]

Below is my test case using the same versions you mentioned. I had to come up with one since yours is not actually runnable. Everything passes here without warnings. What OS and version of Python are you running?

from pyramid.config import Configurator
import unittest
from webtest import TestApp as _TestApp

class FunctionalTests(unittest.TestCase):
    def setUp(self):
        with Configurator() as config:
            config.add_view(lambda r: 'foo', renderer='string')
            app = config.make_wsgi_app()
        self.test_app = _TestApp(app)

    def test_success(self):
        response = self.test_app.get('/')
        self.assertEqual(response.text, 'foo')

❯ env/bin/pytest test_webob_384.py
==================================== test session starts ====================================
platform darwin -- Python 3.6.6, pytest-3.8.2, py-1.7.0, pluggy-0.8.0
rootdir: /Users/michael/work/oss/pyramid, inifile:
collected 1 item

test_webob_384.py .                                                                   [100%]

================================= 1 passed in 0.15 seconds ==================================

[josherickson]

Yep, looks to be the same. This is with WebOb==1.8.3.

=============================== warnings summary ===============================
/usr/local/lib/python3.6/site-packages/webob/acceptparse.py:373: DeprecationWarning: invalid escape sequence \`
"""

/usr/local/lib/python3.6/site-packages/webob/acceptparse.py:526: DeprecationWarning: invalid escape sequence \`
"""

/usr/local/lib/python3.6/site-packages/webob/acceptparse.py:577: DeprecationWarning: invalid escape sequence \ 
"""

/usr/local/lib/python3.6/site-packages/webob/acceptparse.py:1488: DeprecationWarning: invalid escape sequence \ 
"""

/usr/local/lib/python3.6/site-packages/webob/acceptparse.py:1594: DeprecationWarning: invalid escape sequence \ 
"""

/usr/local/lib/python3.6/site-packages/webob/acceptparse.py:1703: DeprecationWarning: invalid escape sequence \ 
"""

/usr/local/lib/python3.6/site-packages/webob/acceptparse.py:1855: DeprecationWarning: invalid escape sequence \ 
"""

/usr/local/lib/python3.6/site-packages/webob/acceptparse.py:2456: DeprecationWarning: invalid escape sequence \ 
"""

/usr/local/lib/python3.6/site-packages/webob/acceptparse.py:2560: DeprecationWarning: invalid escape sequence \ 
"""

/usr/local/lib/python3.6/site-packages/webob/acceptparse.py:2664: DeprecationWarning: invalid escape sequence \ 
"""

/usr/local/lib/python3.6/site-packages/webob/acceptparse.py:2833: DeprecationWarning: invalid escape sequence \ 
"""

/usr/local/lib/python3.6/site-packages/webob/acceptparse.py:3034: DeprecationWarning: invalid escape sequence \*
"""

/usr/local/lib/python3.6/site-packages/webob/acceptparse.py:3469: DeprecationWarning: invalid escape sequence \ 
"""

/usr/local/lib/python3.6/site-packages/webob/acceptparse.py:3574: DeprecationWarning: invalid escape sequence \ 
"""

/usr/local/lib/python3.6/site-packages/webob/acceptparse.py:3680: DeprecationWarning: invalid escape sequence \ 
"""

/usr/local/lib/python3.6/site-packages/webob/acceptparse.py:3854: DeprecationWarning: invalid escape sequence \ 
"""

/usr/local/lib/python3.6/site-packages/webob/acceptparse.py:5035: DeprecationWarning: invalid escape sequence \ 
"""

/usr/local/lib/python3.6/site-packages/webob/acceptparse.py:5140: DeprecationWarning: invalid escape sequence \ 
"""

/usr/local/lib/python3.6/site-packages/webob/acceptparse.py:5245: DeprecationWarning: invalid escape sequence \ 
"""

/usr/local/lib/python3.6/site-packages/webob/byterange.py:5: DeprecationWarning: invalid escape sequence \d
_rx_range = re.compile('bytes *= *(\d*) *- *(\d*)', flags=re.I)

/usr/local/lib/python3.6/site-packages/webtest/forms.py:408: DeprecationWarning: invalid escape sequence \s
'''<label\s+(?:[^>]*)for=(?:"|')([a-z0-9_\-]+)(?:"|')(?:[^>]*)>''',

-- Docs: https://docs.pytest.org/en/latest/warnings.html
=============== 1 failed, 1 passed, 21 warnings in 6.89 seconds ================

[mmerickel]

Does my example above repro the issue for you? Can you edit it to repro the issue?

[mmerickel]

Okay I think we found some invalid backslashes, I'm annoyed that I can't get my box to output the warnings though.

[digitalresistor]

I've cleaned these up in ce914a1

However neither @mmerickel or I are about to reproduce the output you saw, even while running the WebOb test suite.

Would you be so kind as to paste the whole of your pytest output? Specifically I am looking for the lines that look like:

platform darwin -- Python 3.6.6, pytest-3.8.2, py-1.7.0, pluggy-0.7.1
rootdir: /Users/xistence/Projects/Pylons/webob, inifile: setup.cfg
plugins: xdist-1.23.2, forked-0.2, cov-2.6.0

[digitalresistor]

I've been able to reproduce the errors you are seeing (though not easily, and the above fix did fix it). Do you have PYTHONDONTWRITEBYTECODE=1 or -B enabled for your Python interpreter?

How did you install Pyramid/WebOb?

[josherickson]

Hey guys, sorry for the delay on this. Here's a full solution that gives the warnings. Copy each bit of code into the filename I've put at the top. Note that the first file is in an "app" directory, and everything else is in the root directory.

Build and run via Docker using the following commands:
docker build -t warnings-test . && docker run -t warnings-test

@bertjwregeer:
I am not setting PYTHONDONTWRITEBYTECODE=1 or -B that I know of. Originally I was not installing WebOb, I believe it was being installed as part of WebTest. However, now I'm installing everything including WebOb via pip in this example, to make sure I'm getting the specified version of WebOb.

# app/__init__.py
from paste.translogger import TransLogger
from pyramid.config import Configurator


def main(_global_config, **settings):
    config = Configurator(settings=settings)
    config.scan()
    app = config.make_wsgi_app()
    return TransLogger(app, setup_console_handler=False)

# app.ini
[server:main]
use = egg:waitress#main
listen = 0.0.0.0:80
url_scheme = http

[app:main]
use = egg:warnings_test

[loggers]
keys = root

[handlers]
keys = console

[formatters]
keys = generic

[logger_root]
level = INFO
handlers = console

[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = NOTSET
formatter = generic

[formatter_generic]
format = %(asctime)s.%(msecs)03d %(levelname)-8s [%(name)s] %(message)s
datefmt = %H:%M:%S

# Dockerfile
FROM python:3.6-alpine
COPY requirements.txt requirements.txt
RUN pip3 install -r requirements.txt
COPY ./ ./
RUN pip3 install -e .
RUN pip3 freeze
CMD ["pytest", "-c", "test.ini", "-s", "--cov-report=term-missing", "--cov=app", "test.py"]

# requirements.txt
paste==2.0.3
pyramid==1.9.2
pytest==3.9.1
pytest-cov==2.6.0
waitress==1.1.0
WebOb==1.8.3
WebTest==2.0.32

# setup.py
from setuptools import setup, find_packages


setup(
    name='warnings_test',
    packages=find_packages(),
    entry_points={
        'paste.app_factory': [
            'main = app:main',
        ],
    },
)

# test.ini
[pytest]
testpaths = .
python_files = *.py

# test.py
from unittest import TestCase

from pyramid.paster import get_appsettings
from webtest import TestApp as _TestApp

from app import main


class FunctionalTests(TestCase):
    def setUp(self):
        settings = get_appsettings('app.ini', name='main')
        app_ = main({}, **settings)
        self.test_app = _TestApp(app_)

    def tearDown(self):
        pass

    def test_success(self):
        self.assertTrue(True)

    def test_failure(self):
        self.assertTrue(False)

The output of this for me includes:

platform linux -- Python 3.6.6, pytest-3.9.1, py-1.7.0, pluggy-0.8.0
rootdir: /, inifile: test.ini
plugins: cov-2.6.0

and

/usr/local/lib/python3.6/site-packages/webob/acceptparse.py:373: DeprecationWarning: invalid escape sequence \`
  """

/usr/local/lib/python3.6/site-packages/webob/acceptparse.py:526: DeprecationWarning: invalid escape sequence \`
  """

/usr/local/lib/python3.6/site-packages/webob/acceptparse.py:577: DeprecationWarning: invalid escape sequence \ 
  """

/usr/local/lib/python3.6/site-packages/webob/acceptparse.py:1488: DeprecationWarning: invalid escape sequence \ 
  """

/usr/local/lib/python3.6/site-packages/webob/acceptparse.py:1594: DeprecationWarning: invalid escape sequence \ 
  """

/usr/local/lib/python3.6/site-packages/webob/acceptparse.py:1703: DeprecationWarning: invalid escape sequence \ 
  """

/usr/local/lib/python3.6/site-packages/webob/acceptparse.py:1855: DeprecationWarning: invalid escape sequence \ 
  """

/usr/local/lib/python3.6/site-packages/webob/acceptparse.py:2456: DeprecationWarning: invalid escape sequence \ 
  """

/usr/local/lib/python3.6/site-packages/webob/acceptparse.py:2560: DeprecationWarning: invalid escape sequence \ 
  """

/usr/local/lib/python3.6/site-packages/webob/acceptparse.py:2664: DeprecationWarning: invalid escape sequence \ 
  """

/usr/local/lib/python3.6/site-packages/webob/acceptparse.py:2833: DeprecationWarning: invalid escape sequence \ 
  """

/usr/local/lib/python3.6/site-packages/webob/acceptparse.py:3034: DeprecationWarning: invalid escape sequence \*
  """

/usr/local/lib/python3.6/site-packages/webob/acceptparse.py:3469: DeprecationWarning: invalid escape sequence \ 
  """

/usr/local/lib/python3.6/site-packages/webob/acceptparse.py:3574: DeprecationWarning: invalid escape sequence \ 
  """

/usr/local/lib/python3.6/site-packages/webob/acceptparse.py:3680: DeprecationWarning: invalid escape sequence \ 
  """

/usr/local/lib/python3.6/site-packages/webob/acceptparse.py:3854: DeprecationWarning: invalid escape sequence \ 
  """

/usr/local/lib/python3.6/site-packages/webob/acceptparse.py:5035: DeprecationWarning: invalid escape sequence \ 
  """

/usr/local/lib/python3.6/site-packages/webob/acceptparse.py:5140: DeprecationWarning: invalid escape sequence \ 
  """

/usr/local/lib/python3.6/site-packages/webob/acceptparse.py:5245: DeprecationWarning: invalid escape sequence \ 
  """

/usr/local/lib/python3.6/site-packages/webob/byterange.py:5: DeprecationWarning: invalid escape sequence \d
  _rx_range = re.compile('bytes *= *(\d*) *- *(\d*)', flags=re.I)

/usr/local/lib/python3.6/site-packages/webtest/forms.py:408: DeprecationWarning: invalid escape sequence \s
  '''<label\s+(?:[^>]*)for=(?:"|')([a-z0-9_\-]+)(?:"|')(?:[^>]*)>''',

-- Docs: https://docs.pytest.org/en/latest/warnings.html
============================================================================ 1 failed, 1 passed, 21 warnings in 7.39 seconds

[josherickson]

Also, not sure if you guys are on the WebTest team as well, but looks like the last warning there is from WebTest.

/usr/local/lib/python3.6/site-packages/webtest/forms.py:408: DeprecationWarning: invalid escape sequence \s '''<label\s+(?:[^>]*)for=(?:"|')([a-z0-9_\-]+)(?:"|')(?:[^>]*)>'''

[digitalresistor]

@josherickson yeah, the only time those errors are shown is when Python runs the compile step to turn the file into a *.pyc, and then never again.

This is why both @mmerickel and I weren't able to replicate it, as pip should during the installation phase turn the sdist into a wheel, and then install it, that last step I believe will also compile the .py files.

Which is why I was asking if you were turning bytecode compilation/saving off because after the first run you wouldn't see it again. I've fixed the issues in WebOb, and will get them back ported. I am going to try your steps too and see if I can replicate the issue and understand why you are not getting the .py files compiled and thus seeing the error each time you run your tests.

[kiddten]

same here

[digitalresistor]

I've back ported this change to the 1.8-branch and will be releasing a new version shortly.