New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Create profile loading mechanism #154
Changes from all commits
b274417
7c87a26
686203f
7e5d568
664e137
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -6,3 +6,6 @@ venv* | |
.hypothesis | ||
docs/_build | ||
*.egg-info | ||
_build | ||
.tox | ||
.coverage |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -285,6 +285,64 @@ You can also override the default by setting the environment variable | |
setting ``HYPOTHESIS_VERBOSITY_LEVEL=verbose`` will run all your tests printing | ||
intermediate results and errors. | ||
|
||
.. _settings_profiles: | ||
|
||
~~~~~~~~~~~~~~~~~ | ||
Settings Profiles | ||
~~~~~~~~~~~~~~~~~ | ||
|
||
Depending on your environment you may want different default settings. | ||
For example: during development you may want to lower the number of examples | ||
to speed up the tests. However, in a CI environment you may want more examples | ||
so you are more likely to find bugs. | ||
|
||
Hypothesis allows you to define different settings profiles. These profiles | ||
can be loaded at any time. | ||
|
||
Loading a profile changes the default settings but will not change the behavior | ||
of tests that explicitly change the settings. | ||
|
||
.. code:: python | ||
|
||
>>> from hypothesis import Settings | ||
>>> Settings.register_profile("ci", Settings(max_examples=1000)) | ||
>>> Settings().max_examples | ||
200 | ||
>>> Settings.load_profile("ci") | ||
>>> Settings().max_examples | ||
1000 | ||
|
||
Instead of loading the profile and overriding the defaults you can retrieve profiles for | ||
specific tests. | ||
|
||
.. code:: python | ||
|
||
>>> with Settings.get_profile("ci"): | ||
... print(Settings().max_examples) | ||
... | ||
1000 | ||
|
||
Optionally, you may define the environment variable to load a profile for you. | ||
This is the suggested pattern for running your tests on CI. | ||
The code below should run in a `conftest.py` or any setup/initialization section of your test suite. | ||
If this variable is not defined the Hypothesis defined defaults will be loaded. | ||
|
||
.. code:: python | ||
|
||
>>> from hypothesis import Settings | ||
>>> Settings.register_profile("ci", Settings(max_examples=1000)) | ||
>>> Settings.register_profile("dev", Settings(max_examples=10)) | ||
>>> Settings.register_profile("debug", Settings(max_examples=10, verbosity=Verbosity.verbose)) | ||
>>> Settings.load_profile(os.getenv(u'HYPOTHESIS_PROFILE', 'default')) | ||
|
||
If you are using the hypothesis pytest plugin. If the profile was registered | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. These two sentences are confusing me - shouldn't this be something like "If you are using the hypothesis pytest plugin and the profile ..."? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Sure, I can edit that tonight. and open a quick PR |
||
while loading in a conftest file you can load it with the command line | ||
option ``--hypothesis-profile`` | ||
|
||
.. code:: bash | ||
|
||
$ py.test tests --hypothesis-profile <profile-name> | ||
|
||
--------------------- | ||
Defining strategies | ||
--------------------- | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -19,6 +19,9 @@ | |
import pytest | ||
|
||
PYTEST_VERSION = tuple(map(int, pytest.__version__.split('.')[:3])) | ||
LOAD_PROFILE_OPTION = '--hypothesis-profile' | ||
|
||
PYTEST_VERSION = tuple(map(int, pytest.__version__.split('.'))) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. That addition seems wrong, as it's already defined above (with the There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yeah, I noticed this right after merging. It's now fixed in master. I think this was a hiccup caused by the last rebase. |
||
if PYTEST_VERSION >= (2, 7, 0): | ||
class StoringReporter(object): | ||
|
||
|
@@ -31,6 +34,19 @@ def __call__(self, msg): | |
print(msg) | ||
self.results.append(msg) | ||
|
||
def pytest_addoption(parser): | ||
parser.addoption( | ||
LOAD_PROFILE_OPTION, | ||
action='store', | ||
help='Load in a registered hypothesis settings profile' | ||
) | ||
|
||
def pytest_configure(config): | ||
from hypothesis import settings | ||
profile = config.getoption(LOAD_PROFILE_OPTION) | ||
if profile: | ||
settings.Settings.load_profile(profile) | ||
|
||
@pytest.mark.hookwrapper | ||
def pytest_pyfunc_call(pyfuncitem): | ||
from hypothesis.reporting import with_reporter | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -24,6 +24,7 @@ | |
from __future__ import division, print_function, absolute_import | ||
|
||
import os | ||
import copy | ||
import inspect | ||
import threading | ||
from collections import namedtuple | ||
|
@@ -139,6 +140,8 @@ class Settings(SettingsMeta('Settings', (object,), {})): | |
|
||
""" | ||
|
||
_profiles = {} | ||
|
||
def __getattr__(self, name): | ||
if name in all_settings: | ||
d = all_settings[name].default | ||
|
@@ -198,6 +201,10 @@ def define_setting(cls, name, description, default, options=None): | |
all_settings[name] = Setting( | ||
name, description.strip(), default, options) | ||
setattr(cls, name, SettingsProperty(name)) | ||
if cls.default: | ||
setattr(cls.default, name, default) | ||
for profile in cls._profiles.values(): | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This doesn't look correct to me. What happens if I do: x = Settings()
define_setting('blah', default=3, ...)
Settings.register_profile('hi', Settings(blah=2))
assert x.blah == 3
assert Settings.get_profile('hi') == 3 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Ok so I added a test
So I make a settings object I define a new setting x gets the default value while any new profiles get the value I set. Seems right to me. Though this PR is the lat thing I do before bed so I would not be surprised if i'm just being dense There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm very confused what's going on here, but this PR is the first thing I do when drinking my coffee so I also would not be surprised if I were being dense. :-) Why is the setattr(profile, name, default) not clobbering this? There must be an attribute lookup happening before the assignment I guess, but relying on that seems very suspect. OTOH with the test there I don't mind too much. The settings system is due for a significant rework and I'm happy to get this merged and do the internals cleanup myself. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. honestly im not 100% sure. You mentioned attribute lookup. When I register/load profiles I perform a copy so that may do it |
||
setattr(profile, name, default) | ||
|
||
def __setattr__(self, name, value): | ||
if name in all_settings: | ||
|
@@ -259,6 +266,48 @@ def __exit__(self, *args, **kwargs): | |
default_context_manager = self.defaults_stack().pop() | ||
return default_context_manager.__exit__(*args, **kwargs) | ||
|
||
@staticmethod | ||
def register_profile(name, settings): | ||
"""registers a collection of values to be used as a settings profile. | ||
These settings can be loaded in by name. Enable different defaults for | ||
different settings. | ||
|
||
- settings is a Settings object | ||
|
||
""" | ||
Settings._profiles[name] = copy.copy(settings) | ||
|
||
@staticmethod | ||
def get_profile(name): | ||
"""Return the profile with the given name. | ||
|
||
- name is a string representing the name of the profile | ||
to load | ||
A InvalidArgument exception will be thrown if the | ||
profile does not exist | ||
|
||
""" | ||
try: | ||
return copy.copy(Settings._profiles[name]) | ||
except KeyError: | ||
raise InvalidArgument( | ||
"Profile '{0}' has not been registered".format( | ||
name | ||
) | ||
) | ||
|
||
@staticmethod | ||
def load_profile(name): | ||
"""Loads in the settings defined in the profile provided If the profile | ||
does not exist an InvalidArgument will be thrown. | ||
|
||
Any setting not defined in the profile will be the library | ||
defined default for that setting | ||
|
||
""" | ||
Settings.default = Settings.get_profile(name) | ||
|
||
|
||
Setting = namedtuple( | ||
u'Setting', (u'name', u'description', u'default', u'options')) | ||
|
||
|
@@ -390,8 +439,6 @@ def by_name(cls, key): | |
return result | ||
raise InvalidArgument(u'No such verbosity level %r' % (key,)) | ||
|
||
Settings.default = Settings() | ||
|
||
Verbosity.quiet = Verbosity(u'quiet', 0) | ||
Verbosity.normal = Verbosity(u'normal', 1) | ||
Verbosity.verbose = Verbosity(u'verbose', 2) | ||
|
@@ -414,3 +461,6 @@ def by_name(cls, key): | |
default=DEFAULT_VERBOSITY, | ||
description=u'Control the verbosity level of Hypothesis messages', | ||
) | ||
|
||
Settings.register_profile('default', Settings()) | ||
Settings.load_profile('default') |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,43 @@ | ||
# coding=utf-8 | ||
|
||
# This file is part of Hypothesis (https://github.com/DRMacIver/hypothesis) | ||
|
||
# Most of this work is copyright (C) 2013-2015 David R. MacIver | ||
# (david@drmaciver.com), but it contains contributions by others. See | ||
# https://github.com/DRMacIver/hypothesis/blob/master/CONTRIBUTING.rst for a | ||
# full list of people who may hold copyright, and consult the git log if you | ||
# need to determine who owns an individual contribution. | ||
|
||
# This Source Code Form is subject to the terms of the Mozilla Public License, | ||
# v. 2.0. If a copy of the MPL was not distributed with this file, You can | ||
# obtain one at http://mozilla.org/MPL/2.0/. | ||
|
||
# END HEADER | ||
|
||
from __future__ import division, print_function, absolute_import | ||
|
||
from hypothesis.extra.pytestplugin import LOAD_PROFILE_OPTION | ||
|
||
pytest_plugins = str('pytester') | ||
|
||
CONFTEST = """ | ||
from hypothesis.settings import Settings | ||
Settings.register_profile("test", Settings(max_examples=1)) | ||
""" | ||
|
||
TESTSUITE = """ | ||
from hypothesis import given | ||
from hypothesis.strategies import integers | ||
from hypothesis.settings import Settings | ||
|
||
def test_this_one_is_ok(): | ||
assert Settings().max_examples == 1 | ||
""" | ||
|
||
|
||
def test_runs_reporting_hook(testdir): | ||
script = testdir.makepyfile(TESTSUITE) | ||
testdir.makeconftest(CONFTEST) | ||
result = testdir.runpytest(script, LOAD_PROFILE_OPTION, 'test') | ||
out = '\n'.join(result.stdout.lines) | ||
assert '1 passed' in out |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The documentation should also mention that there's an argument you can use if you're using the pytest plugin.