Folder ".hypothesis" appears after "pytest" run but I don't use hypothesis

[buhtz]

I don't use hypothesis. But when I run pytest on my own package the "hypothesis" plugin seems to be active. At the end a folder named .hypothesis appears in my project folder.

A assume that some other pip packages do have hypothesis as an dependency and that is why it exists in my system. Anyway I don't want to have it run on ever use of pytest.

Why does it run? I never configured it or import it in my current project.

[buhtz]

#2106

[Zac-HD]

We're very careful to only create that folder when Hypothesis is used, not just imported, so my guess is that you have a pytest plugin which actually invokes some Hypothesis code.

Can you share a minimal reproducible example, or at least paste in all the output from running pytest? If we can identify the plugin, we can go ask them to fix this.

(in fact one quick way to check would be to pip uninstall -y hypothesis, run pytest, and see where the error comes from!)

[buhtz]

Thanks for your reply.

====================================================== test session starts ======================================================
platform linux -- Python 3.9.2, pytest-7.4.2, pluggy-1.3.0 -- /usr/bin/python3
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase(PosixPath('/home/user/ownCloud/my.work/buhtzology/src/buhtzology/.hypothesis/examples'))
rootdir: /home/user/ownCloud/my.work/buhtzology/src/buhtzology
plugins: hypothesis-6.92.2, pyfakefs-5.2.4, anyio-4.2.0
collected 0 items                                                                                                               
run-last-failure: no previously failed tests, not deselecting items.
Clearing the cache

It might be anyio. I don't know what it is and never used it. I assume that it is not pyfakefs because I use it in several projects and never observed that behavior before.

The docu itself says "AnyIO also comes with its own pytest plugin which also supports asynchronous fixtures. It even works with the popular Hypothesis library."

[jobh]

That sounds like a likely culprit. You might find some additional information in the stacktrace by forcing directory creation to fail: rm -rf .hypothesis; touch .hypothesis; pytest <...>.

[buhtz]

That sounds like a likely culprit. You might find some additional information in the stacktrace by forcing directory creation to fail: rm -rf .hypothesis; touch .hypothesis; pytest <...>.

No stacktrace just a fallback behavior.

/home/user/.local/lib/python3.9/site-packages/hypothesis/_settings.py:74: HypothesisWarning: The database setting is not configured, and the default location is unusable - falling back to an in-memory database for this session.  path=PosixPath('/home/user/ownCloud/my.work/buhtzology/.hypothesis/examples')
  result = ExampleDatabase(not_set)

I opened an Issue at anyio project.

[jobh]

/home/user/.local/lib/python3.9/site-packages/hypothesis/_settings.py:74: HypothesisWarning: The database setting > 

Try pytest -Werror .... But not necessary, let's see what anyio says.

[buhtz]

let's see what anyio says.

Maybe you can jump in there. It seems I lowered my reputation there. 😄

[jobh]

...actually, @Zac-HD, the minimal reproducer:

$ rm -rf .hypothesis; python -c "import hypothesis"; find .hypothesis
.hypothesis
.hypothesis/unicode_data
.hypothesis/unicode_data/14.0.0
.hypothesis/unicode_data/14.0.0/codec-utf-8.json.gz
.hypothesis/unicode_data/14.0.0/charmap.json.gz

[edit] Running with HYPOTHESIS_NO_PLUGINS=1 fixes it, so it's a plugin alright. Just not a pytest plugin, so it's not logged by pytest.

[buhtz]

The anyio maintainers seems to be not happy.
What is the result of the analysis? You can reproduce by importing hypothesis?
But the question is who does import hypothesis...?

[jobh]

@buhtz Don't worry, the anyio was closed as "not our fault/problem" which seems right. Just figuring out where the source of the problem lies. The create-on-import behaviour is a bug and will be fixed.

[jobh]

FWIW: In my case, it seems to be the pydantic plugin that triggers this through st.register_type_strategy(..., st.from_regex(...)), where regex_strategy builds (and caches) unicode categories.

This seems like fairly standard behaviour for a plugin.

[buhtz]

Sorry I don't understand the last comment.
I really don't get it what goes on here. I don't know how pytest plugin system works. I use pytest only because of the nice colorful output. My unit tests are real "unittest" based tests. I refuse to use pytest-like unit tests.

In the end I put this into my pyproject.toml to block this one specific plugin. But I don't know who installed or imported it.

[tool.pytest.ini_options]
addopts = "-p no:hypothesispytest -p no:hypothesis"

[Zac-HD]

Thanks for tracking this down @jobh! Two mitigations to avoid this situation in future:

• Avoid checking strategy.is_empty for LazyStrategys in st.register_type_strategy. While it's a good thing to check, it's better to defer the cost of materializing the strategies in question so that plugins have minimal impact (files, latency, etc) on people not using those features. That's the fix for this particular problem.
• For each Hypothesis plugin we load, check whether it accessed the storage_directory or materialized a LazyStrategy, and emit a deprecation warning if so. That will help downstream authors notice if they're having such impacts, and avoid them in future.

@buhtz, thanks for reporting this and bearing with us as we worked out what combination of interacting libraries were responsible! It sounds like disabling Hypothesis' pytest plugin is working for now, and once we get a fix out you shouldn't even need that.

[jobh]

Sounds good @Zac-HD!

We can even make the is_empty check conditional on the hypothesis import being completed if we want to keep the check for user code. (f.x., setting a flag at the end of hypothesis.__init__ or checking existence of hypothesis.run)

The same flag could be used for the warnings btw; it would improve locality of the warning by warning immediately instead of afterwards.

[jobh]

I'll throw together a PR later