AssertionError in floats(max_value=0, exclude_max=True) when "denormals-are-zero" processor flag is set

[flyte]

From the normal Python shell and ipython, this works:

$ ipython
Python 3.8.0 (default, Feb 25 2021, 22:10:10) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.27.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from hypothesis import strategies as st

In [2]: st.floats(max_value=0, exclude_max=True).example()
Out[2]: -5e-324

But from ./manage.py shell and while actually running Django normally, it fails:

$ ./manage.py shell
Python 3.8.0 (default, Feb 25 2021, 22:10:10) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.27.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from hypothesis import strategies as st

In [2]: st.floats(max_value=0, exclude_max=True).example()
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-2-324381a1006a> in <module>
----> 1 st.floats(max_value=0, exclude_max=True).example()

~/.local/share/virtualenvs/myproject-tDErfRdz/lib/python3.8/site-packages/hypothesis/strategies/_internal/strategies.py in example(self)
    330 
    331         examples: List[Ex] = []
--> 332         example_generating_inner_function()
    333         return random_choice(examples)
    334 

~/.local/share/virtualenvs/myproject-tDErfRdz/lib/python3.8/site-packages/hypothesis/strategies/_internal/strategies.py in example_generating_inner_function()
    318         # tracebacks, and we want users to know that they can ignore it.
    319         @given(self)
--> 320         @settings(
    321             database=None,
    322             max_examples=10,

    [... skipping hidden 1 frame]

~/.local/share/virtualenvs/myproject-tDErfRdz/lib/python3.8/site-packages/hypothesis/core.py in process_arguments_to_given(wrapped_test, arguments, kwargs, given_kwargs, argspec)
    447         search_strategy = WithRunner(search_strategy, selfy)
    448 
--> 449     search_strategy.validate()
    450 
    451     return arguments, kwargs, test_runner, search_strategy

~/.local/share/virtualenvs/myproject-tDErfRdz/lib/python3.8/site-packages/hypothesis/strategies/_internal/strategies.py in validate(self)
    399         try:
    400             self.validate_called = True
--> 401             self.do_validate()
    402             self.is_empty
    403             self.has_reusable_values

~/.local/share/virtualenvs/myproject-tDErfRdz/lib/python3.8/site-packages/hypothesis/strategies/_internal/collections.py in do_validate(self)
     44     def do_validate(self):
     45         for s in self.element_strategies:
---> 46             s.validate()
     47 
     48     def calc_label(self):

~/.local/share/virtualenvs/myproject-tDErfRdz/lib/python3.8/site-packages/hypothesis/strategies/_internal/strategies.py in validate(self)
    399         try:
    400             self.validate_called = True
--> 401             self.do_validate()
    402             self.is_empty
    403             self.has_reusable_values

~/.local/share/virtualenvs/myproject-tDErfRdz/lib/python3.8/site-packages/hypothesis/strategies/_internal/strategies.py in do_validate(self)
    799 
    800     def do_validate(self):
--> 801         self.mapped_strategy.validate()
    802 
    803     def pack(self, x):

~/.local/share/virtualenvs/myproject-tDErfRdz/lib/python3.8/site-packages/hypothesis/strategies/_internal/strategies.py in validate(self)
    399         try:
    400             self.validate_called = True
--> 401             self.do_validate()
    402             self.is_empty
    403             self.has_reusable_values

~/.local/share/virtualenvs/myproject-tDErfRdz/lib/python3.8/site-packages/hypothesis/strategies/_internal/lazy.py in do_validate(self)
    133         w = self.wrapped_strategy
    134         assert isinstance(w, SearchStrategy), f"{self!r} returned non-strategy {w!r}"
--> 135         w.validate()
    136 
    137     def __repr__(self):

~/.local/share/virtualenvs/myproject-tDErfRdz/lib/python3.8/site-packages/hypothesis/strategies/_internal/strategies.py in validate(self)
    399         try:
    400             self.validate_called = True
--> 401             self.do_validate()
    402             self.is_empty
    403             self.has_reusable_values

~/.local/share/virtualenvs/myproject-tDErfRdz/lib/python3.8/site-packages/hypothesis/strategies/_internal/strategies.py in do_validate(self)
    799 
    800     def do_validate(self):
--> 801         self.mapped_strategy.validate()
    802 
    803     def pack(self, x):

~/.local/share/virtualenvs/myproject-tDErfRdz/lib/python3.8/site-packages/hypothesis/strategies/_internal/strategies.py in validate(self)
    399         try:
    400             self.validate_called = True
--> 401             self.do_validate()
    402             self.is_empty
    403             self.has_reusable_values

~/.local/share/virtualenvs/myproject-tDErfRdz/lib/python3.8/site-packages/hypothesis/strategies/_internal/collections.py in do_validate(self)
     44     def do_validate(self):
     45         for s in self.element_strategies:
---> 46             s.validate()
     47 
     48     def calc_label(self):

~/.local/share/virtualenvs/myproject-tDErfRdz/lib/python3.8/site-packages/hypothesis/strategies/_internal/strategies.py in validate(self)
    399         try:
    400             self.validate_called = True
--> 401             self.do_validate()
    402             self.is_empty
    403             self.has_reusable_values

~/.local/share/virtualenvs/myproject-tDErfRdz/lib/python3.8/site-packages/hypothesis/strategies/_internal/lazy.py in do_validate(self)
    131 
    132     def do_validate(self):
--> 133         w = self.wrapped_strategy
    134         assert isinstance(w, SearchStrategy), f"{self!r} returned non-strategy {w!r}"
    135         w.validate()

~/.local/share/virtualenvs/myproject-tDErfRdz/lib/python3.8/site-packages/hypothesis/strategies/_internal/lazy.py in wrapped_strategy(self)
    110             }
    111 
--> 112             base = self.function(*self.__args, **self.__kwargs)
    113             if unwrapped_args == self.__args and unwrapped_kwargs == self.__kwargs:
    114                 self.__wrapped_strategy = base

~/.local/share/virtualenvs/myproject-tDErfRdz/lib/python3.8/site-packages/hypothesis/strategies/_internal/numbers.py in floats(min_value, max_value, allow_nan, allow_infinity, width, exclude_min, exclude_max)
    372             assert is_negative(max_value) and not is_negative(max_arg)
    373             max_value = next_down(max_value, width)
--> 374         assert max_value < max_arg  # type: ignore
    375 
    376     if min_value == -math.inf:

AssertionError: 

I'm definitely using the same installation of hypothesis from the same virtualenv:

$ ipython
Python 3.8.0 (default, Feb 25 2021, 22:10:10) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.27.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from hypothesis import version

In [2]: version
Out[2]: <module 'hypothesis.version' from '/home/flyte/.local/share/virtualenvs/myproject-tDErfRdz/lib/python3.8/site-packages/hypothesis/version.py'>

In [3]: version.__version__
Out[3]: '6.19.0'

$ ./manage.py shell
Python 3.8.0 (default, Feb 25 2021, 22:10:10) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.27.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from hypothesis import version

In [2]: version
Out[2]: <module 'hypothesis.version' from '/home/flyte/.local/share/virtualenvs/myproject-tDErfRdz/lib/python3.8/site-packages/hypothesis/version.py'>

In [3]: version.__version__
Out[3]: '6.19.0'

I'm using Django==3.1.13

[Zac-HD]

Oh no. This is going to be "fun" for whoever tracks it down - likely to include int/float casting and the difference between 0.0 and -0.0, for a start...

[Zac-HD]

I've spent a while trying to reproduce this, without any success - perhaps some of our recent changes for subnormal numbers have fixed it? It's also possible that your problem was due to using a Python interpreter compiled with unsafe optimizations like e.g. -ffast-math.

[ento]

I was seeing the same issue and boiled it down to something about gevent:

# requirements.txt - reproduced with python version 3.9.6
gevent==21.12.0
hypothesis==6.36.0

import gevent
from hypothesis import strategies as st

# raises the same assertion error
print(st.floats(max_value=0, exclude_max=True).example())

One workaround is to specify width=16.

I'm not sure what exactly about gevent would cause this. It might be something about its C extensions, as the change in behavior seems to happen right after gevent.libev.corecext is imported.

I tried building and importing a helloworld module from cython's tutorial and it didn't cause the assertion error.

---

How I got here:

Boiled it down to a difference in the result of this expression from hypothsis.internal.floats.next_up():

$ python -c 'import struct; print(struct.unpack("!d", struct.pack("!q", 1)))'
(5e-324,)
$ python manage.py shell --command 'import struct; print(struct.unpack("!d", struct.pack("!q", 1)))'
(0.0,)

Got a list of imported modules:

python manage.py shell --command 'import sys; [print(m) for m in sys.modules]' > imports.txt

and bisected that by inserting this snippet at the top of the imported module at each bisection point:

import struct, sys                                                                                                                                                                              
print(struct.unpack("!d", struct.pack("!q", 1)), __name__, "*" * 30)                                                                                                 
print(list(sys.modules.keys()))

The unpacked value is non-zero at least until https://github.com/gevent/gevent/blob/21.12.0/src/gevent/_config.py#L699

[Zac-HD]

🙏 Thank you so much for tracking this down! A small repro like this makes all the difference 🙏

[Zac-HD]

Unfortunately it looks like something is setting a flush-to-zero flag in the floating-point module somewhere, which I had hoped was a statically configured thing. See also #3152, #3155, and #3156 - the allow_subnormal=False argument might also fix this?

Either way we should also report this upstream, I'd expect Gevent to want to fix this even if we and Django both work around it.

(Equally unfortunately, I can't reproduce this on my machine 😭)

[ento]

Found a handy package that lets you set the flush-to-zero flag on the fly, and wrote test cases that fail in GitHub Actions. (The branch modifies the main workflow to be able to quickly run those tests in my forked repo. test-win/test-osx jobs also fail because the aforementioned package hasn't been added to a shared requirements.txt file)

The tests show that allow_subnormal=False is helpful but seemingly doesn't fix all cases. I tested allow_subnormal=False with the gevent repro code above too and it still errored.

gevent's latest release is actually compiled with a flag (-fno-fast-math) that should keep the flush-to-zero flag from being set: gevent/gevent#1820

Either it's not taking effect, or something else is setting some flag. Adding import daz; daz.unset_daz() is enough to make the gevent repro code pass, so I'm guessing the flag still being set is probably denormals-are-zero. I'm going to investigate a bit more before opening an issue (if no one else has by then).

I can't reproduce this on my machine

Might have something to do with the CPU in use? I'm on Intel Core i7-10810U.

[Zac-HD]

OK, sounds like this is already fixed with the latest versions of upstream... and all that's left is to decide what we should do about it in Hypothesis.

I think the best available option is to make st.floats(...) raise an error if denormals-are-zero is ever set and allow_subnormal=True. I'd strongly prefer that our internals work the same way regardless of DAZ (and that's required for a reliable database), and weakly prefer not to make this error dependent on the specific values of the endpoints (for UX consistency). Any other opinions?

[honno]

I think the best available option is to make st.floats(...) raise an error if denormals-are-zero is ever set and allow_subnormal=True. I'd strongly prefer that our internals work the same way regardless of DAZ (and that's required for a reliable database), and weakly prefer not to make this error dependent on the specific values of the endpoints (for UX consistency). Any other opinions?

Yep I like that. I had tried setting allow_subnormal via detecting DAZ behavior, but we concluded that was too much—think an error here makes more sense.

[ento]

I'm new to thinking about this problem space; my confidence level isn't high and this comment is more like thinking out loud:

make st.floats(...) raise an error if denormals-are-zero is ever set and allow_subnormal=True

In my case, the AssertionError is coming from Pydantic's Hypothesis plugin, which uses st.floats(...) to generate values for its PositiveFloat and NegativeFloat types, etc. What would be the recommendation for such libraries - libraries that provide types that users can use to annotate their objects while store values of their choosing? Arguments like max_value and exclude_max are something that the library would want to specify, but allow_subnormal would be something left to the library user, since what kind of values is expected is different from user to user. Libraries could provide an initialization API that the user should call, at the expense of the convenience of just adding them as a dependency..? Maybe it can be a re-initialization API that will overwrite previously registered strategies if a user wanted to set a non-default value for allow_subnormal.

I'd strongly prefer that our internals work the same way regardless of DAZ (and that's required for a reliable database)

I first thought of automatically adjusting the default width to the largest value (out of 16, 32, 64) that meets next_up(0.0, width=width) > 0.0. I guess this is at odds with this preference.

---

gevent's latest release is actually compiled with a flag (-fno-fast-math) that should keep the flush-to-zero flag from being set

It appears to be that this flag isn't actually taking effect in terms of disabling DAZ/FTZ flags because of how gcc links in internal code that enables them. I'm going to report upstream later.

[Zac-HD]

[How should downstream libraries handle this?] Maybe it can be a re-initialization API that will overwrite previously registered strategies if a user wanted to set a non-default value for allow_subnormal.

Yeah, I think the best answer here is to encourage end-users to use st.register_type_strategy() directly if they want to customize the results of st.from_type().

FWIW I also expect this to be very rare; the only use-case I've ever had reported was relatively recently - "GPUs don't support subnormals in the default mode" - and that seems unlikely to intersect with downstream library strategies in e.g. Pydantic.

It appears to be that this [-fno-fast-math] flag isn't actually taking effect in terms of disabling DAZ/FTZ flags because of how gcc links in internal code that enables them. I'm going to report upstream later.

Ah, yeah, that would do it! On one level I do understand why people turn to undefined behaviour (performance! seems to work!), but it always ends up being a source of deep sadness...

[Zac-HD]

I don't think we should touch these process-wide flags, but we could reasonably check for them on any use of floats with allow_subnormal=True (the default) and raise a verbose FloatingPointError to explain what's going on.