Bad charmap.pickle.gz file causes user-unfriendly Hypothesis crash

[PiDelport]

Problem

A coworker of mine ran into the following EOFError: Ran out of input crash, with the latest Hypothesis: see the traceback below.

Analysis

It appears that the crash happened while reading .hypothesis/unicodedata/8.0.0/charmap.pickle.gz. Presumably, either the pickle or gzip data were malformed for some reason: deleting the file resolved the problem.

However, it seems like Hypothesis could handle this situation better. Two possible improvements:

1. Wrap the charmap file read with something that produces a friendlier error message, that at least mentions the file name and possible problem?
2. If reading the charmap file fails like this, simply ignore and recreate it?

Thoughts?

Traceback

…/site-packages/hypothesis/core.py:439: in process_arguments_to_given
    search_strategy.validate()
…/site-packages/hypothesis/searchstrategy/strategies.py:210: in validate
    self.do_validate()
…/site-packages/hypothesis/searchstrategy/lazy.py:103: in do_validate
    w = self.wrapped_strategy
…/site-packages/hypothesis/searchstrategy/lazy.py:99: in wrapped_strategy
    for k, v in self.__kwargs.items()})
…/site-packages/hypothesis/strategies.py:386: in tuples
    if arg.is_empty:
…/site-packages/hypothesis/searchstrategy/strategies.py:112: in is_empty
    self.cached_is_empty = self.calc_is_empty()
…/site-packages/hypothesis/searchstrategy/strategies.py:323: in calc_is_empty
    return self.mapped_strategy.is_empty
…/site-packages/hypothesis/searchstrategy/strategies.py:112: in is_empty
    self.cached_is_empty = self.calc_is_empty()
…/site-packages/hypothesis/searchstrategy/lazy.py:90: in calc_is_empty
    return self.wrapped_strategy.is_empty
…/site-packages/hypothesis/searchstrategy/lazy.py:99: in wrapped_strategy
    for k, v in self.__kwargs.items()})
…/site-packages/hypothesis/strategies.py:581: in fixed_dictionaries
    if v.is_empty:
…/site-packages/hypothesis/searchstrategy/strategies.py:112: in is_empty
    self.cached_is_empty = self.calc_is_empty()
…/site-packages/hypothesis/searchstrategy/lazy.py:90: in calc_is_empty
    return self.wrapped_strategy.is_empty
…/site-packages/hypothesis/searchstrategy/lazy.py:99: in wrapped_strategy
    for k, v in self.__kwargs.items()})
…/site-packages/hypothesis/strategies.py:446: in lists
    if elements.is_empty:
…/site-packages/hypothesis/searchstrategy/strategies.py:112: in is_empty
    self.cached_is_empty = self.calc_is_empty()
…/site-packages/hypothesis/searchstrategy/strategies.py:323: in calc_is_empty
    return self.mapped_strategy.is_empty
…/site-packages/hypothesis/searchstrategy/strategies.py:112: in is_empty
    self.cached_is_empty = self.calc_is_empty()
…/site-packages/hypothesis/searchstrategy/lazy.py:90: in calc_is_empty
    return self.wrapped_strategy.is_empty
…/site-packages/hypothesis/searchstrategy/lazy.py:97: in wrapped_strategy
    *[unwrap_strategies(s) for s in self.__args],
…/site-packages/hypothesis/searchstrategy/lazy.py:97: in <listcomp>
    *[unwrap_strategies(s) for s in self.__args],
…/site-packages/hypothesis/searchstrategy/lazy.py:53: in unwrap_strategies
    result = unwrap_strategies(s.wrapped_strategy)
…/site-packages/hypothesis/searchstrategy/lazy.py:99: in wrapped_strategy
    for k, v in self.__kwargs.items()})
…/site-packages/hypothesis/strategies.py:660: in characters
    max_codepoint=max_codepoint)
…/site-packages/hypothesis/searchstrategy/strings.py:45: in __init__
    max_codepoint=max_codepoint,
…/site-packages/hypothesis/internal/charmap.py:155: in query
    catkey = _category_key(exclude_categories, include_categories)
…/site-packages/hypothesis/internal/charmap.py:116: in _category_key
    cs = categories()
…/site-packages/hypothesis/internal/charmap.py:82: in categories
    cm = charmap()
…/site-packages/hypothesis/internal/charmap.py:71: EOFError
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

    def charmap():
        …
            with GzipFile(f, 'rb') as i:
>               _charmap = dict(pickle.loads(i.read()))

EOFError: Ran out of input

[alexwlchan]

I’d lean towards option (2) – exposing details of this file to the user seems like a mistake. It’s created by Hypothesis, we shouldn’t tell the user about it unless things really fall over.

[Zac-HD]

Definitely prefer option 2; having looked at the source we can always handle this with at most a once-per-run performance hit (regenerating the content). I'll offer this as a good first patch for someone at the PyConAU sprints tomorrow; and otherwise get to it soon myself.

[PiDelport]

Thank you!