Hyperopts Crash

[gitgitgat-zz]

Hyperopts crashes, even with low number of epochs.

Step 2: Describe your environment

Python 3.7.3
ccxt==1.18.523
Develop
2463f02

Most common crash report:

*   10/100:      0 trades. Avg profit   nan%. Total profit  0.00000000 BTC (   0.00Σ%). Avg duration   nan mins. Objective: 100000.00000
*   11/100:      0 trades. Avg profit   nan%. Total profit  0.00000000 BTC (   0.00Σ%). Avg duration   nan mins. Objective: 100000.00000
*   12/100:      0 trades. Avg profit   nan%. Total profit  0.00000000 BTC (   0.00Σ%). Avg duration   nan mins. Objective: 100000.00000
joblib.externals.loky.process_executor._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/Users/user/freqtrade/.env/lib/python3.7/site-packages/joblib/externals/loky/process_executor.py", line 418, in _process_worker
    r = call_item()
  File "/Users/user/freqtrade/.env/lib/python3.7/site-packages/joblib/externals/loky/process_executor.py", line 272, in __call__
    return self.fn(*self.args, **self.kwargs)
  File "/Users/user/freqtrade/.env/lib/python3.7/site-packages/joblib/_parallel_backends.py", line 567, in __call__
    return self.func(*args, **kwargs)
  File "/Users/user/freqtrade/.env/lib/python3.7/site-packages/joblib/parallel.py", line 225, in __call__
    for func, args, kwargs in self.items]
  File "/Users/user/freqtrade/.env/lib/python3.7/site-packages/joblib/parallel.py", line 225, in <listcomp>
    for func, args, kwargs in self.items]
  File "/Users/user/freqtrade/.env/lib/python3.7/site-packages/joblib/externals/loky/cloudpickle_wrapper.py", line 38, in __call__
    return self._obj(*args, **kwargs)
  File "/Users/user/freqtrade/freqtrade/optimize/hyperopt.py", line 208, in generate_optimizer
    'end_date': max_date,
  File "/Users/nsm/freqtrade/freqtrade/optimize/backtesting.py", line 344, in backtest
    ticker: Dict = self._get_ticker_list(processed)
  File "/Users/user/freqtrade/freqtrade/optimize/backtesting.py", line 228, in _get_ticker_list
    self.advise_buy(pair_data, {'pair': pair}), {'pair': pair})[headers].copy()
  File "/Users/user/freqtrade/user_data/hyperopts/test_hyperopt.py", line 252, in populate_sell_trend
    'sell'] = 1
TypeError: reduce() of empty sequence with no initial value
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "freqtrade/__main__.py", line 15, in <module>
    main.main(sys.argv[1:])
  File "/Users/user/freqtrade/freqtrade/main.py", line 34, in main
    args.func(args)
  File "/Users/user/freqtrade/freqtrade/optimize/hyperopt.py", line 394, in start
    hyperopt.start()
  File "/Users/user/freqtrade/freqtrade/optimize/hyperopt.py", line 325, in start
    f_val = self.run_optimizer_parallel(parallel, asked)
  File "/Users/user/freqtrade/freqtrade/optimize/hyperopt.py", line 263, in run_optimizer_parallel
    wrap_non_picklable_objects(self.generate_optimizer))(v) for v in asked)
  File "/Users/user/freqtrade/.env/lib/python3.7/site-packages/joblib/parallel.py", line 934, in __call__
    self.retrieve()
  File "/Users/user/freqtrade/.env/lib/python3.7/site-packages/joblib/parallel.py", line 833, in retrieve
    self._output.extend(job.get(timeout=self.timeout))
  File "/Users/user/freqtrade/.env/lib/python3.7/site-packages/joblib/_parallel_backends.py", line 521, in wrap_future_result
    return future.result(timeout=timeout)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/concurrent/futures/_base.py", line 425, in result
    return self.__get_result()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
TypeError: reduce() of empty sequence with no initial value

[xmatthias]

Please post the command you are using that cause this.

Also, if you did modify the hyperopts file (which i assume you did - otherwise hyperopt does not make sense), we'll need that too (in private, if you prefer not to share it publicly), since this is VERY likely a problem with the modifications.

[gitgitgat-zz]

this is the command:

python3 freqtrade --config config.json hyperopt --timerange -100 --customhyperopt TestHyperOpts --eps --dmmp -e 100 --print-all

it happens irrespective of --timerange

[hroff-1902]

What is TestHyperOpts? Your custom hyperopt class? I do not see it in the freqtrade codebase.

[gitgitgat-zz]

any private email I could send it to?, preferably not gmail or the like

[hroff-1902]

better ask @xmatthias, I guess...

(why not gmail? o_O)

[hroff-1902]

a PM at the Slack channel can also be an option, as I see...

[gitgitgat-zz]

ok, slack would be alright

[hroff-1902]

I've been working for an anti-virus company for some period of time...

Do you know, how to send dangerous or sensitive content and to avoid scanning the mail by the machine filter or looking at by a human (if you concern about it)? Just send it via e-mail in a password-protected archive with a non-trivial password and send the password via different communication channel (or in a different e-mail, probably to second e-mail address)... Voila...

[gitgitgat-zz]

where could I find a link to the slack channel, or which address is the most suitable?

[hroff-1902]

where could I find a link to the slack channel, or which address is the most suitable?

See invitation link at https://github.com/freqtrade/freqtrade. somewhere above the Requirements section...

[hroff-1902]

@xmatthias the reason of the exception found, I wonder why this bug has not still been noticed...

Will issue a PR for the quick and dirty resolution and some further considerations.

[xmatthias]

#1875 should fix this problem (but you need to include the if condition: check in your hyperopt file to have it applied).