Merge pull request #2146 from freqtrade/download_module

Download module

docs/backtesting.md

@@ -3,9 +3,43 @@
 This page explains how to validate your strategy performance by using
 Backtesting.
 
+## Getting data for backtesting and hyperopt
+
+To download backtesting data (candles / OHLCV) and hyperoptimization using the `freqtrade download-data` command.
+
+If no additional parameter is specified, freqtrade will download data for `"1m"` and `"5m"` timeframes.
+Exchange and pairs will come from `config.json` (if specified using `-c/--config`). Otherwise `--exchange` becomes mandatory.
+
+Alternatively, a `pairs.json` file can be used.
+
+If you are using Binance for example:
+
+- create a directory `user_data/data/binance` and copy `pairs.json` in that directory.
+- update the `pairs.json` to contain the currency pairs you are interested in.
+
+```bash
+mkdir -p user_data/data/binance
+cp freqtrade/tests/testdata/pairs.json user_data/data/binance
+```
+
+Then run:
+
+```bash
+freqtrade download-data --exchange binance
+```
+
+This will download ticker data for all the currency pairs you defined in `pairs.json`.
+
+- To use a different directory than the exchange specific default, use `--datadir user_data/data/some_directory`.
+- To change the exchange used to download the tickers, please use a different configuration file (you'll probably need to adjust ratelimits etc.)
+- To use `pairs.json` from some other directory, use `--pairs-file some_other_dir/pairs.json`.
+- To download ticker data for only 10 days, use `--days 10` (defaults to 30 days).
+- Use `--timeframes` to specify which tickers to download. Default is `--timeframes 1m 5m` which will download 1-minute and 5-minute tickers.
+- To use exchange, timeframe and list of pairs as defined in your configuration file, use the `-c/--config` option. With this, the script uses the whitelist defined in the config as the list of currency pairs to download data for and does not require the pairs.json file. You can combine `-c/--config` with most other options.
+
 ## Test your strategy with Backtesting
 
-Now you have good Buy and Sell strategies, you want to test it against
+Now you have good Buy and Sell strategies and some historic data, you want to test it against
 real data. This is what we call
 [backtesting](https://en.wikipedia.org/wiki/Backtesting).
 
@@ -109,37 +143,6 @@ The full timerange specification:
 - Use tickframes between POSIX timestamps 1527595200 1527618600:
                                                 `--timerange=1527595200-1527618600`
 
-#### Downloading new set of ticker data
-
-To download new set of backtesting ticker data, you can use a download script.
-
-If you are using Binance for example:
-
-- create a directory `user_data/data/binance` and copy `pairs.json` in that directory.
-- update the `pairs.json` to contain the currency pairs you are interested in.
-
-```bash
-mkdir -p user_data/data/binance
-cp freqtrade/tests/testdata/pairs.json user_data/data/binance
-```
-
-Then run:
-
-```bash
-python scripts/download_backtest_data.py --exchange binance
-```
-
-This will download ticker data for all the currency pairs you defined in `pairs.json`.
-
-- To use a different directory than the exchange specific default, use `--datadir user_data/data/some_directory`.
-- To change the exchange used to download the tickers, use `--exchange`. Default is `bittrex`.
-- To use `pairs.json` from some other directory, use `--pairs-file some_other_dir/pairs.json`.
-- To download ticker data for only 10 days, use `--days 10`.
-- Use `--timeframes` to specify which tickers to download. Default is `--timeframes 1m 5m` which will download 1-minute and 5-minute tickers.
-- To use exchange, timeframe and list of pairs as defined in your configuration file, use the `-c/--config` option. With this, the script uses the whitelist defined in the config as the list of currency pairs to download data for and does not require the pairs.json file. You can combine `-c/--config` with other options.
-
-For help about backtesting usage, please refer to [Backtesting commands](#backtesting-commands).
-
 ## Understand the backtesting result
 
 The most important in the backtesting is to understand the result.

docs/bot-usage.md

@@ -184,19 +184,11 @@ optional arguments:
                         result.json)
 ```
 
-### How to use **--refresh-pairs-cached** parameter?
+### Getting historic data for backtesting
 
-The first time your run Backtesting, it will take the pairs you have
-set in your config file and download data from the Exchange.
-
-If for any reason you want to update your data set, you use
-`--refresh-pairs-cached` to force Backtesting to update the data it has.
-
-!!! Note
-    Use it only if you want to update your data set. You will not be able to come back to the previous version.
-
-To test your strategy with latest data, we recommend continuing using
-the parameter `-l` or `--live`.
+The first time your run Backtesting, you will need to download some historic data first.
+This can be accomplished by using `freqtrade download-data`.  
+Check the corresponding [help page section](backtesting.md#Getting-data-for-backtesting-and-hyperopt) for more details
 
 ## Hyperopt commands
 

freqtrade/configuration/arguments.py

@@ -30,7 +30,7 @@
 
 ARGS_LIST_EXCHANGES = ["print_one_column"]
 
-ARGS_DOWNLOADER = ARGS_COMMON + ["pairs", "pairs_file", "days", "exchange", "timeframes", "erase"]
+ARGS_DOWNLOAD_DATA = ["pairs", "pairs_file", "days", "exchange", "timeframes", "erase"]
 
 ARGS_PLOT_DATAFRAME = (ARGS_COMMON + ARGS_STRATEGY +
                        ["pairs", "indicators1", "indicators2", "plot_limit", "db_url",
@@ -40,6 +40,8 @@
 ARGS_PLOT_PROFIT = (ARGS_COMMON + ARGS_STRATEGY +
                     ["pairs", "timerange", "export", "exportfilename", "db_url", "trade_source"])
 
+NO_CONF_REQURIED = ["start_download_data"]
+
 
 class Arguments(object):
     """
@@ -75,7 +77,10 @@ def _parse_args(self) -> argparse.Namespace:
 
         # Workaround issue in argparse with action='append' and default value
         # (see https://bugs.python.org/issue16399)
-        if not self._no_default_config and parsed_arg.config is None:
+        # Allow no-config for certain commands (like downloading / plotting)
+        if (not self._no_default_config and parsed_arg.config is None
+                and not (hasattr(parsed_arg, 'func')
+                         and parsed_arg.func.__name__ in NO_CONF_REQURIED)):
             parsed_arg.config = [constants.DEFAULT_CONFIG]
 
         return parsed_arg
@@ -93,7 +98,7 @@ def _build_subcommands(self) -> None:
         :return: None
         """
         from freqtrade.optimize import start_backtesting, start_hyperopt, start_edge
-        from freqtrade.utils import start_list_exchanges
+        from freqtrade.utils import start_download_data, start_list_exchanges
 
         subparsers = self.parser.add_subparsers(dest='subparser')
 
@@ -119,3 +124,11 @@ def _build_subcommands(self) -> None:
         )
         list_exchanges_cmd.set_defaults(func=start_list_exchanges)
         self._build_args(optionlist=ARGS_LIST_EXCHANGES, parser=list_exchanges_cmd)
+
+        # Add download-data subcommand
+        download_data_cmd = subparsers.add_parser(
+            'download-data',
+            help='Download backtesting data.'
+        )
+        download_data_cmd.set_defaults(func=start_download_data)
+        self._build_args(optionlist=ARGS_DOWNLOAD_DATA, parser=download_data_cmd)

freqtrade/configuration/cli_options.py

@@ -254,7 +254,8 @@ def __init__(self, *args, **kwargs):
     # Script options
     "pairs": Arg(
         '-p', '--pairs',
-        help='Show profits for only these pairs. Pairs are comma-separated.',
+        help='Show profits for only these pairs. Pairs are space-separated.',
+        nargs='+',
     ),
     # Download data
     "pairs_file": Arg(
@@ -276,9 +277,10 @@ def __init__(self, *args, **kwargs):
     "timeframes": Arg(
         '-t', '--timeframes',
         help=f'Specify which tickers to download. Space-separated list. '
-        f'Default: `{constants.DEFAULT_DOWNLOAD_TICKER_INTERVALS}`.',
+        f'Default: `1m 5m`.',
         choices=['1m', '3m', '5m', '15m', '30m', '1h', '2h', '4h',
                  '6h', '8h', '12h', '1d', '3d', '1w'],
+        default=['1m', '5m'],
         nargs='+',
     ),
     "erase": Arg(

freqtrade/configuration/configuration.py

@@ -4,16 +4,17 @@
 import logging
 import warnings
 from argparse import Namespace
+from pathlib import Path
 from typing import Any, Callable, Dict, List, Optional
 
-from freqtrade import constants
+from freqtrade import constants, OperationalException
 from freqtrade.configuration.check_exchange import check_exchange
 from freqtrade.configuration.create_datadir import create_datadir
 from freqtrade.configuration.config_validation import (validate_config_schema,
                                                        validate_config_consistency)
 from freqtrade.configuration.load_config import load_config_file
 from freqtrade.loggers import setup_logging
-from freqtrade.misc import deep_merge_dicts
+from freqtrade.misc import deep_merge_dicts, json_load
 from freqtrade.state import RunMode
 
 logger = logging.getLogger(__name__)
@@ -53,6 +54,9 @@ def from_files(files: List[str]) -> Dict[str, Any]:
         # Keep this method as staticmethod, so it can be used from interactive environments
         config: Dict[str, Any] = {}
 
+        if not files:
+            return constants.MINIMAL_CONFIG.copy()
+
         # We expect here a list of config filenames
         for path in files:
             logger.info(f'Using config: {path} ...')
@@ -86,6 +90,11 @@ def load_config(self) -> Dict[str, Any]:
 
         self._process_runmode(config)
 
+        # Check if the exchange set by the user is supported
+        check_exchange(config, config.get('experimental', {}).get('block_bad_exchanges', True))
+
+        self._resolve_pairs_list(config)
+
         validate_config_consistency(config)
 
         return config
@@ -148,9 +157,6 @@ def _process_common_options(self, config: Dict[str, Any]) -> None:
         if 'sd_notify' in self.args and self.args.sd_notify:
             config['internals'].update({'sd_notify': True})
 
-        # Check if the exchange set by the user is supported
-        check_exchange(config, config.get('experimental', {}).get('block_bad_exchanges', True))
-
     def _process_datadir_options(self, config: Dict[str, Any]) -> None:
         """
         Extract information for sys.argv and load datadir configuration:
@@ -277,6 +283,19 @@ def _process_plot_options(self, config: Dict[str, Any]) -> None:
         self._args_to_config(config, argname='trade_source',
                              logstring='Using trades from: {}')
 
+        self._args_to_config(config, argname='erase',
+                             logstring='Erase detected. Deleting existing data.')
+
+        self._args_to_config(config, argname='timeframes',
+                             logstring='timeframes --timeframes: {}')
+
+        self._args_to_config(config, argname='days',
+                             logstring='Detected --days: {}')
+
+        if "exchange" in self.args and self.args.exchange:
+            config['exchange']['name'] = self.args.exchange
+            logger.info(f"Using exchange {config['exchange']['name']}")
+
     def _process_runmode(self, config: Dict[str, Any]) -> None:
 
         if not self.runmode:
@@ -307,3 +326,38 @@ def _args_to_config(self, config: Dict[str, Any], argname: str,
                 logger.info(logstring.format(config[argname]))
             if deprecated_msg:
                 warnings.warn(f"DEPRECATED: {deprecated_msg}", DeprecationWarning)
+
+    def _resolve_pairs_list(self, config: Dict[str, Any]) -> None:
+        """
+        Helper for download script.
+        Takes first found:
+        * -p (pairs argument)
+        * --pairs-file
+        * whitelist from config
+        """
+
+        if "pairs" in config:
+            return
+
+        if "pairs_file" in self.args and self.args.pairs_file:
+            pairs_file = Path(self.args.pairs_file)
+            logger.info(f'Reading pairs file "{pairs_file}".')
+            # Download pairs from the pairs file if no config is specified
+            # or if pairs file is specified explicitely
+            if not pairs_file.exists():
+                raise OperationalException(f'No pairs file found with path "{pairs_file}".')
+
+            config['pairs'] = json_load(pairs_file)
+
+            config['pairs'].sort()
+            return
+
+        if "config" in self.args and self.args.config:
+            logger.info("Using pairlist from configuration.")
+            config['pairs'] = config.get('exchange', {}).get('pair_whitelist')
+        else:
+            # Fall back to /dl_path/pairs.json
+            pairs_file = Path(config['datadir']) / "pairs.json"
+            if pairs_file.exists():
+                config['pairs'] = json_load(pairs_file)
+            config['pairs'].sort()

freqtrade/constants.py

@@ -22,7 +22,6 @@
 ORDERTIF_POSSIBILITIES = ['gtc', 'fok', 'ioc']
 AVAILABLE_PAIRLISTS = ['StaticPairList', 'VolumePairList']
 DRY_RUN_WALLET = 999.9
-DEFAULT_DOWNLOAD_TICKER_INTERVALS = '1m 5m'
 
 TICKER_INTERVALS = [
     '1m', '3m', '5m', '15m', '30m',
@@ -38,6 +37,20 @@
     "BTC", "XBT", "ETH", "XRP", "LTC", "BCH", "USDT"
 ]
 
+MINIMAL_CONFIG = {
+    'stake_currency': '',
+    'dry_run': True,
+    'exchange': {
+        'name': '',
+        'key': '',
+        'secret': '',
+        'pair_whitelist': [],
+        'ccxt_async_config': {
+            'enableRateLimit': True,
+        }
+    }
+}
+
 # Required json-schema for user specified config
 CONF_SCHEMA = {
     'type': 'object',

freqtrade/data/history.py

@@ -129,7 +129,7 @@ def load_pair_history(pair: str,
     else:
         logger.warning(
             f'No history data for pair: "{pair}", interval: {ticker_interval}. '
-            'Use --refresh-pairs-cached option or download_backtest_data.py '
+            'Use --refresh-pairs-cached option or `freqtrade download-data` '
             'script to download the data'
         )
         return None

freqtrade/plot/plotting.py

@@ -37,7 +37,7 @@ def init_plotscript(config):
 
     strategy = StrategyResolver(config).strategy
     if "pairs" in config:
-        pairs = config["pairs"].split(',')
+        pairs = config["pairs"]
     else:
         pairs = config["exchange"]["pair_whitelist"]
 

freqtrade/tests/data/test_history.py

@@ -74,7 +74,7 @@ def test_load_data_7min_ticker(mocker, caplog, default_conf) -> None:
     assert ld is None
     assert log_has(
         'No history data for pair: "UNITTEST/BTC", interval: 7m. '
-        'Use --refresh-pairs-cached option or download_backtest_data.py '
+        'Use --refresh-pairs-cached option or `freqtrade download-data` '
         'script to download the data', caplog
     )
 
@@ -109,7 +109,7 @@ def test_load_data_with_new_pair_1min(ticker_history_list, mocker, caplog, defau
     assert os.path.isfile(file) is False
     assert log_has(
         'No history data for pair: "MEME/BTC", interval: 1m. '
-        'Use --refresh-pairs-cached option or download_backtest_data.py '
+        'Use --refresh-pairs-cached option or `freqtrade download-data` '
         'script to download the data', caplog
     )