Feat / candle feed from Coinbase Advanced Trade v2.1

[MementoRC]

Before submitting this PR, please make sure:

• Your code builds clean without any errors or warnings
• You are using approved title ("feat/", "fix/", "docs/", "refactor/")

A description of the changes proposed in the pull request:
Candle feed from Coinbase Advanced Trade

Tests performed by the developer:
ran tests in select coinbase spot_candles

Tips for QA testing:

[david-hummingbot]

Hey @MementoRC, just a reminder this PR and also #6888 is pointing to master instead of development.

[MementoRC]

@david-hummingbot thank you. corrected

[MementoRC]

Locally, I get:

hummingbot/data_feed/candles_feed/coinbase_advanced_trade_spot_candles/coinbase_advanced_trade_spot_candles.py     190     24     56      7  85.77%

[MementoRC]

@david-hummingbot The test_coinbase_advanced_trade_spot_candles.py was not run in the checks, is there something I am missing?

[notion-workspace]

Feat / candle feed from Coinbase Advanced Trade v2.1 #6889

[notion-workspace]

Feat / candle feed from Coinbase Advanced Trade v2.1 #6889

[notion-workspace]

Feat / candle feed from Coinbase Advanced Trade v2.1 #6889

[rapcmia]

PR update:

• When running the script for candles_example, we are getting this error

  2024-05-01 11:32:58,413 - 29518 - hummingbot.core.utils.async_utils - ERROR - Unhandled error in background task: The connector coinbase_advanced_trade is not available. Please select another one.
  Traceback (most recent call last):
    File "/Users/rapcomia/github/hummingbot/6889/hummingbot/core/utils/async_utils.py", line 9, in safe_wrapper
      return await c
    File "/Users/rapcomia/github/hummingbot/6889/hummingbot/client/command/start_command.py", line 125, in start_check
      self._initialize_strategy(self.strategy_name)
    File "/Users/rapcomia/github/hummingbot/6889/hummingbot/client/command/start_command.py", line 280, in _initialize_strategy
      self.start_script_strategy()
    File "/Users/rapcomia/github/hummingbot/6889/hummingbot/client/command/start_command.py", line 191, in start_script_strategy
      script_strategy, config = self.load_script_class()
    File "/Users/rapcomia/github/hummingbot/6889/hummingbot/client/command/start_command.py", line 213, in load_script_class
      script_module = importlib.import_module(f".{script_name}", package=settings.SCRIPT_STRATEGIES_MODULE)
    File "/Users/rapcomia/miniconda3/envs/hummingbot/lib/python3.10/importlib/__init__.py", line 126, in import_module
      return _bootstrap._gcd_import(name[level:], package, level)
    File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
    File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
    File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
    File "<frozen importlib._bootstrap_external>", line 883, in exec_module
    File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
    File "/Users/rapcomia/github/hummingbot/6889/scripts/candles1.py", line 11, in <module>
      class CandlesExample(ScriptStrategyBase):
    File "/Users/rapcomia/github/hummingbot/6889/scripts/candles1.py", line 28, in CandlesExample
      eth_1m_candles = CandlesFactory.get_candle(CandlesConfig(connector=exc, trading_pair="ETH-USDT", interval="1m", max_records=700))
    File "/Users/rapcomia/github/hummingbot/6889/hummingbot/data_feed/candles_feed/candles_factory.py", line 73, in get_candle
      raise UnsupportedConnectorException(candles_config.connector)
  hummingbot.data_feed.candles_feed.candles_factory.UnsupportedConnectorException: The connector coinbase_advanced_trade is not available. Please select another one.
  

  • The connector candles is not yet added to candles_factory.py

[rapcmia]

• Setup a candles_example.py script for coinbase_advanced_trade
• Some data on OHLC matched however every interval has duplicated data
  
  
  • Observed on this 1m interval has been stuck and not updating for 4mins
  • This is the same for 15mins interval running for 21mins
  • On some instances we are getting this error

    2024-05-02 21:43:16,043 - 75524 - hummingbot.connector.exchange.coinbase_advanced_trade.coinbase_advanced_trade_exchange.CoinbaseAdvancedTradeExchange - NETWORK - Error getting server time.
    Traceback (most recent call last):
      File "/Users/rapcomia/github/hummingbot/6889/hummingbot/connector/time_synchronizer.py", line 65, in update_server_time_offset_with_time_provider
        server_time_ms: float = await time_provider
      File "/Users/rapcomia/github/hummingbot/6889/hummingbot/connector/exchange/coinbase_advanced_trade/coinbase_advanced_trade_web_utils.py", line 126, in get_current_server_time_ms
        server_time_s = await get_current_server_time_s(throttler=throttler, domain=domain)
      File "/Users/rapcomia/github/hummingbot/6889/hummingbot/connector/exchange/coinbase_advanced_trade/coinbase_advanced_trade_web_utils.py", line 105, in get_current_server_time_s
        response: Dict = await rest_assistant.execute_request(
      File "/Users/rapcomia/github/hummingbot/6889/hummingbot/core/web_assistant/rest_assistant.py", line 47, in execute_request
        response = await self.execute_request_and_get_response(
      File "/Users/rapcomia/github/hummingbot/6889/hummingbot/core/web_assistant/rest_assistant.py", line 100, in execute_request_and_get_response
        raise IOError(f"Error executing request {method.name} {url}. HTTP status is {response.status}. "
    OSError: Error executing request GET https://api.coinbase.com/api/v3/brokerage/time. HTTP status is 429. Error: go/sg/d2464495-7852-4454-a5c4-fa0e207c15ec
    

  • There are also instances of No data collected when stop then start the script again
    

[MementoRC]

@rapcmia The 429 error could be a temporary issue, the candles uses the time_synchronizer, which itself is controlled by the rate limit of the /time endpoint - note that it is not a rate limit error and could come from intermediate server when the bot send the http request (maybe a cloud server this go/sg/... is odd)

The candle feed only has the 5mn websocket feed on Coinbase, all other intervals are done with a custom REST call, I'd need a bit of time to get back into what was implemented, the repeating entries could be a mismatch between strat/end of each looped request

[MementoRC]

@rapcmia I think I found a bug: for each REST request the end_time was incorrectly set

[rapcmia]

PR update on commit c0fa5b3b429331089eba5cb8c7e4e52d3428bb41

• Test download_candles script ✅
  • Setup 3 markets and 1m,15m and 1hr interval
  • All downloaded on csv in ./data/
• Test candles_example script
  • Confirmed duplicate candles feeds data have been fixed ✅
  • All fetched candle feed from the connector matched to exchange’s OHLC ✅
  • However after 18mins running time, there have been no changes on candles feed since its started
    

[rapcmia]

However after 18mins running time, there have been no changes on candles feed since its started

• PR update to latest commit, ran tests today and check after 2hrs but same behavior occurred, intervals 1m, 5m and 1hr still showing the same data when started the script candles_example 🙇🏼

[MementoRC]

@rapcmia ok, only the websocket changes (the 5m interval). The REST requests go through for the non-WS, but they do not update on the status page. It must be a issue with how the dataframe is updated by the REST listener. I am debugging

[MementoRC]

@rapcmia Silly mistake, I tried to re-use the historical fill method, but it is shielded by the self.ready() and thus never get the latest candles. I separated the REST call and now it seems to be fixed. Thank you