Fuel Canary Watchtower

A tool to monitor both the Fuel and Ethereum chains and the bridge activities occuring between the two

Project Layout

├── fuel_watcher: handles a thread that watches the Fuel chain
│   ├── fuel_chain: reads basic data from the Fuel chain
│   ├── fungible_token_contract: handles monitoring events related to the bridge fungible token contracts
├── ethereum_watcher: handles a thread that watches the Ethereum chain
│   ├── ethereum_chain: reads basic data from the Ethereum chain
│   ├── state_contract: handles interacting with and monitoring events from the Fuel chain state contract
│   ├── portal_contract: handles interacting with and monitoring events from the Fuel message portal contract
│   ├── gateway_contract: handles interacting with and monitoring events from the ERC-20 gateway contract
├── ethereum_actions: handles interactions with the Ethereum chain (pausing contracts)
├── alerts: handles logging and pushing out info/alerts
├── config: reads configuration set in the watchtower_config.json file

Config File

An example config file can be found at watchtower_config.json.example. The following options are available for configuration.

fuel_graphql: <fuel chain graphql endpoint>
ethereum_rpc: <ethereum chain rpc endpoint>
ethereum_wallet_key: <optional private key for an ethereum wallet>
state_contract_address: <address of the fuel chain state contract>
portal_contract_address: <address of the fuel message portal contract>
gateway_contract_address: <address of the ERC20 gateway contract>
duplicate_alert_delay: <delay in seconds before pushing the same alert>
fuel_client_watcher: {
  connection_alert: {
    alert_level: <level of alert [None, Info, Warn, Error]>
    alert_action: <(optional) action to take [None, PauseState, PauseGateway, PausePortal, PauseAll]>
  }
  block_production_alert: {
    alert_level: <level of alert [None, Info, Warn, Error]>
    alert_action: <(optional) action to take [None, PauseState, PauseGateway, PausePortal, PauseAll]>
  }
  portal_withdraw_alerts: [{
      alert_level: <level of alert [None, Info, Warn, Error]>
      alert_action: <(optional) action to take [None, PauseState, PauseGateway, PausePortal, PauseAll]>
      time_frame: <window of time to check for threshold>
      amount: <threshold value which triggers the alert>
    }
    <aditional withdraw alert configs>
  ]
  gateway_withdraw_alerts: [{
      alert_level: <level of alert [None, Info, Warn, Error]>
      alert_action: <(optional) action to take [None, PauseState, PauseGateway, PausePortal, PauseAll]>
      token_name: <name of token for reporting purposes>
      token_address: <address of the fuel token to monitor>
      time_frame: <window of time to check for threshold>
      amount: <threshold value which triggers the alert>
    }
    <aditional withdraw alert configs>
  ]
}
ethereum_client_watcher: {
  connection_alert: {
    alert_level: <level of alert [None, Info, Warn, Error]>
    alert_action: <(optional) action to take [None, PauseState, PauseGateway, PausePortal, PauseAll]>
  }
  block_production_alert: {
    alert_level: <level of alert [None, Info, Warn, Error]>
    alert_action: <(optional) action to take [None, PauseState, PauseGateway, PausePortal, PauseAll]>
    max_block_time: <max seconds between blocks which triggers alert>
  }
  account_funds_alert: {
    alert_level: <level of alert [None, Info, Warn, Error]>
    alert_action: <(optional) action to take [None, PauseState, PauseGateway, PausePortal, PauseAll]>
    min_balance: <min balance which will trigger alert>
  }
  invalid_state_commit_alert: {
    alert_level: <level of alert [None, Info, Warn, Error]>
    alert_action: <(optional) action to take [None, PauseState, PauseGateway, PausePortal, PauseAll]>
  }
  portal_deposit_alerts: [{
      alert_level: <level of alert [None, Info, Warn, Error]>
      alert_action: <(optional) action to take [None, PauseState, PauseGateway, PausePortal, PauseAll]>
      time_frame: <window of time to check for threshold>
      amount: <threshold value which triggers the alert>
    }
    <aditional deposit alert configs>
  ]
  gateway_deposit_alerts: [{
      alert_level: <level of alert [None, Info, Warn, Error]>
      alert_action: <(optional) action to take [None, PauseState, PauseGateway, PausePortal, PauseAll]>
      token_name: <name of token for reporting purposes>
      token_address: <address of the ethereum token to monitor>
      time_frame: <window of time to check for threshold>
      amount: <threshold value which triggers the alert>
    }
    <aditional deposit alert configs>
  ]
}

Alerts Module

The alerts module is responsible for pushing alerts through to some monitoring service as well as logging data to a log file. Logging is configured in logging_config.yaml.

TODOs

• Fuel Watcher:
  • implement polling thread
  • Fuel Chain:
    • verify blocks
    • check chain connection
    • check regular block production
    • check eth withdrawals
  • Fungible Token Contract:
    • check token withdrawals
• Ethereum Watcher:
  • implement polling thread
  • Ethereum Chain:
    • check chain connection
    • check regular block production
    • check eth withdrawals
  • Fuel Chain State Contract:
    • check valid state commits
    • implement alert on pause action timeout
    • implement gas escalator on pause action
  • Fuel Message Portal Contract:
    • check eth deposits
    • implement alert on pause action timeout
    • implement gas escalator on pause action
  • ERC20 Gateway deposits:
    • check token withdrawals
    • implement alert on pause action timeout
    • implement gas escalator on pause action
• Config module
• Ethereum Actions module:
  • implement separate thread singleton
  • implement pausing on ethereum contracts
• Alerts module:
  • implement separate thread singleton
  • set a timer on client startup (gives user some time to quickly fix a bad config before everyone gets alerted of an incorrect error)
  • buffer message alerts to avoid duplicates
  • send alerts through a broadcasting service like PagerDuty

Might Want to Add

• We currently only check that committed blocks match what's in the fuel chain. This does not protect us from a bug in the client that might screw up MessageOut receipts and allow for more ETH or tokens to be withdrawn than should be. We might want a setup that keeps a running log of all asset balances that have been approved for withdrawal and then trigger a pause if more are somehow withdrawn than expected. This would require some kind of persistent data store to work efficiently (like the current "alert" concept but with a timing window that spans from the start of the chain to now).