feat: Add Celestia blobs support to Optimism batches fetcher

[varasev]

Closes #10198, #9982, #8402.

Motivation

This PR adds Celestia blobs handling to Indexer.Fetcher.Optimism.TxnBatch module and saving blob's metadata to the database (for EIP-4844 blobs the metadata is also saved).

Also:

• new API endpoints were added (see the Changelog below).
• Optimism configuration was reduced (most of env variables were removed in favor of SystemConfig contract). See Upgrading section below).

Changelog

New or extended API routes

• /api/v2/optimism/batches (and /api/v2/optimism/batches/count) in addition to the existing /api/v2/optimism/txn-batches. The new endpoint outputs a list of batches with internal Blockscout IDs. Example:

  {
    "items": [
      {
        "internal_id": 2798,
        "l1_timestamp": "2024-01-02T20:11:00.000000Z",
        "l1_tx_hashes": [
          "0xa2255a896d703da9ad9ad98ef1ed2161e3f96f72be539cddc532f39419e3f4ee"
        ],
        "l2_block_start": 616880,
        "l2_block_end": 617053,
        "tx_count": 0
      },
      ...
      {
        "internal_id": 2746,
        "l1_timestamp": "2024-01-02T15:21:12.000000Z",
        "l1_tx_hashes": [
          "0x39b09ec2f0fc6c1f784cd18a6e7c845ada22c8fdcd108118c282c0aa6b75d44f"
        ],
        "l2_block_start": 608210,
        "l2_block_end": 608371,
        "tx_count": 0
      }
    ],
    "next_page_params": {
      "id": 2746,
      "items_count": 50
    }
  }
  

• /api/v2/optimism/batches/da/celestia/:height/:commitment outputs L1 batch info by Celestia blob's height and commitment. Requests and response examples (for Optimism Celestia Raspberry):

  http://localhost:4000/api/v2/optimism/batches/da/celestia/760960/0xf1a51990b5a358a2376e85648b489138ca38533e2b86e0283d41ceeebcf058ea
  

  or

  http://localhost:4000/api/v2/optimism/batches/da/celestia/760961/0x3834d3a92ede97db07defc291c848b6085389c236f88c52b67a933271f316fee
  

  requests will respond with

  {
    "batch_data_container": "in_celestia",
    "blobs": [
      {
        "commitment": "0xf1a51990b5a358a2376e85648b489138ca38533e2b86e0283d41ceeebcf058ea",
        "height": 760960,
        "l1_timestamp": "2023-12-20T10:17:12.000000Z",
        "l1_transaction_hash": "0xf41211e966ec23032dde713d1f775ae5cb07dc5e15951281e6844d74cc02a930",
        "namespace": "0x00000000000000000000000000000000000000000008e5f679bf7116cb"
      },
      {
        "commitment": "0x3834d3a92ede97db07defc291c848b6085389c236f88c52b67a933271f316fee",
        "height": 760961,
        "l1_timestamp": "2023-12-20T10:17:24.000000Z",
        "l1_transaction_hash": "0x9abc0df13890e8c0818b448b15056ecd96368dc2b4f625c1232285e05e5b3826",
        "namespace": "0x00000000000000000000000000000000000000000008e5f679bf7116cb"
      }
    ],
    "internal_id": 5,
    "l1_timestamp": "2023-12-20T10:17:24.000000Z",
    "l1_tx_hashes": [
      "0xf41211e966ec23032dde713d1f775ae5cb07dc5e15951281e6844d74cc02a930",
      "0x9abc0df13890e8c0818b448b15056ecd96368dc2b4f625c1232285e05e5b3826"
    ],
    "l2_block_start": 29996,
    "l2_block_end": 33082,
    "tx_count": 1
  }
  

  • The blobs array represents the list of L1 transactions with the corresponding blob metadata (commitment and height), sorted chronologically.
  • The internal_id field contains the internal Blockscout ID of the batch.
  • The l1_timestamp contains the timestamp of the last L1 transaction in the list.
  • The l1_tx_hashes list can have more than one L1 tx hash (the batch consists of).

• /api/v2/optimism/batches/:internal_id. The output is the same as above for /api/v2/optimism/batches/da/celestia/:height/:commitment, but the batch is requested by its internal ID.

• /api/v2/transactions/optimism-batch/:batch_number outputs a list of L2 transactions related to the specified Optimism batch (by its internal ID). Example of the output:

  {
    "items": [
      {
        "timestamp": "2024-01-04T09:00:24.000000Z",
        "fee": {
          "type": "actual",
          "value": "945824354861"
        },
        "gas_limit": "20000000",
        "block": 683364,
        "status": "ok",
        "method": null,
        "confirmations": 7223670,
        "type": 2,
        "exchange_rate": "3443.61",
        "to": {
          "ens_domain_name": null,
          "hash": "0x2E2D82649acd78e1799195cd8A8EE8aDEcdc3b1b",
          "implementation_address": null,
          "implementation_name": null,
          "implementations": [],
          "is_contract": false,
          "is_verified": false,
          "metadata": null,
          "name": null,
          "private_tags": [],
          "public_tags": [],
          "watchlist_names": []
        },
        "tx_burnt_fee": "3042150",
        "max_fee_per_gas": "1000050",
        "result": "success",
        "hash": "0xfc461d501e860250fe684238034d962d9f86ce643e9186d81fc6c90bf8bce644",
        "gas_price": "1000050",
        "priority_fee": "60843000000",
        "base_fee_per_gas": "50",
        "from": {
          "ens_domain_name": null,
          "hash": "0x7b5159E9bFd7C81b8fB5f7b4FEB1eD4C245Dd588",
          "implementation_address": null,
          "implementation_name": null,
          "implementations": [],
          "is_contract": false,
          "is_verified": false,
          "metadata": null,
          "name": null,
          "private_tags": [],
          "public_tags": [],
          "watchlist_names": []
        },
        "token_transfers": null,
        "tx_types": [],
        "gas_used": "60843",
        "created_contract": null,
        "position": 1,
        "nonce": 37551,
        "has_error_in_internal_txs": null,
        "actions": [],
        "decoded_input": null,
        "token_transfers_overflow": null,
        "raw_input": "0x7b06424000000000000000000000000000000000000000000000000000000000000000c000000000000000000000000000000000000000000000000000060e1af61258c000000000000000000000000000000000000000000000000000000000659673b400000000000000000000000000000000000000000000000000000000659673d200000000000000000000000000000000000000000000000000000000198cac440000000000000000000000000000000000000000000000000000000000000064000000000000000000000000000000000000000000000000000000000000000533303a3135000000000000000000000000000000000000000000000000000000",
        "value": "0",
        "max_priority_fee_per_gas": "1000050",
        "revert_reason": null,
        "confirmation_duration": [
          0,
          2000
        ],
        "tx_tag": null
      },
      ...
    ],
    "next_page_params": {
      "block_number": 683319,
      "index": 1,
      "items_count": 50
    }
  }
  

• /api/v2/blocks/:block_hash_or_number API route was extended to show L1 batch info (where the specified L2 block relates to). Example:

    "optimism": {
      "batch_data_container": "in_celestia",
      "blobs": [
        {
          "commitment": "0xf1a51990b5a358a2376e85648b489138ca38533e2b86e0283d41ceeebcf058ea",
          "height": 760960,
          "l1_timestamp": "2023-12-20T10:17:12.000000Z",
          "l1_transaction_hash": "0xf41211e966ec23032dde713d1f775ae5cb07dc5e15951281e6844d74cc02a930",
          "namespace": "0x00000000000000000000000000000000000000000008e5f679bf7116cb"
        },
        {
          "commitment": "0x3834d3a92ede97db07defc291c848b6085389c236f88c52b67a933271f316fee",
          "height": 760961,
          "l1_timestamp": "2023-12-20T10:17:24.000000Z",
          "l1_transaction_hash": "0x9abc0df13890e8c0818b448b15056ecd96368dc2b4f625c1232285e05e5b3826",
          "namespace": "0x00000000000000000000000000000000000000000008e5f679bf7116cb"
        }
      ],
      "internal_id": 5,
      "l1_timestamp": "2023-12-20T10:17:24.000000Z",
      "l1_tx_hashes": [
        "0xf41211e966ec23032dde713d1f775ae5cb07dc5e15951281e6844d74cc02a930",
        "0x9abc0df13890e8c0818b448b15056ecd96368dc2b4f625c1232285e05e5b3826"
      ]
    }
  

  • blobs array contains the blobs (with metadata) related to the L1 batch, sorted chronologically. In case of EIP-4844 blobs the batch_data_container field will be in_blob4844. In case of Celestia blobs the field will be in_celestia. If there are no blobs, the field will be equal to in_calldata.
  • internal_id designates the internal Blockscout ID of the batch.
  • l1_tx_hashes list can have more than one L1 tx hash (the batch consists of).
  • l1_timestamp contains the timestamp of the last L1 transaction in the list.

• /api/v2/optimism/txn-batches/:l2_block_range_start/:l2_block_range_end allows to limit displaying items by the specified L2 block range. There is no /api/v2/optimism/txn-batches/:l2_block_range_start/:l2_block_range_end/count handle, so UI page should hide the total number of items when using the explicit range. l2_block_range_start must be less than or equal to l2_block_range_end.

New or extended UI pages

The new pages in UI (based on the above API endpoints) should have the following columns and fields:

Batches (instead of Txn batches)

API endpoints for this page:

• /api/v2/optimism/batches
• /api/v2/optimism/batches/count

This should be a table with the following columns:

• Internal ID (should be a link to the Batch details page described below)
• Age
• L1 txn hash (can have more than one hash for the same batch, maybe comma-separated)
• L2 block count
• Txn count (should be a link to the Transactions tab on the Batch details page)

Batch details

API endpoints for this page:

• /api/v2/optimism/batches/:internal_id
• /api/v2/optimism/batches/da/celestia/:height/:commitment

Should have the following fields:

• Internal ID
• Timestamp
• L1 txn hash (can have more than one hash for the same batch)
  (we don't display it here because the hashes are visible in the blob list below)
• L2 block range (and L2 block count in parentheses)
• Txn count (should be a link to the Transactions tab)
• Blob list with the following columns for Celestia blobs: Commitment, Height, Timestamp, L1 txn hash, Namespace
  or
• Blob list with the following columns for EIP-4844 blobs: Versioned hash, Timestamp, L1 txn hash

Transactions tab on the Batch details page

API endpoint for this tab: /api/v2/transactions/optimism-batch/:batch_number

Should be the same table as for zkSync or zkEVM. The transaction list is paginated (as there can be a lot of transactions in one batch).

Block page

API endpoint for this page: /api/v2/blocks/:block_hash_or_number

This page can be extended by the batch info (either by displaying full batch info on a separate tab or just giving the user a link to the batch by its Internal ID).

Upgrading

A new optional env variable INDEXER_OPTIMISM_L1_BATCH_CELESTIA_BLOBS_API_URL was added to define URL to Celestia blobs indexer service. Corresponding PR in blockscout/docs: https://github.com/blockscout/docs/pull/292. Example: http://localhost:8050/api/v1/celestia/blob. If the URL is not defined, Celestia blobs will be skipped.

Env variable INDEXER_OPTIMISM_L1_BATCH_BLOCKSCOUT_BLOBS_API_URL now is not required. If it's not defined, EIP-4844 blobs are ignored (with an error message in logs for skipped blobs).

In part of env variables optimization, the following ones were removed:

• INDEXER_OPTIMISM_L1_PORTAL_CONTRACT
• INDEXER_OPTIMISM_L1_DEPOSITS_START_BLOCK
• INDEXER_OPTIMISM_L1_WITHDRAWALS_START_BLOCK
• INDEXER_OPTIMISM_L1_OUTPUT_ROOTS_START_BLOCK
• INDEXER_OPTIMISM_L1_BATCH_START_BLOCK
• INDEXER_OPTIMISM_L1_BATCH_INBOX
• INDEXER_OPTIMISM_L1_BATCH_SUBMITTER

The two variables now are optional (have default values) and can be omitted:

• INDEXER_OPTIMISM_L2_WITHDRAWALS_START_BLOCK
• INDEXER_OPTIMISM_L2_MESSAGE_PASSER_CONTRACT

The new INDEXER_OPTIMISM_L1_SYSTEM_CONFIG_CONTRACT env var was added to replace the variables listed above. This variable must be defined after instance upgrade (and before its restart).

Checklist for your Pull Request (PR)

• If I added new functionality, I added tests covering it.
• If I fixed a bug, I added a regression test to prevent the bug from silently reappearing again.
• I checked whether I should update the docs and did so by submitting a PR to https://github.com/blockscout/docs. PR: https://github.com/blockscout/docs/pull/292
• If I added/changed/removed ENV var, I submitted a PR to https://github.com/blockscout/docs to update the list of env vars at https://github.com/blockscout/docs/blob/master/for-developers/information-and-settings/env-variables.md and I updated the version to master in the Version column. Changes will be reflected in this table: https://docs.blockscout.com/for-developers/information-and-settings/env-variables.
• If I added new DB indices, I checked, that they are not redundant with PGHero or other tools.
• If I added/removed chain type, I modified the Github CI matrix and PR labels accordingly.