Polygon zkEVM Bridge indexer and API v2 extension #9098

varasev · 2024-01-04T07:22:35Z

Closes #8268.

Motivation

This PR adds an indexer for Polygon zkEVM Bridge operations (Deposits and Withdrawals) and extends API v2 for the corresponding views on UI.

Please note that this PR renames the following env variables:

INDEXER_ZKEVM_BATCHES_ENABLED ➡️ INDEXER_POLYGON_ZKEVM_BATCHES_ENABLED
INDEXER_ZKEVM_BATCHES_CHUNK_SIZE ➡️ INDEXER_POLYGON_ZKEVM_BATCHES_CHUNK_SIZE
INDEXER_ZKEVM_BATCHES_RECHECK_INTERVAL ➡️ INDEXER_POLYGON_ZKEVM_BATCHES_RECHECK_INTERVAL

Other env variables are added in blockscout/docs#217.

Checklist for your Pull Request (PR)

I added an entry to CHANGELOG.md with this PR
If I added new functionality, I added tests covering it.
If I fixed a bug, I added a regression test to prevent the bug from silently reappearing again.
I checked whether I should update the docs and did so by submitting a PR to https://github.com/blockscout/docs
If I added/changed/removed ENV var, I submitted a PR to https://github.com/blockscout/docs to update the list of env vars at https://github.com/blockscout/docs/blob/master/for-developers/information-and-settings/env-variables.md and I updated the version to master in the Version column. Changes will be reflected in this table: https://docs.blockscout.com/for-developers/information-and-settings/env-variables.
If I add new indices into DB, I checked, that they are not redundant with PGHero or other tools

apps/indexer/lib/indexer/fetcher/zkevm/transaction_batch.ex

apps/indexer/lib/indexer/fetcher/zkevm/bridge_l1.ex

apps/indexer/lib/indexer/fetcher/zkevm/bridge_l2.ex

akolotov · 2024-01-08T22:43:03Z

apps/indexer/lib/indexer/fetcher/zkevm/bridge_l1.ex

+    end
+  end
+
+  defp reorg_queue_get(table_name) do


Looks like code duplication with Indexer.Fetcher.PolygonEdge. Does it make sense to introduce a library for this?

Polygon Edge is not supported anymore by Polygon team and Blockscout doesn't have such instances. We are going to remove Polygon Edge modules from Blockscout soon, so I'd leave it as it is for now

Is my initial comment still applicable because we have now Shibarium related code?

Moreover similar code exists in Optimism branch. Does it make sense to make a framework of re-orgs handling re-usable in all L2-chains?

Yes, I will move these functions to a separate module

Done this in 9888f59 and in 869c924

apps/indexer/lib/indexer/fetcher/zkevm/bridge_l1.ex

vbaranov

@varasev please rebase to master branch. After merging #8929 there are multiple conflicts.

varasev · 2024-01-25T06:50:22Z

@vbaranov Rebased

akolotov

Please provide @doc comments for every public function introduced. If a changed public function does not have @doc comments it will be great if you add it there as well.

akolotov · 2024-01-25T21:21:00Z

apps/indexer/lib/indexer/block/fetcher.ex

@@ -157,6 +158,11 @@ defmodule Indexer.Block.Fetcher do
             do: ShibariumBridge.parse(blocks, transactions_with_receipts, logs),
             else: []
           ),
+         zkevm_bridge_operations =
+           if(callback_module == Indexer.Block.Realtime.Fetcher,
+             do: ZkevmBridge.parse(blocks, logs),


Since this call could make lots of RPC calls, it could delay the block importing, which could cause latency in the new block appearance on UI side. Consider to move logic of getting blocks timestamps and token data (both from DB and RPC) to an async worker (see async_import_remaining_block_data of the realtime fetcher).

Yes, done this in affe983

akolotov · 2024-01-25T21:32:52Z

apps/indexer/lib/indexer/fetcher/zkevm/bridge.ex

+    query = from(t in BridgeL1Token, select: {t.address, t.id}, where: t.address in ^addresses)
+
+    query
+    |> Repo.all(timeout: :infinity)


Does it make sense to move this to the Reader module?

Yes, moved in f4f68ea

akolotov · 2024-01-25T21:33:19Z

apps/indexer/lib/indexer/fetcher/zkevm/bridge.ex

+    query =
+      from(
+        t in BridgeL1Token,
+        where: t.address in ^token_addresses,
+        select: {t.address, t.decimals, t.symbol}
+      )
+
+    token_data =
+      query
+      |> Repo.all()


Does it make sense to move this to the Reader module?

Yes, moved in f4f68ea

akolotov · 2024-01-25T21:35:40Z

apps/indexer/lib/indexer/fetcher/zkevm/bridge.ex

+      token_addresses
+      |> Enum.map(fn address ->
+        # we will call symbol() and decimals() public getters
+        Enum.map(["95d89b41", "313ce567"], fn method_id ->


Does it make sense to define the methods selectors as the module attributes?

Yes, done in f4f68ea

akolotov · 2024-01-25T21:39:04Z

apps/indexer/lib/indexer/fetcher/zkevm/bridge.ex

+      if status == :ok do
+        response = parse_response(response)
+
+        address = String.downcase(request.contract_address)


There is a function address_hash_to_string in Indexer.Helper

Refactored in f4f68ea

akolotov · 2024-01-25T21:39:42Z

apps/indexer/lib/indexer/fetcher/zkevm/bridge.ex

+      query
+      |> Repo.all()
+      |> Enum.reduce(%{}, fn {address, decimals, symbol}, acc ->
+        token_address = String.downcase(Hash.to_string(address))


There is a function address_hash_to_string in Indexer.Helper

Refactored in f4f68ea

akolotov · 2024-01-25T21:42:04Z

apps/indexer/lib/indexer/fetcher/zkevm/bridge.ex

+    tokens_inserted_outside = token_addresses_to_ids_from_db(tokens_not_inserted)
+
+    tokens_inserted
+    |> Enum.reduce(%{}, fn t, acc -> Map.put(acc, Hash.to_string(t.address), t.id) end)


Is Hash.to_string(<address>) equal Helper.address_hash_to_string(<address>, false)?

Yes, refactored in f4f68ea

akolotov · 2024-01-25T21:42:14Z

apps/indexer/lib/indexer/fetcher/zkevm/bridge.ex

+    tokens_not_inserted =
+      tokens_to_insert
+      |> Enum.reject(fn token ->
+        Enum.any?(tokens_inserted, fn inserted -> token.address == Hash.to_string(inserted.address) end)


Is Hash.to_string(<address>) equal Helper.address_hash_to_string(<address>, false)?

Yes, refactored in f4f68ea

apps/indexer/lib/indexer/fetcher/zkevm/bridge.ex

akolotov · 2024-02-01T15:03:31Z

@varasev, a general question about the approach to execute L1 RPC requests during the block fetcher job asynchronously. My understanding is that initiating bridge L1 messages happens first, and the corresponding L2 messages appear after some 'finalization' time, correct? So, if we have an independent fetcher (Indexer.Fetcher.Zkevm.BridgeL1?) that looks for L1 messages first, then handling L2 messages will be quite fast during the block fetcher job, without the necessity to make RPC calls to the L1 endpoint.

varasev · 2024-02-02T07:31:01Z

Please provide @doc comments for every public function introduced. If a changed public function does not have @doc comments it will be great if you add it there as well.

Done

varasev · 2024-02-02T07:59:04Z

My understanding is that initiating bridge L1 messages happens first, and the corresponding L2 messages appear after some 'finalization' time, correct?

@akolotov Yes, if you mean Deposit operation. For Withdrawals it's vice versa: L2 message first, then L1 one.

For realtime case on L1:

If it's Deposit operation (L1 -> L2), Indexer.Fetcher.Zkevm.BridgeL1 catches BridgeEvent and initiates separate requests to RPC node (to get block timestamp and token data from L1).
If it's Withdrawal operation (L2 -> L1), Indexer.Fetcher.Zkevm.BridgeL1 catches ClaimEvent, but doesn't send separate requests to RPC as they only needed for BridgeEvent.

For realtime case on L2:

Indexer.Fetcher.Zkevm.BridgeL2 is only responsible for finding historical operations, so in real time it's off.

For the realtime we have Indexer.Transform.Zkevm.Bridge instead of Indexer.Fetcher.Zkevm.BridgeL2. The transformer catches BridgeEvent and ClaimEvent events without sending RPC requests to L1 for getting block timestamp (as they are already sent by the block fetcher). But for getting L1 token data we still need to send separate RPC requests on L1 for BridgeEvent only. They are now sent asynchronously through Indexer.Fetcher.Zkevm.BridgeL1Tokens, so they don't affect the block fetcher.

These token data RPC requests are not sent by Indexer.Fetcher.Zkevm.BridgeL1Tokens if the token data is already in the database (it could be written there earlier by Indexer.Fetcher.Zkevm.BridgeL1 which works in real time, separately and asynchronously).

akolotov · 2024-02-07T21:33:13Z

apps/indexer/lib/indexer/helper.ex

+  Performs a specified number of retries (up to) if the first attempt returns error.
+  """
+  @spec get_blocks_by_events(list(), list(), non_neg_integer()) :: list()
+  def get_blocks_by_events(events, json_rpc_named_arguments, retries) do


Is my understanding correct and if the event list contains 200 records, in the worst case 200 of eth_getBlockByNumber will be sent in one batch? Does it make sense to divide it on chunks?

Yes, added chunks in 95e522b

akolotov · 2024-02-07T21:38:06Z

apps/indexer/lib/indexer/fetcher/zkevm/bridge.ex

+    |> Helper.get_blocks_by_events(json_rpc_named_arguments, 100_000_000)
+    |> Enum.reduce(%{}, fn block, acc ->
+      block_number = quantity_to_integer(Map.get(block, "number"))
+      {:ok, timestamp} = DateTime.from_unix(quantity_to_integer(Map.get(block, "timestamp")))


Consider to use EthereumJSONRPC.timestamp_to_datetime here.

Done in c4ad14b

vbaranov

Now, since we have INDEXER_POLYGON_EDGE_ and INDEXER_POLYGON_ZKEVM_ envs set families I would suggest change the structure of folders. It would be natural if everywhere instead polygon_edge and zkevm folders we'd have:

polygon
  edge
  zkevm

And I'd rename corresponging modules to have Polygon.Edge, Polygon.Zkevm.

CHANGELOG.md

… code

…dule of the same rollup

varasev · 2024-02-14T11:00:33Z

Now, since we have INDEXER_POLYGON_EDGE_ and INDEXER_POLYGON_ZKEVM_ envs set families I would suggest change the structure of folders. It would be natural if everywhere instead polygon_edge and zkevm folders we'd have:
polygon
  edge
  zkevm
And I'd rename corresponging modules to have Polygon.Edge, Polygon.Zkevm.

I renamed Zkevm to PolygonZkevm: d8fd9b2

As PolygonEdge will be removed soon, I left it as it is

varasev force-pushed the va-zkevm-bridge branch from 0694f1c to 7a48513 Compare January 4, 2024 07:36

varasev mentioned this pull request Jan 4, 2024

Add env variables and update sharelock.md for Polygon zkEVM modules blockscout/docs#217

Merged

varasev requested review from vbaranov, sl1depengwyn, Qwerty5Uiop and nikitosing January 4, 2024 07:59

akolotov reviewed Jan 8, 2024

View reviewed changes

apps/indexer/lib/indexer/fetcher/zkevm/transaction_batch.ex Outdated Show resolved Hide resolved

akolotov reviewed Jan 8, 2024

View reviewed changes

apps/indexer/lib/indexer/fetcher/zkevm/bridge_l1.ex Outdated Show resolved Hide resolved

apps/indexer/lib/indexer/fetcher/zkevm/bridge_l2.ex Outdated Show resolved Hide resolved

apps/indexer/lib/indexer/fetcher/zkevm/bridge_l2.ex Outdated Show resolved Hide resolved

akolotov requested changes Jan 8, 2024

View reviewed changes

akolotov reviewed Jan 8, 2024

View reviewed changes

apps/indexer/lib/indexer/fetcher/zkevm/bridge_l1.ex Outdated Show resolved Hide resolved

varasev force-pushed the va-zkevm-bridge branch from 1877b04 to 6c5e1d8 Compare January 12, 2024 07:45

varasev force-pushed the va-zkevm-bridge branch from bd2ca0b to d61c9d3 Compare January 22, 2024 06:39

vbaranov reviewed Jan 24, 2024

View reviewed changes

varasev force-pushed the va-zkevm-bridge branch from 667ddb1 to b8ebd9e Compare January 24, 2024 11:54

akolotov requested changes Jan 25, 2024

View reviewed changes

varasev force-pushed the va-zkevm-bridge branch from 26ff09b to affe983 Compare January 31, 2024 12:57

akolotov reviewed Feb 7, 2024

View reviewed changes

akolotov requested changes Feb 7, 2024

View reviewed changes

varasev force-pushed the va-zkevm-bridge branch from 95e522b to bddcb3d Compare February 13, 2024 10:07

vbaranov reviewed Feb 13, 2024

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

varasev added 6 commits February 14, 2024 10:05

Minor changes

1eb0647

Define chain type

b6859d3

Prepare zkevm_bridge table and its importer

8cf0f63

Add zkevm_bridge_l1_tokens table

1515055

Small changes

1fda56e

Draft

0092b1a

varasev added 21 commits February 14, 2024 10:05

Add envs to common-blockscout.env

5931ec8

Rename uninserted to not inserted

d6a34da

Small refactoring

85ff545

Remove redundant function

98c31ee

Fixes after rebase

7e6b81f

Fixes for Chain.import related to CHAIN_TYPE

811f69e

Small refactoring

ae812aa

Move L1 RPC requests from realtime block handler to a separate GenServer

a6d2a80

Refactoring

5ffb2ea

Add doc comments for public functions

42314a6

Fix spelling

2949f02

Add Indexer.Fetcher.RollupL1ReorgMonitor module and remove duplicated…

ce06032

… code

Update spelling

033ffb5

Allow using Indexer.Fetcher.RollupL1ReorgMonitor for more than one mo…

e57ce96

…dule of the same rollup

mix format and update spelling

816b5d1

Use timestamp_to_datetime

0ecf3fe

Split RPC requests for blocks into chunks

ec76391

Refactor Indexer.Block.Fetcher

121092a

Reset GA cache

b19e196

Update chain.ex

98e80b3

Update changelog

357a7d1

varasev force-pushed the va-zkevm-bridge branch from c415462 to 357a7d1 Compare February 14, 2024 07:05

Rename Zkevm to PolygonZkevm

d8fd9b2

vbaranov requested a review from akolotov February 14, 2024 12:08

vbaranov approved these changes Feb 14, 2024

View reviewed changes

vbaranov merged commit 6ae6692 into master Feb 14, 2024
41 checks passed

vbaranov deleted the va-zkevm-bridge branch February 14, 2024 15:42

varasev mentioned this pull request Mar 14, 2024

feat: Support the 2nd version of L2<->L1 Polygon zkEVM Bridge #9637

Merged

6 tasks

isstuev mentioned this pull request Mar 25, 2024

zkEVM deposits and withdrawals blockscout/frontend#1741

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Polygon zkEVM Bridge indexer and API v2 extension #9098

Polygon zkEVM Bridge indexer and API v2 extension #9098

varasev commented Jan 4, 2024 •

edited

akolotov Jan 8, 2024

varasev Jan 12, 2024

akolotov Jan 24, 2024

akolotov Jan 24, 2024

varasev Jan 25, 2024 •

edited

varasev Feb 5, 2024 •

edited

vbaranov left a comment

varasev commented Jan 25, 2024

akolotov left a comment

akolotov Jan 25, 2024

varasev Jan 31, 2024

akolotov Jan 25, 2024

varasev Feb 1, 2024

akolotov Jan 25, 2024

varasev Feb 1, 2024

akolotov Jan 25, 2024

varasev Feb 1, 2024

akolotov Jan 25, 2024

varasev Feb 1, 2024

akolotov Jan 25, 2024

varasev Feb 1, 2024

akolotov Jan 25, 2024

varasev Feb 1, 2024

akolotov Jan 25, 2024

varasev Feb 1, 2024

akolotov commented Feb 1, 2024

varasev commented Feb 2, 2024

varasev commented Feb 2, 2024 •

edited

akolotov Feb 7, 2024

varasev Feb 8, 2024

akolotov Feb 7, 2024

varasev Feb 8, 2024

vbaranov left a comment

varasev commented Feb 14, 2024

Polygon zkEVM Bridge indexer and API v2 extension #9098

Polygon zkEVM Bridge indexer and API v2 extension #9098

Conversation

varasev commented Jan 4, 2024 • edited

Motivation

Checklist for your Pull Request (PR)

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

varasev Jan 25, 2024 • edited

Choose a reason for hiding this comment

varasev Feb 5, 2024 • edited

Choose a reason for hiding this comment

vbaranov left a comment

Choose a reason for hiding this comment

varasev commented Jan 25, 2024

akolotov left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

akolotov commented Feb 1, 2024

varasev commented Feb 2, 2024

varasev commented Feb 2, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vbaranov left a comment

Choose a reason for hiding this comment

varasev commented Feb 14, 2024

varasev commented Jan 4, 2024 •

edited

varasev Jan 25, 2024 •

edited

varasev Feb 5, 2024 •

edited

varasev commented Feb 2, 2024 •

edited