Madara <> Celestia

[sfyll]

Integrate Celestia as Data Availability Layer:

Continuation of Lambda ZK Week hackathon project Madara <> Celestia.

Integrate Celestia into the Madara client by using it as a Data Availability Layer. To do so, we implement a Celestia module. Within that module, we use the builder pattern to spawn a CelestiaClient which is a wrapper around the celestia-node-rs library. We then make available the public method publish_state_diff_and_verify_inclusion, with self-explanatory functionality with the caveat that we verify inclusion by checking that our blob is present in the Namespaced Merkle Tree (NMT), but we don't verify the merkle tree integrity itself by running a merkle inclusion proof.

As such, on every update_state function call within the DataAvailabilityWorker struct, we’ll first push the state diff into Celestia and then verify that our Blobs are in the NMT at a specific block height. Since update_state is spawned as a non blocking task, we take an optimistic approach whereby it can accumulates n state diffs and fail anywhere in the range [current, n] states.

The user can point to different http and ws endpoints by passing these as parameters when launching the Madara client as these will be parsed by command.rs using the ExtendedRunCommand struct. By the same token, users will be able to pass in their auth_token as parameters so that they can call Celestia RPC endpoints with different access rights.

One caveat is that Madara now relies on an external party which might impact its robustness. Nonetheless, given Celestia significantly lower costs for storing data, exploring design patterns so that the client can have some kind of fallback logic might be more than amortized for in the long run.

Summary of Changes

Component	Change
CLI	New optional flags in ExtendedRunCommand: http_endpoint, ws_endpoint, auth_token
service	On state_diff notifications, push state_diff to Celestia and verify that the state_diff blob is included in the given height

Open Questions

1. State_diff Posting: Should state_diff be posted to Celestia asynchronously? Should there be no communication between the STF and the state_diff poster? What would be a good fallback strategy in case of an issue when posting state_diff and verifying their inclusion?

2. Client Scaling: As the client scales, should we implement the Merkle inclusion proof on the client side to minimize the number of HTTP requests? This would allow for testing the integrity of the Merkle tree.

3. DA Layer Change: If the Madara user wants to change the DA layer, what would be a good migration mechanism? Should that be a Madara or Celestia task? While Celestia offers some modularity properties such as separation of concern, one cannot switch away from it, breaking the so-called "modular continuity".

How To Run Tests

To set up and run a Celestia light node

Requires the installation of the celestia CLI tool (available here). Once you have that set up, run the following:

celestia light start --core.ip consensus-full-arabica-9.celestia-arabica.com --gateway.deprecated-endpoints --gateway --gateway.addr 127.0.0.1 --gateway.port 26659 --p2p.network arabica-9

Next, we need to generate a JWT to access our light node:

celestia light auth admin --p2p.network arabica-9

To run a Madara node

We'll just add the generated JWT to the usual command so that we can have the needed access rights. From root, run the following:

cargo run --release -- --dev --auth-token JWT

[tdelabro]

Hey @sfyll,
Given the recent decision taken regarding the DA layers integration and the common PR solution we decided, can I close this PR?

[drspacemn]

yes, let's close this. I'll open another one unifying the 3 PRs.