Skip to content

Logbook

Jump to bottom
Jean-Philippe Raynaud edited this page Apr 19, 2022 · 1382 revisions

This page records a log of the team’s work, thinking, discussions, pains, joys, events, and experiences that happen on a daily basis. It’s supposed to be a kind of Stream of consciousness that can later be searched, analyzed, reviewed in more formal retrospectives.

Rules:

  • Log days in reverse chronological order, recent first

  • When the stream is getting too big, such that github refuses to render do the following:

    • extract biggest logical section (half-year or quarter) out of this into a correspondingly named Logbook-Year-Hn.md or Logbook-Year-Qn.md page
    • update links on top of this page to make it easier to navigate try to fix all links in the wiki at least
    • notify the team

  • Move (and link) recurring sections to relevant dedicated pages (TBD).

  • April 2022

  • March 2022

  • 2021

April 2022

2022-04-19

Mithril session

  • We have paired on many PRs in order to be ready for a end to end demo at the end of the sprint:

    • Reviewing/finalizing the SnapshotStorer implementation of the aggregator
    • Reviewing the Snapshotter implementation of the aggregator and the Terraform deployment to Google Cloud
    • Reviewing/finalizing the Unpack snapshot archive implementation of the client

  • We have also talked about how to showcase the e2e demo (given the delays needed to download & restore for the testnet)

  • During the reviews we talked about architectural question that arose during the latest developments:

    • Which pattern(s) to adopt to handle best errors?
    • Which implementation(s) to use for a clean architecture?
    • How to keep code readable, concise and not too complex?

  • In order to keep track of these decisions, we will add an ADR topic to the documentation website that will help us keep track of them #128

  • As it is not necessary at the moment, and as it adds complexity in the CI pipelines, we will remove the triggering of the CI for PRs #121

2022-04-15

Mithril session

  • There is an issue with the Docker images build pipelines that are not very reliable. After investigation, it appears that they should be fixed/enhanced. We have created an issue for this #121. The problem was apparently due to not using the latest version of the github actions: we have released a fix and monitor to see if this happens again

  • As the snapshot creation and storage on Google Cloud is now alive, we have created a task to implement a SnapshotStore in the Mithril Aggregator #120

  • A pairing session was done to work on this task and we achieved to retrieve the list of remote snapshots from the aggregator 🎉. The corresponding PR is #123

2022-04-14

Mithril session

  • We have paired in order to fix multiple issues:

    • The branch of the Init Mithril Client was corrupted after a conflict resolution #110
    • After merging the PR, we found out that the documentation workflow of the CI did not work. It was due to publishing build artifacts from the cargo doc processes to the gh-pages branch which reached its limit of 50MB. The fix was published in #118

  • We had talks about what is the best way to deploy the Mithril Aggregator server to public cloud:

    • Several solutions were investigated (Kubernetes, Custom deployment Terraform, Google Cloud Run, ...)
    • As we need to embed a Cardano Node with the Aggregator that has access to its local storage, we decided to start with a Terraform deployment of a Docker Compose setup
    • This hosting setup will also allow us to produce easily and regularly snapshot archives (scheduled by a cron)
    • At each merge of the main branch a workflow will trigger a deployment of the latest Docker image to the cloud

2022-04-13

Mithril session

  • We talked about the questions raised by Inigo about yesterday's meeting with Galois. A Q&A meeting session will be organized in that matter

  • The documentation job needs to be changed. This will be done in #104

    • The documentation should be produced in the builds jobs (for more efficiency)
    • Artifacts (such as compressed archive of the generated files) should be uploaded as artifacts
    • The documentation job should use the previously produced artifacts to prepare the new release of the documentation website

  • We decided not to use jobs templates for now in the CI jobs definition

  • We worked on a Feature Map for the increments of the components of the Mithril network

  • We have reviewed the graceful shutdown fix for the Mithril Aggregator #116

  • We have also reviewed the code of the Mithril Client #110

  • We had some thoughts on how to produce snapshots and use the remote storage as a database at first and have the Mithril Aggregator make this information available through its API

2022-04-12

Mithril session

  • The aggregator has a few issues and fixes/optimizations that will be done in #109

  • We have worked on the CI pipeline architecture in order to get a clear view of it

  • We should publish the executable files and make them available on Github (Linux only, except for the Mithril Client that needs to be on macOS, Linux and Windows)

  • Question: Do we need to do the end to end testing on the executable files or the Docker images?

  • We have decided to work with the executable files at first, and then with Docker. We may need to change the way the Docker images are built and embed the previously built executable file.

  • We have successfully created a first ETE test scenario where we check that a freshly started aggregator effectively produces a snapshot and makes it available at its /snapshots route in assertClientCanVerifySnapshot. It will not work with the current aggregator implementation until the number of fake snasphots displayed is reduced to 1

Galois meeting

  • We had a fruitful discussion detailing the state machine view of the signer's registration process that will be formalised in the monitor framework.
  • The signer should be essentially stateless regarding the certificates, and retrieve all data from the aggregator, eg. pending certificate and certificates chain. If there are multiple aggregators it becomes possible to have multiple chains, which is probably not a big deal but could become quite messy
  • Aggregators are essentially infrastructure provider: They setup some infrastructure to efficiently distribute snapshots and they have an incentive to provide as up-to-date as possible snapshots to their "Customers"
  • In the long run, both signers and aggregators could use the blockchain itself as a communication medium in order to synchronise and ensure consensus over the certificates chain: The tip of the certificate could be part of a UTxO locked by a script enforcing single chain production, possibly with time bounds in order to prevent arbitrary locking of the certificate chain and thus DoS?
    • The amount of transactions involved is not huge: 1000-2000 signatures + 1 aggregation per epoch (5 days) would be enough but this implies signers and aggregators spend money for posting the txs
    • The problem of rewarding mithril network is still open
  • The previous certificate hash is missing from the /certificate-pending route. An issue has been created for this task: #112

2022-04-11

Mithril session

Pairing
Documentation

  • We have worked on some refinements of the openapi.yaml file so that the documentation generated by Docusaurus does not truncate the description sentences

  • In order to keep the generation process not too cumbersome, we have decided to keep the source .md files in the docs/root folder, mixed with the website files. When we need to refer to an external README.md file (for example mithril-network/mithril-aggregator/REAME.md) the easiest way to do it is to provide a direct Github link such as (https://github.com/input-output-hk/mithril/blob/main/mithril-network/mithril-aggregator/README.md)

  • The documentation is now live and accessible at (https://mithril.network/) and the CI updates it along the way

  • As releasing production Github actions can be tricky (especially when specific operations are launched only on the *main branch), it looks like a good idea to do it in pairing

Mithril Client

2022-04-07

Mithril session

  • We have created the tickets/tasks on the project Kanban board following the sprint planning that took place yesterday

  • In order to facilitate the pairing sessions, we will setup Live Sharing on VS Code next week

  • Question: Do we move the openapi.yaml file from the root to the mithril-network/mithril-aggregator folder?

  • Question: In order to have as fast as possible CI workflows completion, what are the best parallelization/caching strategies of the jobs?

2022-04-06

Mithril session

Demo

  • We have talked about the sprint demo and reviewed the demo path

  • Questions asked during the demo:

    • Can we cache the digest computation on the signer side? (Yes, good idea. It will lower the load on the signer side)
    • Can we pipeline the download and the decompression of the snapshot? (Yes, good idea. We can do that as an optimization later)
    • What happens with the Genesis certificate when the Genesis keys of the node are updated? (We need to investigate this)
    • How long does it take to verify the certificate chain? (This could be done in parallel of the snapshot download/uncompress to avoid further delays)
    • How do we handle the discovery of the aggregators? (This could be done on-chain. We will need to investigate this later)
How to produce deterministic ledger state
  • It appears that the tool db-analyser provides a good way to create deterministic snapshots of the ledger state:
    • It will be much easier to use than modifying the Cardano Node snapshot policy
    • We will investigate further in that direction
    • We need to check if we can rely on the slot number or the block number to produce consistent snapshots
Documentation / Open sourcing

  • We have worked on the documentation and reviewed the first version of the docusaurus website

  • We still need to define precisely the final documentation structure

  • A session with the communication department will be setup to review the documentation and prepare open sourcing

CI / Docker registry

  • A new CI workflow has been released that better separates the steps of the process

  • We still struggle to get access to the Docker registry. IT department is working on it

  • Question: How can we get delete previously created packages? (mithril package now and mithril-node-poc later)

2022-04-06

Mithril session

Mithril Aggregator

  • The OpenAPI specification validation unit tests in Rust are almost done

  • They allow to validate that the request and the response sent to the aggregator server are compliant with the specification (for the routes we decide to test)

  • There are a few modifications/upgrade under development that should be finalized shortly:

    • Create a separate module for the APISpec
    • Create a Docker image of the aggregator
    • Create a workflow for the aggregator server in the CI
    • Make some refinements on the specification (include 400 errors for POST method, and handle more precisely the size of the string params)

  • Next steps are:

    • create an integration test for the server
    • implement the server in the ETE test lab
    • optimize the tests code if necessary

  • Question: do we need to move the openapi.yaml file to the mithril-network/mithril-aggregator folder?

Documentation

  • We have now a better understanding of how Docusaurus works

  • As a start, we will try to reproduce the Hydra documentation website (when it applies)

  • We will update the docs directory structure and create Under construction pages where it makes sense

How to produce snapshots

  • Here is a new breakdown of the timings to produce/certify/restore snapshots on the mainnet:

Mithril

2022-04-05

Mithril session

Mithril Aggregator

  • We have finally found a way to test that the actual implementation of the Mithril Aggregator is conform to the OpenAPI specification

  • We have worked in order to make the testing work for a response object and with explicit error messages when tests fail in rust tests

  • We are currently trying to implement the validation of the requestBody according to the specification

  • We have also discussed about the several layers of test and how to implement them:

    • At a upper level, we'd like to have an integration (or several) that would:
      • test the global conformity of the aggregator server implementation to the specification
      • use the specification as the source to generate tests: fuzzy tests and/or stress tests
      • we still need to investigate and find the best tooling for this and work on a CI implementation
    • At a lower level, in the Rust server, we will need to test that each route is correctly implemented in unit tests:
      • use the actual implementation and test it to validate that it matches the specification
      • each route should be tested for every possible response status code (this will be easier when mocks will be implemented)
      • each of these tests must also validate the conformity of the request/response to the OpenAPI specification
      • an other unit test should also check that all the routes implemented in the server match exactly all the routes defined in the specification (we still need to find a way to list all these routes on the server - we will investigate further warp documentation)

Some more tools that could be useful for testing/checking REST APIs:

Documentation

  • We have started to review the way the documentation is implemented on Hydra and the integration that is made with Docusaurus and the CI worklows

  • We will continue this investigation tomorrow

  • We will design a tree structure for the website that will be hosted on (https://mithril.network) and work on restructuring the docs folder of the repository

Snapshot Generation

Working on snapshot generation in the cardano-node, adding parameter to be able to generate snapshot every X blocks. Not sure how to test this though, as the test infrastructure of consensus is quite hard to grok

I was unable to build cardano-node using nix-shell -A dev nor nix develop as instructed in build instructions, but managed to build it installing the needed OS dependencies (forked libsodium and libsecp256k essentially). Adding the ouroboros-network packages as local packages in cardano-node's cabal.project enables incremental build whereby changes to the former entail rebuild of the later.

2022-04-04

On-boarding Denis on Mithril

  • Intros
  • Explanation of Mithril
  • Explanation of the POC and MVP architecture and scope
  • Demonstration of the cryptographic library
  • Plan next days/weeks
    • Dig into repository codebase and documentation
    • Development environment setup
    • Setup of daily stand-up and added Denis to other meetings

Mithril Aggregator

  • First version of the server is pending review (in its fake version)
  • We need to enhance the tests and make sure the server implements correctly the OpenAPI specification
  • For these tests we are scouting some libraries and utilies: OpenAPI Fuzzer crate and/or OpenAPI crate
  • We cannot use openapi-fuzzer as a lib so we try to get inspiration from it instead. Package openapiv3 seems mature enough to be used to read the specification and use that to write simple tests to check output of various routes, and we can later generate arbitrary data to input.

Created an APISpec structure that will help us define what we test and what to expect

Struggling with validating our dummy aggregator server against OpenApi specification using rust tooling, seems like jsonschema-rs does not work as easily as expected. Turns out things are more complicated than we would want: The OpenAPI specification is not really a proper JSONSchema, only some fragments of it are (eg. types in requests/responses) so we would need to find the correct part of the schema to check our code against.

Possible options from there:

  • Stick to use openapi-fuzzer as an external binary, running basic integration tests against a running server. Should be fine at first but will quickly break as we flesh out the internals of the server and need to tighten what's valid input
  • Understand how jsonschema-rs library works and make sure we check requests against it
  • Use another tool like https://github.com/Rhosys/openapi-data-validator.js/blob/main/README.md. This one is in node so this would require us to run it as an integration test. Hydra uses a similar approach based on anothoer jsonschema executable in node.

Test Lab

  • Question: Should we develop the ETE tester of the test lab in Rust instead of Haskell (for simplicity and maintainability)?

How to produce snapshots

  • New tests have been run and it outcomes that they are ~2x faster than the previous ones:

Mainnet

Data Node Full Archive Snapshot Upload Download Restore Startup
Immutable Only standard 43GB 24GB ~28m ~45m ~25m ~12m ~420m
With Ledger State modified 45GB 25GB ~28m ~45m ~25m ~12m ~65m

Testnet

Data Node Full Archive Snapshot Upload Download Restore Startup
Immutable Only standard 9.5 GB 3.5 GB ~7m ~5m ~3m ~2m ~130m
With Ledger State modified 10 GB 3.5 GB ~7m ~5m ~3m ~2m ~6m

Host: x86 / +2 cores / +8GB RAM / +100GB HDD Network: Download 150Mbps / Upload 75Mbps Compression: gzip

2022-04-01

Mithril's fool day 😆

  • The network is now live on the mainnet 🚀

Wiki

  • The Project Charter page is now available here and open to comments

AB on ETE Tests

Added basic ETE test to spin up a cardano-node cluster of 3 nodes, now adding the test to CI probably in another parallel job as it will take a while to build...

Having trouble building on CI

trace: WARNING: No sha256 found for source-repository-package https://github.com/input-output-hk/hydra-poc 60d1e3217a9f2ae557c5abf7c8c3c2001f7c2887 download may fail in restricted mode (hydra)
fatal: couldn't find remote ref refs/heads/60d1e3217a9f2ae557c5abf7c8c3c2001f7c2887

Probably should try adding the sha256 field:

  • the nar executable in https://github.com/input-output-hk/nix-archive/ will do it 
    nar  git-nar-sha --git-dir ../whatever --hash XXXXXX
  • Alternative command to compute sha256 for nix: 
    nix-prefetch-git <repo> <git-hash>
  • When ran nix just tells you the missing sha256's value 🤦

Got a weird error when building: Cabal cannot find commit 60d1e3217a9f2ae557c5abf7c8c3c2001f7c2887 for hydra-poc which is weird as it's definitely available on github. Removing dependency and rerunning cabal again works 🤔 🤷

Created an archive for testnet and uploaded it to google storage bucket

  • Not sure how to handle authentication properly, I had to upload a service account key file to impersonate the hydra-poc-builder service account, I should have been able to do it using the credentials/auths attached to the VM which is a resource

March 2022

2022-03-31

OpenAPI specification

  • A first version of the OpenAPI specification has been created
  • An OpenAPI UI allowing us to interact with the routes is accessible and updated by the CI
  • We will work on a basic REST server in Rust that implements this specification

Repository reorganization

  • The rust folder containing the cryptographic library has been moved to mithril-core

Mithril session

How to produce snapshots
  • We don't need to copy the files somewhere when we create a snapshot (or when a signer needs to compute the digest of the snapshot)
  • We will make some tests with an uncompressed archive (and/or slice the archive in smaller chunks) in order to see if this can make the restoration faster
  • We will make a test to see what's happening when we restore a snapshot with only ledger state (last time it worked 1 out of 2 times)
  • The following diagram shows a breakdown of the timing for the snapshots creation/restoration on the mainnet:

Mithril Snapshot timings

Cryptography
  • We need to conduct an external audit of the crypto library (it should take ~3 months)
  • The library will be published on crates.io once it is open sourced
  • How to sign genesis Mithril certificate:
    • we will use Cardano genesis private key to sign it for the mainnet (manually)
    • we will use test genesis keys published on the repository for the testnet

Next steps

  • The repository is now getting ready to being open sourced. Few verifications/update must be completed before going public #92
  • We can start working on a fake aggregator server that follows the OpenAPI specification
  • We have talked about the Project Charter page content (that will be completed soon)
  • The CI should be optimized/reorganized in order to better reflect the new structure of the repository and maybe operate faster

End-to-End Test

Starting to build an ETE test infrastructure based on the work we did for Hydra project

  • Moving mithril-test-lab to mithril-monitor package and adding a new mithril-end-to-end package to host the needed code for managing ETE test infra. Compilation of zlib dependency is failing due to missing zlib-dev deps -> adding it to shell.nix
  • Toplevel mithril-test-lab is now a cabal.project containing 2 packages: One for the monitor and one for the end-to-end test that will depend on cardano nodes et al.
  • Trying to get the dependencies right reusing Hydra's nix files so that we can benefit from iohk's nix caching too, which implies using haskell.nix 🤷
  • Tried to remove use of hydra-cardano-api but getting the types right now becomes way too complicated, so I just added hydra-poc as a dependency for the moment

2022-03-30

How to produce snapshots

  • Finished tests to create a snapshot from mainnet Cardano Node and restoring the snapshot. Currently running same tests on testnet:

Mainnet

Data Node Full Archive Snapshot Upload Download Restore Startup
Immutable Only standard 41GB 25GB ~47m ~45m ~25m ~18m ~420m
With Ledger State modified 43GB 26GB ~47m ~45m ~25m ~18m ~65m

Host: x86 / +2 cores / +8GB RAM / +100GB HD Network: Download 150Mbps / Upload 75Mbps Compression: gzip

  • A simple cli has been developped in order to conduct the tests

  • Here are the informations needed to create a working snapshot:

    • the whole immutable folder of the database (required)
    • the protocolMagicId file (required)
    • the latest ledger state snapshot file in the ledger folder (optional)

  • Question: Do we need to stop the node when the snapshot is done? (the test cli currently makes a copy of the files to snapshot in a separate folder)

  • In order to create a deterministic digest, it appears that we will need to rely on the binary content of the snapshotted files (digest calculation from snapshot archive file is not enough)

Repository reorganization

  • The rust-node folder has been moved to mithril-prototype/test-node
  • The monitor folder has been renamed mithril-test-lab

2022-03-29

Worked on CEO update presentation: https://drive.google.com/file/d/19_Lrr5sYAhVatxdiMws6OGwvsVRn8p0Q/view?usp=sharing

Mainnet machine is still synchronizing the network, currently at slot 53M while tip's slot is 57M, hopefully should be done by noon.

  • Mainnet node crashed when it ran out of disk space... Current disk space usage: 
    3.2M    cardano-configurations
4.0K    configure-mainnet.sh
4.0K    docker-compose-mainnet.yaml
4.0K    install-nix-2.3.10
4.0K    install-nix-2.3.10.asc
4.0K    ipc
4.0K    ledger.gpg
30G     mainnet
8.4G    mainnet.bak
4.8G    mainnet.tar.bz2
  • Compression is about 60% with bzip2 which is onpar with pkzip, and mainnet directory is now at 30G with nearly completed sync up. Going to remove the backup to regain some space
cardano-node_1  | [d77879a4:cardano.node.ChainDB:Error:5] [2022-03-29 08:21:59.82 UTC] Invalid snapshot DiskSnapshot {dsNumber = 54107040, dsSuffix = Nothing}InitFailureRead (ReadFailed (DeserialiseFailure 400482271 "end of input"))

This error is annoying: This means the snapshot needs to be reconstructed :( The error does not say which file failed to be deserialised though

It's reconstructing the ledger but from the latest snapshot file:

cardano-node_1  | [d77879a4:cardano.node.ChainDB:Info:5] [2022-03-29 08:25:44.52 UTC] Replaying ledger from snapshot at 2e90104a43cd8ecfbd8d16f03ce17ac3e46ffdff0546f93079e4b3a9e298f8ed at slot 53558101
cardano-node_1  | [d77879a4:cardano.node.ChainDB:Info:5] [2022-03-29 08:25:44.72 UTC] Replayed block: slot 53558151 out of 54107040. Progress: 0.01%
cardano-node_1  | [d77879a4:cardano.node.ChainDB:Info:5] [2022-03-29 08:26:15.85 UTC] Replayed block: slot 53567929 out of 54107040. Progress: 1.79%
cardano-node_1  | [d77879a4:cardano.node.ChainDB:Info:5] [2022-03-29 08:27:27.14 UTC] Replayed block: slot 53589561 out of 54107040. Progress: 5.73%

Seems like this should happen relatively fast as the gap is not huge?

Interestingly there's no concurrency in the ledger replay logic: Everything happens sequentially hence CPU is at 100%, eg. 1 core is occupied

So node starts up by:

  • checking immutable DB and extracting the last known blocks from there
  • opening volatile DB at the point where immutable DB is
  • opening ledger DB and restoring it from snapshot
    • in case there's a Δ between the ledger state and the immutable DB, replay the blocks to udpate ledger's state -> That's the part that takes a while (about 29 minutes for 548889 slots)
  • updating ledger DB with "new" blocks
    • when node crashed it was at slot 54162434, so ledger is also updated from volatile DB's stuff passed the immutable one
  • then it connects to relays and start pulling new blocks and extend its state

current rate of slot validations is 95 slots/second, would be interesting to provide that information in the logs? => That could be extracted from parsing the logs obviously, or as a dashboard

  • Seems I will still need about 7 hours to catch up with tip, which is not exactly true because tip keeps growing 😬

Closer to the tip here are the current disk size:

~$ du -sh mainnet/node.db/*
33G     mainnet/node.db/immutable
2.8G    mainnet/node.db/ledger
0       mainnet/node.db/lock
4.0K    mainnet/node.db/protocolMagicId
175M    mainnet/node.db/volatile

Mithril session

Options:

  • Work on the "restore" side using testnet archive

  • Work on the consensus code to be able to generate snapshot reliably at some block X

  • how about epoch boundaries? Can we know when change epoch in the consensus DB?

  • modify cardano-node to emit snapshot at fixed intervals

  • run cardano-node on a private testnet

Acceptance test:

  • run a cardano-node on a private testnet or a cluster of cardano-nodes using withCardanoCluster from Hydra
  • we feed the private testnet a bunch of transactions
    • generate n arbitrary transactions in a sequence, possibly not depending on each other
  • we should end up with identical snapshots on every node of the cluster
Discussion w/ Galois

  • What properties can we express and test?

  • We could start w/ high-level property stating that:

    given signatures of a snapshot then aggregation produces a valid certificate of the expected snapshot

    Once an aggregator receives valid signatures, it should produce valid certificate ultimately

Decentralised vision:

  • several aggregators
  • signers produce signatures in a decentralised manner

2022-03-28

Mithril Sessions

Morning session:

  • We updated the Story Mapping (added details for 1st, 2nd and 6th increments)
  • First investigation on the Cardano node database snapshots structure: Snapshots are located at mainnet/node.db/ledger/{block_no}. Need to review code to be able to produce snapshots at epoch boundary or at regular block number (this afternoon)
  • We need to get access to a project on a public cloud (GCP preferred or AWS). Arnaud asks Charles for credentials
  • We need 2 separate credentials on the public repository: Read/Write for producing snapshots and Read for restoring them
  • Later, we will also host a Mithril Aggregator on the project (Compute resources will be needed). Containers could be orchestrated by Nomad/Hashicorp (preferred) or Kubernetes. We will also need Devops resources at this time
  • In order to get a clear project organization, a specific "Project Charter" page will be created on the Wiki. It will list all the resources (rituals, links, ...)
  • In order to get the genesis state verification (first certificate in the Mithril chain), we will need to get it signed by someone who has access to the private key (the public key is packaged with the Cardano Node)
  • First step in repository reorganization has been completed: the go-node has been moved to mithril-proto/mithril-node-poc
  • Target repository structure is :
    • .github
    • demo
      • protocol-demo
    • mithril-core < rust (TBD by JP)
    • mithril-network (TB Created during MVP)
      • mithril-aggregator
      • mithril-client
      • mithril-signer
    • mithril-proto
      • mithril-poc-node
      • test-node < rust-node (TBD by James)
    • mithril-test-lab < monitor (TBD by James)

Afternoon pairing session, dedicated at investigating the production of Mithril compatible snapshots from a modified Cardano Node:

  • LedgerDB Disk Policy
  • Committed Code: Start hacking on disk snapshot
  • When we will produce the digest of the snapshot, we need to make sure that the file will have deterministic names or rely on the binary content
  • The minimum information required to bootstrap a Cardano Node is the immutable folder
    • We have tried to bootstrap with the ledger snapshot only (no immutable data) and it did not work. The node started from the first block
    • A faster bootstrap is possible with the ledger snapshot in the ledger folder. (If not, the node needs to recompute the ledger state first)
    • A rapid test showed that it took ~20 minutes to recompute ~12,000,000 slots
    • The blocks in the volatile folder are uncommitted to the immutable state (commit will occur after 2,160 blocks, the security parameter)
  • Currently, the snapshot policy of the node is to take snapshots at regular time intervals (take block_number < chain_tip - security_parameter)
  • New snapshot policy selected is to take snapshot at regular block intervals (e.g. every 10,000 blocks): (tip - security_parameter) % block_interval == 0
  • We will need some support from the consensus team to validate the new snapshot policy

In-memory LedgerDB works like a sliding window of k ledger states so that it can be rolled-back at most k blocks in the past.

  • Snapshots written is the oldest (aka. anchor) state of a given in-mem ledger state
  • Snapshotting on epoch boundary might prove difficult because it's not an easy information to compute in the snapshot policy
  • We might want to produce snapshots at some fixed block number

Mithril Galois discussion

  • Talked about new API for monitors which allow to express properties in a more succinct and legible way
  • everywhere combinator is reminiscnet of cooked-validators approach, but that's perhaps only by analogy?
  • All in all, everything that improves the expressiveness of properties goes in the right direction
what's next
  • We want to define a first or a handful of interesting properties and test them
  • We need to make sure the formalism makes sense even in the context of drastically simplified network layer whereby mithril signers would communicate with a single "certifier" or through on-chain transactions posting
  • at an abstract level, the information a node sends and receives from the certifier or the chain could be modelled as broadcast channel because we expect all nodes to see the same information. Then we can model and test interesting behaviour where signers have different world views

2022-03-24

Mithril Session

Discussions/thoughts about MVP and +:

  • In order to simplify the registration process, each signer must (re)register its keys at epoch n-1 in order to be able to sign at epoch n
  • The aggregator server could later keep track of the operations taking place and thus be able to produce reports & statistics
  • How can we verify the genesis state? The genesis state (first certificate in chain) must be signed by a genesis key. It will be verified by the associated public key delivered on the Cardano node (TBV)
  • Signers could store their single signatures on-chain
  • Aggregator could store the protocol parameters on-chain
  • Certificates (or hash of them) could be stored on-chain
  • Verification could use the on-chain storage (after restoring it on a node) to proceed to full verification (all the previous certificates chain)
  • The drawback of using on-chain transactions, is that it has a cost. We could maybe think of a "cashback" mechanism where the aggregator could refund (+reward) these transactions later by using the treasury
  • Question: should we hard code the protocol parameters or allow them to be modified, and thus include them into the certificate?
  • Question: do we need to prepare different snapshots for each targeted OS ?

2022-03-22

Mithril Session

We talked about the project organization with Roy:

  • We need to work on an estimated roadmap for the Mithril project:

    • This will help check if we are on track with the goals
    • For now let's focus on the MVP, from the end goal: allow people who use Cardano to bootstrap a node fast and securely
    • The MVP is expected at +6 months on the real testnet (+3 months on the development testnet)
    • Let's focus on priorities instead of deadlines
    • We can work as for Hydra with a matrix of Features vs Priorities (must have, should have, could have, nice to have)
    • Short term goal: have something working for people to start using (even though it is not full features yet)
    • We will focus on the layout of the plan during the next session on Thursday

  • Agile organization:

    • Let's work with agile sprints and ceremonies (duration 2 weeks)
    • Ceremonies: 30 min for the demo (together with Galois for the Test Lab), 1h for the backlog grooming and sprint planning (on Thursday)
    • First sprint would start on Thursday, March 24
    • First demo would be on Thursday, April 07

  • Target features for the MVP would be:

    • Create a snapshot from the Cardano node
    • Create a certificate for a snapshot
    • Store a certificate and a snapshot (centralized on a server)
    • Verify a certificate and download a snapshot
    • Bootstrap a Cardano node from a downloaded & verified snapshot

Mithril architecture meeting

Questions:

  1. what is the thing we want to certify? => that's the thing a node would need to bootstrap
    • point on the chain + corresponding ledger state at some epoch boundary
    • node.db does contain stuff we don't need -> volatile db
    • it contains the blocks up to some point + ledger state

Right now node assumes it has all blocks => can't run without previous blocks

  • In principle, it could "forget" past of the chain

Need to select parts of the node.db:

  • Node can already take a snapshot whenever we like -> current policy is snapshot 10K blocks, could be adapted Takes in memory state in a non-blocking way and serialises it to disk: contains only the ledger state ledger-state directory in the node.db
  • ledger state is internal format but block format is not block format is guaranteed to be portable

SPOs could post their signatures on the chain -> follow the chain to produce the

TODO:

  • look at consensus code (LedgerDB) for the snapshot policy, need to understand what are the data structures stored on need

  • details on how to make sure what does the node need to set up faster

  • use Mithril certificate to sign just segment from the previous certificate -> could download/verify segments in parallel

  • Sign point = (slot nr, hash) -> enough to ensure we have the valid chain

  • include list of hashes for all epoch files?

  • Index files can be reconstructed from the epoch files -> needed for fast bootstrap

  • tradeoff between speed and robustness

  • sign db immutable files independently from the ledger state

Q: epoch files?

  • Praos => large chunks (X nr of slots) per file
  • Need to check alignment on the epoch boundary -> https://hydra.iohk.io/build/13272534/download/1/report.pdf We could use a 10K or whatever limit as long as it includes the epoch boundary -> it's ok to have more blocks worst case = epoch boundary + 10k ?

2022-03-21

First Session with JP, some Q&A:

  • What's the role of the TestLab?
  • What about the repository's organisation? It feels a bit messy right now, with lot of inconsistencies here andther
    • JP should feel free to reorganise the code to be cleaner

Work from past week:

  • Understanding how it works by Looking at the go node, it's interesting to see a first stab at the node, even though it's definitely not suited for production and there are quite a few problems
  • Developed a simple CLI utility in rust-demo to be an easy to use executable for mithril primitives
    • currently, it can "run" mithril protocol with several parties, replicating what's done in the integration tests but more configurable
    • Could be the basis for a more elaborate tool to be used by SPOs without needing a full blown network

Big question: What's a "signable state" from a cardano-node?

  • The node's DB might not be identical between nodes even at the same block no.
  • How do we ensure all potential signers sign the same "thing"?
  • We need to add stake distribution to the signed "thing" as this is part of the validity of the certificate?

How does bootstrapping would work:

  • Sign certificate with genesis keys at epoch X => Then parties can produce signature and certificates starting from epoch X + 1
  • Issuing a new genesis certificate could be a "version boundary". A new genesis certificate is issued with metadata for versions, that makes it clearer when protocol changes, nodes get upgraded...

What if nodes come and go?

  • Mithril nodes need Stake + Keys to produce signatures => Nodes that come within an epoch boundary need to wait next epoch
  • Each epoch boundary the parameters are recomputed and redistributed to all participants

2022-03-18

TestLab Demo

  • Had a first demo integrating the monitor framework with a primitive rust-node: The test lab spins up a number of nodes (actual processes), connects them and intercepts all messages between them, and checks basic properties.
    • Tried verifying a simple fault injection during registration process
    • We noted it's perfectly fine to "extend" the node's API in order to make testing easier, esp. as there aren't any node right now, as long as we take care of not breaking the "black box" abstraction: The test lab should talk to the node through an API that regular clients could make sense of
  • We decided to postpone next meeting till 2022-03-28 in order for everyone to have time to regroup, and think about what the next steps would look like

2022-03-10

On-boarding JP on Mithril

  • Intros
  • Explanation of Mithril
    • details about the certificate validation -> chain de validation
    • We should be able to run some program representing the protocol working -> Rust CLI program?
    • local registration -> register all stake owners, all possible lottery winners
    • signatures need to be broadcast or sent to aggregator -> could be anyone
    • Q.: Certificates can be different?
    • Test Lab
  • Q&A
    • Incentives? paying for snapshot is valid only for large amount of sync
  • Plan next days/weeks
    • Codebase
    • Goal: Opensource repo end of march
    • Goal: Write CLI simulating Mithril
    • Have a recurring meeting -> 2hours block 3 times a day
    • have an @iohk.io address -> Roy Nakakawa

Mithril Product Catch-up

  • Talking to Exchanges about Mithril

    • Vitor talking to them abouyt Scientia
    • Discussing problems about node bootstrap (run db-sync, lot of issues), exploring solutions
    • Hard to find "friendly" exchanges

  • How about wallet providers?

  • Trying to talk to ops to be able to deploy a Mithril node on testnet/mainnet?

  • DApp developers could make use of a snapshot-enabled wallet

  • We could have a progressive strategy to increase %age of trust

    • Maybe be cautious for mainnet?

  • Start product development from the consumption side: How would people use the snapshots/certificates, generate the certiciates "manually"

    • We don't need a full fledged mithril node and network to be able to produce signatures and certificates

  • Certificates are signed using specific keys

  • We need to link the stake keys with the mithril keys

  • Cold keys -> sign -> KES keys

    • Cold keys validates new KES every 2^6 epochs
    • KES blocks are put on-chain

Should not be too hard to certify Mithril signing keys https://docs.cardano.org/core-concepts/cardano-keys

2021

2021-12-01 - Weekly

Hello, Here comes agenda for today meeting:

  1. Mithril node status discussion:
    • Private network setup (WIP).
    • Resolve comments on github PR's (WIP).
    • Docker composer for Mithril node
    • Switch to GraphQL
    • Working on the Mithril client
    • Start performance tests
    • Implement github autotests
  2. Additional part-time engineer for the project to help Sasha with the project backlog.
  3. Misc

PR's comments:

  • ulonglong issue: Seems like it's a Linux specific problem? Replacing with ulong does not work on Mac OS => file an issue
  • DevOps engineer looking at Docker -> perhaps Monday?
  • launching a testnet -> https://github.com/input-output-hk/cardano-node/tree/master/cardano-testnet
  • launching network + docker-compose => next week
  • Help on spinning up a devnet network => write a script to do that and maybe inject transactions

TODOs:

2021-11-17 - Demo

  • 3 nodes for the demo, connect to each other directly using libp2p for connecting the nodes

  • DB stores the certificates -> can retrieve certificates from API with block number + MT root for UTXO + multisig

  • multisign = base64 of binary code from library, how about what happens in client?

    • provide a better representation for aggregate?
    • mithril has multisig + STMs -> the one that rely on the MT, contains the paths of the MT
    • we want to define the serialisation format but not structure it when we send it

  • not clear how the library works?

    • define criteria defining when the test should pass
    • explanation: signers will play a lottery, parameter defining how many signatures are required => there might be some messages where the parties don't win enough lottery
      • more parties with all parameters being equal => more succesful signatures
      • inigo working on benchmarks varying the parameters

  • next steps:

    • fix bugs -> file issues
    • what kind of benchmarks we need -> brainstorm
    • finish the API
    • node live check
    • aggregate process
    • write documentation
    • build instructions + CI
    • end-to-end test

  • actions:

    • (IV) file bugs
    • (IOG) Fix bugs
    • (IV) provide a PR with go code and build/run instructions
    • (IV) internal brainstorm on benchmarks
    • (both) reconvene on Mon/Tue to plan next sprint

2021-11-16 - Mithril Dev Q&A

  • deployment strategy? CI mandatory example testnet: https://github.com/input-output-hk/cardano-node/tree/master/scripts/byron-to-alonzo even better packaged: https://github.com/woofpool/cardano-private-testnet-setup => why I said we need to interact more...!
  • security? =>
  • sizing? => amount of Utxo
  • persisting keys? => Mithril keys for multisig keys can be serialised and exported but cannot be passed initialiser can be serialised with the initial keys provide an API to be able to update the keys? -> better to update the stake
  • process: load the initialiser if it exists, else generate new one
  • leave stake cahnge aside for the moment
  • getting errors from "not enough indices" phi_f = 1 => all indices are valid not enough indices is fine, we can tweak phi_f to reduce it
  • don't need to keep the individual signatures of each node around -> aggregated by the aggregator
  • how does the mithril client verifies a certificate? Clerk initialises all parameters verifier of aggregate verifies signatures wrt to Clerk
  • hardcoded elements? leadership? -> height of nodes? => we don't need a leader to start the signing the process the entity generating the certificate is not trusted => use
  • implementing MT => 1M Utxo = 1GB
  • now understanding the library and can start asking questions!

2021-11-10 - Parallel Epoch Validation

Parallel Epoch Validation: something Mithril does well without any engineering

  • we are verifying message according to some state => verifying several messages against several states

  • each message includes the next state state is very small : H(MT) + some parameters that don't change often

  • things are hard if we have a large state -> implies we need to attach large (n,n+1) state in each verification

  • parallelism should be in the node -> state used for verification is very small

  • useful for catching up with a chain of certificates -> we can download a whole bunch of certificates w/ proofs and verify all of them in parallel

  • we need to measure concretely!!

  • do we need the crypto library for verification? => probably yes

  • we could skip some part of the logic -> use wasm to ship it to light client

  • separate library in 2 parts, signer + verifier and only ship the verifier for light clients

2021-10-27 - Discussion w/ Cardano node

2 problems:

  • end-users have a hard time folllowing the chain
  • enterprise users need to resync db-sync every so often which requires hours/days
    • additional ops requirements
    • majority of people running db-sync don't care about history of chain -> make a transaction, get some level of certainty it got "validated"
    • "eventual history" is good enough
    • low-hanging fruit, 1st increment?
  • diff. between mithril node/client
    • mithril takes whole Utxo set -> do not solve hardware solutions for "light" clients?
  • Q. how long is too long?
    • days... => might be enough to sync up faster with db-sync
    • start w/ current Utxo snapshot then continue from some point (checkpointing)
    • bootstrap the node itself is hard
    • < 6 hours
  • real questoin: how fast can I get node and db-sync up and running to be able to make requests on them?
  • given an up-to-date node -> have a db-sync w/o history but w/ Utxo set
    • don't need to replay the full history to populate db-sync
    • requires change to db schema -> it does not store Utxo, only a way to replay them (history)
    • requires analysis about queries
  • what people want is mithril !
  • db-sync is just slow right now, not doing any validation, only populating data
  • could use mithril to get some state
  • majority of clients use combination of node + db-sync + graphql api
  • 2 modes of mithril:
    • get me a validated point -> I want to spend some point
    • get me a validated point + sync the chain in the background (takes hours to download history, particluar choice of DB, lookups instead of writes...)
  • mithril in and of itself is not very useful in the enterprise setting -> need more work to do to use the blessed Utxo state
  • one use-case: Exchange that use WBE could benefit directly from this -> put the Utxo
  • requirements on trust?
    • provide "blessed" checkpoints to exchanges as long as it's quick? -> don't know
  • why is it too long?
    • every offline day -> 50M loss for an exchange!
    • every upgrade -> resyncing the whole state
    • more a DevOps best practice problem -> we could provide solutions/advices on how to operate stuff
  • let's the PoC/demonstrator live its life until December
    • continue work on the Product side -> have high-level requirements/architecture by EOY and then do proper product design
  • why l2 solution wont' work
    • checkpoint approach is cool, great feature to add
    • difficulty : what do you sign? Need a ledger state that's useful -> hash(MT(Utxo))
    • every node has to maintain exact same representation -> not an impl detail anymore, part of the on-chain format -> standardize the format of the state
    • specification ---> implementation dependency
    • have to be part of the ledger's rules, enforced
    • is it so much work to do it on L1?

2021-10-05 - Mithril Product Q&A

Separate proof production, Mithril signatures of the state from the actual storage and delivery of UTXO Mithril lightwallet should be able to connect to any (trusted?) node

  • need not keep all the certificates -> only need to keep it to some depth

how about verifying proof on a mobile phone?

  • This is doable, a cert is about 1-200KB

we need initial trusted point / result of previous certificate

How to boostrap a node with mithril?

  • should we be signing past data depending on the stake of the past? But what about keys
  • key evolving signature -> key rotation over time => avoid an adversary tracking nodes and resigning past data
  • we need to be starting from some definite point in time (Eg. mithril hard-fork?)
  • there's some misconceptions floating around that mithril certificate should start at genesis, eg. retrofitting "block headers" in the past but it does not seem possible to sign past blocks using past stakes and signatures

2021-09-07 - Roman/Dmytrov

Introductions:

  • Arnaud is tech architect for Hydra
  • Roman worked for IOG for years, reasearch on consensus/ouroboros, work on midnight
  • Dmytro involved in the stablecoin project

Current situation:

  • Charles asked RO to find a team in Ukraine to implement Mithril node -> IdyllicVision, used to work w/ several types of ledger
  • Idyllic vision will implement network/storage, waiting for SoW
  • NDA is signed
  • AB produced some docs from the paper and discussions

Information needed:

  • API to get access to cardano ledger, to get the full chain, interface to cryptographic functions
  • Scorex and Mantis frameworks in Scala
  • Links to Haskell libraries, entrypoint to read the blockchain?
  • Impossible to build the UTXO Set outside of Haskell => need DB Sync to read the UTXO set from PG
  • Initially, Mithril node does not need to post TXs to the Cardano node => just read only access so no need to have an API to post txs

2021-09-07 - Mithril Protocol Details

Setup

  • group, signature scheme => can be fixed/static for each instance of the protocol

  • bulletproof -> requires reference string, something sufficiently random -> hash that

  • catalyst => uses hash of header block

  • If I select a weird seed string, people would be suspicious -> need to be impredictable

2 different proof systems:

  • concatenation
  • bulletproof : size much better, 5KB/epoch rather than 200KB, verification non trivial

Key registration

  • overlap with staking, could happen on-chain
    • initially there were thoughts of having mithril signatures be something slot leader does in slot generation

Operations

  • msg = hash of UtxO + hash of stake distribution for next round
  • index: k is threshold value, m is maximum index
    • m lotteries, need to "win" k out of m lotteries to produce a valid multisignature
    • example m = 2500, k = 300
      • every user genertaes a signature
      • does 300 trials, 1 .. 300 -> get ev (eligibility evaluation) for each index => you broadcast the signature(s) at the index for which ev < threshold
  • But we need k signatures to produce a certificate

Use case

  • new client shows up, mithrhil will help them get a new UTxO state
    • Client needs to verify all certificates up to some point
  • mithril is not useful for the producers except to agree on the next epoch's distribution

Details

frequency of signatures: one per epoch, more frequently might be more expensive

how to bootstrap? => does not make sense to retrofit, start from some point in time, some epoch

what's next:

  • submission to PKC in 10 days,
  • alternative proof system named Halo => might be relevant for mithril
  • group setting? signature scheme requires a pairing, proof system requies embedded curve => costly requirements => dense mapping for bulletproof quite unwieldy
  • find a smarter mapping

size of 1 signature: 1 group element, but we need the MT to verify the stake distribution => signer needs to provide the path in the MT alongside the signature

  • Bulletproof: use poseidon that behaves better with MT, MT are popular with ZK proofs researchers and users
  • Could optimise the size of proofs in concatenation scheme

musig is simpler: set of pubkeys is known

  • mithril adss the complication of verifying the stake which adds much more information to verify[<0;125;28M[<0;125;28m

2021-09-03 - Mithril Product Meeting

Agenda:

  • What can we do with Mithril?
  • Something has been moved forward with Galois
  • Coherent product scope and desired outcome

Discussion revolves mostly about https://input-output.atlassian.net/wiki/spaces/SPS/pages/3057713293/Technical+Architecture, with AB presenting the thing and people reacting/asking questions.

  • What about whales running node? => lower threshold
  • Rewriting to put it in the node => monetary incentive, better from an engineering and product perspective
    • We don't have the resources right now, esp. as lead architect for node is leaving
    • Be clear about what we expect and define a proper business case
  • We don't have a running distribution of the stakes -> issue a snapshot
  • Clients (and Mithril signers) wille need the UTxO set
  • Can I run this in a browser or a mobile, e.g for a light wallet?
    • What people need is a snapshot at some point in time, not live data
    • A node could provide a subset of UTxOs by signing the hash of those as part of a MerkleTree of known hashes

Clone this wiki locally