Skip to content

Logbook

Jump to bottom
Jean-Philippe Raynaud edited this page Sep 5, 2023 · 1381 revisions

This page records a log of the team’s work, thinking, discussions, pains, joys, events, and experiences that happen on a daily basis. It’s supposed to be a kind of Stream of consciousness that can later be searched, analyzed, reviewed in more formal retrospectives.

Rules:

  • Log days in reverse chronological order, recent first

  • When the stream is getting too big, such that github refuses to render do the following:

    • extract biggest logical section (half-year or quarter) out of this into a correspondingly named Logbook-Year-Hn.md or Logbook-Year-Qn.md page
    • update links on top of this page to make it easier to navigate try to fix all links in the wiki at least
    • notify the team

  • Move (and link) recurring sections to relevant dedicated pages.

History

2023-H1

2022-H2

2022-H1

2021

September 2023

2023-09-05

2023-09-04

2023-09-01

August 2023

2023-08-31

2023-08-30

2023-08-29

2023-08-28

2023-08-25

  • Today we have worked on the following issues:
    • Use zstandard compression for snapshot archives#876: More work on the POC has been made, we explored ZStandard dictionary & multithreading (only the later seems to have value in our case) + we also added a "stream" scenario in order to keep the capacity to report the download/unpack progress.

The POC findings are summarized in the following file: https://github.com/dlachaumepalo/download-extract-poc/blob/main/perf.md

2023-08-24

2023-08-23

2023-08-22

Github actions are back to normal.

2023-08-21

Today was marked by massive slowdown of the upload-artifact and download-artifact that we use a lot in our CI pipelines, making our CI barely usable (ie: a workflow that usually take 5 minutes is at 7 hours of running time right as I'm writting those lines). The problem is github side, as other users have reported the same issue, so we can do nothing but wait.

  • Today we have worked on the following issues:

Also a PR [#1165] have been submitted by Yvan Sraka that should stabilize the hydra CI build (those have been flaky for some weeks). Many thanks to him for his time.

2023-08-18

  • Today we have worked on the following issues:
    • Refactor (de)serialization of crypto entities #668:
      • A PR for the verification key signature #1164 have been submitted, reviewed and merged.
      • A regression was identified on testing-preview after merging #1164, a fix have been made with #1166 but we need to monitor the environment to see if this indeed solve the issue.
      • Advancement has also be made for two other types (GenesisSecretKey and GenesisVerificationKey), a PR should be submitted next Monday. With those two types we will be at 8/13 types converted.
    • Simplify logs on the Signer/Aggregator #1106: The main work is finished, the related PR #1165 is awaiting review.

Also we merged a PR for the routine task of upgrading our dependencies (#1163).

2023-08-17

We also split the logbook, extracting the second half of 2022 into a new page, and extracting the first half of 2023 into a new page too.

2023-08-16

2023-08-11

2023-08-10

2023-08-09

2023-08-08

2023-08-07

2023-08-04

2023-08-03

2023-08-02

2023-08-01

July 2023

2023-07-31

2023-07-28

2023-07-27

2023-07-26

2023-07-25

2023-07-24

2023-07-21 🚀

  • Today, we have officially launched the Mithril Protocol’s Mainnet Beta 💪 🎉. The aggregator of the release-mainnet network is up and running and runs the newly releasedd 2329.0 distribution. The first step is to get signer registrations before we can genesis its certificate chain. This is likely to occur as early next week. We have defined an expected timeline for the network to start producing certificates (as explained in this dev blog post):

mainnet-launch-timeline

  • We have talked about some scenario that could happen on the mainnet, such as having signers registered but not signing 2 epochs later. Depending on the protocol parameters and their shares in the total stakes registered, this could prevent the network to sign properly (specially when the number of signers registered is low). We all agreed that this must be considered as an edge case that is part of the risk of running the protocol on the mainnet.However, if this occurs, we will probably have to re-genesis the chain and/or update protocol parameters. This lead us to decide that we will work on the redaction of specific maintenance procedure that will be compiled in a Runbook. Also, we will work on designing a mechanism that facilitates the retirement of a signer from the network in a fashion close to what is done on the Cardano network

  • Some SPO have reported a bug for which the issue Shared library error in CI binaries #1073 has been created. It appears that the problem can be easily fixed by embedding the openssl in the binary builds (with a configuration in the Cargo.toml). We will implement this fix and probably create a fix for the 2329 distribution

  • Additionally, we have worked on the following PRs:

  • Finally, we have verified if the KES keys of the SPOs of the Mithril networks should be rotated: this was not needed this week

2023-07-20

2023-07-19

  • Today, we have made two pairing sessions on the issue Refactor (de)serialization of crypto entities #668. We have tried 2 different approache:

    • Using a custom serde attribute with to use a specific for serializing a ProtocolVerificationKey as verification_key instead of its serialized string version
    • And another that wraps the native STM StmVerificationKeyPoP in the ProtocolVerificationKey type that would be more than just an alias
    • Both approaches have advantages and drawbacks: we will keep working in an exploratory mode to find the optimal approach that will be then implemented on the other concerned types
    • We have pushed our work on this ensemble/668/refacto_crypto_entities branch

  • We have also created the pre-release for the new distribution 2329.0-pre. It has been successfully deployed on the pre-release-preview network. This is also the release candidate for the first Mithril distribution that runs on Cardano mainnet that is scheduled to be launched in beta by end of week

  • Finally, we have worked on the following PRs:

2023-07-18

2023-07-17

  • We have spent most of the day pairing on the issue Design & implement basic stress test tool for aggregator#991:

    • We have consolidated the foundations of the stress tester in a separated binary directory
    • We have implemented custom arguments (with working default values)
    • The scenario execution is now cleaner and displays logs and loaders during execution
    • We have worked on a general wait function for calls on aggregator REST API as it a recurrent need in the scenario
    • We have discussed about how to compute the stake distribution for the generated signers: we have agreed that during the signer key registration preparation step, we will also generate the associated stake distribution in the expected JSON format. This stake distribution will be used as is by the mock cardano cli
    • We have created the draft PR Aggregator Stress Test #1062

  • We have also worked on:

    • Fixing new clippy warnings with the release of Rust 1.71.0 with the PR Upgrade Rust 1.71.0 #1060 which has been closed and merged
    • Preparing the mainnet infrastructure in the new GCP project. We will resume once we have enough rights to givemandatory roles to the service account

2023-07-13

2023-07-12

2023-07-11

2023-07-10

2023-07-06

2023-07-05

2023-07-04

2023-07-03

  • We have worked on the following issues:
    • testing-preview network does not create certificates #1015: after investigation, it appears that this problem that occurs sporadically is due to the current implementation of the verification key storer. We are working on a fix and a PR should be created tomorrow
    • Add infrastructure monitoring #987: the prometheus endpoint is now able to scrape metrics from Cardano nodes. We are currently implementing logs gathering with promtail and loki. The draft PR has been updated and it should be ready to be reviewed tomorrow
    • Remove certificate hash from Artifact #932: a PR has been created, but we need to do some refactoring on the message adapters to avoid using Serialize/DeSerialize traits when it is not mandatory. It should be ready for review tomorrow
    • Deploy mainnet infrastructure #988: we have started to prepare the new production project on GCP for the mainnet aggregator
    • We have also started preparing the next distribution: we have made a test on a dev network to make sure that the re-computation of the hashes of the certificate chain is working properly on real networks data. This is the case: this will avoid to re-genesis the certificate chain. We will continue preparing the distribution tomorrow

Clone this wiki locally