-
Notifications
You must be signed in to change notification settings - Fork 52
Logbook
This page records a log of the team’s work, thinking, discussions, pains, joys, events, and experiences that happen on a daily basis. It’s supposed to be a kind of Stream of consciousness that can later be searched, analyzed, reviewed in more formal retrospectives.
Rules:
-
Log days in reverse chronological order, recent first
-
When the stream is getting too big, such that github refuses to render do the following:
- extract biggest logical section (half-year or quarter) out of this into a correspondingly named
Logbook-Year-Hn.mdorLogbook-Year-Qn.mdpage - update links on top of this page to make it easier to navigate try to fix all links in the wiki at least
- notify the team
- extract biggest logical section (half-year or quarter) out of this into a correspondingly named
-
Move (and link) recurring sections to relevant dedicated pages.
- Today we have worked on the following issues:
- Refactor (de)serialization of crypto entities #668: A PR have been submited and reviewed and merged.
- Errors refactoring #798: A PR has been submitted, reviewed and merged for the first task (making our common result and error alias of their anyhow equivalent).
We also splitted the logbook, extracting the second half of 2022 into a new page, and extracting the first half of 2023 into a new page too.
-
Today we have worked on the following issues:
- ECS-3156 Review glossary. Adjust capitalization and bolding. #1158 :the PR has been reviewed and merged.
- fnv/core verifier #939: the PR has been reviewed and merged.
- Download latest snapshot & stake distribution if latest is mentioned in Mithril Client commands #1110: the PR has been reviewed and merged.
- Refactor (de)serialization of crypto entities #668: Works have started, and good progress was made, for the Operational Certificates.
- Errors refactoring #798: Works have started, and good progress was made, on the first task (making our common result and error alias of their anyhow equivalent).
-
Also the following issue have been reported:
- Failed to restore recent snapshot with strange error #1160: investigation have started but this will probably be difficult to solve.
- Today we have worked on the following issues:
- Create explorer page with registered signers #1097: After some polishing and the addition of a Stake component the PR have been merged.
- fnv/core verifier #939: We provided some help on a issue with hashes of the aggregate signatures making some tests fails.
- ECS-3156 Review glossary. Adjust capitalization and bolding. #1158 :the PR has been created and is awaiting reviews.
-
Today we have worked on the following issues:
- Create Production Infrastructure Runbook #1085: the issue has been closed with the merge of the PR Add network production runbooks for Aggregator #1086
- Create explorer page with registered signers #1097: good progress have been made and a PR should be created tomorrow
- Design & implement stress test tool for aggregator - phase 1 #991: the issue has been closed
- Design & implement stress test tool for aggregator - phase 2 #1155: the issue has been created
- Update stm README - Core verifier #1156: the issue has been created
-
We have also worked on the demo preparation and the sprint planning preparation
-
Today, we have released the new
2331.1distribution, and deployed it to therelease-mainnetandrelease-preprodnetworks 💪 -
We have also worked on the following PRs:
- Upgrade explorer to next 13 #1149: the PR has been reviewed and merged. This also closes the bug issue The release-mainnet network does not show up in the explorer #1111
- Implement prometheus monitor for end to end tests #1139: the PR has been reviewed and merged
- Reorganize docs folder #1150: the PR has been reviewed and merged
- Better unpack error in client #1148: the PR has been reviewed and merged
- Add network production runbooks for Aggregator #1086: the PR will be ready to be reviewed and merged tomorrow
-
Additionally, we have created the following issues:
-
Finally, we have worked on the demo path: (https://miro.com/app/board/o9J_lKq5Szc=/?moveToWidget=3458764561204820777&cot=14)
- Mithril Mainnet Beta Release: figures and showcase
- Even Faster Bootstrap: showcase
- Aggregator Stress Test: design and showcase
-
Today we have created a new distribution pre-release
2331.1-pre. We have deployed it to thepre-release-previewnetwork, and it works as expected. We will release it tomorrow -
We have also worked on the following PRs:
- Add snapshot archive verification #1138: the PR has been reviewed and merged. It fixes the bug issue Client fails to unpack some snapshot archive #1137
- Clean ssh keys of infrastructure #1142: the PR has been reviewed and merged
- Implement prometheus monitor for end to end tests #1139: the PR is ready to be reviewed. It will be merged shortly
- Add network production runbooks for Aggregator #1086: the PR is in draft and will be ready to be reviewed shortly
- Better unpack error in client #1148: the PR is ready to be reviewed and will be merged shortly
- Documentation PRs:
- TW review of references.md #1134: the PR has been reviewed and merged
- TW review of: aggregator.md & signer.md #1145: the PR has been reviewed and merged
- ECS-3077 Apply style guide, tighten some sentences. #1130: the PR has been reviewed and merged
- ECS-3078 review certificates.md, the description of the certificate c… #1135: the PR has been reviewed and merged
- ECS-3079 Review simulation document, apply style guide, some suggesti… #1136: the PR has been reviewed and merged
- ECS-3111 Apply style guide, minor changes. #1143: the PR has been reviewed and merged
- ECS-3112 Review architecture.md, apply style guide. #1144: the PR has been reviewed and merged
- ECS-3115 Review client.md, apply IOG style guide. #1146: the PR has been reviewed and merged
-
Additionally, we have worked on the following issues:
- Create explorer page with registered signers#1097: a new PR for the updated explorer will be created shortly
- Stabilize snapshot archive production in aggregator#1140: we have groomed the issue
- Make mithril-client crate a library#1141: the issue was created and groomed
- Some end to end tests are flaky in the CI#1147: the issue has been created
-
Finally, we have discussed about the explorer for which many ideas have been raised lately. We think that we should probably use these ideas to design a v2. We will probably need support from UI designer for this
-
Today we have worked on the following PRs:
- Add snapshot archive verification #1138: the PR is ready to be reviewed and includes a fix for the bug Client fails to unpack some snapshot archive #1137
- Add network production runbooks for Aggregator #1086: the PR is almost ready to be reviewed, and should be merged shortly
- Implement prometheus monitor for end to end tests #1139: the draft PR has been created and will be ready for reviwe shortly
- ECS-3077 Apply style guide, tighten some sentences. #1130: the PR has been reviewed and will be merged shortly
- TW review of references.md #1134: the PR has been reviewed and will be merged shortly
- ECS-3078 review certificates.md, the description of the certificate c… #1135: the PR has been reviewed and will be merged shortly
- ECS-3079 Review simulation document, apply style guide, some suggesti… #1136: the PR has been reviewed and will be merged shortly
-
Also, we have created the issue Stabilize snapshot archive production in aggregator #1140 that will provide along term fix to the creation of corrupted snapshot archives
-
Today we have created the following issues:
-
Client fails to unpack some snapshot archive #1137: we have discovered a problem on the archives that are produced by the
release-mainnetnetwork. We have been investigating the problem: we are able to reproduce the problem locally and are trying to understand the origin of the corruption of the archive. We will keep investigating next week - Display a summary of signers before the full list #1133
-
Client fails to unpack some snapshot archive #1137: we have discovered a problem on the archives that are produced by the
-
We have also worked on the following PRs:
- Speedup node bootstrap after client download #1132: reviewed and merged
- ECS-3077 Apply style guide, tighten some sentences. #1130: ready to be reviewed
- TW review of references.md #1134: ready to be reviewed
- ECS-3078 review certificates.md, the description of the certificate c… #1135: ready to be reviewed
- ECS-3079 Review simulation document, apply style guide, some suggesti… #1136: weady to be reviewed
-
We had planned to create a new pre-release for the
2331distribution, but we decide to postpone it during the investigation of the bug with the corrupted archives. We will probably create this new pre-release early next week, which will not impact the expected date of release -
Additionally, we have groomed the issues:
-
Finally, we have checked if KES keys needed to be rotated on the test networks, which was not the case
-
Today we have created a new pre-relase distribution
2331.0-prethat we are currently testing and that we expect to be released early next week -
Additionally, we have worked on the following PRs:
- Ingest metrics/logs in Grafana #1128: the PR has been reviewed and merged, this closes issue Ingest mainnet metrics/logs in Grafana #1122
- Add signers registered route to aggregator #1129: the PR has been created, reviewed and merged 💪
- Speedup node bootstrap after client download #1132: the PR has been partially reviwed and should be merged shortly
- ECS-3077 Apply style guide, tighten some sentences. #1130: the PR has been created and partially reviewed. It should be merged shortly
-
We have also created the new issue Add a clean file to extracted database in client #1131: with this issue, the client will add a
cleanfile to the extracted database folder, and thus allow the Cardano node to startup much rapidly by skipping the immutable files validation (which is not mandatory). Early tests have shown a drastic improvement in the startup time from more than1hto less than5min🥳 💪 -
Finally, we have worked with the Hydra team to prepare a workshop where Mithril andHydar will be showcased
-
Today we have worked on the following PRs:
- Aggregator Stress Test - Phase 1 #1062: we have paired on the PR and merged it. It closes the phase 1 of the issue Design & implement stress test tool for aggregator #991
- Simplify epoch type by implementing Deref & TryInto #1126: the PR has been reviewed and merged
- Ingest metrics/logs in Grafana #1128: the PR is ready to be reviewed
-
We have also worked on the following issues:
- Create explorer page with registered signers #1097: we have groomed the issue, and we have also started implementing the new route of the aggregator, and we will attempt to include it in the next distribution
- Record statistics about downloaded snapshot in aggregator #1127: we have created and groomed the issue
- Simplify logs on the Signer/Aggregator #1106: we have groomed the issue
- Use zstandard compression for snapshot archives #876: we have groomed the issue
- Signer can't sign with Invalid signature for party error#1125: we have closed the issue
-
Today we have worked on the following PRs:
- Protocol key for Multi and Genesis signature #1124: we have partially paired on this PR which has been reviewed and merged
- Update README #1118: the PR has been reviewed and merged. This closes the epic Release mainnet Mithril network #918
- TW review of mithril-aggregator.md #1117: the PR has been reviewed and merged
- TW review of mithril-signer.md #1120: the PR has been reviewed and merged
- TW review of mithril-client.md #1121: the PR has been reviewed and merged
- Getting started: capitalization edits #1119: the PR has been reviewed and merged
- Update current documentation #1123: the PR has been reviewed and merged
- Simplify epoch type by implementing Deref & TryInto #1126: the PR has been created and is ready to be reviewed
-
We have also worked on the following issues:
-
release-preprod network stopped producing certificates #1114: a quick fix has been implemented , and a long term fix (Cardano node upgrade to
8.1.2) will be delivered with the next distribution -
Signer can't sign with Invalid signature for party error #1125: the issue was created following the report of a problem by a SPO on the
mainnet. The problem was identified and a fix has been implemented. We are waiting for a confirmation that the fix works with the next snapshots produced by the network. We have also crated a new Troubleshooting label for issues of this type that will help us build a knowledge base - Ingest mainnet metrics/logs in Grafana #1122: a PR will be created shortly
-
release-preprod network stopped producing certificates #1114: a quick fix has been implemented , and a long term fix (Cardano node upgrade to
-
As expected, the
release-mainnethas produced its first multi-signatures and certificates on themainnetyesterday 🎉 💪 🚀 !!! -
We have noticed a bug on the
release-preprodnetwork which is no longer producing certificates as detailed in this bug issue release-preprod network stopped producing certificates #1114. The problem has been mitigated and 2 certificates have been produced for the epoch 85. However, we are still investigating some troubles with the Cradano nodes not synchronizing properly -
We have worked on the following PRs:
- Aggregator Stress Test #1062: we have mob programmed on the PR and made good progress on signing immutable files and verifying that the certificate & artifacts are produced. We'll keep working on it this week during another session
- Factorize hardcoded keys #1109: the PR was reviewed and merged
- Update Mainnet Beta Launch dev blog post #1113: The PR has been reviewed and merged
- TW review of mithril-aggregator.md #1117: the PR is ready to be reviewed and will be merged shortly
-
Additionally, we have created the following issues:
-
Today we have worked on the following PRs:
- Upgrade Cardano node to 8.1.2 #1092: the PR has been reviewed and merged and closes the issue Upgrade Cardano node to 8.1.2 #1090
- Common protocol key type for serialization of Stm types #1108: we paired and merged this PR
- Factorize hardcoded keys #1109: we paired on this PR which is ready to be reviewed
- Download last snapshot if latest is mentioned as digest in Snapshot download command #1110: we paired on this PR which is still in draft
-
We have created an issue related to a minor bug of the explorer The release-mainnet network does not show up in the explorer #1111
-
We have also noticed soem flakiness in the CI end to end tests. We will investigate further next week
-
Additionally, We have also discussed about a new feature that would allow warm-up of the cache used by the signers to compute the hash of the immutable files. This could happen during the 2 epochs period during which the signer is not eligible yet to sign
-
We have also checked that no KES keys needed to be rotated on the test SPOs that we run on the test networks
-
Today, we have mainly paired on the issue Refactor (de)serialization of crypto entities #668 during 2 separate sessions:
- We have added the new
ProtocolSingleSignatureimplementation and replaced theHexEncodetype where needed - We have refactored the
typesmodule in thecrypto_helpermodule of themithril-commoncrate - We have started implementing a type factorization for elegantly wrapping the STM native types into new generic
ProtocolKey<T>types. This will avoid a lot of boilerplate code and a good framework for refactoring all the other types of this refactoring - The PR single signature #1107 has been reviewed and merged
- We will keep working on this ticket next week
- We have added the new
-
We have also worked on these issues:
- Enhance Run a Mithril Signer node (SPO) guide #1055: the PR Enhance Run a Mithril Signer as an SPO guide #1071 has been reviewed and merged
-
Simplify logs on the Signer/Aggregator #1106: the issue has been created and labelled with a new
UX/DevXlabel
-
Finally, we have prepared the sprint planning for the next iteration
-
Today we have worked on the following PRs:
- fix verification key (de)serialization #1088: the PR was reviewed and merged, and fixed a critical bug
- Aggregator Stress Test #1062: we paired on this PR and made good progress. We are now able to handle seamlessly 1000 signer registration in seconds. We'll keep
- TW review #1087: the PR was reviewed and merged
- Update Mainnet Beta Launch dev blog post #1084: the PR was reviewed and merged
- Enhance Mithril relay configuration #1089: the PR was reviewed and merged
- Remove legacy store adapters & fix prune of stake pool & epoch settings #1076: the PR was reviewed and merged
- Add sign sub-command in aggregator genesis command #1082: the PR was reviewed and merged
- Update dependencies #1093: the PR was reviewed and merged
- Enhance Run a Mithril Signer as an SPO guide #1071: the PR has been reviewed and is almost ready to be merged
- Upgrade Cardano node to 8.1.2 #1092: the PR has been reviewed and ready to be merged
-
We have also created the following issues:
- Upgrade Cardano node to 8.1.2 #1090
- Post deployment mainnet infrastructure #1091
- Make Multi Signer stateless in Aggregator #1094
- Provide progress information with --json option in Client #1095
- Expose Prometheus metrics for Mithril Signer/Relay/Aggregator #1096
- Create explorer page with registered signers #1097
- Design Signer Retirement mechanism #1098
- Document complete footprint of signer #1099
- Implement a resume download mechanism for client #1100
- Add a rollback workflow in CI for testing-preview #1101
- Get rid of unwrap/panic in database providers in aggregator #1102
- Add a non 404 status code on the aggregator endpoint #1103
- Resolve flaky test in the stm library on the CI #3853 #1104
-
Today, we have created the genesis certificate for the
release-mainnetnetwork 🥳 💪. We expect the first certificates to be produced at the end of the week -
We have closed the following issues:
-
And we have created the following issues:
-
Aggregator panics with new (de)serialization of ProtocolVerificationKey #1083: a critical bug that prevented
testing-previewaggregator from working - Create Production Infrastructure Runbook #1085
-
Aggregator panics with new (de)serialization of ProtocolVerificationKey #1083: a critical bug that prevented
-
Additionally, we have worked on the following PRs:
- Aggregator Stress Test #1062: we paired on this PR and fixed the problem that prevented the load tester to work for more than 50 signers. We have optimized and refactored the code to have a cleaner bootstrap sequence. Also we need to fix a bug that blocks signature registrations when more than 256 signers are used. We will keep working on it tomorrow
- Enhance Run a Mithril Signer as an SPO guide #1071: the PR is pending review
- Remove legacy store adapters & fix prune of stake pool & epoch settings #1076: the PR is ready to be merged
- Add sign sub-command in aggregator genesis command #1082: the PR is ready to be merged
- Update Mainnet Beta Launch dev blog post #1084: the PR is ready to be merged
- Add network production runbooks for Aggregator #1086: a draft PR with the runbook written for creating the genesis certificate in production
- TW review #1087: the PR is ready to be merged
-
fix verification key (de)serialization #1088: the PR that should fix the crashing aggregator on
testing-preview. Once this is merged, we will be able to merge all the other pending PRs - Enhance Mithril relay configuration #1089: the PR is ready to be reviewed
- Today we have worked on the following PRs:
- Aggregator Stress Test #1062: we have paired on this PR during the team session and we have completed 2 new steps in the stress test scenario
-
Ensemble/668/refacto crypto entities #1075: we have reviewed and merged the PR. Unfortunately, it created panics on the
testing-previewwhich is now down. We try to fix the problem in order to avoid a regenesis of the network 🔥 - Bundle openssl with released binaries #1079: the PR was reviewed and merged, and attempts to fix the bug issue Shared library error in CI binaries #1073. We will check with some SPO to confirm that the problem is fixed on their end
- Remove legacy store adapters & fix prune of stake pool & epoch settings #1076: the PR was reviewed and will be merged shortly
- Add sign sub-command in aggregator genesis command #1082: the PR is pending approval and will be merged shortly
- nix: improve caching strategy to share deps among artifacts. #1069: the PR was reviewed and merged
- Enhance Run a Mithril Signer as an SPO guide #1071: the PR is still in draft and we will keep working on it
- Today, we have officially launched the Mithril Protocol’s Mainnet Beta 💪 🎉. The aggregator of the
release-mainnetnetwork is up and running and runs the newly releasedd2329.0distribution. The first step is to get signer registrations before we can genesis its certificate chain. This is likely to occur as early next week. We have defined an expected timeline for the network to start producing certificates (as explained in this dev blog post):

-
We have talked about some scenario that could happen on the
mainnet, such as having signers registered but not signing 2 epochs later. Depending on the protocol parameters and their shares in the total stakes registered, this could prevent the network to sign properly (specially when the number of signers registered is low). We all agreed that this must be considered as an edge case that is part of the risk of running the protocol on themainnet.However, if this occurs, we will probably have to re-genesis the chain and/or update protocol parameters. This lead us to decide that we will work on the redaction of specific maintenance procedure that will be compiled in a Runbook. Also, we will work on designing a mechanism that facilitates the retirement of a signer from the network in a fashion close to what is done on the Cardano network -
Some SPO have reported a bug for which the issue Shared library error in CI binaries #1073 has been created. It appears that the problem can be easily fixed by embedding the openssl in the binary builds (with a configuration in the
Cargo.toml). We will implement this fix and probably create a fix for the2329distribution -
Additionally, we have worked on the following PRs:
- refacto crypto entities #1075: pairing on the PR that needs to be reviewed before being merged
- Remove legacy store adapters & fix prune of stake pool & epoch settings #1076: the PR was created and will be reviewed shortly
- Add release-mainnet configuration in documentation and explorer #1065: the PR was reviewed and merged
- Update current documentation #1072: the PR was reviewed and merged
- Add dev blog post for mainnet Beta launch #1074: the PR was reviewed and merged
-
Finally, we have verified if the KES keys of the SPOs of the Mithril networks should be rotated: this was not needed this week
-
Today we have paired on the following issues:
- Design & implement basic stress test tool for aggregator #991: we have made good progress and are now able to register signatures from simulated signers in the scenario implementation. We have noticed few problems with the number of signers (generation of the crypto materials) and with the handling of the signer registration that currently works only with a low runtime update frequency. We will keep working on this issue shortly
-
Refactor (de)serialization of crypto entities #668: we also have made good progress and are close to having a first running version of the PoC we are working on with the
ProtocolVerificationKeytype. We will keep working on it shortly
-
We have also worked on the following PRs:
- Fix crates versions not shown in release notes #1070: reviewed and merged
- Minor editing tweaks to 'Welcome' #1068: reviewed and merged
- Tech writing review #1067: reviewed and merged
- nix: improve caching strategy to share deps among artifacts. #1069: reviewed and merged
- Enhance Run a Mithril Signer as an SPO guide #1071: draft PR was created
-
Finally, we have monitored the
2329.0-prepre-release which is working as expected. The multi platform client test has been launched and was successful. We should be able to release it and open the betamainnetnetwork tomorrow 💪
-
Today, we have made two pairing sessions on the issue Refactor (de)serialization of crypto entities #668. We have tried 2 different approache:
- Using a custom
serdeattributewithto use a specific for serializing aProtocolVerificationKeyasverification_keyinstead of its serialized string version - And another that wraps the native STM
StmVerificationKeyPoPin theProtocolVerificationKeytype that would be more than just an alias - Both approaches have advantages and drawbacks: we will keep working in an exploratory mode to find the optimal approach that will be then implemented on the other concerned types
- We have pushed our work on this ensemble/668/refacto_crypto_entities branch
- Using a custom
-
We have also created the pre-release for the new distribution
2329.0-pre. It has been successfully deployed on thepre-release-previewnetwork. This is also the release candidate for the first Mithril distribution that runs on Cardanomainnetthat is scheduled to be launched in beta by end of week -
Finally, we have worked on the following PRs:
- Add mainnet verification keys & address in infrastructure #1063: the PR was reviewed and merged
- Add release-mainnet deployment to CD #1066: the PR was reviewed and merged
- Tech writing review #1064: the PR was reviewed and merged
-
Today, we have groomed the issue Refactor (de)serialization of crypto entities #668. We have decided to split it in 3 steps:
- Step 1: PoC on
ProtocolVerificationKeytype withJsonHex(de)serialization only - Step 2: PoC on
ProtocolVerificationKeytype withJsonHexandCborHex(de)serializations (if it does not introduce too many complications) - Step 3: Apply new (de)serialization to all other crypto types
- Step 1: PoC on
-
We have also paired on the issue Design & implement basic stress test tool for aggregator #991:
- Finalized step 1: with the dynamic creation of a valid stake distribution, and used it in the cardano cli mock
- Finalized step 4: with the successful signer registration of
100signers 💪 - Finalized step 5: with controlling the epoch from the mock cardano cli
- We have updated the draft PR Aggregator Stress Test #1062
- We will keep pairing on this issue in the following days
-
We have also worked on the followign PRs:
- Tech writing review #1061: the PR has been reviewed and merged, and the onboarding guide for SPO has been aligned with Cardano docs standards
- Add mainnet verification keys & address in infrastructure #1063: the PR is ready for review
- Add mainnet configuration in documentation #1065: the draft PR has been created and will be ready for review shortly
-
We have spent most of the day pairing on the issue Design & implement basic stress test tool for aggregator#991:
- We have consolidated the foundations of the stress tester in a separated binary directory
- We have implemented custom arguments (with working default values)
- The scenario execution is now cleaner and displays logs and loaders during execution
- We have worked on a general wait function for calls on aggregator REST API as it a recurrent need in the scenario
- We have discussed about how to compute the stake distribution for the generated signers: we have agreed that during the signer key registration preparation step, we will also generate the associated stake distribution in the expected JSON format. This stake distribution will be used as is by the mock cardano cli
- We have created the draft PR Aggregator Stress Test #1062
-
We have also worked on:
- Fixing new clippy warnings with the release of Rust
1.71.0with the PR Upgrade Rust 1.71.0 #1060 which has been closed and merged - Preparing the
mainnetinfrastructure in the new GCP project. We will resume once we have enough rights to givemandatory roles to the service account
- Fixing new clippy warnings with the release of Rust
-
Today we have mostly paired on the issue Design & implement basic stress test tool for aggregator#991:
- We have fixed some problems when trying to run the load tester on devs computers
- We have groomed the issue after the first exploratory phase, and we have defined the final expectations for the first version of the stress test
- We will resume implementing the tool next week
-
We have also worked on the following PRs:
- Aggregator refactoring: services and dependencies container #1057: the PR has been reviewed and merged, this closes the issue Refactor aggregator dependency injection and services #1058
- Add run-only option in mithril-end-to-end test #1052: the PR has been reviwed and merged. This closes the issue Add a run-only option in end to end test #1048
-
Add min libc version to distribution release note #1056: the PR has been reviewed and merged, and it adds a clear mention of the supported
glibcversions in the release notes in GitHub
-
Today, we have worked on these PRs:
- Add monitoring loki endpoint #1046: the PR has been closed and the loki endpoint has been added to the Grafan instance
- CI: Build linux binaries using ubuntu 20-04 and add vulnerabilities check #1043: the PR has been merged and this closes the issues Build static binaries in CI #874, Check vulnerabilities in CI #1037, SQLite compatibility in aggregator #837 and should also close the bug Debian package does not install cleanly on older ubuntu versions #834 💪
- Factorize crypto operation #1045: the PR has been reviewed and merged. This closes the issue Factorize protocol crypto operations #669
- Add SPO On-boarding Guide #1051: the PR has been reviewed and merged. The associated issue Prepare SPO on-boarding guide #1049 is now closed
- Add run-only option in mithril-end-to-end test #1052:we paired on this PR that will be merged shortly
-
We have also created the issue Enhance 'Run a Mithril Signer node (SPO)' guide #1055 in order to enhance the documentation of the setup of a signer node: we received feedback from some SPOs who were having troubles understanding the steps to accomplish with the setup
-
We have worked on the following PRs:
- fix flaky end-to-end tests #1050: the PR has been reviewed and merged and fixes the bug issue E2E tests are flaky in CI #1023 💪
- Add created_at field in Mithril Stake Distribution messages #1047: the PR has been reviewed and merged. It closes the issue Add created_at in Mithril Stake Distribution messages #1030
- CI: Build linux binaries using ubuntu 20-04 and add vulnerabilities check #1043: the PR has been reviewed and validated. It will be merged tomorrow
- Add monitoring loki endpoint #1046: the PR has been reviewed and is ready to be merged
- Factorize crypto operation #1045: the PR needs final review and will be merged shortly
- Add SPO On-boarding Guide #1051: the PR needs some fixes and will be ready to be merged tomorrow
-
The following issues have been created:
-
We have worked on the following PRs:
- check epoch when validating certificate chain #1033: the PR has been reviewed and merged. This closes the issue Aggregator does not detect certificate chain epoch gap #952
- Update Signer Deployment Models Firewall documentation #1042: the PR has been reviewed and merged. It closes the issues Add instructions to set firewall using iptables #1040 and Update ufw command to set firewall on Mithril Signer installation instructions #1041
- Add monitoring persistent storage #1039: the PR has been reviewed and merged, and it adds persistent storage for monitoring data
- Add monitoring loki endpoint #1046: a draft PR has been created in order to expose an external endpoint for gathering logs in Grafana
- CI: Build linux binaries using ubuntu 20-04 and add vulnerabilities check: this PR has been drafted and is related to the issues Build static binaries in CI #874 and Check vulnerabilities in CI #1037
-
We have also paired on the issue Factorize protocol crypto operations #669: we have created the PR Factorize crypto operation #1045 that will be reviewed and merged shortly 💪
-
We have worked on the following PRs:
- Fix QA issues #1034: reviewed and merged the PR. It closes the newly created issue Q&A followup fixes #1035
- Update dependencies #1036: reviewed and merged the PR. It is an attempt to fix some vulnerabilities that were triggered in GitHub dependabot alert. We have also created a new issue Check vulnerabilities in CI #1037 that will help us know what vulnerabilities are fixed inside a PR
- check epoch when validating certificate chain #1033: we have reviewed the PR which is almost ready to be merged (will be after fiew typos are fixed)
-
We have also team paired on the following issues:
- Factorize protocol crypto operations #669: we have almost finished implementing the single signer builder, after we completed tests on the multi signer builder. We have scheduled a last pairing session early next week
- Design & implement basic stress test tool for aggregator #991: we have completed a first load test on the signer registration with more than 100 signers. We will conitnue working on this PoC shortly
-
Finally, we have released the new
2327.0(https://github.com/input-output-hk/mithril/releases/edit/2327.0) release 🎉, and we have updated the current version of the documentation with the PR Update current documentation #1038 which has been merged
-
We have worked on the following PRs:
- Fix documentation #1032: this PR has been reviewed and merged, and it fixes some mistakes in the documentation (especially a configuration error of the new Mithril relay that was reported by a SPO)
- Fix failing end to end test in Hydra CI #1031: this PR fixes a test that consistently failed only on the nix CI. It has been reviewed and merged
- Remove certificate hash from Artifact #1027: this PR has been reviewed and merged, it closes the issue Remove certificate hash from Artifact #932
-
We have kept working on the issues:
- Factorize protocol crypto operations #669: we paired on the issue, a PR will be available shortly (commits have been published on the branch https://github.com/input-output-hk/mithril/tree/ensemble/669/factorize-crypto-operation)
- Aggregator does not detect certificate chain epoch gap #952: a PR will be created shortly
- Design & implement basic stress test tool for aggregator #991: we paired on this issue and we will keep pairing tomorrow. We are still in an exploratory mode. We have committed on the branch https://github.com/input-output-hk/mithril/tree/ensemble/991-stress-test-aggregator
-
Additionally, we have monitored the pre-release of the distribution
2327.0-pre: everything looks good, we will keep monitoring tomorrow and expect the release to occur by end of week) -
Finally, we have created the issue Add created_at in Mithril Stake Distribution messages #1030
-
We have worked on the following PRs:
- Add infrastructure monitoring #1026: reviewed and merged the PR, which closes the issue Add infrastructure monitoring #987
- Signer verification key discrepancy #1028: reviewed and merged the PR, which closes the bug issue testing-preview network does not create certificates #1015. We will keep monitoring that the problem does not occur again
- nix: increase cargo verbosity to avoid hitting max-silent-time #1029: reviewed and merged the PR that should fix the timeouts on some hydra ci builds when Rust compilation takes longer than usual
- Remove certificate hash from Artifact #1027: we have reviewed and worked on this PR. It should be merged shortly
-
We have also kept working on the issues:
- Factorize protocol crypto operations #669: we have continued pairing on the issue. We will continue shortly and create a PR shortly
- Aggregator does not detect certificate chain epoch gap #952: a PR will be created shortly
-
We have also created the new distribution pre-release
2327.0-prethat we expect to release by end of week
- We have worked on the following issues:
- testing-preview network does not create certificates #1015: after investigation, it appears that this problem that occurs sporadically is due to the current implementation of the verification key storer. We are working on a fix and a PR should be created tomorrow
- Add infrastructure monitoring #987: the prometheus endpoint is now able to scrape metrics from Cardano nodes. We are currently implementing logs gathering with promtail and loki. The draft PR has been updated and it should be ready to be reviewed tomorrow
- Remove certificate hash from Artifact #932: a PR has been created, but we need to do some refactoring on the message adapters to avoid using Serialize/DeSerialize traits when it is not mandatory. It should be ready for review tomorrow
-
Deploy mainnet infrastructure #988: we have started to prepare the new production project on GCP for the
mainnetaggregator - We have also started preparing the next distribution: we have made a test on a dev network to make sure that the re-computation of the hashes of the certificate chain is working properly on real networks data. This is the case: this will avoid to re-genesis the certificate chain. We will continue preparing the distribution tomorrow