Skip to content

Logbook January 2026

Jump to bottom
paolino edited this page Jan 30, 2026 · 19 revisions

Tuesday 13th

RocksDB rollback support (PV)

Wednesday 14th

RocksDB rollback support (PV)

  • Refined the 'Update' abstraction that represents the database changes FSM
    • Work on Switch chainsync application to rocksdb backend #6
      • update type change
      • chainsync change
      • This required to add a Reset command to the ChainSync protocol as Rewind is taking [Point] while we actually need to reset to Origin
      • Now the state of the database is a tagged Update object where we specify if
        • we can proceed forward of backward or
        • we have to find an intersection among a set of valid points before rolling back to one of them
        • we have to reset to origin
      • Only rollback can really send the state to the reset state and intersection. This is not encoded in the types yet.

Thursday 15th

RocksDB full support (PV)

  • Work on PR: Switch chainsync application to rocksdb backend #6
    • Implemented the rollForwardFinality for rocks-db backend
      • Required a fix in csmt library to allow seeking column-type encoded keys
    • Introduced the Point type to containe hash and slot
    • Point type is the rocksdb slot. Maybe we should rename slot as point in the Update / State type ?
      • In memory is unsafe on that side and could rollback to points on a different fork. OTOH running in memory implies trust on the node and running once on one node.
    • Introduced an armageddon concept. This is breaking atomicity of the Update application. OTOH it should never happen in practice. It happens only if the node rolls back more than our inverse operation queue, which is dimensioned at 2160 blocks, covering the instability window.
      • Armageddon is when we have to reset to origin because we cannot find a common intersection point. It deletes all the database content, but is done in batches to avoid unbound memory usage to build the complete deletion in one transaction.
      • Because it happens only as a reaction to when the node request to rollback to Origin (and not straight at the time when it proposes an impossible rollback), it's also happening when we start with an empty inverse operation queue, which is fine.
    • Funny time with fibonacci when sampling the possible rollback points.

Friday 16th

RocksDB full support (PV)

  • Work on PR: Switch chainsync application to rocksdb backend #6
    • Generalized implementation to be independent from rocksdb backend
      • actually was already independent, just lower the capital letter in the types did it
    • cleanup the implementation by using a deeper monad stack
    • Added the implementation runner for the rocksdb backend, both update and query, based on the transactional interface.
    • Copy over the properties from the InMemorySpec to the RocksDBSpec
    • It's not needed for the specs themselves so I will make a common spec later
    • Prepare the spec runner for the rocksdb backend
    • Got some success but found out already a bad assumption that makes the specs unusable:
      • I assumed it was ok to use the inverse operation queue to identify both the tip and the finality point, but in reality I am not storing the finality when it is at origin (i.e. when the queue is empty), and because it's always computed as the last point in the queue, it results moved from Origin to the first slot after the first forward, which is wrong (it should move only when required). I will need to store the finality point separately or encode the elements in the queue differently, with origin as a special value. The second solution is more complex but cleaner. The empty database will then have a queue with one element, the Origin, representing both tip and finality at origin.

Monday 19th

RocksDB full support (PV)

  • Work on PR: Switch chainsync application to rocksdb backend #6
    • Fixed the finality point tracking by storing finality and tip as Origin at bootstrap
    • Implemented an Application module that took the logic from the in memory application module
      • Using it for the in-memory application would require some work to unlift the StateT ... to IO. The in-memory application can die as soon as the rocksdb is working ..
    • Implemented the rocksdb run module that provides prisms nad context to run the application module
    • It runs! But it fails to restart from the tip. Plus there is no interface to query proofs and stuff yet.

Tuesday 20th

  • Work on PR: Switch chainsync application to rocksdb backend #6
    • Removed the in-memory implementation as rocksdb is now working
    • Fixed the restarting issue that was always triggering the armageddon. The kv interface is not perfect: we cannot set the cursor over the last key, so iterating with prev-key is not able to pick up the last entry. The solution for now is picking up the last entry and then going into scanning.
    • Added logging support, to file or to stdout
    • Investigated the space leak more, not fixed yet.
    • Simplified the block-fetch module by removing the IORef around the points and breaking up the big functions.

Wednesday 21st

Thursday 22nd

  • Merged PR Add HTTP service with merkle-roots endpoint #13
    • implemented the HTTP server with servant
    • added the merkle-roots endpoint
    • added the swagger UI on a separate port
    • added the swagger UI to the documentation and CI swagger UI
    • tried both copilot agent with github PRs, interesting but not very interactive
    • I decided to evaluate Claude Code pro for 1 year, the pro plan seems not enough for me, but let's see how it goes
      • today's PR was written with Claude assistance, I must say it was quite helpful in writing the servant boilerplate and the swagger integration. Seeing the mistake it made I have hard time trusting it to write any businnes logic, but for boilerplate it's quite good.

Friday 23rd

Saturday 24th

Base checkpoint and code quality (PV)

Sunday 25th

Mithril integration for fast bootstrapping (PV)

Monday 26th

Mithril streaming and bootstrap improvements (PV)

Tuesday 27th

API sync gating and streaming improvements (PV)

Wednesday 28th

Documentation and planning (PV)

haskell-csmt dependency bump (PV)

Executable cleanup (PV)

Test coverage (PV)

Bootstrap progress metrics (PV)

  • Opened PR feat: expose bootstrap progress metrics via HTTP API #66
    • Adds bootstrap phase tracking: Downloading, Counting, Extracting, SyncingHeaders, Synced
    • Adds extraction progress with current count, total, percentage, and rate
    • Adds header sync progress with current/target slots
    • Metrics available via existing /metrics endpoint
    • Work in progress: metricsTracer fold not accumulating events correctly

New issues (PV)

Thursday 29th

Bootstrap progress metrics (PV)

New issues (PV)

Friday 30th

Config file support (PV)

Remove PartialHistory type (PV)

Node connection validation (PV)

Bootstrap recovery (PV)

Documentation (PV)

  • Merged PR docs: add bootstrap time estimates for all networks #80
    • Add bootstrap time estimates for preview, preprod, mainnet
    • Based on actual measurements: extraction ~1h for 2.9M UTxOs, header sync ~1.5h
    • Added detailed bootstrap design documentation to architecture.md
  • Merged PR docs: remove time estimates for non-mithril sync #81
    • Remove "Comparison with Full Sync" section
    • Document planned direct chain sync alternative (issue #58)
    • Added architecture.md: Bootstrapping section with double-buffer design
    • Added mithril-bootstrap.md: Alternative Direct Chain Sync section

Fix /ready endpoint (PV)

Ouroboros network question (PV)

New issues (PV)

Logbook 📓

Clone this wiki locally