The HardFork protocol combinator

[edsko]

This only introduces the protocol combinator, not yet the ledger
integration. We might still end up tweaking this a bit.

Co-authored-by: Rupert Horlick rupert.horlick@iohk.io

[edsko]

This rewind window might be too large (P2 might restrict the window, but if we have a snapshot, we could rollback anyway). Think about whether we could tighten this (related: dropSnapshotIfNeeded).

[edsko]

Will need rebasing on #1227 .

[edsko]

This still uses

  type ForkedBefore hdr :: Type
  type ForkedAfter  hdr :: Type

but actually the hard fork combinator should use a concrete type for blocks and for headers, which adds the necessary envelope. This isn't implemented yet, but we did discuss the binary format:

Binary format for the hard fork combinator

The binary format used for one ledger (say, Byron) should not have to take into
account the binary format used for another (say, Shelley). Such a requirement
would after all be inherently non-compositional.

Nonetheless, hard forks -- points in the block chain where we switch over from
one ledger to another -- will of course happen, and after they do happen,
blocks and headers that we receive over the network may come from before the
hard fork or after, and we must be able to distinguish between them.

One option is to hope that by a stroke of luck, the binary formats of the
various ledgers are such that we can unambiguously distinguish between them.
Since we use CBOR, there is actually some reason to believe that this might be
feasible, due to the built-in redundancy in CBOR (for example, tuple lengths).
However, a generic envelope that the hard fork combinator adds around the
the binary formats for the individual ledgers seems a lot cleaner. Such an
envelope could very simply consist of a tag indicating where we are in the
hard fork history.

Note that such an envelope must be anticipated from the start. We cannot
omit the envelope now, and only introduce it after the hard fork. Since the
envelope would change the binary format of what we send across the network,
introducing such an envelope ahead of the hard fork combinator would mean that
all nodes would have to switch binary format all at once (unless we can
piggy-back on some other kind of versioning).

Unfortunately, of course, the current Byron chain does not include this
envelope, and so it must remain a special case. However, this reduces the
problem of "can we distinguish the binary formats of these n ledgers" to
"can we distinguish a Byron block from something wrapped in the hard fork
envelope", which is a much easier problem, especially also because we can
design the hard fork envelope specifically with this requirement in mind.

Very concretely, the decoder for a Byron block is

fromCBORABlockOrBoundary
  :: EpochSlots -> Decoder s (ABlockOrBoundary ByteSpan)
fromCBORABlockOrBoundary epochSlots = do
  enforceSize "Block" 2
  fromCBOR @Word >>= \case
    0 -> ABOBBoundary <$> fromCBORABoundaryBlock
    1 -> ABOBBlock <$> fromCBORABlock epochSlots
    t -> cborError $ DecoderErrorUnknownTag "Block" (fromIntegral t)

and for a Byron header

fromCBORABlockOrBoundaryHdr :: CC.EpochSlots
                            -> Decoder s (ABlockOrBoundaryHdr ByteSpan)
fromCBORABlockOrBoundaryHdr epochSlots = do
    enforceSize "ABlockOrBoundaryHdr" 2
    fromCBOR @Word >>= \case
      0 -> ABOBBoundaryHdr <$> CC.fromCBORABoundaryHeader
      1 -> ABOBBlockHdr    <$> CC.fromCBORAHeader epochSlots
      t -> error $ "Unknown tag in encoded HeaderOrBoundary" <> show t

That is, both consist of a CBOR 2-tuple, with either 0 or 1 in the first
field and the actual payload in the second.
This means that for the hard fork envelope, we can use the same format: CBOR
2-tuple with a tag in the first field and the actual payload in the second. If
that tag is 0 or 1 we pass the whole thing to the Byron decoder, but if the tag
is higher than 1, we pass take just the payload and pass it to the appropriate
decoder.

This makes the hard fork envelope entirely backwards compatible with the Byron
block format, and Shelley as well future ledger designs don't need to worry
about any kind of envelope. The node itself will always run with the hard
fork combinator as the top-level protocol (even if, say, running a pure Shelley
node with no support for Byron), to allow for future hard forks -- and this
format is then both used at the network level as well as on-disk (indeed, we
very much want the on-disk representation to equal the network one so that we
can stream blocks without any processing whatsoever).

[edsko]

This design currently assumes the SlotNo of the hard fork is a constant (not ledger dependent). This may not be correct, and may depend on update proposals. This will require some generalizations; in particular, it means that the block chain time needs to be able to cope with updates to SlotLengths. To be continued.

[edsko]

We might have to rethink the EpochInfo, or at least, it's integration in the Shelley ledger. See #1557 (comment) and #1557 (comment) .

[edsko]

After discussing this with @dnadales , useful to be very explicit: the hard fork combinator _must be n-ary. Of course, it could be binary as long as we only have two forks, and then become tertiary, or whatever, but crucially, it should not be nested. If you started with Wrap '[B, S], say, and then wanted to go to Wrap '[B, Wrap '[S, G]], you'd change the wrapper for the existing Shelley chain, which wouldn't work.

[edsko]

Sketching out what the header should look like:

data HardForkHeader before after = 
    BeforeHeader  (Header before)
  | BoundaryHeader (HeaderHash before) (Header after)
  | AfterHeader (Header after)

The HeaderHash before in AfterHeader is necessary in order to be able to give a proper blockPrevHash instance: if the "after" ledger reports its genesis hash as the previous hash, we should instead report the hash of the final "before" ledger:

data HardForkHeaderHash before after =
    BeforeHeaderHash (HeaderHash before)
  | AfterHeaderHash (HeaderHash after)
 
type instance HeaderHash (HardForkHeader before after) = 
    HardForkHeaderHash before after

instance HasHeader (HardForkHeader before after) where
  blockPrevHash (BoundaryHeader finalBefore hdr) =
    case blockPrevHash hdr of
      BlockHash _ -> error "invariant violation"
      GenesisHash -> BlockHash (BeforeHeaderHash finalBefore)
  blockPrevHash (AfterHeader hdr) =
    case blockPrevHash hdr of
      GenesisHash -> error "invariant violation"
      BlockHash b -> BlockHash (AfterHeaderHash b)

(and exactly the same for blocks).

[edsko]

The Shelley ledger will need to be updated so that the prev hash it uses is something like this:

data PrevHash crypto
    -- | The first Shelley block 
    -- 
    -- For a Shelley only chain, the hash here would be the hash of the genesis config.
    -- For a Byron/Shelley chain, the hash here would be the hash of the final Byron block.
    Genesis !(HashHeader crypto)

   -- | Any subsequent block
  | Block !(HashHeader crypto)`

[edsko]

Change of plan re Shelley. See IntersectMBO/cardano-ledger#1266 (comment) .

[edsko]

Obsoleted by #2034.