017-api-version-6-major-rewrite.md

Number	Title	Category	Status
17	API Version 6's major rewrite	Server	accepted

Context

Ogmios was created circa 2018, prior to the Shelley era, as a way to bridge Cardano to developers. One of Ogmios' main goal has been to maximise the developer experience and ease of interaction with the Cardano blockchain.

True to its core goal, its API hasn't changed much in the course of the parse years. However, having crossed all the Cardano eras, it has accumulated few technical debts. Ogmios often integrate eras and features ahead of time and often, before they properly settle in the ledger and the ecosystem. Hence, some names chosen initially now sound a little awkward (e.g. extra data hash instead of script integrity hash or, metadata vs auxiliary data).

In addition, the Byron era which was integrated first and well before the Shelley era came out still present some oddities. While most eras that followed Shelley were built upon Shelley and shared an extensive part of its source code, Byron has been seggregated out and is different. At the time, it wasn't clear that all eras following Shelley would be similar so we did not bother making Byron and Shelley too similar. Now however, Byron feels very much disconnected from other eras.

More, there were also regrettable decisions made from the start regarding choices of serialization in the API. In particular, the use of singleton object instead of discriminated unions made it hard down the line for client to parse data. This particularly the case for recursive structures such as timelocks scripts from the Allegra/Mary eras.

Finally, over time, Ogmios has tried to abstract away the complexity of the hard-fork combinator. While all mini-protocols in Cardano are era-dependent, Ogmios tries as much as possible to be era-independant. Yet, Ogmios would still return different responses based on era sometimes which create a strange duality between the input interface and the server's output. This era separation on the client side also adds complexity as it forces client to consider era differences even when it isn't needed (e.g. accessing a block header hash that is present in ALL eras shouldn't require a complex dance).

Decision

The outer API of Ogmios will be rewritten, aiming to solve the various quirks that have been identified since it was created. In particular, we will follow a few principles while doing so:

1. The API should maximise consistency.
   If two fields have same names, they should refer to the same entity. And vice-versa, two entities that are semantically equivalent should have the same name. Similarly, if a particular approach is used for structuring a certain object, it is reasonable to expect any object that are structurally equivalent to also be serialised in the same way.

2. The API should be as unambiguous and as self-explanatory as possible.
   This means, for example, avoiding acronyms such as tx, vk, blk and favor their complete alternatives transaction, verificationKey, block. The only two exceptions in the API to this are:

   • id (→ identifier);
   • vrf (→ verifiable random function);

because they are more often seen in these forms than in their full form. The new API shall also try to avoid ambiguity arising from context. If something is called hash or id, it should be nested in a parent object that removes any ambiguity. For example: { "transaction": { "id": "..." } }, { "header": { "hash": "..." } } .

3. The API should favor composability.
   By composability, we mean composability within the API itself. Thus, if a method of the API expects a an object X, and some other method returns an object X, it should be straightforward to lift the result as an input. On a concrete example, this means avoiding coumpound keys such as transactionId or headerHash and prefer them the form of a partial object (e.g. { "transaction": { "id": "..." } }). This also means that whenever possible, the API should try to align the representation of objects that share similar fields. This is particularly the case for blocks and transactions which are now represented in one common format, with optional fields.

Consequences

• The API is easier to parse and comprehend overall.

• The JSON-schema definition has been split into two. One file still lives in this repository, while the other has been moved to CardanoSolutions/cardanonical. The latter contains definitions of ALL data-types present in the Cardano blockchain for reuse across multiple services of the ecosystem.

• Every single client application is now broken :) ... but here is a (hopefully exhaustive) migration guide:

Migration guide

Tip

There are still many test vectors available for every element of the Ogmios API. Use them!

JSON-WSP → JSON-RPC 2.0

JSON-WSP has been ditched and replaced by JSON-RPC 2.0 with which Ogmios is now fully compatible. In particular, this means that request and response payloads are a bit more lightweight.

Old	New
{ "type": "jsonwsp/request", "version": "1.0", "servicename": "ogmios", "methodname": "query", "args": { "query": "genesisConfig" }, "mirror": { "id": "foo" } }	{ "jsonrpc": "2.0", "method": "queryNetwork/genesisConfiguration", "params": { "era": "shelley" }, "id": "foo" }

Note

Ogmios' implementation of JSON-RPC 2.0 is slightly more flexible as the specification w.r.t to the id field. While the specification indicates that this field should be a string, Ogmios will still accept anything as an id field (string, number, object, etc..). So it is essentially a drop-in replacement for mirror.

Similarly, Ogmios will always return the method as part of the response field as it often help with context and parsing.

Requests

All methods names have been adjusted as well. Here's a translation table (beware the casing, methods are now lowerCamelCase):

Old	New
RequestNext	nextBlock
FindIntersect	findIntersection
---	---
SubmitTx	submitTransaction
EvaluateTx	evaluateTransaction
---	---
Acquire	acquireLedgerState
Query	queryLedgerState/* queryNetwork/*
Release	releaseLedgerState
---	---
AwaitAcquire	acquireMempool
NextTx	nextTransaction
HasTx	hasTransaction
SizeAndCapacity	sizeOfMempool
ReleaseMempool	releaseMempool

4. Similarly, many ledger state queries have been renamed. Here's a recap table:

Old	New
currentEpoch	epoch
currentProtocolParameters	protocolParameters
delegationsAndRewards	rewardAccountSummaries
eraStart	eraStart
eraSummaries	eraSummaries
ledgerTip	tip
nonMyopicMemberRewards	projectedRewards
proposedProtocolParameters	proposedProtocolParameters
rewardsProvenance	N/A
rewardsProvenance'	rewardsProvenance
stakeDistribution	liveStakeDistribution
utxo	utxo

Also, some queries have been moved under queryNetwork and are always available, in any era:

Old	New
blockHeight	blockHeight
chainTip	tip
genesisConfig	genesisConfiguration
systemStart	startTime

Warning

The queryNetwork/genesis local-state-query now expects one era as argument (either 'byron', 'shelley' or 'alonzo') to retrieve the corresponding genesis configuration.

Responses

Query responses from the local-state-query protocol are now properly linked to their parent query. In prior version of ogmios, the response will simply have query as a method, not giving much information about which query was it a response for.

Request	Response
{ "jsonrpc": "2.0", "method": "queryLedgerState/tip", }	{ "jsonrpc": "2.0", "method": "queryLedgerState/tip", "result": { "slot": 1234, "id": "1234567890abcdef" } }

Errors

All protocol errors are now identified using unique error codes. Beyond the codes specific to JSON-RPC 2.0 (i.e. -32700, -32600, -32601, -32602, -32603), Ogmios returns errors in different ranges depending on the mini-protocol. Hence the range 1000-1999 is reserved to the chain synchronization, 2000-2999 for the ledger/network state queries, 3000-3999 for the transaction submission / evaluation and 4000-4999 for the mempool monitoring.

You don't have to worry too much about those ranges as it sufficient to know that each error has a unique code. The code can thus be used as a discriminant for parsing the error details, if any. With this, all the submission errors have been greatly reworked to provide extensive descriptions as 'message' and useful details when possible. In addition, Ogmios no longer returns a list of ledger errors. The list turned out to be often quite confusing as some error would trigger more error in cascade. To cope with this, Ogmios now has a built-in heuristic to figure out which error came first and is the most relevant to tackle next and thus, will only show one error at a time.

Block

The block model has also been reworked and merged together into one comprehensive block type. Fields have been renamed, some have been nested and other unnested. Here's a recap:

Old	New
hash	N/A (removed)
header.blockSize	size
header.blockHeight	height
header.slot	slot
header.blockHash	N/A (removed)
headerHash	id
header.previousHash	ancestor
header.opCert	issuer.operationalCertificate
header.issuerVk	issuer.verificationKey
header.issuerVrf	issuer.vrfVerificationKey
body	transactions
header.signature	N/A (removed)

Byron blocks are also now less "weird" than the rest of the blocks. So few changes concern only (old) Byron blocks to align them with the new model:

Old	New
header.genesisKey	issuer.verificationKey
header.prevHash	ancestor
header.signature.signature	N/A (removed)
header.signature.dlgCertificate.delegateVk	delegate.verificationKey
header.protocolVersion	protocol.version
header.protocolMagicId	protocol.magic
header.softwareVersion	protocol.software
header.proof	N/A (removed)
body.txPayload	transactions
body.dlgPayload	operationalCertificates
body.updatePayload	governanceAction

Transaction

The transaction model has been greatly reworked. The main changes are:

• The fields previously nested under body and witnesses have been flattened out and are now part of the top level object (e.g. inputs are no longer nested under body).

• Metadata now only contains user-defined labelled metadata instead of also containing extra scripts. Extra scripts have been moved to the scripts field and merged with witness scripts. The hash digests of those metadata-scripts are now listed as requiredExtraScripts. The naming of metadata is now also a bit less awkward as body → blob for accessing user-defined metadata is now simply labels.

• Few fields have been renamed, here's a recap:

  Old	New
  body.certificates	certificates
  body.collateralReturn	collateralReturn
  body.collaterals	collaterals
  body.fee	fee
  body.inputs	inputs
  body.mint	mint
  body.network	network
  body.outputs	outputs
  body.references	references
  body.requiredExtraSignatures	requiredExtraSignatories
  body.scriptIntegrityHash	scriptIntegrityHash
  body.totalCollateral	totalCollateral
  body.update	proposals
  body.validityInterval	validityInterval
  body.withdrawals	withdrawals
  body	N/A
  id	id
  inputSource	spends
  metadata	metadata
  raw	cbor
  witness.bootstrap	signatories1
  witness.datums	datums
  witness.redeemers	redeemers2
  witness.scripts	scripts
  witness.signatures	signatories1
  witness	N/A
  N/A	requiredExtraScripts
  N/A	votes

Tip

[1] The bootstrap and signatures fields have been merged under a single signatories since they were both Ed25519 signatures of the transaction body. However, a bootstrap signature contains some extra field needed to verify the signature. Hence the chainCode and addressAttributes are only present on bootstrap signatures (when spending from a Byron/Bootstrap address)."

Warning

[2] The format of the redeemers field has changed from an map to an array. Map keys have been turned into object, nested under a field validator.

Output references

Transaction's output references have been aligned to follow the new context-nesting strategy on Ogmios and avoid acronyms. Thus, there's no txId field anymore but instead a transaction and nested id fields. Similarly, the index is now scoped under output.

Old	New
txId	transaction.id
index	output.index

Protocol parameters

Starting from version 6, Ogmios has only one protocol parameter data model. Era-dependent models from Shelley, Alonzo and Babbage have been merged into one. Names have been made more uniform across eras. Consequently, some parameters have been made optional as they may only be present after a certain era. And some only exist in old eras. Yet overall, if something is meant to exist across all eras it will be present and has one single name.

Shelley

Old	New	Presence
decentralizationParameter	federatedBlockProductionRatio	Up to Babbage
desiredNumberOfPools	desiredNumberOfStakePools	All eras
extraEntropy	extraEntropy	Up to Babbage
maxBlockBodySize	maxBlockBodySize	All eras
maxBlockHeaderSize	maxBlockHeaderSize	All eras
maxTxSize	maxTransactionSize	All eras
minFeeCoefficient	minFeeCoefficient	All eras
minFeeConstant	minFeeConstant	All eras
minPoolCost	minStakePoolCost	All eras
minUtxoValue	minUtxoDepositConstant	All eras
monetaryExpansion	monetaryExpansion	All eras
poolDeposit	stakePoolDeposit	All eras
poolInfluence	stakePoolPledgeInfluence	All eras
poolRetirementEpochBound	stakePoolRetirementEpochBound	All eras
protocolVersion	version	All eras
stakeKeyDeposit	stakeCredentialDeposit	All eras
treasuryExpansion	treasuryExpansion	All eras

Alonzo

Old	New	Presence
coinsPerUtxoWord	minUtxoDepositCoefficient	All eras
collateralPercentage	collateralPercentage	Alonzo onwards
costModels	plutusCostModels	Alonzo onwards
maxCollateralInputs	maxCollateralInputs	Alonzo onwards
maxExecutionUnitsPerBlock	maxExecutionUnitsPerBlock	Alonzo onwards
maxExecutionUnitsPerTransaction	maxExecutionUnitsPerTransaction	Alonzo onwards
maxValueSize	maxValueSize	Alonzo onwards
prices	scriptExecutionPrices	Alonzo onwards

Babbage

Old	New	Presence
coinsPerUtxoByte	minUtxoDepositCoefficient	All eras

Stake pool parameters / Stake pools

The stake pools data model has been unified across the entire API. In particular, queries that used to return only a list of stake pool identifiers will now return a more complete list of stake pool objects, with the registered pool parameters. Parameters have therefore been lifted up to the stake pool model. Some have been slightly renamed too. So here's a translation table of the new 'StakePool' object:

Old	New
cost	cost
id	id
margin	margin
metadata	metadata
owners	owners
pledge	pledge
rewardAccount	rewardAccount
vrf	vrfVerificationKeyHash

Phase-1 (native) scripts

Great rework of the representation of phase-1 monetary scripts (a.k.a native script). They are now easier to parse thanks to a type discriminant clause at the fields level which indicates how the overall object should be interpret; rather than being at the key level. Below is shown how the old model translates to the new.

In addition, scripts (including Plutus scripts) are now wrapped in an object with three properties: language, json and cbor. The language is used as a discriminant for the whole script. The field json is only present for native scripts and contains the format detailed below. The field cbor contains the serialized representation of that script.

Old	New
"<credential-digest>"	{ "clause": "signature", "from": "<credential-digest>", }

Old	New
{ "all": [ "<native-script>" ] }	{ "clause": "all", "from": [ "<native-script>" ] }

Old	New
{ "any": [ "<native-script>" ] }	{ "clause": "any", "from": [ "<native-script>" ] }

Old	New
{ "NOf": { "<integer>": [ "<native-script>" ] } }	{ "clause": "some", "atLeast": "<integer>", "from": [ "<native-script>" ] }

Old	New
{ "expiresAt": "<slot>" }	{ "clause": "before", "slot": "<slot>" }

Old	New
{ "startsAt": "<slot>" }	{ "clause": "after", "slot": "<slot>" }

Certificate

A discriminant value field (type) has been introduced to all certificate to allow parsing them with more ease. Consequently, certificates are no longer prefixed with a discriminant key. For fields have also been renamed along the way.

Stake delegation

Old	New
{ "stakeDelegation": { "delegate": "<credential-digest>", "delegatee": "<stake-pool-id>" } }	{ "type": "stakeDelegation", "credential": "<credential-digest>", "stakePool": { "id": "<stake-pool-id>" } }

Stake credential registration

Old	New
{ "stakeKeyRegistration": "<credential-digest>" }	{ "type": "stakeCredentialRegistration", "credential": "<credential-digest>", }

Stake credential deregistration

Old	New
{ "stakeKeyDeregistration": "<credential-digest>" }	{ "type": "stakeCredentialDeregistration", "credential": "<credential-digest>", }

Stake pool registration

Old	New
{ "poolRegistration": "<stake-pool-parameters>" }	{ "type": "stakePoolRegistration", "stakePool": { "id": "<stake-pool-id>", "parameters": "<stake-pool-parameters>" } }

Stake pool retirement

Old	New
{ "poolRetirement": { "poolId": "<stake-pool-id>", "retirementEpoch": "<epoch>" } }	{ "type": "stakePoolRetirement", "stakePool": { "id": "<stake-pool-id>", "retirementEpoch": "<epoch>" } }

Genesis delegation

Old	New
{ "genesisDelegation": { "delegateKeyHash": "<credential-digest>", "verificationKeyHash": "<credential-digest>", "vrfVerificationKeyHash": "<vrf-digest>" } }	{ "type": "genesisDelegation", "issuer": { "verificationKeyHash": "<credential-digest>", "vrfVerificationKeyHash": "<vrf-digest>" }, "delegate": { "verificationKeyHash": "<credential-digest>" } }

Treasury transfers (a.k.a MIR certificates)

Treasury transfers have been converted into governance actions, so you'll now find them in the proposals field of transactions.

Value

The representation of Value has been changed to be more compact, more extensible and clearer. Values are now encoded as nested objects, where keys are respectively asset's policy id and asset name. Leaves are plain integers. The special case of Ada is encoded as a special policy id ada and asset name lovelace. This behavior is consistently applied to any amount that refers to a lovelace quantity. Transaction fees for example are now encoded as: { "ada": { "lovelace": 1234 } }.

Transaction's Metadata

The representation of transaction metadata has been both simplified and made more user-friendly, while remaining safe for more complex use-cases. In fact, many people in the community have grown to expect transaction metadata to be JSON objects. However, they aren't. Or more specifically, they aren't necessarily. There are actually plenty of transaction metadata on-chain that aren't representable as valid JSON. Prior to version 6, Ogmios would give a so-called detailed JSON schema representation of those metadata, by encoding the binary encoding as a JSON object. This has created a lot of confusion for rookie users not yet familiar with Cardano entrails who would be expecting a plain JSON object. Plus, the format was unpractical to parse for client down the line as it used object keys as type discriminant, leaving decoders no choice to try various encoding alternatively.

Starting from version 6, by default (see note below) Ogmios returns transaction metadata as JSON object when possible and fallback to CBOR otherwise. In fact, when metadata aren't representable as JSON object, this is probably because they are some elaborated binary encoding and users consuming them are most seemingly capable of decoding that themselves in the way they intended. Ogmios can be configured to always return the CBOR output using the --include-metadata-cbor flag on start.

To give a concrete example:

Old	New
{ "14": { "map": [ { "k": { "string": "foo" }, "v": { "int": 42 } }, { "k": { "string": "bar" }, "v": { "list": [ { "int": 1 }, { "int": 2 } ] } } ] } }	{ "14": { "cbor": "A263666F6F182A63626172820102", "json": { "foo": 42, "bar": [ 1, 2 ] }, } }

When it isn't possible to represent the metadata as a plain JSON object, the json field is simply omitted and the metadata is only provided as CBOR.

[!INFO]

The old behavior can be requested on-demand by enabling the --metadata-detailed-schema flag. When enabled, the json metadata will always be present and use the old declarative representation. This can be used in combination with the new --include-metadata-cbor flag as well.