0037 XLS-37d: Improved Concise Transaction Identifier (CTID)

[interc0der]

This spec has been moved to Pull Request #111.

Title:       Improved Concise Transaction Identifier (CTID)
Revision:    1 (2023-02-11)
Authors:     Richard Holland (XRPL-Labs, XRPLF)
             Ryan Molley (Crossmark)

This proposal replaces the original proposal for Concise Transaction Identifiers XLS-15d

Quickstart

If you are a developer and want to get started quickly with integrating CTID, please visit the quickstart repo.

Abstract

This standard provides a way to locate a validated transaction on any XRP Ledger Protocol Chain using its ledger sequence number, transaction index, and network ID rather than its transaction hash.

This identifier is only applicable for validated transactions. Non-validated or unsubmitted transactions cannot be identified using a CTID.

Specification

Format

CTIDs are composed of 16 uppercase hex nibbles, and always begin with a C.
The identifier is divided into three fields and a single lead-in nibble:

C [ XXXXXXX ] [ YYYY ] [ ZZZZ ]

Fields

Character Offset	Field	Size (bits)	Description
0	C	4	Lead-in
1-7	XXXXXXX	28	Ledger Index / Sequence Number
8-11	YYYY	16	Transaction Index
12-15	ZZZZ	16	Network ID

1. Background

1.1 Existing Transaction Identifiers

The XRP Ledger historically identifies ledgers and transactions (and other objects) using a namespace-biased 'SHA-512Half' hashing function, which results in a 64 hex nibble unique identifier.[1]

Since these hashes are derived from the contents of the data, each identifier is completely independent of consensus.

Example Transaction Hash (ID):

C4E284010276F8457C4BF96D0C534B7383087680C159F9B8C18D5EE876F7EFE7

1.2 An Alternative: Indexing

Ledgers and transactions can be identified by their sequenced position.

As new ledgers are validated on XRP Ledger Protocol Chains, they are assigned a sequence number, which is always the previous ledger sequence plus one. The first ledger sequence is the genesis ledger with a value of one.

Ledgers can therefore be uniquely identified by a ledger_index (sequence). [2] [3] The only limitation is that the ledger needs to be closed before a sequence number can be used for identification.

Example Ledger Hash:

F8A87917637D476E871D22A1376D7C129DAC9E25D45AD4B67D1E75EA4418654C

Example Ledger Index:

62084722

During consensus, all nodes on an XRP Ledger Chain will agree on the order of transactions within a given ledger. This unique sequence of transactions is referred to as the canonical order. [4] This means that transactions, like ledgers, can also be identified by their index. This index is present in the transaction metadata as TransactionIndex.[5]

Example:

...
    "TransactionIndex":7
    "TransactionResult":"tesSUCCESS"
  }
"hash":"53726E6D021522A97602004841A2A7477D1900BD348ADA4D4DF812B466E56454"
"ledger_index":62084722
"date":1615278490000
}

1.3 Motivation

The XRP Ledger is poised to become (or depending on the time of reading: has already become) an ecosystem of cooperatively interconnected XRPL Protocol Chains. It is imperative that users of these chains can efficiently locate a specific transaction on a specific chain. Therefore a network-aware transaction identifier is necessary.

2. Considerations

2.1 Bit Allocations

To future-proof CTID identifiers, the parameters and their sizes and lifespans are considered:

Field	Size (bits)	Limit (decimal)	Lifespan	Explanation
Ledger Index	28	268,435,455	34 years from genesis	This field would otherwise be 32 bits but for the C lead-in nibble. We feel the easily identified C is more useful than an extremely long lifespan.
Transaction Index	16	65,535	∞ / until there are more than 65,535 transactions per ledger	It is very unlikely there will be more than 65535 transactions per ledger in any XRPL Protocol Chain for a long time. If there are then those above this limit still exist but cannot be identified as CTID.
Network ID	16	65,535	∞ / until there are more than 65535 ports allowed in TCP	In XRPL Protocol Chains the Network ID should match the chosen peer port. Thus the natural limitation on Network ID is that of the TCP port (65536).

2.2 Extensible

A leading C provides easy human identification of the format, but also room for growth. If the number of closed ledgers on a particular chain exceeds 268,435,455, the leading C may be incremented to D, E and F allowing for an additional 100 years of use. It is not expected this will be necessary as another standard will likely replace it by this time. As of 2023, all CTIDs always start with a C and this will almost certainly be the case for at least the next 20 years.

2.3 Space Reduction and Savings

Improved Concise Transaction Identifiers use a quarter of the space when compared to transaction hashes. This can save considerable space in databases containing references to millions of transactions.

3. Specification

This section is enclosed in the header of the proposal.

4. Implementations

Diverse implementations (in multiple languages) with test cases and explanations are available at the quickstart repo.

Two different implementations for the Improved Concise Transaction Identifier (CTID) are presented in this XLS.

• Simple
• Advanced

The first is a simplified method which is intended for easier self-implementation and adoption. The second is a more robust version with type checking and error handling.

4.1 Simple

An example encoding routine in javascript follows:

const encodeCTID = (lgrIndex, txnIndex, networkId) => {
  return (
    ((BigInt(0xc0000000) + BigInt(lgrIndex)) << 32n) +
    (BigInt(txnIndex) << 16n) +
    BigInt(networkId)
  )
    .toString(16)
    .toUpperCase();
};

const decodeCTID = (ctid) => {
  if (typeof ctid == 'string') ctid = BigInt('0x' + ctid);
  return {
    lgrIndex: (ctid >> 32n) & ~0xc0000000n,
    txnIndex: (ctid >> 16n) & 0xffffn,
    networkId: ctid & 0xffffn,
  };
};

4.2 Advanced

See here for the source code.

Reference documentation is available here.

4.2.1 Getting Started

In an existing project (with package.json), install xls-37d with:

npm install xls-37d

or with yarn:

yarn add xls-37d

4.2.2 Encoding

An example encoding routine in typescript follows:

import xls37d from 'xls-37d';

const { ctid } = new xls37d.encode({
  networkId,
  lgrIndex,
  txnIndex,
});

4.2.3 Decoding

An example decoding routine in typescript follows:

import xls37d from 'xls-37d';

const { networkId, lgrIndex, txnIndex } = new xls37d.decode(ctid);

References

[1] https://xrpl.org/basic-data-types.html#hashes

[2] https://xrpl.org/basic-data-types.html#ledger-index

[3] https://xrpl.org/ledger-header.html

[4] https://xrpl.org/consensus.html

[5] https://xrpl.org/transaction-metadata.html

Changelog

2023-02-11

• Published draft for open discussion

[dangell7]

Can you give an example of how you see an explorer, a developer and a user using this standard?

How would we use this? Just some examples might help.

[interc0der]

Sure. You can think of these Improved Concise Transaction Identifiers (CTIM) as a replacement for transaction hashes. In most situations, where you are querying information about a validated transaction, you can use this identifier over a hash.

Maybe you are a front-end developer and you want to cache an array of validated transactions for a given wallet. You could store an unfiltered array of concise identifiers for any and all networks using the XRP Ledger Protocol Chain. Each identifier gives information about the validating network (chain) and ledger sequence. This would not be possible with transaction hashes.

Maybe you are an explorer, and you want to set up a map of ledger indexes with nested transactions. Your transaction query would be far most efficient if you know the ledger index. Sure, you may do this with a transaction hash ( decode hash and get sequence number ), but since these concise identifiers are not encoded using the contents of the transaction, they take up a quarter of the space as compared to the hashes.

Just some ideas. Hope this helps.

[dangell7]

Thank you,

This is only for the transaction hash right? Not the following hashs?

• Channel
• Escrow
• Offer
• NFTokenOffer
• NFTokenID
• URITokenID
• Address

[interc0der]

CTIMs would be valid for all transaction types such as NFTokenCreateOffer, PaymentChannelCreate, etc. However, concise transaction identifiers would not apply to things like Addresses, NFtokenId, ChannelId, etc.

Sure, the networkId could be added to the base transaction and I am not directly opposed to that. However, having a slim identifier holding only the ledger sequence, transaction id, and network id, seems like a valuable asset to all developers building on the ledger, especially as more sidechains and parallel networks are operating in tandem.

Whether this identifier sits on the ledger is still up for vote. ( I have not seen Richard's PR, but rather than something that sits on the ledger, its probably going to be a "calculated" value injected in API responses )

[interc0der]

I should also note:
A standard proposal (and PR) pertaining to networkId's is in the works. I would love to get your feedback about that standard proposal and how networkId's could be structured to enhance interoperability

[mvadari]

What does the M stand for in CTIM?

[RichardAH]

Improved like in VIM. ICTI didn't really roll off the tongue

[RichardAH]

Updated the name from CTIM -> CTID on @mvadari recommendation

[pkcs8]

All other artifacts (r-address, hashes, ledger index etc) in XRPL are network agnostic. Should CTIDs be network aware?

[AlexanderBuzz]

I am currently evaluating this CTID thing. Use case is transaction syncing in E-Commerce, where it may likely save some DB resources. Let's say someone uses the plugin in test mode and decides to switch to live, there would be transactions from Testnet and Mainnet in the table. So the requirement "get me the last ledger index and update from there" would result in two last ledger indexes in that case. Plus that it allows to use sidechains ad libitum.

[interc0der]

Exactly!
If you are using the ledger index ( sequence number ) and transaction index to identify transactions (rather than the transaction hash), there will be an overlap between parallel chains. For this reason, the CTID needs to be network aware.

If using a transaction hash, the network ID would be included in the hashed content, therefore making it unique and identifiable across parallel networks.

[AlexanderBuzz]

Hey, what about putting the network id at the first position? It would allow for indexing in databases (performance) and even fit the acronym, now being "CanonicalTransaction Identifier" - hehehe

[interc0der]

Could you give us an example of how you would index your database if the network_id was in the first position?
I would be curious to know what the performance benefits would be if indexed by network_id compared to ledger_index.

[AlexanderBuzz]

Ok, so the hands-on use case is someone paying with XRP in an online shop. The payment page polls the shop periodically to check wether the transaction is already settled. The Shop in turn fetches the latest transactions from the ledger. For performance reasons, and maybe they are just academic, it is desirable to get the $lastLedgerIndex like "snip". Currently I would have to save the network and the ledger index and index both. Would be something like SELECT MAX(ledger_index) from xrpl_tx WHERE network_id = 'testnet'. Having it all in one column might increase performance, but I would have to coroborrate that by tests. Is there a tool to create boarloads of XRPL testdata? If not, I create one, I need one in the future anyways.

For those that don't know, in relational databases like MySQL columns can get indexed, which means that internally those colums are stored in a pre-sorted way so that searching and joining besomes much faster that search operations in randomly distributed datasets

[cjcobb23]

Why not just add a chain id to the transaction itself, which would cause transactions on different chains to always have different hashes? This is what the various EVM chains do to prevent replaying transactions on different chains and to keep hashes unique.

[mvadari]

That's already a separate proposal: XRPLF/rippled#4370

[interc0der]

@dangell7 had a similar idea. See #91 (reply in thread) was my response.

Long story short, CTIDs need to be network-aware since they are not derived from the contents of the transaction, but rather the sequential order. Adding the Network ID to all transaction objects is independent of CTIDs

[dangell7]

@interc0der It was already in there when I said that 😂 lol

I did make this the other day though.

Its a global xrpl id. If you paste this into a search bar, you would be able to parse go directly to the tx or ledger object/entry. Kinda cool, but not sure anyone would use it lol

its long right now but when it uses the ctid it could be pretty short.

[AlexanderBuzz]

Stupid question perhaps, but how to I transfer a NetworkId like "testnet" to a int/hex value?

[AlexanderBuzz]

Um, not really yet. I could arbitrarily assign NetworkIds like I do now, but when they will get standardized in the future and those mappings are not correct, bunk goes the CTID. Perhaps this proposal should also include a mapping of the currently existing networks.

Which would take some consideration, as I could see there might be reserved spaces like in networking ports. For example, I will need to test at some point with a local rippled, and there might me test cases where more than one instance is needed, so a reserved space for 10-100 values to be used for local testing would be nice. all the possible XLS0815d's would also require some space, as they migh involve the creation of a new network for testing purposes etc.

[interc0der]

"In XRPL Protocol Chains the Network ID should match the chosen peer port. Thus the natural limitation on Network ID is that of the TCP port (65536)."

[AlexanderBuzz]

When I look here devnet / testnet /amm-devnet all have the same port :(

[mvadari]

First of all, that link is referring to the HTTPS and WebSocket ports, not the peer port. Second of all, you can run server_info on any network to find out its network ID - mainnet is 0, testnet is 1, and devnet is 2 I believe.

[AlexanderBuzz]

Thanks for clarifying this, I cobbled together peer port and network port and'm probably not the only one.

[nbougalis]

CTIDs are composed of 16 uppercase hex nibbles, and always begin with a C.
The identifier is divided into three fields and a single lead-in nibble:

C [ XXXXXXX ] [ YYYY ] [ ZZZZ ]

Here, X represents, Y represents and Z represents is the ledger sequence number, Y is the transaction ID and Z is the network ID.

I'd recommend changing the format to either ZZZZ-XXXXXXXX-YYYY which makes identifiers more "natural" for humans, or ZZZZ-YYYY-XXXXXXX which is likely to mismatch sooner, making distinguishing two separate CTIDs easier. I feel that human readability is an important consideration.

[AlexanderBuzz]

CanonicalTransactionIDentifier :)

[sappenin]

@RichardAH Do you mind pushing this spec along through new CONTRIBUTING process that was recently merged?
In particular, I think the next step here is to publish this spec as a DRAFT via a PR to this repo (read more about how to do that here).

One intention with the new process is that all specs would move through the process in a more overt fashion before code is merged into rippled develop branch, so I think this will have bearing on #4418.

Of course, this is a somewhat new process, so feedback welcome as we try it out in this repo (possibly as a PR to the CONTRIBUTING.md if you want).

Just today, I initiated this process for the CFT spec -- see it in action here.

[dangell7]

@sappenin The standard spec has been added to the PR's. Thank you

[tequdev]

[dangell7]

Thats documentation drift. We will get that updated. No flag

[tequdev]

I made a mistake in the comment place. sorry.

[dangell7]

Hello group: per the new XLS Contributing process, it is my opinion that we have reach a "well-refined standard."

As such, I propose that we move this discussion to a file (via #111) and work on final changes using additional PRs, for better change-tracking.

Please comment here if you would like to object to moving this spec/discussion forward in the process into a DRAFT spec.

The official PR in the rippled repository is XRPLF/rippled#4418.

(Note that per the Contributing guidelines, moving a spec into the DRAFT state does not mean any kind of endorsement, nor does it mean that this specification will become adopted. It is solely meant as a mechanism to enable better change tracking using PRs.)

[intelliot]

Closing this discussion in order to direct future comments to these PRs:

• Specification PR: XLS-37d: Improved Concise Transaction Identifier (CTID) #111
• Implementation PR: feat: support Concise Transaction Identifier (CTID) (XLS-37) rippled#4418