RFC: adding a new TDP message for Pool/JDS custom job validation

[plebhash]

This discussion explores adding a Template Distribution Protocol (TDP) message that lets a Job Declarator Server (JDS) validate a Custom Job against a Template Provider (TP), so that the Pool side of the Job Declaration Protocol (JDP) no longer depends on mechanisms the spec leaves undefined.

It grew out of #217 and supersedes it.

Where the spec stands today

Section 6.1 says that "in order to fully implement the Server side of the Job Declaration Protocol, the JDS also needs to exchange RPCs (or similar) with a Bitcoin Node", and lists "maintaining an internal mempool (via RPCs (or similar) to a Bitcoin Node)" among the JDS responsibilities. The blue arrows in the diagram below are that loosely defined channel.

Two more JDS duties ride on the same undefined channel and are missing from the diagram: deciding whether a DeclareMiningJob gets a .Success or an .Error, and propagating the block it reconstructs from PushSolution (section 6.4.9).

None of this has a wire format in the spec, so every JDS defines its own. Meanwhile TDP, the protocol that exists precisely so a node can serve mining software without RPC, plays no part on the Pool side of job declaration.

Suggested design

JDS opens a TDP connection to a TP. When a DeclareMiningJob arrives, JDS forwards the declared job to the TP through a new TDP message:

ProposeTemplate

ProposeTemplate.Error tells JDS what is wrong (for example which transactions the node does not know, which JDS turns into a JDP ProvideMissingTransactions), and ProposeTemplate.Success is the gate for DeclareMiningJob.Success.

If ProposeTemplate.Success also carries a template_id, JDS can later relay a PushSolution as a plain TDP SubmitSolution, so block propagation on the Pool side also goes through TDP and the JDS no longer needs any direct link to a node.

Thread	Topic
1 · ProposeTemplate	what it carries, what .Success and .Error mean
2 · Solution propagation	template_id in ProposeTemplate.Success so JDS can send SubmitSolution
3 · A validation-only TDP connection	a SetupConnection flag that turns off the template stream
4 · Coinbase-only mode	whether ProposeTemplate has anything to validate when no tx data is revealed
5 · Spec and implementation fallout	what JDP and TDP text changes, what happens to bitcoin_core_sv2

Note: drafted with help from Claude Fable 5.1 with sv2-spec loaded into the context window, guided by SRI core contributor @plebhash

Note: this Discussion is organized in threads, which should evolve around specific topics. Please avoid starting a new thread if your comment is directly related to a specific topic that already has a dedicated thread. Otherwise there's no point in using Github Discussions and this is just a Github Issue.

[plebhash]

1 · The validation message (ProposeTemplate)

Context from the spec

DeclareMiningJob (6.4.4) carries version, coinbase_tx_prefix, coinbase_tx_suffix, wtxid_list and excess_data. The wtxid_list description says "JDS checks the list against its mempool and requests missing txs via ProvideMissingTransactions", and 6.4.5 says "JDS MAY request txdata via ProvideMissingTransactions before making this commitment". Nowhere does the spec say what "checking" means or what JDS must be satisfied of before sending DeclareMiningJob.Success.

What ProposeTemplate carries

The subset of DeclareMiningJob a node needs to assemble and check a candidate block: version, coinbase_tx_prefix and coinbase_tx_suffix, wtxid_list and excess_data, plus a request_id for pairing.

DeclareMiningJob has no prev_hash, so neither does ProposeTemplate. The TP validates the template against its own current tip and commits to that tip by returning its prev_hash in ProposeTemplate.Success. JDS keeps that prev_hash with the job, so PushSolution.prev_hash has something concrete to be checked against before the solution is relayed. A template that is valid on the TP's tip but was declared on a stale one is a race the fallback rules in 6.2 already cover.

What ProposeTemplate.Success returns

ProposeTemplate.Success asserts that the block assembled from the proposal is consensus-valid on the TP's tip except for proof-of-work. That assertion is what closes the attack 6.3.1 describes for Full-Template mode (a coinbase claiming more than the template earns), and it is what the 6.3.3 table attributes to JDS as "verifying against revealed tx data on behalf of Pool".

It carries two values: a template_id under which the TP now tracks the template, and the prev_hash it was validated against. The template_id is what lets JDS turn a later PushSolution into a TDP SubmitSolution for this exact template. Thread 2 covers that path and the tracking obligation it puts on the TP.

Missing transactions

A node can only validate transactions it has. Two shapes seem possible. In the first, ProposeTemplate.Error carries the positions of unknown wtxids (mirroring ProvideMissingTransactions.unknown_tx_position_list), JDS relays the request to JDC, and once ProvideMissingTransactions.Success arrives JDS sends the full transactions to the TP, either in a repeated ProposeTemplate with an optional full-tx field or in a companion message. In the second, ProposeTemplate always carries full transaction data, which defeats the point of declaring by wtxid. The first shape fits well with 6.4.8, which already requires the relayed transactions to be the exact TP serialization, so the JDS to TP leg can forward the same bytes untouched.

Open questions

• Should missing transactions be reported by ProposeTemplate.Error and supplied through a second ProposeTemplate, or through a dedicated pair of messages?
• Should ProposeTemplate.Error distinguish "unknown transactions" from "invalid block" so JDS knows whether to ask JDC for more data or to send DeclareMiningJob.Error?

[Sjors]

I would not shove missing transactions in ProposeTemplate.Error. The TP should request them with a proper message, and they should be provided with a proper message.

Conceptually this would reuse existing spec messages, but maybe there's some detail (different field) that requires a new, but similar, message.

This should fit quite naturally with the proposed TxCollection:

• mining: add TxCollection to bandwidth-efficiently validate external block templates bitcoin/bitcoin#35671

[Sjors]

Not a strong preference, but I think error messages about the block should go in ProposeTemplate.Error. When submitting transactions, if there's an error in e.g. serialization, duplication, size limits of those transactions, that should be reported in the [HereAreYourTransactions].Error. Any validation errors for transactions would still end up in ProposeTemplate.Error.

[Sjors]

There's potentially a long delay between ProposeTemplate and its .Error / .Success reply. Is that an existing pattern in the spec? It's worth considering if .Error should be limited to problems that can immediately be discovered, while having a dedicated message in which the TP announces a validation result.

[average-gary]

Agree with the field set and with no prev_hash. One consequence worth writing into the text: a job declared on a stale tip fails the TP's check on the BIP34 height push (bad-cb-height), so JDS needs no dedicated stale code.

I implemented the TP side twice to test the shape (rbitcoin's native TP, and sv2-tp #137).

Missing transactions. Agree with Sjors: proper messages, and I'd reuse the existing ones. 6.4.7 and 6.4.8 already have the right fields and direction: ProvideMissingTransactions { request_id, unknown_tx_position_list } is server → client, .Success { request_id, transaction_list } is client → server. Give them TDP message numbers (the spec already does this for SetNewPrevHash, 0x20 and 0x72) and keep names and layout. Then JDS relays byte-identical payloads in both directions, only the msg_type changes, and request_id pairs the whole exchange with the originating ProposeTemplate, the way 6.4.7 pairs it with DeclareMiningJob. The TP holds the pending proposal until the .Success arrives or a timeout; that is a TxCollection handle in Core terms, and a bounded pending set elsewhere. Both implementations currently repeat the whole ProposeTemplate with a transaction_list instead; I'll switch them once this settles.

On the delay question: this is the existing JDP pattern. DeclareMiningJob is answered by DeclareMiningJob.Success/.Error only after the ProvideMissingTransactions round trip and clients handle it already. ProposeTemplate mirrors that one level down, so I would not add a separate result message.

On where errors go: no strong preference either. If the provide message has no .Error (6.4.8 has none today), ProposeTemplate.Error with distinct codes covers both: duplicate-wtxid for the proposal, bad-missing-tx for a supplied transaction that doesn't hash to a requested wtxid, the node's reject reason for the block. If we want a ProvideMissingTransactions.Error, it would follow adding in JDP too, which is a bigger change.

Placeholder coinbase. To assemble the block the TP fills the extranonce gap with placeholder bytes of the right length. That length is only derivable if the extranonce is the tail of the scriptSig, i.e. coinbase_tx_suffix starts at nSequence: parse the prefix to the scriptSig length L, count the P scriptSig bytes present, fill L − P. SRI's JDS already assumes this when it rebuilds the coinbase for Core's checkBlock (sv2-apps#645), but JDP never says so. I'd state it in 6.4.4 as part of this change. The placeholder value is irrelevant: with merkle root and PoW skipped, every remaining coinbase check is independent of the extranonce bytes.

Rules I'd make normative, from the JDS hardening issues (#597/#610/#645/#795/#796) and both implementations: no duplicate wtxids; every supplied tx must hash to a requested wtxid; both checked before any decode, copy, or node lookup. The TP applies every consensus check it would apply to a received block except proof-of-work and merkle root (getblocktemplate proposal mode; Core's checkBlock(checkMerkleRoot=false, checkPow=false)) and MUST NOT reject on local policy. Concurrency is the TP's business, not the spec's: proposals are independent read-only checks and each .Success names the tip it was checked on, so a TP MAY validate them in parallel or queue them, and a client SHOULD apply its own timeout. The one thing that needs text is the pending state in the request/provide shape: a TP MUST be able to expire a proposal that is waiting for ProvideMissingTransactions.Success, and a late provide is answered ProposeTemplate.Error with unknown-request-id. (Operationally, on Core every checkBlock holds cs_main, so a TP on the node that also serves the Pool's templates delays its own block processing; that is a deployment note, not a protocol rule.)

One field I'd drop, one I'd add. excess_data has no TP-side use: the witness reserved value is already inside coinbase_tx_suffix, and 7.6 defines excess_data as Pool-side material; I'd drop it unless someone has a consumer. And I'd add fees: U64 to .Success: the sum of the declared transactions' fees as the TP computed them. The TP is the only party that can calculate it: fee = Σ inputs − Σ outputs needs the input values, which live in the UTXO set, and the node computes the total anyway to enforce bad-cb-amount. Without it the Pool only has the claimed amount from the coinbase, which bad-cb-amount makes a lower bound, not the value of the template it is being asked to pay for (sv2-apps#610). One caveat: Core's mining IPC doesn't return the total today (checkBlock gives reason/debug/result; the TxCollection.makeTemplate templates in bitcoin#35671 throw on getTxFees), so sv2-tp #137 sends 0 until it does. That's a one-field IPC addition I'm raising on #35671

Generated using Fable 5.1 and review/edited by human eyeballs. :)

[plebhash]

2 · Solution propagation through TDP

Context from the spec

Section 6.4.9 says that on PushSolution "JDS MUST attempt to reconstruct and propagate the block using the job data associated with its most recently sent DeclareMiningJob.Success", and that regardless of JDS-side propagation the block always reaches the network through JDC's SubmitSolution. How JDS propagates is unspecified. SRI JDS bypasses TDP and talks to a node directly, which is exactly the loosely defined mechanism this discussion wants to eliminate.

TDP's SubmitSolution (7.8) carries template_id, version, ntime, nonce and coinbase_tx. The intro to chapter 7 says "Template Providers MUST attempt to broadcast blocks which are mined using templates they provided, and thus MUST track the work which they provided to clients".

Suggested design

ProposeTemplate.Success carries a template_id. From then on the TP treats the validated job like any template it provided, including the tracking obligation. When PushSolution arrives, JDS rebuilds coinbase_tx from coinbase_tx_prefix, PushSolution.extranonce and coinbase_tx_suffix, copies version, ntime and nonce from PushSolution, and sends SubmitSolution with the template_id paired to the DeclareMiningJob.Success the solution corresponds to.

Friction with existing text

SubmitSolution.ntime "MUST be greater than or equal to the ntime_start field in the latest SetNewPrevHash message". A validation-only connection (thread 3) receives no SetNewPrevHash, so this sentence needs a second branch for templates created by ProposeTemplate. PushSolution carries prev_hash and nbits, which SubmitSolution does not; the TP already knows them from validation, so nothing is lost. The spec never says whether template_id is scoped per connection; with the TP minting ids for both NewTemplate and ProposeTemplate.Success, that should be stated.

Having a template_id per validated job also makes the "most recently sent DeclareMiningJob.Success" rule in 6.4.9 less load-bearing: JDS can key solutions to jobs instead of assuming the last one, which is what the current MAY clause about older jobs gestures at.

Open questions

• Should template_id be mandatory in ProposeTemplate.Success, or optional for TPs that validate but do not want to track?
• How long MUST a TP keep a template created through ProposeTemplate? The same rule as NewTemplate, or until the next ProposeTemplate on the connection?
• Does this change the "most recently sent DeclareMiningJob.Success" rule in 6.4.9?

[Sjors]

Should template_id be mandatory

Given the potentially long delay between ProposeTemplate and a .Success / .Error response, having an identifier seems useful for the JDS.

Also, if you go along with my preference above #239 (reply in thread) to request transactions with a dedicated message, the TP side also needs an identifier to known which submitted transactions need to go to which template.

[average-gary]

Agree template_id should be mandatory in .Success. On Sjors's routing point: request_id already pairs the pending exchange (the proposal, the TP's ProvideMissingTransactions, the JDS's .Success), exactly as 6.4.7 pairs them with DeclareMiningJob, so the TP doesn't need to mint anything before validation finishes. template_id arrives with .Success and from then on the job is tracked like any template. In both implementations a validated job becomes the same tracked object as a pushed template, so SubmitSolution against it needed no new code path.

On the other open questions:

• Retention. Not "until the next ProposeTemplate on the connection": a JDS multiplexes many JDCs over one TDP connection, so the latest .Success is not the only live job. The TP MUST retain a validated job under the same rules as its own templates (retired after the stale grace that follows a tip change; evictable oldest-first under a per-connection cap, which SHOULD be at least 64 jobs). JDS keys solutions by template_id and treats template-id-not-found as "JDC's own SubmitSolution path propagates".
• template_id scope. "Unique within a connection, opaque to the client" covers both implementations: rbitcoin mints per-session ids from the counter it shares with NewTemplate, sv2-tp uses a server-global counter.
• ntime. If the JDS connection receives no SetNewPrevHash (thread 3), 7.8's ntime_start sentence is simply not applicable to a proposed template; the node's own nTime checks run when the block is submitted. No need for an ntime_start in .Success.

Generated with Fable 5.1 and reviewed by the human steering the LLM. 👨🏻‍💻

[plebhash]

3 · A validation-only TDP connection

Context from the spec

The intro to chapter 7 says "after the initial common handshake, the client MUST immediately send a CoinbaseOutputConstraints message to indicate the space it requires for coinbase output addition, to which the server MUST immediately reply with the current best block template", and that thereafter the server pushes NewTemplate and SetNewPrevHash. Section 7.1 defines no flags. The JDP already has a precedent for a mode-selecting flag: DECLARE_TX_DATA in 6.4.1.

Suggested design

A SetupConnection.flags bit for TDP meaning that the client will not send CoinbaseOutputConstraints and does not want NewTemplate or SetNewPrevHash; the connection carries only ProposeTemplate, its responses and SubmitSolution. A TP that does not support it answers SetupConnection.Error with that bit in flags, which is the existing convention from section 3.3 for unsupported features, and lets a JDS detect an older TP before sending anything.

Trade-offs

A JDS might still want SetNewPrevHash even without templates: knowing the tip lets it reject a DeclareMiningJob built on a stale prev_hash without a round trip to the TP, and lets it detect that a pending validation became moot. That argues for two flags (no CoinbaseOutputConstraints and no NewTemplate, versus no SetNewPrevHash) or for keeping SetNewPrevHash in the validation-only mode. The counter-argument is that a tip change surfaces as ProposeTemplate.Error anyway.

Whether a TP supports ProposeTemplate at all is a separate question from whether it streams templates. A TP could stream templates and also validate. If validation support is not implied by the protocol version, it needs its own signal, either a flag in SetupConnection.Success or the existing unknown-message handling.

Open questions

• One flag or two? Does a validation-only connection still receive SetNewPrevHash?
• Is ProposeTemplate support a mandatory part of TDP once specified, or negotiated per connection?
• Does the "client MUST immediately send CoinbaseOutputConstraints" sentence need rewording regardless, or is the flag enough

[Sjors]

I would not introduce a feature flag for this. Instead, we could state that CoinbaseOutputConstraints MUST be sent before the TP sends NewTemplate messages.

That way a JDS can simply not send CoinbaseOutputConstraints.

It's not actually required for anything else, and I already found myself relaxing the constraint in practice, e.g.:

• Never drop SubmitSolution in the network layer  sv2-tp#120

[Sjors]

A JDS might still want SetNewPrevHash even without templates

We could make a breaking spec change to make template_id an optional U64. The TP would send it unconditionally.

The counter-argument is that a tip change surfaces as ProposeTemplate.Error anyway.

Yes, this is probably good (and quick) enough.

https://stratumprotocol.org/specification/07-template-distribution-protocol/#74-setnewprevhash-server---client

[average-gary]

Agree with Sjors on the stream side: no flag, just "CoinbaseOutputConstraints MUST be sent before the TP sends NewTemplate", and a JDS doesn't send it. Validation-only then falls out of the existing rules: no constraints → no NewTemplate → no SetNewPrevHash either, since 7.4 defines its template_id as one from a previous NewTemplate. The connection carries ProposeTemplate, its replies, and SubmitSolution, and every template_id the JDS holds came from a ProposeTemplate.Success. That is already how both implementations behave (sv2-tp #120 relaxed the ordering; rbitcoin builds nothing until constraints arrive), so the spec text would match that.

I'd still keep one bit for "does this TP validate?": REQUIRES_JOB_VALIDATION in SetupConnection.flags, echoed in SetupConnection.Success.flags when accepted, otherwise SetupConnection.Error / unsupported-feature-flags. Without it a JDS sending ProposeTemplate to an old TP can't distinguish "unsupported" from "slow", and the fallback rules in 6.2 need that distinction to fire quickly. It also lets a TP refuse rather than degrade: on Core the lookup needs getTransactionsByWitnessID (v32), so sv2-tp on a v31 node refuses the bit. Both test implementations do this today.

On SetNewPrevHash without templates: agree a tip change surfacing as ProposeTemplate.Error is good enough, so I would not make template_id optional for this. If someone later wants the hint, an optional template_id is the clean way; nothing in the proposal depends on it either way.

Fable 5.1 assisted and human revised.

[plebhash]

4 · Coinbase-only mode

Context from the spec

Under Coinbase-only mode (6.3.1) "JDC never reveals the tx data contained in the template", "the DeclareMiningJob message is never used", and proposed work is acknowledged by Pool via SetCustomMiningJob.Success. The spec openly accepts that this leaves Pool exposed to a coinbase claiming fee revenue the template does not have, treats it as a block-withholding-class problem, and points to ZK-based extensions as a possible mitigation.

What could be validated

In Coinbase-only mode there is no JDS-side validation to formalize, because there is nothing to validate against. The only party that could ask a TP anything is Pool, and the only material it has is what SetCustomMiningJob carries: version, prev_hash, ntime_start, nbits, the coinbase pieces and merkle_path. A TP could check that prev_hash is its current tip, that nbits and version match consensus, and that the coinbase is well-formed, but it cannot check the fee claim or that merkle_path commits to valid transactions. That is a real but thin benefit.

Why this matters for the message design

If ProposeTemplate is meant to also serve Coinbase-only, its transaction fields must be optional and the TDP client could be Pool rather than JDS, which changes what ProposeTemplate.Success can assert (the TP would be confirming header-level sanity, not block validity). If ProposeTemplate is Full-Template only, the message stays simpler and SetCustomMiningJob.Success remains the only gate in Coinbase-only mode, as today.

Open questions

• Is header-plus-coinbase sanity checking worth a message variant, or is it better left out of scope?
• If Coinbase-only is supported, who is the TDP client: Pool, JDS, or either?
• Should the 6.3.1 attack paragraph mention this as a partial mitigation if it is adopted?

[average-gary]

Full-Template only. Both implementations treat the tx fields as required. SetCustomMiningJob covers it, imho.

No Fable needed :)

[plebhash]

5 · Spec and implementation fallout

Spec text that would change

• 6.1: the "exchange RPCs (or similar) with a Bitcoin Node" sentence and the "maintaining an internal mempool" bullet become a TDP relationship with a TP; the TP's node is the mempool.
• 6.4.5: DeclareMiningJob.Success gains an explicit precondition (a ProposeTemplate.Success for the declared job).
• 6.4.7: ProvideMissingTransactions is triggered by ProposeTemplate.Error, rather than by an undefined mempool lookup.
• 6.4.9: PushSolution handling gains the SubmitSolution relay from thread 2.
• 6.3.3: the "ability to broadcast mined block" row for JDS now goes through TDP.
• 7.1: first TDP flag(s), thread 3.
• 7.x: new sections for ProposeTemplate, ProposeTemplate.Success, ProposeTemplate.Error, and 7.8 SubmitSolution gets the ntime branch from thread 2.
• 08-Message-Types: new ids after 0x76.

Implementation impact

• Node integration layers for SRI only need their template_distribution_protocol surface; a dedicated job_declaration_protocol one becomes redundant. In other words: bitcoin_core_sv2 wouldn't need a JDP module anymore.
• JDS implementations drop their node client and gain a TDP client, and stop requiring a co-located node.
• TP implementations (sv2-tp and any other) gain ProposeTemplate handling, which is essentially what getblocktemplate proposal mode already does inside the node.

Open questions

• Is "JDS MUST hold a ProposeTemplate.Success" the right strength, or should existing RPC-based validation remain tolerated?
• Does sv2-tp want to own this before the spec text lands, or the other way around?

[average-gary]

On the strength question: require ProposeTemplate.Success when the connected TP supports it; leave RPC-based validation as an unspecified transition rather than spec it.

rbitcoin implementation will support whatever the spec lands on.

From sjors comment below, it looks like sv2-tp can pick up these changes in tandem.

[Sjors]

0 · General Feedback

so that the Pool side of the Job Declaration Protocol (JDP) no longer depends on mechanisms the spec leaves undefined.

I agree it makes sense to spec out this important part of the protocol, rather than requiring implementers to study the JDS implementation.

Additionally, it makes it easier to use alternative node software with the JDS. Its direct use of the Bitcoin Core IPC, especially once bitcoin/bitcoin#35671 lands, is currently not (easily) portable.

As I mentioned in stratum-mining/sv2-tp#137 (comment) I'm happy to (review an) implement(ion of) these new messages in sv2-tp.

[average-gary]

The portability point is a strong motivation: rbitcoin has a native TP but no Core IPC, so without this a JDS can't use it at all. sv2-tp #137 is the Core-IPC version of the same messages and I'll keep it in draft until the field set settles and I'll bring anything that turns out to be missing from TxCollection to bitcoin#35671 as you suggested.

bitcoin/bitcoin#35671 (comment)