DataStore error while re-executing network transactions

[SantiagoPittella]

Seems like after #1521 the NTX builder started to fail to execute the increase counter network transactions.

To reproduce I started the node as usual:

$ miden-node bundled bootstrap --data-directory data --accounts-directory accounts --genesis-config-file ./simple.toml

$ miden-node bundled start --rpc.url http://0.0.0.0:57291 --data-directory data --enable-otel

And in another terminal:

$ miden-network-monitor start --rpc-url http://localhost:57291 --counter-increment-interval 10s

Then I get this logs:

2026-01-22T22:45:27.434246Z  INFO ntx.state.select_candidate: miden-ntx-builder: crates/ntx-builder/src/actor/account_state.rs:104: close, time.busy: 3.75µs, time.idle: 750ns
2026-01-22T22:45:27.435248Z ERROR rpc: tower_http::trace::on_failure: /Users/spittella/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tower-http-0.6.8/src/trace/on_failure.rs:93: response failed, classification: Code: 13, latency: 0 ms rpc.service: "store.NtxBuilder", rpc.method: "GetVaultAssetWitnesses", otel.name: "store.NtxBuilder/GetVaultAssetWitnesses"
2026-01-22T22:45:27.435264Z  INFO rpc: miden_node_utils::tracing::grpc: crates/utils/src/tracing/grpc.rs:21: close, time.busy: 39.5µs, time.idle: 11.3µs rpc.service: "store.NtxBuilder", rpc.method: "GetVaultAssetWitnesses", otel.name: "store.NtxBuilder/GetVaultAssetWitnesses"
2026-01-22T22:45:27.435353Z ERROR ntx.actor.execute_transactions:ntx.execute_transaction:ntx.execute_transaction.filter_notes:store.client.get_vault_asset_witnesses: miden-ntx-builder: crates/ntx-builder/src/store.rs:388: error: gRPC client error
2026-01-22T22:45:27.435366Z  INFO ntx.actor.execute_transactions:ntx.execute_transaction:ntx.execute_transaction.filter_notes:store.client.get_vault_asset_witnesses: miden-ntx-builder: crates/ntx-builder/src/store.rs:388: close, time.busy: 35.2µs, time.idle: 255µs
2026-01-22T22:45:27.435378Z ERROR ntx.actor.execute_transactions:ntx.execute_transaction:ntx.execute_transaction.filter_notes: miden-ntx-builder: crates/ntx-builder/src/actor/execute.rs:217: error: failed to filter notes
2026-01-22T22:45:27.435384Z  INFO ntx.actor.execute_transactions:ntx.execute_transaction:ntx.execute_transaction.filter_notes: miden-ntx-builder: crates/ntx-builder/src/actor/execute.rs:217: close, time.busy: 710µs, time.idle: 260µs
2026-01-22T22:45:27.435412Z ERROR ntx.actor.execute_transactions:ntx.execute_transaction: miden-ntx-builder: crates/ntx-builder/src/actor/execute.rs:145: error: failed to filter notes
2026-01-22T22:45:27.435419Z  INFO ntx.actor.execute_transactions:ntx.execute_transaction: miden-ntx-builder: crates/ntx-builder/src/actor/execute.rs:145: close, time.busy: 902µs, time.idle: 258µs
2026-01-22T22:45:27.435428Z ERROR ntx.actor.execute_transactions: miden_node_ntx_builder::actor: crates/ntx-builder/src/actor/mod.rs:313: network transaction failed, err: "failed to filter notes\ncaused by: transaction preparation failed: failed to fetch asset witnesses from the data store\ncaused by: failed to fetch asset witnesses from the data store\ncaused by: failed to get vault asset witnesses\ncaused by: gRPC client error\ncaused by: status: 'Internal error', self: \"root not found\", metadata: {\"content-type\": \"application/grpc\", \"date\": \"Thu, 22 Jan 2026 22:45:27 GMT\"}"

[bobbinth]

It seems like we are getting "root not found" error during the GetVaultAssetWitness request. Is this happening for new account?

[sergerad]

Yea I only see the corresponding btree map being inserted in InnerForest::update_account_vault() here.

Should InnerForest::get_vault_asset_witnesses() not return an error but an empty Vec in such cases?

[bobbinth]

I think this depends on what could be going wrong. I think there are a few possibilities:

1. vault_roots in InnerForest are not being updated correctly. For example, an account is created, but the vault root for this account is never inserted.
2. get_vault_root() in InnerForest is not working correctly. That is, we do have an entry there, say for (account_0, 4), and when we are trying to look up an entry for (account_0, 10), it says there is no root for some reason.
3. There is a legitimate reason for why the vault root is not inserted - though, off the top of my head, I don't know what such a reason could be. My assumption here that we should have vault roots inserted for all network accounts.

So, I guess the main question is which one of these issues we are facing.

[sergerad]

I am seeing that the inner forest's vault roots map is empty when the error is occurring

key query: (V0(AccountIdV0 { prefix: 14502093677457293392, suffix: 30850463768591360 }), 221)
roots: {}

[sergerad]

It appears that InnerForest::update_account_vault() is never being called for some reason. The delta vault must be empty here.

        if !delta.vault().is_empty() {
            self.update_account_vault(block_num, account_id, delta.vault(), is_full_state)?;
        }

[igamigo]

It appears that InnerForest::update_account_vault() is never being called for some reason. The delta vault must be empty here.

    if !delta.vault().is_empty() {
        self.update_account_vault(block_num, account_id, delta.vault(), is_full_state)?;
    }

Given that this is the network counter account from the network monitor, it very likely is empty (I think it just uses a storage slot for the counter). I think the issue is that this doesn't update for full deltas as well, and that the forest is not recording the empty root for the account (currently update_account_vault() bails for empty updates). I'm not sure if this is the better fix but seems to work:

diff --git a/crates/store/src/inner_forest/mod.rs b/crates/store/src/inner_forest/mod.rs
index 0a4bd00d..fca3b6fe 100644
--- a/crates/store/src/inner_forest/mod.rs
+++ b/crates/store/src/inner_forest/mod.rs
@@ -320,8 +320,9 @@ impl InnerForest {
     ) -> Result<(), InnerForestError> {
         let account_id = delta.id();
         let is_full_state = delta.is_full_state();
-
-        if !delta.vault().is_empty() {
+        // Update for new accounts or valid delta
+        if is_full_state || !delta.vault().is_empty() {
             self.update_account_vault(block_num, account_id, delta.vault(), is_full_state)?;
         }

@@ -408,14 +409,9 @@ impl InnerForest {
             entries.push((asset.vault_key().into(), value));
         }

-        if entries.is_empty() {
-            return Ok(());
-        }
-
-        let updated_root = self
-            .forest
-            .batch_insert(prev_root, entries.iter().copied())
-            .expect("forest insertion should succeed");
+        let new_root = if entries.is_empty() {
+            prev_root
+        } else {
+            self
+            .forest
+            .batch_insert(prev_root, entries.iter().copied())
+            .expect("forest insertion should succeed")
+        };

-         self.vault_roots.insert((account_id, block_num), updated_root);
+         self.vault_roots.insert((account_id, block_num), new_root);

Along the lines of what I mentioned in the other issue, investing in more tests would help surface these earlier.

[Mirko-von-Leipzig]

If I'm understanding correctly, this is because we use the same type and the same flow for a brand new account and an existing account's delta? I agree more tests would be good, but we can't cover all the combinations possible -- this should really be enforced by the type system, or at least hinted at.

[PhilippGackstatter]

this should really be enforced by the type system, or at least hinted at.

Something like this might help:

pub enum AccountDelta {
    Full(FullStateAccountDelta),
    Partial(PartialStateAccountDelta),
}

I'm not sure if we should do this at miden-protocol level because FullStateAccountDelta and PartialStateAccountDelta would (eventually) look identical afaict. So would it be sufficient for this to be a wrapper at the node level?

[Mirko-von-Leipzig]

this should really be enforced by the type system, or at least hinted at.

Something like this might help:

pub enum AccountDelta {
Full(FullStateAccountDelta),
Partial(PartialStateAccountDelta),
}

I'm not sure if we should do this at miden-protocol level because FullStateAccountDelta and PartialStateAccountDelta would (eventually) look identical afaict. So would it be sufficient for this to be a wrapper at the node level?

I think part of our problem is that Full is used as a proxy for New and that ideally we wouldn't re-use delta for both. I'm fairly sure that you've just done the opposite transition in protocol (e.g. remove New in favor of just using delta's for all of it).

This sort of follows my general feel for our types at the moment where things are bundled into the same "type" which require introspection via methods to understand what they are for our use cases in the node. e.g. note should be an enum and not some opaque type.

We'll figure out how we should represent things on our end; I don't think we can really afford to do this at the protocol level anymore.