Error "missing utxo" using example .toml

[CatspersCoffee]

Hi,

running scrolls with the example .toml file (cutdown slightly) on mainnet for Filter type = "UtxoByAddress" gives error:

Note: this is from compiing from scratch with redis on standard localhost and port (6379). The docker image works as expected.

ERROR gasket::runtime] STAGE: reducers, WORK, PANIC: missing utxo: ba1950d0cf9e6a551215701f85a209a9a47fb64032efa14e21fcfbf37c125a9a#0

[2022-10-16T20:45:11Z INFO  scrolls::daemon] scrolls is running...
[2022-10-16T20:45:11Z INFO  scrolls::sources::n2n::transport] handshake output: Accepted(9, VersionData { network_magic: 764824073, initiator_and_responder_diffusion_mode: true })
[2022-10-16T20:45:11Z INFO  scrolls::sources::utils] no cursor found in storage plugin
[2022-10-16T20:45:11Z INFO  scrolls::sources::n2n::transport] handshake output: Accepted(9, VersionData { network_magic: 764824073, initiator_and_responder_diffusion_mode: true })
[2022-10-16T20:45:12Z WARN  scrolls::reducers::worker] rollback requested for (57867490, c491c5006192de2c55a95fb3544f60b96bd1665accaf2dfa2ab12fc7191f016b)
[2022-10-16T20:45:13Z ERROR gasket::runtime] STAGE: reducers, WORK, PANIC: missing utxo: ba1950d0cf9e6a551215701f85a209a9a47fb64032efa14e21fcfbf37c125a9a#0
[2022-10-16T20:45:14Z INFO  scrolls::daemon] Scrolls is stopping...
[2022-10-16T20:45:14Z WARN  scrolls::daemon] dismissing stage: n2n-headers with state Alive(Working)
[2022-10-16T20:45:14Z WARN  scrolls::daemon] dismissing stage: n2n-blocks with state Alive(Working)
[2022-10-16T20:45:14Z WARN  scrolls::daemon] dismissing stage: enrich-sled with state Alive(Working)
[2022-10-16T20:45:14Z WARN  scrolls::daemon] dismissing stage: reducers with state Alive(StandBy)
[2022-10-16T20:45:14Z WARN  scrolls::daemon] dismissing stage: redis with state Alive(Working)

.toml file:

[source]
type = "N2N"
address = "relays-new.cardano-mainnet.iohk.io:3001"

[enrich]
type = "Sled"
db_path = "/blah02/sled_db"

# enable the "UTXO by Address" collection
[[reducers]]
type = "UtxoByAddress"
# you can optionally prefix the keys in the collection
key_prefix = "c1"
# you can optionally only process UTXO from a set of predetermined addresses
filter = ["addr1qy8jecz3nal788f8t2zy6vj2l9ply3trpnkn2xuvv5rgu4m7y853av2nt8wc33agu3kuakvg0kaee0tfqhgelh2eeyyqg"]

[storage]
type = "Redis"
connection_params = "redis://127.0.0.1:6379"

[intersect]
type = "Point"
value = [57867490, "c491c5006192de2c55a95fb3544f60b96bd1665accaf2dfa2ab12fc7191f016b"]

[chain]
type = "Mainnet"

I have also tried a similar .toml file on preprod testnet and get the same error. Im either doing something stupidly wrong or missing something in the setup? could anyone advise?

[CatspersCoffee]

Turns out im missing the skip missing data

[policy]
missing_data = "Skip"

[CatspersCoffee]

Closing.

[scarmuega]

@CatspersCoffee yes, exactly, that's a way to avoid the panic and allow Scrolls to keep going.

Just to give a little context...

Some reducers in Scrolls require data from previous transactions. To allow those use-cases, there's a local DB that gets populated with UTxO as the chain gets crawled. If Scrolls starts from a particular point in the chain, other than origin, the past UTxO aren't available for the reducers requiring them, thus the error message.

If this is expected by the user, the [policy] configuration section provides a way to downgrade the error into a warning or directly omit the processing of that unit in the reducer:

[policy]
missing_data = "Skip" | "Warn"

[CatspersCoffee]

@scarmuega Thanks for the reply. I have a couple questions regarding the adding and/or removing of a UTXO smember from the redis DB. When a UTXO is added as an smember (per key basis) if the UTXO is consumed by a future transaction, the UTXO still stays in the DB. In "utxo_by_address.rs", in function process_consumed_txo() there is:

        let utxo = match utxo {
            Some(x) => x,
            None => return Ok(())
        };

But this always returns without matching MultiEraOutput, (or atleast this is what im getting in my testing) and execution never makes it to the CRDTCommand to set_remove and thus a consumed UTXO stays in the DB. Is this by design and if so, how to get a list of only available UTXOs at the current key?

Second question (related to the original question):

running form Origin with a blank DB

[[reducers]]
type = "UtxoByAddress"

[chain]
type = "PreProd"

[intersect]
type = "Origin"

I still get the Missing UTXO Error from the Origin and not a point intersect. Without using missing_data = "Skip" It seems like there is no way the reducer will continue?

output:

Running `/Scrolls/scrolls/target/debug/scrolls daemon --console plain --config ./utxo.toml`
[2022-10-18T17:29:13Z INFO  scrolls::daemon] scrolls is running...
[2022-10-18T17:29:28Z INFO  scrolls::sources::n2n::transport] handshake output: Accepted(10, VersionData { network_magic: 1, initiator_and_responder_diffusion_mode: false })
[2022-10-18T17:29:28Z INFO  scrolls::sources::utils] no cursor found in storage plugin
[2022-10-18T17:29:31Z INFO  scrolls::sources::n2n::transport] handshake output: Accepted(10, VersionData { network_magic: 1, initiator_and_responder_diffusion_mode: false })
[2022-10-18T17:29:31Z WARN  scrolls::reducers::worker] rollback requested for Origin
[2022-10-18T17:29:31Z INFO  scrolls::storage::redis] new cursor saved to redis 0,9ad7ff320c9cf74e0f5ee78d22a85ce42bb0a487d0506bf60cfb5a91ea4497d2
[2022-10-18T17:29:31Z INFO  scrolls::storage::redis] new cursor saved to redis 2,1d031daf47281f69cd95ab929c269fd26b1434a56a5bbbd65b7afe85ef96b233
.
.
.

[2022-10-18T17:29:31Z INFO  scrolls::storage::redis] new cursor saved to redis 87020,d9a96d7e3c20d30efc0bcb1d004546d07e63477d95dcf6eea5dfd557664eb554
[2022-10-18T17:29:31Z ERROR gasket::runtime] STAGE: reducers, WORK, PANIC: missing utxo: 5526b1373acfc774794a62122f95583ff17febb2ca8a0fe948d097e29cf99099#0