[Bug] XCH-USDS offer acceptance returns change in unhinted form (no memo) which wallet loses

[trgarrett]

What happened?

I accepted an offer to trade some of my USDS for 1 XCH. The change coin is visible in the blockchain explorer at an address my wallet believes it controls. However, the change coin was sent back to my wallet address lacking any hinting.

Unspent coin:
https://www.spacescan.io/coin/0x952ed023ace998c6ae5a7b4e78f148376d3db97176fd68c5edfb3fb663a5f0b2

$ cdv rpc coinrecords --by id 0x952ed023ace998c6ae5a7b4e78f148376d3db97176fd68c5edfb3fb663a5f0b2
[
    {
        "coin": {
            "amount": 13400,
            "parent_coin_info": "0x643046c2386aac8328093387e0312877c1bcba2f145639f7e25ce6f5f0254d64",
            "puzzle_hash": "0x88a39d59258fbc9d1289e901cbae82a92a539ccc8dc55af2ac9e9f6c4a4d6430"
        },
        "coinbase": false,
        "confirmed_block_index": 3582554,
        "spent_block_index": 0,
        "timestamp": 1682623178
    }
]

$ cdv rpc blockspends --coinid 643046c2386aac8328093387e0312877c1bcba2f145639f7e25ce6f5f0254d64 -h 3582554
{
    "coin": {
        "amount": 55500,
        "parent_coin_info": "0xae7af921fa909a79e1c4799cc9d51047ee04240f09e5b8212aea9be8769769cc",
        "puzzle_hash": "0x728ca02f361ce0ff4fb7719d64c3c5f21b82d78fdc832297a87ddab13917c21d"
    },
    "puzzle_reveal": "0xff02ffff01ff02ffff01ff02ff5effff04ff02ffff04ffff04ff05ffff04ffff0bff34ff0580ffff04ff0bff80808080ffff04ffff02ff17ff2f80ffff04ff5fffff04ffff02ff2effff04ff02ffff04ff17ff80808080ffff04ffff02ff2affff04ff02ffff04ff82027fffff04ff82057fffff04ff820b7fff808080808080ffff04ff81bfffff04ff82017fffff04ff8202ffffff04ff8205ffffff04ff820bffff80808080808080808080808080ffff04ffff01ffffffff3d46ff02ff333cffff0401ff01ff81cb02ffffff20ff02ffff03ff05ffff01ff02ff32ffff04ff02ffff04ff0dffff04ffff0bff7cffff0bff34ff2480ffff0bff7cffff0bff7cffff0bff34ff2c80ff0980ffff0bff7cff0bffff0bff34ff8080808080ff8080808080ffff010b80ff0180ffff02ffff03ffff22ffff09ffff0dff0580ff2280ffff09ffff0dff0b80ff2280ffff15ff17ffff0181ff8080ffff01ff0bff05ff0bff1780ffff01ff088080ff0180ffff02ffff03ff0bffff01ff02ffff03ffff09ffff02ff2effff04ff02ffff04ff13ff80808080ff820b9f80ffff01ff02ff56ffff04ff02ffff04ffff02ff13ffff04ff5fffff04ff17ffff04ff2fffff04ff81bfffff04ff82017fffff04ff1bff8080808080808080ffff04ff82017fff8080808080ffff01ff088080ff0180ffff01ff02ffff03ff17ffff01ff02ffff03ffff20ff81bf80ffff0182017fffff01ff088080ff0180ffff01ff088080ff018080ff0180ff04ffff04ff05ff2780ffff04ffff10ff0bff5780ff778080ffffff02ffff03ff05ffff01ff02ffff03ffff09ffff02ffff03ffff09ff11ff5880ffff0159ff8080ff0180ffff01818f80ffff01ff02ff26ffff04ff02ffff04ff0dffff04ff0bffff04ffff04ff81b9ff82017980ff808080808080ffff01ff02ff7affff04ff02ffff04ffff02ffff03ffff09ff11ff5880ffff01ff04ff58ffff04ffff02ff76ffff04ff02ffff04ff13ffff04ff29ffff04ffff0bff34ff5b80ffff04ff2bff80808080808080ff398080ffff01ff02ffff03ffff09ff11ff7880ffff01ff02ffff03ffff20ffff02ffff03ffff09ffff0121ffff0dff298080ffff01ff02ffff03ffff09ffff0cff29ff80ff3480ff5c80ffff01ff0101ff8080ff0180ff8080ff018080ffff0109ffff01ff088080ff0180ffff010980ff018080ff0180ffff04ffff02ffff03ffff09ff11ff5880ffff0159ff8080ff0180ffff04ffff02ff26ffff04ff02ffff04ff0dffff04ff0bffff04ff17ff808080808080ff80808080808080ff0180ffff01ff04ff80ffff04ff80ff17808080ff0180ffff02ffff03ff05ffff01ff04ff09ffff02ff56ffff04ff02ffff04ff0dffff04ff0bff808080808080ffff010b80ff0180ff0bff7cffff0bff34ff2880ffff0bff7cffff0bff7cffff0bff34ff2c80ff0580ffff0bff7cffff02ff32ffff04ff02ffff04ff07ffff04ffff0bff34ff3480ff8080808080ffff0bff34ff8080808080ffff02ffff03ffff07ff0580ffff01ff0bffff0102ffff02ff2effff04ff02ffff04ff09ff80808080ffff02ff2effff04ff02ffff04ff0dff8080808080ffff01ff0bffff0101ff058080ff0180ffff04ffff04ff30ffff04ff5fff808080ffff02ff7effff04ff02ffff04ffff04ffff04ff2fff0580ffff04ff5fff82017f8080ffff04ffff02ff26ffff04ff02ffff04ff0bffff04ff05ffff01ff808080808080ffff04ff17ffff04ff81bfffff04ff82017fffff04ffff02ff2affff04ff02ffff04ff8204ffffff04ffff02ff76ffff04ff02ffff04ff09ffff04ff820affffff04ffff0bff34ff2d80ffff04ff15ff80808080808080ffff04ff8216ffff808080808080ffff04ff8205ffffff04ff820bffff808080808080808080808080ff02ff5affff04ff02ffff04ff5fffff04ff3bffff04ffff02ffff03ff17ffff01ff09ff2dffff02ff2affff04ff02ffff04ff27ffff04ffff02ff76ffff04ff02ffff04ff29ffff04ff57ffff04ffff0bff34ff81b980ffff04ff59ff80808080808080ffff04ff81b7ff80808080808080ff8080ff0180ffff04ff17ffff04ff05ffff04ff8202ffffff04ffff04ffff04ff78ffff04ffff0eff5cffff02ff2effff04ff02ffff04ffff04ff2fffff04ff82017fff808080ff8080808080ff808080ffff04ffff04ff20ffff04ffff0bff81bfff5cffff02ff2effff04ff02ffff04ffff04ff15ffff04ffff10ff82017fffff11ff8202dfff2b80ff8202ff80ff808080ff8080808080ff808080ff138080ff80808080808080808080ff018080ffff04ffff01a037bef360ee858133b69d595a906dc45d01af50379dad515eb9518abb7c1d2a7affff04ffff01a06d95dae356e32a71db5ddcb42224754a02524c615c5fc35f568c2af04774e589ffff04ffff01ff02ffff01ff02ffff01ff02ffff03ff0bffff01ff02ffff03ffff09ff05ffff1dff0bffff1effff0bff0bffff02ff06ffff04ff02ffff04ff17ff8080808080808080ffff01ff02ff17ff2f80ffff01ff088080ff0180ffff01ff04ffff04ff04ffff04ff05ffff04ffff02ff06ffff04ff02ffff04ff17ff80808080ff80808080ffff02ff17ff2f808080ff0180ffff04ffff01ff32ff02ffff03ffff07ff0580ffff01ff0bffff0102ffff02ff06ffff04ff02ffff04ff09ff80808080ffff02ff06ffff04ff02ffff04ff0dff8080808080ffff01ff0bffff0101ff058080ff0180ff018080ffff04ffff01b0b0ef6261a8bb29bdbc8e9ebf978e59b48fd98f7257514bb5d4d55b8c006de7ab26c5e1091f94a43d6d516db955c8d773ff018080ff0180808080",
    "solution": "0xffff80ffff01ffff33ffa0cfbfdeed5c4ca2de3d0bf520b9cb4bb7743a359bd2e6a188d19ce7dffc21d3e7ff8300a474ffffa0cfbfdeed5c4ca2de3d0bf520b9cb4bb7743a359bd2e6a188d19ce7dffc21d3e78080ffff33ffa03d0b3c63f526b2ccba2e42d9bbadd4e9dee12909e209e906b78084fd81c9d200ff82345880ffff3cffa050161c2c4f77348d3f64db5c1559afa8d5fea8706766bbff0d2c0a2a25eecda880ffff3fffa0124b75c0e17b8458a6828d2f79e62b897c667b62b518d96083920a6bc0b03b5e8080ff8080ffffa095d79c401512ebc14f464e94989b2584f89a34164fad33625d546e0c08c5589fffa0ffb10cf9c5a0faea483fe291de760e80901cf2e8461a3f91feaa9c4685622e22ff830186a080ffa0643046c2386aac8328093387e0312877c1bcba2f145639f7e25ce6f5f0254d64ffffa0ae7af921fa909a79e1c4799cc9d51047ee04240f09e5b8212aea9be8769769ccffa0728ca02f361ce0ff4fb7719d64c3c5f21b82d78fdc832297a87ddab13917c21dff8300d8cc80ffffa0ae7af921fa909a79e1c4799cc9d51047ee04240f09e5b8212aea9be8769769ccffa0ffb10cf9c5a0faea483fe291de760e80901cf2e8461a3f91feaa9c4685622e22ff8300d8cc80ff80ff8080"
}

The primary spend for redemption was hinted, while the change was not:

((() (q (51 0xcfbfdeed5c4ca2de3d0bf520b9cb4bb7743a359bd2e6a188d19ce7dffc21d3e7 0x00a474 (0xcfbfdeed5c4ca2de3d0bf520b9cb4bb7743a359bd2e6a188d19ce7dffc21d3e7)) (51 0x3d0b3c63f526b2ccba2e42d9bbadd4e9dee12909e209e906b78084fd81c9d200 13400) (60 0x50161c2c4f77348d3f64db5c1559afa8d5fea8706766bbff0d2c0a2a25eecda8) (63 0x124b75c0e17b8458a6828d2f79e62b897c667b62b518d96083920a6bc0b03b5e)) ()) (0x95d79c401512ebc14f464e94989b2584f89a34164fad33625d546e0c08c5589f 0xffb10cf9c5a0faea483fe291de760e80901cf2e8461a3f91feaa9c4685622e22 0x0186a0) 0x643046c2386aac8328093387e0312877c1bcba2f145639f7e25ce6f5f0254d64 (0xae7af921fa909a79e1c4799cc9d51047ee04240f09e5b8212aea9be8769769cc 0x728ca02f361ce0ff4fb7719d64c3c5f21b82d78fdc832297a87ddab13917c21d 0x00d8cc) (0xae7af921fa909a79e1c4799cc9d51047ee04240f09e5b8212aea9be8769769cc 0xffb10cf9c5a0faea483fe291de760e80901cf2e8461a3f91feaa9c4685622e22 0x00d8cc) () ())

I believe this to be the root cause of many of my wallet issues following offer acceptance, possibly for CATs, XCH, and NFTs.

Version

1.7.1

What platform are you using?

macOS

What ui mode are you using?

GUI

Relevant log output

No response

[trgarrett]

Here's a tool I'm building to fix this issue. It has already found some missing XCH. Trying it now on some USDS as well.

https://github.com/trgarrett/chialisp/blob/main/misc/hint_fixer.py

[emlowe]

Change for CATs has historically never been hinted and those coins are normally found via puzzle subscription and not the hint. It's possible you are reaching the limit of puzzle subscriptions (especially when using untrusted nodes aka, light wallet mode) which is why they may appear missing.
The idea originally was to be clever about CAT change and to save space on-chain by not hinting.

We do anticipate changing this going forward to start hinting CAT change (see #15179).
Also, we plan some other fixes to make it more robust to find any existing unhinted change

[trgarrett]

Thanks for the feedback, Earle. Is there a hard-coded or configurable limit somewhere on puzzle subscriptions? I have a synced full node and have still dealt with missing coins regularly. I have plenty of RAM so if it’s just a matter of updating a cache limit that could be really helpful. Sorry for any confusion I caused with incorrect assumptions on the state of hinting for CATs. I knew that hints were pretty critical for finding things like NFTs in other cases, so I may have jumped to some unsupported conclusions.

…

On Thu, May 4, 2023 at 11:56 AM Earle Lowe ***@***.***> wrote: Change for CATs has historically never been hinted and those coins are normally found via puzzle subscription and not the hint. It's possible you are reaching the limit of puzzle subscriptions (especially when using untrusted nodes aka, light wallet mode) which is why they may appear missing. The idea originally was to be clever about CAT change and to save space on-chain by not hinting. We do anticipate changing this going forward to start hinting CAT change (see #15179 <#15179>). Also, we plan some other fixes to make it more robust to find any existing unhinted change — Reply to this email directly, view it on GitHub <#15156 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AI2B4MSHRCVHVTPQA5TYX53XEPNVFANCNFSM6AAAAAAXPX4DKE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[emlowe]

Yes, there is a limit to puz subscriptions in by the full_node config.yaml, but it is very high in 1.7.1 for trusted nodes:

  # Number of coin_ids | puzzle hashes that node will let wallets subscribe to
  max_subscribe_items: 200000

  # the maximum number of CoinStates will be returned by a RegisterForPhUpdates
  # request, for untrusted peers
  max_subscribe_response_items: 100000

and

  # Number of coin_ids | puzzle hashes that node will let local wallets subscribe to
  trusted_max_subscribe_items: 2000000

  # the maximum number of CoinStates will be returned by a RegisterForPhUpdates
  # request, for trusted peers
  trusted_max_subscribe_response_items: 500000

again those are full_node settings. So the default for the local node should be set very high and normally I wouldn't expect any missing in that scenario.

[trgarrett]

Thanks @emlowe. I have probably 70 CAT tokens (unique asset ID), hundreds of NFTs, and both observable and non-observable keys. I'm probably an outlier for the default values, so I will try to bump them up by a couple of orders of magnitude and see if I see better results.

[emlowe]

Marking as closed, as going forward this change will be hinted due to #15179