Startup can fail with certain multi-tenancy subwallets [Multi-tenancy]

[jamshale]

There was an issue reported here #2922 (review) with the anoncreds upgrade feature. For some reason it is unable to query the subwallet and/or create to subwallet profile.

It needs to handle this scenario correctly.

I have tried to reproduce with sqlite DB and normal sub-wallet and base-wallet setting but have been unable to reproduce the issue so far.

[ff137]

Thank you! I'll try modify provision=True and test with that... Will report back tomorrow 👌

[ff137]

I've tested just with the simple change setting provision to True, but we still get the same error:

Traceback (most recent call last):
aries_askar.error.AskarError: Backend error
Caused by: error returned from database: Request returned an error: database "a8f2692b4e4b4312a9ae696d76643362" does not exist.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/aries_cloudagent/commands/start.py", line 72, in init
    await startup
File "/usr/local/lib/python3.9/site-packages/aries_cloudagent/commands/start.py", line 29, in start_app
    await conductor.start()
File "/usr/local/lib/python3.9/site-packages/aries_cloudagent/core/conductor.py", line 522, in start
    await self.check_for_wallet_upgrades_in_progress()
File "/usr/local/lib/python3.9/site-packages/aries_cloudagent/core/conductor.py", line 846, in check_for_wallet_upgrades_in_progress
    subwallet_profiles = await get_subwallet_profiles_from_storage(
File "/usr/local/lib/python3.9/site-packages/aries_cloudagent/utils/profiles.py", line 54, in get_subwallet_profiles_from_storage
    await MultitenantManager(root_profile).get_wallet_profile(
File "/usr/local/lib/python3.9/site-packages/aries_cloudagent/multitenant/manager.py", line 85, in get_wallet_profile
    profile, _ = await wallet_config(context, provision=provision)
File "/usr/local/lib/python3.9/site-packages/aries_cloudagent/config/wallet.py", line 51, in wallet_config
    profile = await mgr.provision(context, profile_cfg)
File "/usr/local/lib/python3.9/site-packages/aries_cloudagent/askar/profile.py", line 293, in provision
    opened = await store_config.open_store(provision=True)
File "/usr/local/lib/python3.9/site-packages/aries_cloudagent/askar/store.py", line 168, in open_store
    raise ProfileError("Error opening store") from err
aries_cloudagent.core.error.ProfileError: Error opening store

It's likely related to some specifics in our config. I'll debug a bit more and see what I can find out

[jamshale]

OK. I'll try and replicate it later today with some other configurations. I can't think of anything off the top of my head.

[jamshale]

It might be easy to fix by just ignoring the subwallets that can't open their store. All this is doing is resuming an in progress upgrade if interrupted by a restart. It's not very likely to happen. I thought it would be a safe operation but apparently there's a bit more to it then I thought.

[jamshale]

I've tried to replicate this with normal configurations and haven't been able to.

However, I was able to reproduce something similar by manually deleting a subwallet DB and restarting with the global auto_provision setting set to False (the default).

I feel like there must be something going on with the configurations and DB to cause this problem. However, setting provision to True in the https://github.com/hyperledger/aries-cloudagent-python/blob/main/aries_cloudagent/utils/profiles.py#L54 call also prevented the error from happening and re-created the DB.

[jamshale]

Could you share your ACAPY_WALLET_STORAGE_CONFIG? Obviously without any sensitive info.

I'm wondering if this has anything to do with the wallet_scheme. I've only tested with the defaults. I can't even remember or find how to configure it without a DatabsePerWallet wallet_scheme.

[jamshale]

I'm going to close this for now as #3048 will prevent this check from crashing on startup. If a more detailed explanation of the scenario that leads to the problem can be produced I can have another look at preventing it.