-
Notifications
You must be signed in to change notification settings - Fork 11
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Investigate growing OrgBook - Dev Wallet #124
Comments
Agent Logs (Set to WARNING):
API Logs (Set to WARN):
|
The last credential records that were added to the OrgBook were added 2022-12-19. |
Info from the
|
One theory is the Credential Verification monitoring is triggering proof exchange records and they are not being cleaned up. To test this, we've paused the Credential Verification monitors, to see if that results fewer records being added to the wallet. |
@ianco, I think we found the issue. Have a look at the numbers under the Summary of size increase between 2023/02/07 and 2023/02/08 - After pausing the monitors section. There is a significant decrease in the number of records created with the monitors turned off. The number of records created could easily be accounted for by the few hours the monitors were running after the data from yesterday was collected. |
Info from the
Summary:
This confirms |
Is this happening in test and prod, or just Dev? |
From #124 (comment) Summary:
This confirms |
If we're running the same monitor, then it's happening in test and prod as well and we just haven't noticed due to the larger records counts in these environments. |
Even though all the data in the database in encrypted, we can use the type column in a where clause, because the encrypted value will always be the same if the type is the same So for example you can run this query to see how many records there are for each record type:
(this works in orgbook dev because we know the most recent record is a proof exchange) ... and if you know which record is of a certain type:
... so we could delete the records directly from the wallet using this kind of filter The trick is we also need to delete all the tags associated with these records. |
Scripts to delete the presentation requests from a wallet; bcgov/von-bc-registries-audit#26 |
To resolve this permanently, the ACA-Py instance for OrgBook needs to be updated to a newer version where presentation records are deleted by default. |
This is the PR containing the desired behavior, deleting the presentation exchange records; hyperledger/aries-cloudagent-python#2352. First available in aca-py v0.10.1 |
The other consideration during the upgrade is the wallet type. OrgBook and BC Reg agents are still using |
Both py3.9-indy-1.16.0-0.10.5 and py3.9-indy-1.16.0-0.11.0 are available. @swcurran, any recommendation on which version, |
Are you updating both the issuer and OrgBook? If not, then yes, you will probably want to miss that breaking change. If you are doing both, then go with 11. I assume the |
Yes, I'll be upgrading both. Correct, neither |
Will be upgrading BC Reg and OrgBook agents to py3.9-indy-1.16.0-0.11.0 |
I've upgraded the BC Reg and OrgBook |
OrgBook Agent Logs: BC Reg Agent Logs: BC Reg Controller Logs: OrgBook API logs (from another run): |
Deployed the |
The credentials were posted successfully using the previous aca-py versions; |
Does the issuer_registration plugin need to be updated to support the new aca-py version(s)? |
Logged bcgov/aries-vcr#766 to address this and upgrading to a newer agent version |
On 2023/02/06 the OrgBook-Dev backup reported an out of diskspace issue. On investigation we found the wallet database was growing daily. This is unexpected since the cron jobs are turned off in
dev
on the BC Registries side so no corporate records are being processed, so no credentials are being issued.On 2023/02/07 another check was done and the issue with the growing wallet was confirmed. Now we need to determine what is causing the issue.
Agents are running
aca-py
v0.7.1
Summary of size increase between 2023/02/06 and 2023/02/07
items_id_seq
increased by 15,230tags_encrypted
table increased by 14,754 records and 10MBitems
table increased by 14,754 records and 8,215MBSummary of size increase between 2023/02/07 and 2023/02/08 - After pausing the monitors
items_id_seq
increased by 1,568tags_encrypted
table increased by 1,540 records and 1MBitems
table increased by 1,540 records and 20MBData from 2023/02/06
wallet-bc-agent_bc_wallet backups:
agent_bc_wallet database info:
Data from 2023/02/07
wallet-bc-agent_bc_wallet backups:
agent_bc_wallet database info:
OrgBook - Dev - Wallet PVC
Data from 2023/02/08 - After pausing the monitors
wallet-bc-agent_bc_wallet backups:
agent_bc_wallet database info:
OrgBook - Dev - Wallet PVC
The text was updated successfully, but these errors were encountered: