feat: add update assistant for storage migrations

[TimoGlastra]

Adds an UpdateAssistant that can help with updating AFJ storage to newer versions. If the agent is initialized with an older storage version than the framework an error is thrown and the agent can't be initialized. See the updating.md doc for examples on how to use the update assistant to update agent storage.

I've also added an autoUpdateOnStartup parameter that will auto update the agent storage on initialization. Would like to have a discussion about this during the WG call whether this is desired or what the most optimal flow would be for updating the agent.

I've added two migrations for now, we can add more once the PRs are merged:

• update the role in mediation record
• update the metadata object in the credential record to the new format

The changes are well documented in the 0.1-to-0.2 document. This currently contains only breaking storage changes, but should be updated with breaking code changes also.

A backup will be created when starting the agent, and it will be restored if the migration failed.

I spent A LOT of time on testing to make sure nothing goes wrong here as that can lead to nasty situations. Every migration has extensive tests and there are also quite some tests for the e2e flow.

[karimStekelenburg]

Is there an option to keep the backup file even if the migration succeeds? I can imagine that some people feel more comfortable doing a migration if they have a way to (although manually), rollback.

[codecov-commenter]

Codecov Report

Merging #690 (0db8629) into main (858253a) will increase coverage by 0.49%.
The diff coverage is 95.86%.

@@            Coverage Diff             @@
##             main     #690      +/-   ##
==========================================
+ Coverage   86.17%   86.66%   +0.48%     
==========================================
  Files         343      355      +12     
  Lines        7443     7694     +251     
  Branches     1153     1194      +41     
==========================================
+ Hits         6414     6668     +254     
+ Misses       1028     1025       -3     
  Partials        1        1              

Impacted Files	Coverage Δ
...modules/credentials/repository/CredentialRecord.ts	91.22% <ø> (ø)
packages/core/src/types.ts	100.00% <ø> (ø)
packages/core/src/wallet/IndyWallet.ts	68.51% <33.33%> (-1.86%)	⬇️
packages/core/src/storage/BaseRecord.ts	95.65% <66.66%> (+8.69%)	⬆️
packages/core/src/agent/Agent.ts	95.90% <89.28%> (-2.16%)	⬇️
...ages/core/src/storage/migration/UpdateAssistant.ts	93.58% <93.58%> (ø)
packages/core/src/agent/AgentConfig.ts	94.73% <100.00%> (+0.19%)	⬆️
...s/core/src/storage/didcomm/DidCommMessageRecord.ts	96.66% <100.00%> (-0.11%)	⬇️
packages/core/src/storage/index.ts	100.00% <100.00%> (ø)
...core/src/storage/migration/StorageUpdateService.ts	100.00% <100.00%> (ø)
... and 18 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 858253a...0db8629. Read the comment docs.

[TimoGlastra]

Is there an option to keep the backup file even if the migration succeeds? I can imagine that some people feel more comfortable doing a migration if they have a way to (although manually), rollback.

Partially. Currently the migration script uses the temp directory to store the backup. This means the backup will stay around after the migration, but there's no guarantee on how long it will stick around.

I think this is a good addition, and we can maybe add this is in a future PR. We'll at least need the following issue to be resolved first: #706

[TimoGlastra]

Per our discussion during the WG today, I'll work on adding migration scripts for OOB / DIDExchange. @JamesKEbert / @jakubkoci if you could give this a review and get it merged I can start working on this.

[JamesKEbert]

This is amazing Timo! Thank you for the incredible work here, I think this will serve AFJ very well.
I left a couple of comments, and one change for a test, but nothing really blocking. :)

[JamesKEbert]

What's the thought process behind this removal?

[TimoGlastra]

I've maybe been to quick to remove them.

The default way for indy to store array values is by storing the value in the key and setting
the tag value to "1".

So when we store a message for a protocol version of 1.1 it will convert it to array when retrieving from storage.

Currently we use the exact '@type' of a message to retrieve it. If we're going to support more versions of protocols I thought we could retrieve the message by protocol name and message name and do the version filtering post.

However it may be useful to retrieve e.g. all 1.x messages in the future.

Will think about it for a bit, but we would have to store the value of '1' as something different so we don't transform it into an array, or check if the key doesn't contain an :

[TimoGlastra]

Looking at it, it's not arrays but booleans. We check if the value === '1' (or '0') and base the boolean of off that.

      } else if (value === '1' || value === '0') {
        transformedTags[key] = value === '1'
      }

What we could do is store the number variant as __1 or something so we know for sure it's the number 1 (or 0) and not the boolean value. Not sure what is best here. Thoughts?

[JamesKEbert]

Sorry, I likely am missing something--I'll admit I'm not quite sure what the issue is. If we store a tag with a key of 'versionMajor' with a value of '1', shouldn't that be fine? What am I missing?

[TimoGlastra]

Indy only allows tag values to be strings, while in AFJ tag values can be string, number, boolean and array of those types. So we have custom processing to go from afj tag to indy tag and vice versa

Boolean values are transformed by us to the string value '1'. So if we're storing the actual value 1 it will also be stored as string value '1'. When retrieved from storage afj will transform the value 1 to a boolean value.

[TimoGlastra]

I've removed this from this PR and created a fix for it in #714.