Refactor DID support to use 3ID's

[tahpot]

Verida currently has its own DID generation approach. This has served it well for a PoC, but it's better to migrate to an alternative DID solution. In this issue I will outline a migration plan to use Ceramic network's 3ID DID provider.

Current architecture

Verida currently generates a DID on a per application basis:

• A blockchain wallet (Ethereum, VeChain, NEAR) is used to sign a consent message (ie: Do you want to login to the Verida Vault application?)
• The signature from this consent message is used as entropy to generate a new hierarchical deterministic wallet
• This HD wallet is then used to generate an encryption keypair and signing keypair
• A new Verida DID document is created that lists the public keys of those keypairs, which is then stored in a centralized API
• This DID document also stores information such as the database address for the user's encrypted data (using DID service definition)
• The encryption key is used to encrypt all data within that application context
• The signing key is used to sign all generated data within that application context

This offers the following benefits:

1. An existing blockchain wallet can be used to unlock the data for any application
2. This unlocking is deterministic. No private key needs to be stored, it can be generated on demand from the master blockchain wallet private key
3. Each application has it's own secure enclave of data, allowing a web application to only gain access to data relevant for its use case

Ceramic architecture

Ceramic provides an alternative approach, whereby multiple blockchain wallets can be used to control a 3ID DID via their 3ID DID Provider. Utilizing the Ceramic network and IDX protocol, it's possible to store public or encrypted data for each 3ID.

Under this model a user will need a 3ID in order to use the Verida network. A user will authenticate (or sign up) as follows:

• A user of a web application will login via 3ID connect (a web based interface for authenticating with a blockchain wallet to unlock an existing 3ID)
• Alternatively, a user of a web application will login via the Verida QR code login system connected to the Verida Vault

In the future, it may be possible for the Verida Vault QR code auth to be integrated into 3ID Connect.

Behind the scenes, the following needs to occur:

• Unlock or create the user's 3ID
• Once authenticated, the user can write data into their stream
• The 3ID provider instance private key can be used to deterministically sign a consent message to authenticate an application
• The 3ID provider instance can be used to generate encryption and signing keys (as per current process)
• The 3ID provider instance can be used to store the public key on a per application basis within the 3ID's IDX profile (instead of storing a Verida DID in a central database)

Why Ceramic?

• It's built to provide this core DID infrastructure
• Utilizes the latest decentralized standards and plays nice with DIF projects
• It has many features (key rotation, multiple blockchains supported, well supported project)
• It unlocks an ecosystem of existing apps that can tap into the power of the Verida network

Risks to eliminate

• Create a web based PoC that validates the above approach to using 3ID to store and unlock per application databases will work as expected
• Ceramic + IDX won't work within React Native environment
• Ceramic + IDX introduces latency issues in web or mobile environments
• Ceramic + IDX adds a significant file size overhead to the Verida Datastore library

[tahpot]

Need to consider how to connect in a server (or mobile) environment v connect in a web environment.

• Server environment will require access to a blockchain private key that is used to authenticate with Ceramic.
• Web environment will use 3ID-connect.

Both separate process will result in the creation of a DID object that can be used to sign and encrypt data via a 3ID.

Proposed architecture

• packages/connect-web: Uses 3ID Connect to authenticate a user and generate a DID instance
• packages/connect-node: Uses EthereumAuthProvider from Ceramic to wrap a private key and blockchain network. This is injected into a @3id/manager instance that can produce a DID instance.
• packages/datastore: Accepts a DID instance and replicates the existing datastore capabilities

However, the above fails to work in a web environment if we are wanting a user to authorize access to a specific application context. Under the above model, the web environment has granted full access to the current website to the 3ID, rather than delegating access to an application context that has limited access (ie: can't write to the user's profile).

I have asked the Ceramic team on Discord about this application context issue.

Can hopefully build into connect-web and connect-node the application specific authorization.

[tahpot]

I have asked the Ceramic team on Discord about this application context issue.

From Ceramic discord:

The plan right now for IDX "spaces" i.e. encrypted definitions/records can be found here: https://github.com/ceramicnetwork/CIP/blob/main/CIPs/CIP-11/CIP-11.md#idx-keychain-definition

A read of this indicates any support will require changes to Ethereum libraries, while the current 3id-connect library doesn't support paths (3id-did-provider does, in theory).

I don't feel we can rely on ceramic / IDX for this capability at this stage, so will need to roll out own.

Security: At the moment web applications using 3ID's gain full access to sign and encrypt using that 3ID, providing no ability to restrict access between different web applications.

We can solve this by enforcing our single sign on to only work via a mobile application, but that's not ideal long term.

[tahpot]

From the proposed architecture above:

packages/connect-web: Uses 3ID Connect to authenticate a user and generate a DID instance

After further investigation 3id-connect does everything we need here, so no need for a separate Verida helper library.

packages/connect-node: Uses EthereumAuthProvider from Ceramic to wrap a private key and blockchain network. This is injected into a @3id/manager instance that can produce a DID instance.

This has been implemented with working tests as packages/3id-utils-node. See 4579439.

[tahpot]

Ceramic + IDX introduces latency issues in web or mobile environments

Doing some basic tests with https://self.id shows a variable time to connect anywhere from 4-8 seconds. This isn't great, however this should only be required on the mobile app when a user first connects and will then be cached.

Tests of fetching an existing user's IDX profile returned a result in ~1 second, which should be fine.

[tahpot]

Ceramic + IDX won't work within React Native environment

According to Ceramic discord others are working on this, so assume it's not an issue.

Create a web based PoC that validates the above approach to using 3ID to store and unlock per application databases will work as expected

Decision is to use Verida's Single Sing on to support this as Ceramic won't support it any time soon.

[tahpot]

The core packages that support this are now complete:

• https://github.com/verida/verida-js/tree/main/packages/3id-utils-node (Utility to generate 3ID's from Ethereum private keys)
• https://github.com/verida/verida-js/tree/main/packages/storage-link (Utility to create named secure storage contexts for 3ID's)
• https://github.com/verida/verida-js/tree/main/packages/storage-keyring (Utility to create a keyring for a given seed. Used by storage-link)