docs: compose the ADR for DID-linked resources. ATL-3186 [skip-ci] #518

yshyn-iohk · 2023-05-02T15:41:08Z

Overview

Fixes ATL-3186

https://hackmd.io/@YuriiShynbuievIOHK/BkqWWk2zh - for reading

Checklist

My PR contains...

No code changes (changes to documentation, CI, metadata, etc.)
Bug fixes (non-breaking change which fixes an issue)
Improvements (misc. changes to existing features)
Features (non-breaking change which adds functionality)

My changes...

are breaking changes
are not breaking changes
If yes to above: I have updated the documentation accordingly

Documentation

My changes do not require a change to the project documentation
My changes require a change to the project documentation
If yes to above: I have updated the documentation accordingly

Tests

My changes can not or do not need to be tested
My changes can and should be tested by unit and/or integration tests
If yes to above: I have added tests to cover my changes
If yes to above: I have taken care to cover edge cases in my tests

github-actions · 2023-05-02T15:48:40Z

Atala PRISM Test Results

  52 files ±0   52 suites ±0 39s ⏱️ +7s
396 tests ±0 396 ✔️ ±0 0 💤 ±0 0 ❌ ±0
397 runs ±0 397 ✔️ ±0 0 💤 ±0 0 ❌ ±0

Results for commit 9df4dc8. ± Comparison against base commit ab9d2b8.

♻️ This comment has been updated with latest results.

github-actions · 2023-05-02T15:48:43Z

Atala PRISM Code Coverage

File	Coverage
All files	`22%`	✅

Minimum allowed coverage is 12%

Generated by 🐒 cobertura-action against 9df4dc8

github-actions · 2023-05-02T16:02:46Z

E2E Test Results

10 files ±0 10 suites ±0 5m 22s ⏱️ -11s
19 tests ±0 19 ✔️ ±0 0 💤 ±0 0 ❌ ±0
30 runs ±0 30 ✔️ ±0 0 💤 ±0 0 ❌ ±0

Results for commit 9df4dc8. ± Comparison against base commit ab9d2b8.

♻️ This comment has been updated with latest results.

bvoiturier

Awesome work @yshyn-iohk!
I actually learned a lot on the subject while reviewing this ADR. Thanks 🙏

bvoiturier · 2023-05-03T11:42:17Z

docs/decisions/20230405-did-linked-resources.md

+- Provide a concrete proposal for what we would like to implement for PRISM.
+- Provide a generic way of storing and linking the resources for the DID in the PRISM platform.
+
+Cheqd's approach for linking the resources to the DID is not a part of the current version of DID specification. Even if it's possible to find some information about `linkedResources` and `linkedResourceMetadata` optional field of the DIDDoc, it looks like the current decision is 


It would be helpful to have direct links to the specs where linkedResource and linkedResourceMetadata are mentioned.

Also, seems the end of the sentence is missing here.

I'm not sure this sentence should be here, I think it is out of place, and I can't quite figure out what it has to do with "Target"

@bvoiturier, as I told, it's not a part of the did-core spec, but if you ask ChatGPT to describe you the usage of the linkedResources fields, it points to the did-code v1.0.1

what part of the DID core specification describes the usage of the fields linkedResource? ChatGPT The linkedResources field is not defined in the original DID core specification (v1.0), but it is introduced in the DID core specification v1.0.1. Specifically, section 5.4.4 of the DID core specification v1.0.1 defines the linkedResources field as an optional field that can be included in a DID document. The section describes the purpose of the linkedResources field as follows: "The linkedResources property expresses relationships between a DID and external resources, such as public keys, verification methods, or other data types. These resources can be described in separate documents or be directly included as part of the DID document." The section provides more detailed information on the structure of the linkedResources field, including the required and optional fields for each linked resource object. It is worth noting that the DID core specification v1.0.1 is an update to the original DID core specification (v1.0) and includes several improvements and clarifications based on feedback from the community. The linkedResources field is one of the additions in the updated specification.

I moved the sentence to the section with Cheqd's solution.

bvoiturier · 2023-05-03T11:46:52Z

docs/decisions/20230405-did-linked-resources.md

+
+## Context and Problem Statement
+
+Atala Prism platform must be able to store and distribute the various resources such as credential schemas, logos, revocation status lists, and documents (aka any text, JSON, images, etc)


To be exhaustive, I would add schema definitions, credential definitions, and revocation registry definitions that are required by AnonCreds.

I would clearly state that we want to leverage this DID-linked resource dereferencing mechanism as a global solution covering all our VDR needs (JWT VC, AnonCreds VC, VC schema, revocation, etc.)

part of the described data, such as images or logos, is something that may be fine to live in servers or client app side, and are likely not treated in the same way as credential or schema definitions

In the scope of the Jira ticket, the task was to define the general way of storing the resources.
But the primary intention is to be prepared for AnonCreds integration.
So, I can reduce the scope to the AnonCreds entities: credential schema and credential definition, but the approach will be the same, and it will be possible to extend it to other type of small resources.

bvoiturier · 2023-05-03T14:02:20Z

docs/decisions/20230405-did-linked-resources.md

+
+While DLT and IPFS are designed for scalability, they can still face issues with scalability when it comes to storing SSI resources. As more users store their SSI resources on these platforms, it can become more difficult to scale the infrastructure to handle the increased demand.
+
+DLT and IPFS and other decentralized storages should not be considered for storing large collections of resources. For these cases, the hybrid solutions should be considered when the payload of the resource is stored off-chain and the hash for the resource is stored on-chain.


Not sure I understand it correctly. IPFS would be a viable solution for storing a single large file (5Gb), but not for storing a large collection of files?

Agree. I will remove these two statements.
The main message concerns the general approach for storing the resources in DLT and IPFS. You always need to spin up the additional infrastructure for DLT and IPFS.

docs/decisions/20230405-did-linked-resources.md

bvoiturier · 2023-05-10T10:07:10Z

docs/decisions/20230405-did-linked-resources.md

+	- by signing the payload with one of the DID's keys or 
+	- by publishing the resource metadata that contains the information about the resource (id, type, name, media type, hash) on-chain or
+	- for the resource that is less than the blockchain limitation (up to 64KB) by publishing the resource together with the hash, and/or signature
+- the resource can be stored in the cloud storage - PostgreSQL database - for indexing and lookup API


On the above paragraph about resource integrity/trust, my feeling is that we will end-up reimplementing what is described in the ToIP specification in terms of Linked Resource Metadata.

We agree that we need a way to prevent resource content tampering. Signing the resource content with the DID key would work, but means that a client dereferencing the resource would get a payload which is proprietary to Atala PRISM (raw data + proof metadata) - except for Credential Schemas that natively contain a proof attribute. As you say, another possibility is to publish resource metadata, including hash/checksum, on-chain, but this is precisely what the ToIP specification proposes, right?

Yes it does, ToIP spec defines the resource that is stored in DLT and resource metadata that is stored in DID document that is also published on-chain.

in this option, we don't need to store the resource metadata together with the DID document.
instead of this, the resource owner must create the DID URL that will be used by other SSI systems.

docs/decisions/20230405-did-linked-resources.md

shotexa · 2023-05-10T14:33:33Z

docs/decisions/20230405-did-linked-resources.md

+
+Known limitations for concrete decentralized storage:
+- Cardano blockchain (the resource is stored in the metadata space of the Tx): up to 64KB
+- Cheqd blockchain: up to 32KB 


Where did this idea come from? why would we add yet another blockchain in our stack, if we already have Cardano metadata and on top of that it can store more data

@shotexa, this ADR describes the existing options that can be implemented using the Cardano blockchain.
We are not going to introduce another blockchain to our stack.

docs/decisions/20230405-did-linked-resources.md

EzequielPostan · 2023-05-10T21:50:27Z

docs/decisions/20230405-did-linked-resources.md

+- Security - resources must be stored in the reliable tamper-proof storage
+- Immutability - resource must be immutable


what is the difference between these two? tamper proof means that the content cannot be altered without detection already

You are right.
immutability is one of the ways to guarantee the security and tamper-proof capability of the resource.
I will remove it for clarify

EzequielPostan · 2023-05-10T21:52:07Z

docs/decisions/20230405-did-linked-resources.md

+- Decentralization - resources must be stored in decentralized storage
+- Security - resources must be stored in the reliable tamper-proof storage
+- Immutability - resource must be immutable
+- Versioning - it should be possible to track the versioning of the resource


isn't this contradicting that the resources are immutable? Maybe the language of this list should be refined

Honestly, I would reconsider the versioning to be the decision maker for this ADR.
versioning is a label or piece of metadata attached to the resource, and it's not essential for making the final decision.

So, if version is a required part of the credential schema, it doesn't mean that it's essential to the SSI ecosystem that there is a previous or next version of the particular resource. Usually, the issuer knows which version is required for the specific case. Other SSI systems should be able to dereference the resource the issuer distributes.

I will remove it.

EzequielPostan · 2023-05-10T21:58:38Z

docs/decisions/20230405-did-linked-resources.md

+All decentralized storage (DLT or IPFS) has storage limitations, and the amount of data that can be stored is limited by the available storage capacity and the way how the resources are stored.
+
+Known limitations for concrete decentralized storage:
+- Cardano blockchain (the resource is stored in the metadata space of the Tx): up to 64KB


the current transaction size limit is 16Kb if I am not mistaken. And I think that blocks have been extended to 80Kb. Details aside, one could "fragment" the data if needed
However, what I would describe as the issue is not exactly max size limits, the inconvenience is the throughput (bytes we can insert to storage per unit of time), latency (multi-second time per insert) and cost (each insertion cost fees)

EzequielPostan · 2023-05-10T22:05:17Z

docs/decisions/20230405-did-linked-resources.md

+Known limitations for concrete decentralized storage:
+- Cardano blockchain (the resource is stored in the metadata space of the Tx): up to 64KB
+- Cheqd blockchain: up to 32KB 
+- IPFS: up to 5GB (Go SDK is limited by 255MB by default per file)


for IPFS, the challenge I would describe is providing incentives to store the data. One could use a distributed hash table (DHT) (IPFS's basic underlying structure) as a protocol to share/query files. The challenge is how to give incentives to people to store the data. If one does not pay to "pin" files in IPFS, they are actually garbage collected
We could discuss ways to use IPFS and provide incentives, but lets avoid doing this in this thread, I just wanted to highlight the challenge of using DHTs

EzequielPostan · 2023-05-10T22:13:10Z

docs/decisions/20230405-did-linked-resources.md

+- Merkle Tree DAG
+- documentation in the markdown format


what is the merkle tree DAG? what are these two needed for? and who would construct them?

Sorry for the inconsistency, I meant Merkle Tree. I will fix it.

a Merkle DAG is a more general structure that can handle non-binary structures and has nodes with multiple parents

docs/decisions/20230405-did-linked-resources.md

EzequielPostan · 2023-05-11T03:00:34Z

docs/decisions/20230405-did-linked-resources.md

+- the resource is published and indexed by the managed VDR layer (prism-node)
+- the resource is available via REST API & SDK for the product-level applications
+- the resource is dereferenced via the DID URL in the DID resolver
+- the changes to the PRISM DID method are minimal or even not required


well, I am not entirely sure about this
First we need to implement information encoding, operations and then both post and index them. This is only on node side
Then, I am not sure what logic we need to add to node/castor to provide the dereferencing logic, not if we need to add this to the spec

I think you are right.
We need to do the following things:

add the section about DID-linked resource dereferencing to the DID method.

implement the encoding and decoding login in the prism-node

implement the resource indexing logic and REST API in the Pollux and

the DID URL dereferencing logic in the DID PRISM resolver

I removed this item.

EzequielPostan

this is a very good document gathering information from multiple places. Great work @yshyn-iohk
I added some comments, and share most of the questions asked by @bvoiturier

docs/decisions/20230405-did-linked-resources.md

…repo subject to the Developer Certificate of Origin (DCO), Version 1.1. 81a2ad9 docs: removed secret-storage from the Tutorials section. (#559) [skip-ci] 130abbd docs: fix secret storage documentation page (#556) 4f413a3 docs: add secret management documentation [skip-ci] (#542) 885870d Fix ATL-4830 verification policies update (#549) f6d1fd4 docs: compose the ADR for DID-linked resources. ATL-3186 [skip-ci] (#518) dc54af8 docs: ADR for data isolation for multi-tenancy (#531) 357273d docs: ADR for HD key derivation in the PRISM v2 (#529) 64b0a2a fix(prism-agent): Alight the error responses according to RFC7807. ATL-3962 (#480) 7fa2e1a fix(prism-agent) Align VerificationPolicy OAS ATL-3909 (#473) 8d42fb1 fix: update mercury to 0.21.0 979b609 chore: update mercury dependency to 0.21.0 (#458) d3a8d15 feat(prism-agent) update schema logic - agent part. ATL-3164 (#452) 6e22bfc feat(pollux): update credential schema logic (#450) ATL-3164 68b8e4a docs: credential schema introduction, creation, update, and delete. ATL-3548 (#443) 91902ce feat(prism-agent): Add OAS specification to the schema registry endpoint. ATL-3164 (#438) 32f9e83 feat(prism-agent): CredentialSchema DAL, model, service and repositor… (#425) 79352f0 feat(pollux): CredentialSchema DAL, model, service and repository #2 (#424) 6e941da docs(pollux): add credential-schema.md to docusaurus (#407) ffa5f7e feat(pollux): CredentialSchema service, repository and sql (#416) c96f804 doc(pollux): verification policy documentation (#384) 1e47ba1 doc(pollux): add schema registry documentation. ATL-1334 (#296) 142ff55 feat(prism-agent): integrate VerificationPolicy DAL into the agent, update OAS and implement REST API for verification policies (#369) b290a18 feat(pollux): implement the DAL for CRUD on the verifiable policy entity. ATL-2478 (#368) edaab33 docs: ADR Quill library for sql statement generation (#318) 3d0c642 feat(prism-agent): implement DAL for the credential schema. ATL-1334 e0831e8 feat(pollux): fix the lookup count in the credential schema DAL (#315) f43320f feat(pollux): add dal for the credential schema ATL-1342 (#298) 6903afa fix(prism-agent): switch datetime format to offsetdatetime. ATL-2723 (#243) e13a1bd fix(connect): bump mercury version to 1.10.1 and touch README.md ee27755 fix(pollux): upgrade mercury lib to 1.10.1 cdd4772 fix(castor): README.md is added to increase the version of the castor library 5ffb0cc fix(mercury): simple commit to increase the version of mercury library 403eb38 feat(prism-agent): verification policies pagination. ATL-1334 (#205) 726e2d9 feat(prism-agent): implement pagination with navigation for schema-registry (#195) 4528c57 feat(pollux): alight the OAS for schema registry (#189) 16d5fdb feat(pollux): cleanup the code of IssueCredentialApi 79170f8 feat(pollux): cleanup the OAS from Issue Credentials and other unused tags d75b36b feat(pollux): schema registry lookup and verification policies REST API ATL-1334 (#168) b3cf828 feat(apollo): add schema registry to the agent using Tapir library. ATL-1334 (#94) 8d8bf56 doc(adr): use Tapir library as a DLS for OAS (#51) 496337b ATL-1334 feat(pollux): add sandbox project to play with Tapir and schema-registry (#62) 95667ba doc(pollux): add JWT encoding and Present Proof endpoints (#37) 234bc06 [ATL-1388] doc(pollux): add Revocation Registry API, remove v1 from the path, apply changes after review (#14) e47242c [ATL-1001] doc: add examples of OpenAPI specification of well-known competitors (#23) 22e37cf doc(pollux): add Pollux open-api specification (schema and issue-credentials) (#8) Signed-off-by: Yurii Shynbuiev <yurii.shynbuiev@iohk.io>

yshyn-iohk requested a review from antonbaliasnikov as a code owner May 2, 2023 15:41

yshyn-iohk requested a review from a user May 2, 2023 15:41

yshyn-iohk requested a review from BS-IO as a code owner May 2, 2023 15:41

atala-dev added the Type: Documentation label May 2, 2023

docs: compose the ADR for DID-linked resources

cd25afc

yshyn-iohk force-pushed the ATL-3186-ADR-DID-linked-resources branch from 372518e to cd25afc Compare May 2, 2023 15:43

yshyn-iohk requested review from bvoiturier, shotexa, mineme0110 and lohanspies May 9, 2023 11:00

bvoiturier reviewed May 10, 2023

View reviewed changes

docs/decisions/20230405-did-linked-resources.md Show resolved Hide resolved

shotexa reviewed May 10, 2023

View reviewed changes

docs/decisions/20230405-did-linked-resources.md Show resolved Hide resolved

shotexa reviewed May 10, 2023

View reviewed changes

docs/decisions/20230405-did-linked-resources.md Show resolved Hide resolved

EzequielPostan reviewed May 10, 2023

View reviewed changes

EzequielPostan reviewed May 11, 2023

View reviewed changes

docs/decisions/20230405-did-linked-resources.md Show resolved Hide resolved

EzequielPostan reviewed May 11, 2023

View reviewed changes

bvoiturier reviewed May 11, 2023

View reviewed changes

docs/decisions/20230405-did-linked-resources.md Show resolved Hide resolved

docs: improve the ADR for DID-linked resources according to the feedback

2f3efe2

yshyn-iohk force-pushed the ATL-3186-ADR-DID-linked-resources branch from 05886da to 2f3efe2 Compare May 15, 2023 08:42

Merge branch 'main' into ATL-3186-ADR-DID-linked-resources

f691844

Merge branch 'main' into ATL-3186-ADR-DID-linked-resources

9df4dc8

yshyn-iohk changed the title ~~docs: compose the ADR for DID-linked resources. ATL-3186~~ docs: compose the ADR for DID-linked resources. ATL-3186 [sckip-ci] Jun 8, 2023

yshyn-iohk changed the title ~~docs: compose the ADR for DID-linked resources. ATL-3186 [sckip-ci]~~ docs: compose the ADR for DID-linked resources. ATL-3186 [skip-ci] Jun 8, 2023

bvoiturier approved these changes Jun 8, 2023

View reviewed changes

yshyn-iohk merged commit f6d1fd4 into main Jun 8, 2023
8 checks passed

yshyn-iohk deleted the ATL-3186-ADR-DID-linked-resources branch June 8, 2023 10:38

yshyn-iohk mentioned this pull request Aug 18, 2023

I, Yurii Shynbuiev hereby sign-off-by all of my past commits to this … #634

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docs: compose the ADR for DID-linked resources. ATL-3186 [skip-ci] #518

docs: compose the ADR for DID-linked resources. ATL-3186 [skip-ci] #518

yshyn-iohk commented May 2, 2023 •

edited

github-actions bot commented May 2, 2023 •

edited

github-actions bot commented May 2, 2023 •

edited

github-actions bot commented May 2, 2023 •

edited

bvoiturier left a comment

bvoiturier May 3, 2023

shotexa May 10, 2023

yshyn-iohk May 11, 2023

yshyn-iohk May 15, 2023

bvoiturier May 3, 2023

EzequielPostan May 10, 2023

yshyn-iohk May 11, 2023

bvoiturier May 3, 2023

yshyn-iohk May 11, 2023

bvoiturier May 10, 2023

yshyn-iohk May 11, 2023

yshyn-iohk May 15, 2023

shotexa May 10, 2023

yshyn-iohk May 11, 2023

EzequielPostan May 10, 2023

yshyn-iohk May 11, 2023

EzequielPostan May 10, 2023

yshyn-iohk May 11, 2023

EzequielPostan May 10, 2023

EzequielPostan May 10, 2023

EzequielPostan May 10, 2023

yshyn-iohk May 11, 2023

EzequielPostan May 11, 2023

yshyn-iohk May 11, 2023

yshyn-iohk May 15, 2023

EzequielPostan left a comment


		## Context and Problem Statement

		Atala Prism platform must be able to store and distribute the various resources such as credential schemas, logos, revocation status lists, and documents (aka any text, JSON, images, etc)


		While DLT and IPFS are designed for scalability, they can still face issues with scalability when it comes to storing SSI resources. As more users store their SSI resources on these platforms, it can become more difficult to scale the infrastructure to handle the increased demand.

		DLT and IPFS and other decentralized storages should not be considered for storing large collections of resources. For these cases, the hybrid solutions should be considered when the payload of the resource is stored off-chain and the hash for the resource is stored on-chain.

		- Security - resources must be stored in the reliable tamper-proof storage
		- Immutability - resource must be immutable

docs: compose the ADR for DID-linked resources. ATL-3186 [skip-ci] #518

docs: compose the ADR for DID-linked resources. ATL-3186 [skip-ci] #518

Conversation

yshyn-iohk commented May 2, 2023 • edited

Overview

Checklist

My PR contains...

My changes...

Documentation

Tests

github-actions bot commented May 2, 2023 • edited

Atala PRISM Test Results

github-actions bot commented May 2, 2023 • edited

github-actions bot commented May 2, 2023 • edited

E2E Test Results

bvoiturier left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

EzequielPostan left a comment

Choose a reason for hiding this comment

yshyn-iohk commented May 2, 2023 •

edited

github-actions bot commented May 2, 2023 •

edited

github-actions bot commented May 2, 2023 •

edited

github-actions bot commented May 2, 2023 •

edited