Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Metadata Key/Value Association & Store #8

Open
kaiyzen opened this issue Jan 17, 2019 · 12 comments

Comments

Projects
None yet
7 participants
@kaiyzen
Copy link
Member

commented Jan 17, 2019

Metadata Key/Value Association & Store

-WORK IN PROGRESS-

Introduction

Currently in the version 2 protocol and storage format there is no formalized way to associate related metadata to accounts, assets, etc. This proposal outlines the aproach to introducing a key/value storage implementations

NIP: ?
Layer: Core Layer, API/SDK
Author: nem eu
Interest
Group:
sig-api
Status: Active
License: ASL v2

Specification

Data Model

Storage of assocaited key/value pairs, target supported objects:

  • Accounts
  • Assets
  • Namespaces
  • Transactions

Key and Value Size Limits

to be finalized

Expiration

It is desired to have expiration functionality in order to have key/value pairs expire and be null'd out and/or pruned

Audit History/Pruning

Current design thinking is that full history will not be stored by default, only the latest value for any asserted key. To accomidate build in audit history it will be designed in such a manner it can be a configurable setting which could take a couple approaches:

One could default to entire metadata history to be stored:

MetaDataAuditHistory=true

Which would perist and make retrievable the entire history, until forceably pruned/truncated out.

Another approach that could be taken is to have a configuration setting that would take an integer value for a maximum number of historical key/value pairs to be kept around:

MetaDataAuditHistoryDepth=3

Which would store the current asserted value, plus the last two asserted value(s).

TODO: decide if any configuration support for expiration, is it baked in by default, can it be turned off, etc

Motivation

Various wants and needs when using the existing protocol is to associate one or more pieces of metadata to various objects to enable a fuller solution.

Typically to date most have leveraged the description field on assets and the message field in transactions

Design Decisions

Some decisions and tradeoffs will need to be made balancing capabilities, developer/user experience, and overall management of a chains size and networks performance over its lifecycle. Some topics that stand out that will need more detailed design and discussion:

  • Expiring key/value pairs
  • History/Pruning handling and implementation
  • Permissions around Account based key/value pairs and allowing any Account to tag any other Account
  • Display/Search UX

Implementation

tbd

Backwards compatibility

This functionality is currently targeted for initial support in the up and coming "E" release. TODO: add link ref here

Drawbacks

Increased storage pressure, possible reduction in TPS support depending on deployment and configuration

References

TODO: add any refs here

History

Date Version
Jan 17 2019 Initial Posting
@jabo38

This comment has been minimized.

Copy link

commented Jan 21, 2019

This is going to add a lot of utility to Catapult, especially for "Blockchain of Information" projects. It would be nice to think of a good name for it.

For the TODOs. Here are my first thoughts.

  1. Expiring Key/Value: Expirations, of course, would be nice. Static and perpetual is more important though. So if it is a choice of one or the other, I choose, static. But of course, if we can have expirations and/or static, then that opens up more options for business use cases.

  2. Pruning: I think businesses can live with or without this case. To me, this is more of a network, engineering, scale issue.

I'm not completely sure the relationship between pruned and expiration is, but assuming it is expired, then it is good to prune it. But given that it is static and perpetual, then I'm assuming it can't and shouldn't be pruned?

Also, about the value being pruned. I'm assuming that if Alice writes on my account with the Key of "email" and Bob writes on my account the Key of "address", and Charlie writes on my account the Key of "phone number" that even though these three were done at different times that each of these keys/values remain? Lastly, then I could go over Alice, Bob, and Charlie's Key/values and make the values all null by leaving the values blank in an edit?

  1. Account Permissions: I think mirroring the Account Properties would work very well here. So for instance.
    A) Anybody/Nodody can do any kind of key/value designation. (By default I would have it set to "Nobody but the account itself")
    B) Only this "key/value" can be assigned by other accounts.
    C) Only a white list of accounts may assign "key/value" to an account

Really, for account permissions, if you just had "only this white list can assign key/values" that will cover most use cases IF we can combine these with aggregates, e.g., all permission are turned off, I make an aggregate with three parts. Part 1: turn on Alice being able to edit key/value data on my account, Part 2: Alice edits say key/value on my account. Part 3: I turn off Alice from being able to edit the key/value. Then basically I am approving Alice each time she edits my account.

  1. UX: I think most of the times wallets and apps would only state the latest key/values.
@daoka

This comment has been minimized.

Copy link

commented Jan 22, 2019

you just had "only this white list can assign key/values" that will cover most use cases

good idea 👍
In addition, I think the white list should edit only account owner or asset owner or namespace owner or transaction or participants in the transaction

UX: I think most of the times wallets and apps would only state the latest key/values.

I think so too.
But, change histories should able to trackable by transaction history API or some other API.
It is important in the auditing of apostille or traceability. IMO.

@daoka

This comment has been minimized.

Copy link

commented Jan 22, 2019

Have inner transactions metadata?

@jabo38

This comment has been minimized.

Copy link

commented Jan 26, 2019

It would be nice to know if inner transactions each can have their own metadata. I'm not sure the implications of this but it makes my mind think of things.

Also, wrt Apostille like transactions, it would be nice if we could have a non-mutable option for metadata on an account. So that it was written and fixed forever, not even nullable.

@jorisadri

This comment has been minimized.

Copy link

commented Feb 11, 2019

Also, about the value being pruned. I'm assuming that if Alice writes on my account with the Key of "email" and Bob writes on my account the Key of "address", and Charlie writes on my account the Key of "phone number" that even though these three were done at different times that each of these keys/values remain? Lastly, then I could go over Alice, Bob, and Charlie's Key/values and make the values all null by leaving the values blank in an edit?

I think who sets the metadata is owner of the metadata. So when A set a metadata with key "X" to B, only A should be able to change the value of "X".

History could be kept by only including the hash of the key/value in the "metadata" transaction and key/value data itself in the state of the object (account, mosaic....) (just an example of storing the history). This allow removing of sensitive data by the owner of the metadata. It can still be validated that it was there at a certain point. I don`t think this is gdpr complient but thats up to users itself.

  1. Account Permissions: I think mirroring the Account Properties would work very well here. So for instance.
    A) Anybody/Nodody can do any kind of key/value designation. (By default I would have it set to "Nobody but the account itself")
    B) Only this "key/value" can be assigned by other accounts.
    C) Only a white list of accounts may assign "key/value" to an account

A) I would set default to "Everybody". This because it gives the capability to use accounts as assets withoud knowing the privatekey.
Metadata could have some privacy issues but thats the same for mosaics and messages.
B) Not sure if this is helpfull.
C) I aggree that it should be possible to whitelist/blacklist accounts to be able to send metadata to your account.

@jorisadri

This comment has been minimized.

Copy link

commented Feb 28, 2019

An idea for metadata

An assumption of metadata, there are three keys used to set the metadata (value). These are: (1) sender account, (2) receiver object type, (3) metadata key.
With multiple keys, it is possible that:

  • a sender can send to multiple objects
  • multiple senders can send to the same object (if rules allow that)
  • a sender can set multiple metadata to the same object

Questions

  1. Expiring key/value pairs
  2. History/Pruning handling and implementation
  3. Permissions around Account based key/value pairs and allowing any Account to tag any other Account
  4. Display/Search UX

Answers

  1. Metadata should expire if the linked object expires too. If the object cannot expire (accounts), there should be a possibility to set expiry after a number of blocks. Keeping data of objects that do not exist anymore in the state cache, can be seen as bloat and will grow the state cache.

objects that can receive metadata are:

Entity Who can set Who can change Info
Namespaces Any account Setter of the metadata pruned* when namespace expires
Mosaics "" "" pruned* if mosaic expires
Transactions "" ""  
Accounts "" "" ability to block other accounts linking metadata to your account. Default: allow anyone to set metadata to account

* pruning means that the data object is removed from the state of the object. The metadata (hash of the key/value pair) is still available in the blockchain history (pruning deserves its own discussion).

  1. Transaction history should always be kept. Imo to slow down blockchain size growth, a hash of the metadata keypair is sufficient. History for the metadata keypair can be held by nodes that choose to do so. Consider a business running their own node doing so. The majority might only hold the current state. If a client wants to get history of the metadata, the client can choose to connect to a node that has the full history.
    An implementation could be something like this: When setting metadata, a transaction is made with all the data. The blockchain validates the transaction and only stores the transaction and the hash of the key/value pair. This still allows the validation of the history of the metadata if you have a copy of the data stored. An extra boolean MetaDataAuditHistory=true can be added to the REST server configuration, to allow storing the history of the data.
  2. By default, setting metadata to another account should be without restrictions. To whitelist/blacklist accounts the account filter feature should be updated to support the metadata entity type.
    This allows for accounts to be used as an asset without the need to have the privatekey. If the key is known. The owner can prevent spamming by blocking incoming metadata transactions.
  3. This is more an API gateway implementation. There should be query possibilities for metadata on keys, objects and senders(more queries if MetaDataAuditHistory=true to get history data).

The key for me is to have metadata as universal to all the objects as possible.

@jorisadri

This comment has been minimized.

Copy link

commented Feb 28, 2019

Examples for metadata

Below I am giving examples for the use of metadata. Before doing so, I need to address the concept of the immutability of metadata:
I assume when metadata is set to an object, it can be changed. In some use-cases, this can result in less trust in the value of the metadata because it can be changed. Having a boolean “immutable” when setting the metadata, it can help to gain trust in the value of the metadata because it cannot be changed.

Account metadata

Non-fungible token (NFT).
Set metadata to an account to make a unique asset. Consent is needed to model this on the blockchain.
This can already be done with transfer transaction messages, but then the model of the NFT is not in the state but in the history of the asset account.
Remarks: in this scenario, the asset account should accept metadata by other accounts as default. Have the possibility to set immutable metadata so when for example the NFT description cannot be changed afterwards.

Access control.
Set metadata to an account that needs to have temporary access to your system. Your system checks metadata and signature. When valid, give access. When expired, no access. Metadata can also be modified by setter so setter can revoke access.
This can already be done with non-transferable mosaics but access cannot be revoked.
Remarks: in this scenario metadata that can expire is needed so expiring of access can be modelled on the blockchain.

DNS.
Set metadata using a specific account to a specific account with key: "domain name" and value:" IP address". This allows for secure DNS on the blockchain with the possibility to change IP addresses.
This can already be done by transfer transaction messages. However, updating IP addresses can become difficult and looking up is more demanding as the transactions grow.
Remarks: It should be possible to change the metadata (only) by setter of the metadata.

Transaction metadata.

Message
Add messages for extra information to any transaction. This can replace the messages that are used in transfer transactions.
Remark: All transaction types should be able to receive metadata including inner transactions

Receipt
When Alice sends a transaction to Bob, Bob can set metadata to that transaction to state what for service will be delivered for the transaction.
Remarks: Bob should be able to set metadata to a transaction that is not signed by Bob.

Namespace metadata

Description of the namespace
Add description to namespaces/subnamespaces.

Mosaic metadata

Description of the mosaic
Add description to mosaic.
Remarks: should be possible to set metadata to immutable so the description cannot be changed if the description tells the purpose of the mosaic.

Please feel free to improve!

@jontey

This comment has been minimized.

Copy link

commented Mar 6, 2019

It should also be possible to set max value size for the <key>:<value> pair of metadata to prevent abuse of the system.

Regarding @jorisadri comments for access control. It is possible to have a simple access control list using account properties to whitelist accounts that can send transactions to it, hence also limiting which accounts can edit or add metadata.

@jorisadri

This comment has been minimized.

Copy link

commented Mar 11, 2019

@jontey in my opinion, when metadata is set, it should always be possible for the setter to edit the <value> from the <key>:<value> pair (when metadata object is set to mutable).

@gimer

This comment has been minimized.

Copy link
Member

commented Mar 12, 2019

I have a feeling discussion above should probably be restarted, because it seems everyone has different understanding of what metadata IS and how it will behave.

Important thing to note here is: server will not parse OR recognize keys in ANY WAY
That has few implications:

  1. That means metadata cannot be used for any kind of RULES, that are supposed to be guarded by the server.
  2. Server could have only simple rule like allow overwrite, disallow overwrite.
  3. Verifiability of that data should still be discussed. (more on that later if there will be need)

If you need any kind of rules, what you're asking about is NOT metadata and should not be discussed within this NIP.

@alvin-reyes

This comment has been minimized.

Copy link

commented Mar 21, 2019

I'd like to weigh my cents here. We've been playing around this metadata idea at ProximaX for a while and I have a very simple approach to it. The entire reason why we want a metadata key-pair is to have an additional data store on all the possible plugins both existing and custom ones. We want to have a generic metadata so that one can attach any association without any rules to it. If one wants to apply some rules, then a custom plugin should be built in accordance to the framework or architecture of catapult.

My suggestion is to just create a key pair object on an abstract object used by any plugins. The size of this object should be limited. In very simple terms, it's like a message field but applied to all plugins :) This is currently what I'm trying to do on our side and we want to it as soon as possible because we see a great deal of use for this.

I'll be able to show some code once it's done it but I'd like to get some thoughts/feedback if this is the intention of this NIP. If not then it's all ok.

@jabo38

This comment has been minimized.

Copy link

commented Mar 24, 2019

Hi Alvin,

I think your concept of metadata is similar to much of the discussion, especially as highlighted by Gimre, but mainly the discussion centers around accounts, namespaces, and mosaics. Can you explain a need for it to be around all plugins?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.