Updates to NFT TokenMetadata

[BenKurrek]

After discussions with @mattlockyer, we wanted to bring to the table potential updates to the NFT TokenMetadata.

Currently, the TokenMetadata might have too many unused fields and could be reduced. Most projects are going to use some decentralized storage to link to an off-chain JSON file that describes the metadata for their tokens.

It seems like we just need a few small things so we can have NFTs that basically state: "I'm not using TokenMetadata and my data is off-chain"

Approximately:

NFT ContractMetadata

• no changes

NFT Standard / Implementation

• include a flag for off-chain metadata in a lighter-weight struct.
• TokenMetadata is optional.
• include optional field directly on the token for reference (perhaps if this is present on token no need for flag)

Then you can have NFTs that follow the existing patterns AND lightweight NFTs that don't use TokenMetdata but instead have:

NFT ContractMetadata
- base_uri
Token
- reference

Where base_uri + reference = full URI to off-chain metadata. This would be a simple bump to the Metadata to include platforms that have stored JSON off-chain and allow for projects to avoid using the TokenMetadata (because it's cumbersome and no one seems to use it anyway).

To outline the proposal more directly --> currently we've got TokenMetadata which has:

type TokenMetadata = {
  title: string|null, 
  description: string|null, 
  media: string|null, 
  media_hash: string|null, 
  copies: number|null,
  issued_at: number|null,
  expires_at: number|null,
  starts_at: number|null,
  updated_at: number|null,
  extra: string|null, 
  reference: string|null,
  reference_hash: string|null 
}

The problem with this model is that not a lot of people are using all the fields. We propose lighter weight metadata which includes:

type LightTokenMetadata = {
  title: Option<String>,
  reference: Option<String>,
}

Where reference is combined with the base_uri (from NFT contract metadata) that points to information off-chain. This off-chain metadata would need to conform to a standard so that it shows up in the wallet correctly and isn't in a disorderly format.

When you query for token information, it would send back information from TokenMetadata, LightTokenMetadata, or if the token implements both, it would send back TokenMetadata and LightTokenMetadata.

The user has the option to pick either the TokenMetadata or the LightTokenMetadata (which includes the possibility for both).

[mattlockyer]

When you query for token information, it would send back information from TokenMetadata, LightTokenMetadata, or if the token implements both, it would send back TokenMetadata and LightTokenMetadata.

To further clarify the proposal and this point specifically.

I think we need to remove a majority of the current TokenMetadata fields that are mentioned by the standard, even if they are optional.

Because right now if we want to use TokenMetadata from https://github.com/near/near-sdk-rs/tree/master/near-contract-standards we simply have to provide None (a lot).

I think it's in our best interest to implement a standard that's minimal in nature e.g. only requiring title, reference, ??? and then allow developers to add their own fields.

To the point about what nft_token returns and composition:

Anyone can add extra fields or structs to their token, and return an "output" struct that satisfies a much more minimum set of standard metadata, with custom fields.

Summary:

I believe the standard and any "standard implementations" reduce TokenMetadata to it's absolute minimum the community agrees on from this discussion.

[ghost]

Thanks for @mattlockyer and @BenKurrek for starting this discussion.

What you propose @BenKurrek, I've been told, closely resembles how Ethereum stores theirs onchain.

I've read through the history of how this TokenMetadata came, which I encourage everyone to read - #171. Initially, metadata was inspired by ERC721 Metadata. Later on, it was further refined here - #177.

Before I read through the history, hearing how your proposal closely resembles ERC721 made me question why NEAR TokenMetadata became bloated but the reason is as I expected. In NEAR, it is feasible to store way more data onchain than Ethereum to associate more data to a token and provide consistent tailored experiences to NFT owners across the ecosytem.

So even though apps may not use most TokenMetadata fields now, it doesn't mean it's not serving intended purposes. NEAR makes it possible to do things that are infeasible elsewhere so I could say that those apps underutilising their potential. Also it'd be good to get statistics on apps using TokenMetadata so we can have a better conversation about its usages after all, this is an experiment to encourage rich dApps.

I wasn't in any discussion where similar solutions were proposed or privy to primary disagreements of this standard. However, I'm aware different NFT marketplace dApps created their TokenMetadata that weren't 100% compliant and use offchain storage services. Changing what's already stored would be a problem for everybody.

Hence it appears to be an offchain compliance issue not onchain. Whatever is taken out of the chain, is in another jurisdiction so compliance can't be enforced. However, storing offchain metadata is a common practice so what I propose is a new field called metadata_type added to TokenMetadata to represent storage type like so

type TokenMetadata = {
    metadata_type: string|null // new. "arweave" | "ipfs" | null 
    title: string|null, // ex. "Arch Nemesis: Mail Carrier" or "Parcel #5055"
    description: string | null, // free-form description
    media: string | null, // URL to associated media, preferably to decentralized, content-addressed storage
    media_hash: string | null, // Base64-encoded sha256 hash of content referenced by the `media` field. Required if `media` is included.
    copies: number | null, // number of copies of this set of metadata in existence when token was minted.
    issued_at: string | null, // When token was issued or minted, Unix epoch in milliseconds
    expires_at: string | null, // When token expires, Unix epoch in milliseconds
    starts_at: string | null, // When token starts being valid, Unix epoch in milliseconds
    updated_at: string | null, // When token was last updated, Unix epoch in milliseconds
    extra: string | null, // anything extra the NFT wants to store on-chain. Can be stringified JSON.
    reference: string | null, // URL to an off-chain JSON file with more info.
    reference_hash: string | null // Base64-encoded sha256 hash of JSON from reference field. Required if `reference` is included.
}

This way we are not removing the possibilities to enrich NFTs with onchain metadata vs. keeping everything useful offchain. I also propose that for a consistent consumer experience, offchain metadata should at least agree on few required fields - the rest can be dApp specific.

Let's keep the standard flexible, as NEAR is itself more flexible than others, so dApps that take advantage of this are not left out. I was due to make a pr for this but didn't finish, I'll mention this for reference.

[mattlockyer]

@evergreen-trading-systems really appreciate your time to break this down and read through the history.

Unfortunately what we've seen already is that despite NEAR being able to store a bunch of this data on-chain, many apps are not taking advantage of this.

Changing what's already stored would be a problem for everybody.

I'm not proposing that we ask anyone to change what's stored. I'm suggesting we lower the requirement for what's stored on-chain going forward.

Newer contracts can return a minimum set of metadata, including your metadata_type which absolutely should be included.

Newer contracts can "optionally" return or store more metadata on-chain if they like, but they are not required to use the large and cumbersome TokenMetadata struct found here: https://github.com/near/near-sdk-rs/blob/e88859955e0bb17c88c64ff59b15b7a9f79756c1/near-contract-standards/src/non_fungible_token/metadata.rs#L25-L38

Many of us are talking about minting thousands or millions of NFTs. The gas and storage implications of maintaining the current TokenMetadata structs, even if they are empty, will add up.

It's also a really poor developer experience. Here is a contract that "extends" the standard implementation:
https://github.com/near-apps/nft-series/blob/3df30ed14db592cdf20519066b595487c553da10/contract/src/lib.rs#L164-L180

Older contracts can still return more metadata then that's fine too.

What should actually be stored on-chain

I don't know how everyone feel about this, but to me it's always been a pretty clear line.

Only properties interacted with by contract methods or cross contract calls to other contract methods should be stored on-chain.

The rest of the data is static and should be hashed and stored off-chain, with the reference to this of course being on-chain.

[BenKurrek]

Thank you so much for taking the time and sharing your opinion and some very relevant history. My main concern, as Matt mentioned, is the storage and gas implications of keeping the current metadata structs. If the user had the option to use a more lightweight metadata struct, it could save on storage and it would scale a lot nicer - especially if you consider minting millions of NFTs.

Users can absolutely still use the heavier version of metadata (the current TokenMetadata struct) and this would be an optional addition to optimize gas and storage costs.

[microchipgnu]

@evergreen-trading-systems comment sums up our internal discussions over the past weeks.

My question is: should we standardize what metadata_type looks like?

@mattlockyer mentioned CAIPs on one of the recent meetings. Is anyone aware of something similar to this for decentralized filestorage systems?

[ghost]

Thanks @mattlockyer and @BenKurrek for sharing your concerns about costs for transacting large amount of tokens. It is a valid point and I was curious to further understand these implications in detail.

To that end, I simulated the scenario of storing 1 million tokens using optional and non-optional metadata. The results can be found here - https://github.com/evergreen-trading-systems/mint-million-tokens

In a nutshell, there are significant savings to be had. I agree that large scale dApps that want to utilise these advantages shouldn't be forced to store data in an unnecessary format.

@chadoh proposed the current metadata standard, would be good to hear an opinion. Regardless, I think it's clear to everyone the benefits to be had.

Based on this, if the current standard were made optional for dApps who want to utilise onchain metadata, it applies the rest would be queried offchain. This is where the base_uri stored on NftContractMetadata and reference_hash stored on TokenMetadata come to play. It now would make more sense for the reference_hash to be stored elsewhere as it is required for offchain access but not useful onchain.

I'm not an expert on offchain storage but I understand given a base_uri of https://abc.com and a reference_hash of 123 the end result could be https://node1.com/123. This is because the base_uri indicates the storage network but URL to the node could be different. Personally, I think this is not a metadata concern and dApps should be responsible for URL resolutions. Currently, base_uri is a string and we need not enforce a list of storage providers. My previous suggestion of metadata_type is not needed.

To sum it up

// Offchain only token - storage efficient
Token {
    metadata:None,
    reference_hash: Some("123"),
    ..Default::default()
}

// Onchain only token - less efficient
Token {
    metadata:Some(TokenMetadata::default()),
    reference_hash:None,
    ..Default::default()
}

// Off and on chain token - least efficient
Token {
    metadata:Some(TokenMetadata::default()),
    reference_hash:Some("123"),
    ..Default::default()
}

Other metadata fields like copies could be extracted out as reference_hash if is necessary. If we don't agree such fields should be extracted to the Token level then the alternative solution would have those fields as metadata and the rest as optional extras. Though I'd argue at that point, the cost savings wouldn't be significant anymore. I don't think we've seen a lot of enterprise momentum for this.

Token {
  TokenMetadata {
    name:"",
    copies:1,
    reference_hash:Some("123"),
    extra:None,
  }
 ..Default::default()
}

[mikedotexe]

Good convo. A couple of things to point out.

I think in the last section here I was finding myself confused and think I'll clarify what's written in Nomicon. The TokenMetadata's reference is the URL location where you can find more details, like the description of what motivated the author to draw this upside-down seahorse depicted in the NFT. The reference_hash is intentionally stored on-chain because we cannot be certain we trust the reference, since it lives in the outside world. Someone could poison DNS, forget to renew a domain, or get man-in-the-middle attacked. The reference_hash is the hash of the JSON file, so if a person downloads the JSON file, hashes it, and compares it to reference_hash and they're different, something is wrong.

Secondly, regarding minting a million tokens, I certainly hope someone doesn't write a smart contract that stores a bunch of ones and zeroes needlessly like this. If you have an NFT and only want to store title and nothing else, you should definitely not store anything else. The way standards work is we specify what the function names, arguments, and return values should look like. In this example, you'd have something like:

/// This is all we want to store, and all we need to store
pub struct StoredTokenMetadata {
  title: String
}

/// We only use this when returning token info, we don't store this
pub struct StandardTokenMetadata {
    pub title: Option<String>, // ex. "Arch Nemesis: Mail Carrier" or "Parcel #5055"
    pub description: Option<String>, // free-form description
    pub media: Option<String>, // URL to associated media, preferably to decentralized, content-addressed storage
    pub media_hash: Option<Base64VecU8>, // Base64-encoded sha256 hash of content referenced by the `media` field. Required if `media` is included.
    pub copies: Option<u64>, // number of copies of this set of metadata in existence when token was minted.
    pub issued_at: Option<String>, // ISO 8601 datetime when token was issued or minted
    pub expires_at: Option<String>, // ISO 8601 datetime when token expires
    pub starts_at: Option<String>, // ISO 8601 datetime when token starts being valid
    pub updated_at: Option<String>, // ISO 8601 datetime when token was last updated
    pub extra: Option<String>, // anything extra the NFT wants to store on-chain. Can be stringified JSON.
    pub reference: Option<String>, // URL to an off-chain JSON file with more info.
    pub reference_hash: Option<Base64VecU8>, // Base64-encoded sha256 hash of JSON from reference field. Required if `reference` is included.  
}

pub struct NFTContract {
  tokens: TreeMap<String, StoredTokenMetadata>
}

impl NonFungibleTokenCore for NFTContract {
  fn nft_token(&self, token_id: String) -> Option<Token> {
    // blah blah blah
    let title = self.tokens.get(token_id).title;
    let metadata = StandardTokenMetadata {
      title,
      None,
      None,
      None,
      None,
      None,
      None,
      None,
      None,
      None,
      None,
      None,
    }
    // or could probably use the …Default::default() dealio above
    Some(Token { 
      token_id, 
      // owner_id (not covered in this example snippet I'm writing, 
      metadata, 
      // approved_account_ids (not covered in this example snippet I'm writing, 
    })
  }
}

This is how you'd write it and a million of these would be negligibly identical in storage. Let's just keep in mind that standards aren't telling anyone how they should store ones and zeroes, but in this case, only what to expect will be returned.

[mikedotexe]

I agree we should come to a consensus on what's reasonable to cut from the TokenMetadata. I'm finding myself thinking how strange it is to fit this into "versions." I don't want to complicate things too much, but I think a lot of discussion is around art.

• If a proof-of-attendance person were here (like https://poap.xyz) they might be in favor of having issued_at, and suggest we have coordinates or zip codes or something.
• Tomorrow I'm meeting with the CEO of https://unlock-protocol.com who use NFTs for subscriptions, who might also value timestamps, including expires_at.
• Uniswap V3 issues liquidity pool NFTs, and might want range values in there, and would definitely need those on-chain.

Speaking to that last bullet point, I think there are useful reasons for folks to have on-chain metadata that hasn't been mentioned yet, and perhaps this is because we're sharing an assumption that a marketplace frontend somewhere will have no problem fetching info from decentralized storage since the end user is already in a browser. That doesn't cover a fully, on-chain workflow, however, so let's just keep that in mind.

But more to the point, perhaps what we're looking for aren't "versions" of the NFT Token Metadata but variants. Like:

• collectible (art, most of the stuff we commonly see with NFTs. You like it but you might part with it for incentives.)
• experiential (high school diploma, proof-of-attendance. You purely like it. It would be silly to trade or sell it.)
• utility (allows you to interact with something, like Uniswap LP token or Unlock membership NFT)

[chadoh]

Great points about NFT variants. It might be useful to make a streamlined process for proposing & quickly standardizing variants. Maybe we could lean on the extra field:

extra: anything extra the NFT wants to store on-chain. Can be stringified JSON.

Maybe we should:

• Make a “Variants” sub-category in Nomicon’s NFT section
• Explain that Variants only attempt to standardize what will show up in the extra field for a given NFT (and maybe the spec field?)
• Invite people building (collectibles|experientials|utilities) to submit Variant proposals
• Each Variant listed on the Variant page can include information about the Variant “owner” (or “champion” or “author” or whatever, please come up with a better word), perhaps with contact information
• Merge them with minimal discussion
• When other teams come along and discover a Variant that is almost what they want, encourage them to discuss and modify existing variants with original authors rather than adding almost-identical Variants.

[VeronikaSemykopna]

👍