KodaDot - Atomic Swap Indexer & API for AssetHub NFTs

[vikiival]

Project Abstract

AssetHub Indexer is a GraphQL service that streamlines interaction with singles, non-fungible NFT assets and fungible assets on AssetHubs. The service was specifically designed to serve as a solid data layer for NFT-oriented dApps and leverages GraphQL and REST API simplicity.

In Milestone 2, we focus on significant improvements to the AssetHub indexer to streamline NFT transactions and improve the collection management of the Polkadot network. Our updates focus on the integration of Atomic Swaps, Offers, Architectural Upgrades, and overall improvement of the developer's experience.

This proposal is a follow-up of Proposal 1 for AssetHub NFT indexer

• KodaDot - AssetsHub NFT indexer - Milestone 1  #1864

Grant level

• Level 1: Up to $10,000, 2 approvals
• Level 2: Up to $30,000, 3 approvals
• Level 3: Unlimited, 5 approvals (for >$100k: Web3 Foundation Council approval)

Application Checklist

• The application template has been copied and aptly renamed (project_name.md).
• I have read the application guidelines.
• Payment details have been provided (Polkadot AssetHub (DOT, USDC & USDT) address in the application and bank details via email, if applicable).
• I understand that 30% of each milestone will be paid in vested DOT, to the Polkadot AssetHub address listed in the application.
• I am aware that, in order to receive a grant, I (and the entity I represent) have to successfully complete a KYC/KYB check.
• The software delivered for this grant will be released under an open-source license specified in the application.
• The initial PR contains only one commit (squash and force-push if needed).
• The grant will only be announced once the first milestone has been accepted (see the announcement guidelines).
• I prefer the discussion of this application to take place in a private Element/Matrix channel. My username is: @_______:matrix.org (change the homeserver if you use a different one)

[PieWol]

Hey @vikiival ,
thank you for the application.

Sadly our recent changes to the PR template were a little late but, as we want to allow grantees to stake their vested DOT, you need to provide us with a native Polkadot (not AssetHub) address. Sorry for the inconvenience.

[vikiival]

Hello @PieWol I have Updated the links 👍

[vikiival]

Hello @PieWol 👋,
I would like to kindly ask we if can move forward with the proposal?

[PieWol]

Ah yes, very well @vikiival
With the changes in place we will start reviewing the application. Thanks for the changes.

[PieWol]

Just one more thing. We usually name the different parts of a single application "Milestone" so it doesn't make a lot of sense to use this term in the title for an application itself. Would you be able to come up with a distinct but fitting name for the follow up work on the indexer? This would be a nice change before the committee looks at the application.

[takahser]

@vikiival I've taken a look your proposal and have the following questions:

1. Can you elaborate on this? What do you mean by "function" here, could you make some examples maybe?

We need to obtain the data for the correct function of our indexer.

2. I also have a couple of comments regarding the DB Model:

• It's rather odd to me that you'd add the Entity suffix to some of the entities (e.g. NFTEntity), but for some you'd skip it (e.g. Offer). Is there any meaning to that?
• Why does the NFTEntity have 2 relations with TokenEntity - one 1-to-many and on many-to-many?
• Not sure if that should be a question:

  Next are Swap and Offer that will represent Atomic Swaps and Offers built on top of the Atomic Swaps?

• Wouldn't it be more straight-forward to skip the TokenEntity and index the DB accordingly to your needs? This would also dev to "perform search operations effectively". In case you disagree (e.g. if that is some new concept that I'm not aware of), please provide us with the detailed data structure for these entities.

  TokenEntity represents a list of NFTEntity grouped by the same MetadataEntity.

3. Could you explain what you're trying to swap in particular? NFT to NFT? Or NFT to FT? Or both?

4. To stay on the topic of Swaps, have you looked into the asset_conversion pallet that is already live on AssetHub? How does it compare with that? If I understand correctly, your solution is not decentralised (while asset_conversion is). Is there any reason you decided to choose this route, rather than a more decentralised approach?

5. What are the exact trust assumptions for user using this centralised swap protocol?

[vikiival]

Can you elaborate on this? What do you mean by "function" here, could you make some examples maybe?

E.g To make your car work, you need gas :)

The data we will use are Events and Extrinsics defined in the nfts-pallet

Is there any meaning to that?

The entity suffix was there for legacy reasons

Therefore, for new entities we decided to drop that (Developer experience :)

Why does the NFTEntity have 2 relations with TokenEntity - one 1-to-many and on many-to-many?

To understand better what is a TokenEntity I have an example with pictures.

Imagine you have distributed more than 800 Poaps with the same picture

This is cool to see on the collection page, but this would be annoying in the case of a "general" explore page.
Therefore, the idea is to "stack" the NFTs (with the same image) under one entity (TokenEntity).
It implies that TokenEntity holds Array<NFTEntity> where NFTEntity can be owned by just one TokenEntity

Please look on the generated TypeORM model

@Entity_()
export class NFTEntity {
    // Rest was skipped

    @Index_()
    @ManyToOne_(() => TokenEntity, {nullable: true})
    token!: TokenEntity | undefined | null
}

@Entity_()
export class TokenEntity {
    @OneToMany_(() => NFTEntity, e => e.token)
    nfts!: NFTEntity[]
}

Wouldn't it be more straight-forward to skip the TokenEntity

Please check the comment above

Could you explain what you're trying to swap in particular? NFT to NFT? Or NFT to FT? Or both?

The logic is built on the top of the nfts-pallet deployed on the AssetHub chain.

As we can observe in the link above, the nfts-pallet offers only swapping Non-fungible to non-fungible token (allowing surcharge in DOT.)

What are the exact trust assumptions for user using this centralised swap protocol?

As core logic lies on the decentralized network (which Polkadot is) I do not see the point of centralization.

Moreover, SubSquid is also a decentralized network, and because 100% of the data comes from the chain (which is replicable), there is also no point of centralization.

Side note

This project aims to bring an easy-to-use solution for projects that want to leverage the power of nft-pallet on the AssetHub.

Hope I clarified all your questions @takahser and we can move forward 👍.

[vikiival]

cc @PieWol @keeganquigley 🥺 🙏

[PieWol]

Hi @vikiival ,
Are there some stats you can point me to that represent the public demand of your indexer since inception?
Also I noticed that the items you listed in your "future plans" section are almost identical to the ones in your previous application in your indexer. I'd assume you'd make reasonable progress on your "future plan" items in this very grant so that you would list new items in this section now. Just curious 😉

[vikiival]

Are there some stats you can point me to that represent the public demand of your indexer since inception?

In last 20 day we have served more than 500k+ queries via M1 indexer alone.

It is a powering @TalismanSociety's NFT UI

@novasamatech NFT section in their mobile wallet.

Both of them has big amount of users ^-^ and we are very happy to make ecosystem awesome by providing them an infrastructure that is out of their capacity.

Also I noticed that the items you listed in your "future plans" section are almost identical to the ones in your previous application in your indexer

Must have slipped while I was shuffling with the files
Updated in e6e2038

[PieWol]

Hey @vikiival ,
thanks for providing the numbers. I'm happy to see that other ecosystem teams are using the indexer already. Thats really compelling.

[PieWol]

@vikiival
I am under the impression that a simple NFT to NFT swap is what you are calling Atomic Swap in this application. Is this correct? Or are you actually preparing the indexer to handle cross-chain swaps as in https://github.com/paritytech/polkadot-sdk/tree/master/substrate/frame/atomic-swap ?

[vikiival]

This milestone focuses on atomic swap functionality implemented in the nfts-pallet.

I will chceck the pallet you have sent, I was not awarr of that one, thanks!

[PieWol]

Got it, thanks for clarifying. I have to say whoever decided on the terminology of the nfts pallet has made a questionable choice by calling a simple swap "atomic swap" instead. Anyway it doesn't change anything about my approval.

Terminology listed here https://docs.rs/crate/pallet-nfts/latest

[takahser]

@vikiival thx for your replies.

The data we will use are Events and Extrinsics defined in the nfts-pallet

Is there any meaning to that?

The entity suffix was there for legacy reasons

Wouldn't it be better practice to refactor the naming? It seems quite messy to me to mix the two terminologies (with and without Entity suffix).

Why does the NFTEntity have 2 relations with TokenEntity - one 1-to-many and on many-to-many?

This is cool to see on the collection page, but this would be annoying in the case of a "general" explore page. Therefore, the idea is to "stack" the NFTs (with the same image) under one entity (TokenEntity). It implies that TokenEntity holds Array<NFTEntity> where NFTEntity can be owned by just one TokenEntity

So the only difference between a collection and a tokenentity is that the latter will display the img only once? If that's the case, I think there's no justification to duplicate the whole logic and you should let the UI handle it instead (and maybe add a binary attribute stackImages or something similar).

Please look on the generated TypeORM model

@Entity_()
export class NFTEntity {
    // Rest was skipped

    @Index_()
    @ManyToOne_(() => TokenEntity, {nullable: true})
    token!: TokenEntity | undefined | null
}

@Entity_()
export class TokenEntity {
    @OneToMany_(() => NFTEntity, e => e.token)
    nfts!: NFTEntity[]
}

You're showing the same relation, just from different perspective: NFTEntity's ManyToOne relationship becomes a OneToMany relationship when viewed from the TokenEntity's PoV. But that doesn't explain why you have two different relationships in the diagram: 1-to-many and 1-to-1.
Also, I noticed the same holds true for the relations between CollectionEntity and NFTEntity.

Could you explain what you're trying to swap in particular? NFT to NFT? Or NFT to FT? Or both?

The logic is built on the top of the nfts-pallet deployed on the AssetHub chain.

As we can observe in the link above, the nfts-pallet offers only swapping Non-fungible to non-fungible token (allowing surcharge in DOT.)

What's "allowing surcharge in DOT"? Does it mean that if the NFTs vary in value, the missing value will be paid in DOT?

[vikiival]

So the only difference between a collection and a tokenentity is that the latter will display the img only once? If that's the case, I think there's no justification to duplicate the whole logic and you should let the UI handle it instead (and maybe add a binary attribute stackImages or something similar).

Updated in 00b1bf8

You're showing the same relation, just from different perspective: NFTEntity's ManyToOne relationship becomes a OneToMany relationship when viewed from the TokenEntity's PoV. But that doesn't explain why you have two different relationships in the diagram: 1-to-many and 1-to-1. Also, I noticed the same holds true for the relations between CollectionEntity and NFTEntity.

why you have two different relationships in the diagram: 1-to-many and 1-to-1.

Because that's how Subsquid handles that :).
Please take a look at the documentation

So the only difference between a collection and a tokenentity is that the latter will display the img only once? If that's the case, I think there's no justification to duplicate the whole logic and you should let the UI handle it instead (and maybe add a binary attribute stackImages or something similar).

Collection (CollectionEntity) Holds a list of Item (NFTEntity)

What could happen is that Item can share same ItemMetadata and it causes heavy performance and UI issues. That's why we have decided to "group" NFTEntity sharing the same MetadataEntity into one bag called TokenEntity.

The logic is not duplicated, it is our engineering approach of making things fast :)

UI handle it instead

We have tried that, but It does not work.
Fetch 12 NFTs per page with the same metadata. Group them on the front end; you are left with one item on the page. Additionally, what about other UIs using indexer? Should we tell them to implement it by themselves? Therefore, we have decided to use the approach: fix once - use everywhere.

What's "allowing surcharge in DOT"? Does it mean that if the NFTs vary in value, the missing value will be paid in DOT?

Let us derive a situation.
I have an iPhone 13, which I want to swap for your iPhone 15.
You have agreed that I will give you an additional 20 DOT.
This is what pallet allows you to do

My surcharge is a representation of PriceWithDirection<Amount>

[takahser]

@vikiival

Updated in 00b1bf8

The diagram doesn't reflect that.

why you have two different relationships in the diagram: 1-to-many and 1-to-1.

Because that's how Subsquid handles that :).
Please take a look at the documentation

No, that's not how subsquid handles that. The link you've shared contains examples for the following relationships:

• One-to-one
• Many-to-one/One-to-many
• Many-to-many (using a joint table)

There is no example of two entities that have both 1-to-many AND 1-to-1 relationships because it doesn't make any sense.

Collection (CollectionEntity) Holds a list of Item (NFTEntity)

What could happen is that Item can share same ItemMetadata and it causes heavy performance and UI issues. That's why we have decided to "group" NFTEntity sharing the same MetadataEntity into one bag called TokenEntity.

The logic is not duplicated, it is our engineering approach of making things fast :)

Just to be clear: I don't think it's a problem to abstract away the MetaData as you do with MetadataEntity. I was just questioning whether both CollectionEntity and TokenEntity are needed, if the only difference is that the TokenEntity will display the img only once (which you didn't answer).

Let me use code to illustrate what I mean.

// TokenEntity contains an array of nfts
@Entity_()
export class TokenEntity {
    @OneToMany_(() => NFTEntity, e => e.token)
    nfts!: NFTEntity[]
}

// Similarly, CollectionEntity contains an array of nfts
@Entity_()
export class CollectionEntity {
    @OneToMany_(() => NFTEntity, e => e.token)
    nfts!: NFTEntity[]
}

// each nft contains metadata, no matter if it's part of a CollectionEntity or TokenEntity
@Entity_()
export class NFTEntity {
    @OneToOne_(() => MetadataEntity, e => e.meta)
    meta: MetaEntity
}

One solution would be to introduce an additional attribute to CollectionEntity that contains the common metadata:

// Similarly, CollectionEntity contains an array of nfts
@Entity_()
export class CollectionEntity {
   // the "meta" attribute in these nfts would be empty, but if not, it's ignored, if "commonMetaData" is populated
    @OneToMany_(() => NFTEntity, e => e.token)
    nfts!: NFTEntity[]

    // we add this
    @OneToOne_(() => MetadataEntity, e => e.meta)
    commonMetaData?: MetaEntity
}

As far as I can see you already have that relationship in your diagram:

That way, the whole TokenEntity would become obsolete. I'm not sure if it's the best solution, since the rule to ignore downstream meta infos within NFTEntity is an ugly dependency as well, but at least you don't need two almost identical entities. I'm also not sure if TokenEntity is a good name for an entity that contains a lot of NTFEntity objects where "T" obviously stands for Token as well - kinda strange, if you ask me.

Fetch 12 NFTs per page with the same metadata. Group them on the front end; you are left with one item on the page. Additionally, what about other UIs using indexer? Should we tell them to implement it by themselves? Therefore, we have decided to use the approach: fix once - use everywhere.

Personally, I think having 2 almost identical entities is much more confusing. It's definitely possible to implement it in other UIs that way as well.

[vikiival]

The diagram doesn't reflect that.

Probably cache?

I will try to explain

• CollectionEntity has one-to-one connection to MetadataEntity because

• CollectionEntity has one-to-many connection NFTEntity because each NFT belongs to some connection

There is no example of two entities that have both 1-to-many AND 1-to-1 relationships because it doesn't make any sense.

• We have learned the hard way

One solution would be to introduce an additional attribute to CollectionEntity that contains the common metadata:

This would not be doable for ERC-1155 tokens

Personally, I think having 2 almost identical entities is much more confusing.

TL;DR: Collection represents on-chain collection where as Token represents a grouping of Items with the same metadata - (you can have n different metadata for)

# Entity to group NFTEntity by common metadata
# grouping is done either by NFTEntity.image or NFTEntity.media 
# https://github.com/paritytech/polkadot-sdk/blob/b8ad0d1f565659f004165c5244acba78828d0bf7/substrate/frame/nfts/src/lib.rs#L293

If you have any questions please send me a message on telegram (same handle)

Also @takahser can I please ask you for swiftier responses?
This proposal is already opened for a month.

[takahser]

@vikiival

Probably cache?

I guess the only one that's not updated is Event:

I will try to explain

• CollectionEntity has one-to-one connection to MetadataEntity because
• CollectionEntity has one-to-many connection NFTEntity because each NFT belongs to some connection

This still doesn't explain the 2 relationships between CollectionEntity and NFTEntity:

• We have learned the hard way

Please elaborate. Sorry for the delays here, but we currently have a very large backlog in both the W3F Grants program as well as the Decentralised Futures program. We're trying to be fair to all teams, and I've repeatedly asked you for an explanation on these things (e.g. the previous point), without having received a detailed explanation. Proving links is fine, as long as it's clear to what you're referring to, however, that's not the case here.

This would not be doable for ERC-1155 tokens

Also here: Feel free to elaborate.

TL;DR: Collection represents on-chain collection where as Token represents a grouping of Items with the same metadata - (you can have n different metadata for)

So Collection is a representation of an on-chain concept, but Token is just a view model, aggregated for the frontend?!

[vikiival]

This still doesn't explain the 2 relationships between CollectionEntity and NFTEntity:

Collection (CollectionEntity) Holds a list of Item (NFTEntity)

So Collection is a representation of an on-chain concept, but Token is just a view model, aggregated for the frontend?!

Collection is therefore "representation of an on-chain entity" where as TokenEntity is very smart table for clients to get data in the better way,

Let us take a look on the biggest NFT collection ever made in Polkadot Ecosystem

select count(*) from nft_entity;
-- 719_296

select collection_id, count(*) from nft_entity group by collection_id order by count(*) desc ;
-- 90c6619c6b94fcfd34-EVRLOOT_RESOURCES, 186_083

To explain
Frontend now needs to process 720 thousand NFTs and select a portion of nfts to be shown as we can agree this a fairly a big chunk of data to process.

If we want to show now a collection detail database needs to go through 186 thousand NFTs to show you this:

However if we scroll to next page UI will try to have a problem and shows you this:

therefore we agreggate nfts by (image) or animation_url or both

select image, count(*) from nft_entity where collection_id = '90c6619c6b94fcfd34-EVRLOOT_RESOURCES' group by image order by count(*) desc;

[
  {
    "image": "ipfs://ipfs/QmfHxMyYpBsDCnL8VVkqyXNCkGZZ5d5v4W4NgKar455HZq/water.png",
    "count": 82387
  },
  {
    "image": "ipfs://ipfs/QmfYEsnN4T3gFgwQqkXZpbxZbLSTj11FMgGhEEfodfsUh7",
    "count": 39428
  },
  {
    "image": "ipfs://ipfs/QmfHxMyYpBsDCnL8VVkqyXNCkGZZ5d5v4W4NgKar455HZq/wood.png",
    "count": 38691
  },
  {
    "image": "ipfs://ipfs/QmfHxMyYpBsDCnL8VVkqyXNCkGZZ5d5v4W4NgKar455HZq/cooked_fish.png",
    "count": 16430
  },
  {
    "image": "ipfs://ipfs/QmPfg4v5Qx9iv4sJ6A1jZkjJrvAVbXfoTMZyBT6aeAxEMB/clam_bait.png",
    "count": 2632
  },
  {
    "image": "ipfs://ipfs/QmPfg4v5Qx9iv4sJ6A1jZkjJrvAVbXfoTMZyBT6aeAxEMB/palm_wood_2.png",
    "count": 1235
  },
  {
    "image": "ipfs://ipfs/QmPfg4v5Qx9iv4sJ6A1jZkjJrvAVbXfoTMZyBT6aeAxEMB/Hyena%20Bones.png",
    "count": 816
  },
  {
    "image": "ipfs://ipfs/QmPfg4v5Qx9iv4sJ6A1jZkjJrvAVbXfoTMZyBT6aeAxEMB/Old%20Rope.png",
    "count": 740
  },
  {
    "image": "ipfs://ipfs/QmPfg4v5Qx9iv4sJ6A1jZkjJrvAVbXfoTMZyBT6aeAxEMB/spicy%20soup.png",
    "count": 720
  },
  {
    "image": "ipfs://ipfs/QmPmMP6gN5SpNxkqn54ov8nJLmnmLRi8BVwwi7ok6MGU7w/bangar_fruit.png",
    "count": 713
  },
  {
    "image": "ipfs://ipfs/QmPfg4v5Qx9iv4sJ6A1jZkjJrvAVbXfoTMZyBT6aeAxEMB/wild_chilli.png",
    "count": 713
  },
  {
    "image": "ipfs://ipfs/QmPfg4v5Qx9iv4sJ6A1jZkjJrvAVbXfoTMZyBT6aeAxEMB/smoked%20fish.png",
    "count": 493
  },
  {
    "image": "ipfs://ipfs/QmTDNVHPg5Kuf2BhbFGej3SW3nkzMCPkFJd9p3npQydpKn",
    "count": 394
  },
  {
    "image": "ipfs://ipfs/QmPfg4v5Qx9iv4sJ6A1jZkjJrvAVbXfoTMZyBT6aeAxEMB/Palm%20Fibers.png",
    "count": 277
  },
  {
    "image": "ipfs://ipfs/QmPfg4v5Qx9iv4sJ6A1jZkjJrvAVbXfoTMZyBT6aeAxEMB/Palm%20Resin.png",
    "count": 246
  }
]

As we can observe the proposed idea of commonMetaData wont works since nfts in this collection variates a lot.

What if we calculate this aggregation on demand?

That is very good question but the database is not very ok with that.

[PieWol]

re-approving

[keeganquigley]

Hi @vikiival as we are now requiring KYB for all grantees, could you submit verification if you haven't done so already? Thanks!

[vikiival]

Hi @vikiival as we are now requiring KYB for all grantees, could you submit verification if you haven't done so already? Thanks!

^-^

[keeganquigley]

Hi @vikiival I see you have personally completed KYC but not seeing that KYB has been passed yet for KodaLabs. Are you able to submit this as well?

[vikiival]

KYB has been passed yet for KodaLabs. Are you able to submit this as well?

Hey, happy to do that, can I ask you for the link please?
Because when I click on the link above I will get

[keeganquigley]

Sure thing @vikiival try this one! Thanks and sorry for the hassle.

[vikiival]

Sure thing @vikiival try this one! Thanks and sorry for the hassle.

[github-actions]

Congratulations and welcome to the Web3 Foundation Grants Program! Please refer to our Milestone Delivery repository for instructions on how to submit milestones and invoices, our FAQ for frequently asked questions and the support section of our README for more ways to find answers to your questions.

Before you start, take a moment to read through our announcement guidelines for all communications related to the grant or make them known to the right person in your organisation. In particular, please don't announce the grant publicly before at least the first milestone of your project has been approved. At that point or shortly before, you can get in touch with us at grantsPR@web3.foundation and we'll be happy to collaborate on an announcement about the work you’re doing.

Lastly, please remember to let us know in case you run into any delays or deviate from the deliverables in your application. You can either leave a comment here or directly request to amend your application via PR. We wish you luck with your project! 🚀