Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ipfs search #32

Merged
merged 22 commits into from
Jan 24, 2023
Merged

Ipfs search #32

merged 22 commits into from
Jan 24, 2023

Conversation

empaemanuel
Copy link
Contributor

@empaemanuel empaemanuel commented Dec 29, 2022

About the update

  • Makes it possible to query services, proposals, reviews, users, and platforms using their data on IPFS.

Here is how it works:
When an entity such as a service, proposal, review, user, or platform is created by a contract it emits an event with a uri to where the ipfs data is located. The convention of which event contains the uri is currently inconsistent between the different entities which has the consequence that it is a little bit unclear regarding when the data fetching mechanism is initialized. Generally, it is initialized in the event handler receiving the uri.

All data fetched from ipfs are stored as entities called <ParentEntity>Description, i.e. ServiceDescription, ProposalDescription, ReviewDescription, etc, where the id of the entity is it's ipfs id (cid).

Each ParentEntity is linked to its description through a field called description. Each Description entity is linked to its parent through a field called . Ex. ServiceDescription.service = Service.

Keywords
User.skills and Service.keywords are fields that stores a list of Keyword entities. These entities are created based on the information from ipfs, which can be viewed in User.skills_raw and Service.keywords_raw.

There are currently a few issues that need to be resolved.

  1. There is no way to get previously created Entities from subgraph store using the file datasource used in ipfs-data.ts. This means that in order to not get duplicated values we need to delete the previously created ones outside of the file datasource.
  2. Keyword entities must be created within file datasource in ipfs-data.ts. Therefore, keyword entities are duplicated.
  3. There is a lot of code duplication in ipfs-data.ts. This should be fixed before production.

Resolution
Waiting for the current graph issue to be solved.
graphprotocol/graph-node#4087

Changes that impact others
The graph schema is updated so front-end implementations need to update according to the new schema..

Makefile Outdated
graph codegen
graph build --network goerli
graph deploy --product hosted-service empaemanuel/talentlayer-goereli
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo in the name "goereli"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed from here

schema.graphql Outdated Show resolved Hide resolved
schema.graphql Outdated Show resolved Hide resolved
schema.graphql Outdated Show resolved Hide resolved
schema.graphql Outdated
name: "serviceDescriptionSearchRank"
language: en
algorithm: proximityRank
include: [{ entity: "ServiceDescription", fields: [{ name: "title" }, { name: "keywords_raw" }, { name: "about" }] }]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is the cost of adding the about "a potential really large text" here ?
It would probably be better to have only title and keywords searchable together. And potentially the about on his own, what do you think ?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for now we are limited at only one Schema, so we can't separate it, and can't create another one for another entity

schema.graphql Outdated Show resolved Hide resolved
schema.graphql Outdated Show resolved Hide resolved
schema.graphql Outdated Show resolved Hide resolved
descriptions: [ReviewDescription!] @derivedFrom(field: "review")
}

type ReviewDescription @entity {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note: in theory, this entity could be marked as immutable https://medium.com/edge-node-engineering/two-simple-subgraph-performance-improvements-a76c6b3e7eac
But as we plan in the future to have dispute on rating, it's ok to keep it like that

schema.graphql Outdated Show resolved Hide resolved
schema.graphql Outdated
rateType: BigInt
description: String
proposal: Proposal! #Proposals that the description describes.
createdAt: BigInt
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add updatedAt

schema.graphql Outdated Show resolved Hide resolved

let description = getOrCreateServiceDescription(ipfsId)

description.service = context.getString('id')
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you would probably need a getOrCreate for each linked entity in case they are not yet created. Don't you have any issues here on goerli ?

Copy link
Contributor Author

@empaemanuel empaemanuel Jan 23, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can use it but it always return a new entity due to open issue. Added comments in code

handler: handleServiceData
entities:
- ServiceDescription
abis:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DO you know why we need abis in this context?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They are not used by the template but they are yet required by the compiler.

@0xRomain 0xRomain added the NEED FIX CODE Dev needs to check feedbacks from review and act on it label Jan 12, 2023

const jsonObject = json.fromBytes(content).toObject();

let description = getOrCreateServiceDescription(ipfsId)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as load didn't work in this ipfs data file, we can add to the context var a bool saying if we are on a creation or an update and use that rather than the load

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We chose to use cid for ID, meaning it is always a new entity.

schema.graphql Outdated
@@ -13,26 +13,54 @@ enum PaymentType {

type Service @entity {
id: ID! # service id
createdAt: BigInt # timestamp of block creation
updatedAt: BigInt # timestamp of the last change
created: BigInt! # timestamp of block creation
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should not change

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

transaction: Transaction @derivedFrom(field: "service") # transaction associated with this service
proposals: [Proposal!] @derivedFrom(field: "service") # proposals for this service
platform: Platform # Platform on which service was created
cid: String
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need this one?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, if id is the cid and we need to remove the cid, then we need to store the cid to access the old description that we want to remove. Using service.description does not work since it's derived and loaded on query time.

}

type Review @entity {
id: ID! # review nft id
createdAt: BigInt # timestamp of block creation
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should not be removed

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
NEED FIX CODE Dev needs to check feedbacks from review and act on it
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants