upgrade lifecycle handler #168
Conversation
src/lifecycle.mjs
Outdated
| ) { | ||
| log(`Ending extractor strategy with name "${strategy.module.name}"`); | ||
| } | ||
| const graph = [ |
There was a problem hiding this comment.
this is amazing, I wanted to have something like this since a while!
|
general feedback is that please continue with your work this is really helpful. However, I haven't had time to look into detail of this change set as I've been trying to fix the crawler |
|
this may also resolve: #170 |
Added issue for this too: #174 |
|
btw I have sequentialized the strategy calls but without defining a graph for now to hot fix the current crawl. But I think you should still go forward with this PR as it is the right thing to do. But here is my intermediate change that sadly causes conflicts on your branch now: 0de7b87 |
|
|
I ran a small crawl using this PR. It ran successfully. I crawled 543 NFTs. The strategy was to limit the NFTs in web3subgraph-transformation. The crawl took 3m40s with a concurrency of 50. For IPFS I used Here is the crawl data if you are interested. data.zip I think only unit tests are remaining to be added. |
The new crawl path schema allows us to only run extractor or transformer of a strategy.
also remove unit tests for functions not used anymore
|
Due to lack of reviewer, merging this branch to dev for now. So I can run the crawler without affecting master. |
This is a proposal for an upgrade in lifecycle handler. It solves two main problems.
Additional benefit:
@TimDaub Have a look at the implementation. I tested it with minimal data and it works. I couldn't test it completely because of no IPFS gateway access right now. I'll try to test more and iron out the bugs.