Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use nodejs worker for block fetching #713

Closed
ianhe8x opened this issue Dec 20, 2021 · 1 comment · Fixed by #1103
Closed

Use nodejs worker for block fetching #713

ianhe8x opened this issue Dec 20, 2021 · 1 comment · Fixed by #1103
Labels
enhancement New feature or request

Comments

@ianhe8x
Copy link
Collaborator

ianhe8x commented Dec 20, 2021

to solve the process hang issue when processing a large ws payload

@fishmonger45
Copy link
Contributor

fishmonger45 commented Jan 20, 2022

I've been working on this a little bit today (https://github.com/wqsz7xn/subql/tree/node-worker). It's fairly simple and just means that you have to offload fetchBlockBatches to a nodejs worker. The only problem that i'm experiencing is that you have to pass the api, block numbers and overallspecVersion to the worker and doing that is difficult without some amount of serialisation (worker.postMessage only accepts some primitives/is limited). This would probably have a hit on performance.

Workers also require a worker file which doesn't support typescript obviously which is proving a bit tricky

Screenshot from 2022-01-28 16-40-01

@lamcc21 lamcc21 added the enhancement New feature or request label Mar 14, 2022
@stwiname stwiname mentioned this issue Jul 13, 2022
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants