Indexer workers #1103

stwiname · 2022-06-10T01:06:39Z

Usage:

--worker=<number> flag to specify number of workers. If this flag is not provided then workers will not be used and everything will run in the same thread.

Fixes #713

TODO:

Get tests working
~~Scale batch size based on free memory~~ Not used and non-trivial to implement
Fix file based MMR not working
Apply Fix pending data being used when it should be reset with new DS #1140 to block dispatcher implementations
Clean up
Fix MMR failure with worker threads enabled

ianhe8x · 2022-07-05T01:52:52Z

ERROR Unable to get dynamic datasources SequelizeDatabaseError: relation "_metadata" does not exist after I use --force-clean an existing project

packages/node/src/indexer/worker/block-dispatcher.service.ts

ianhe8x · 2022-07-05T23:19:45Z

Error: Queue exceeds max size sometimes

stwiname · 2022-07-06T03:08:04Z

I can't find a reasonable way to scale the batch size based on memory, we don't use the feature anyway.

ianhe8x · 2022-07-06T21:44:46Z

packages/node/src/indexer/worker/block-dispatcher.service.ts

+    private eventEmitter: EventEmitter2,
+  ) {
+    this.numWorkers = getMaxWorkers(numWorkers);
+    this.queue = new AutoQueue(this.numWorkers * workerQueueSize);


Suggested change

this.queue = new AutoQueue(this.numWorkers * workerQueueSize);

this.queue = new AutoQueue(workerQueueSize);

I think make batchSize the global batchSize is easier to understand.

ianhe8x · 2022-07-07T02:44:48Z

packages/node/src/indexer/worker/block-dispatcher.service.ts

+    private eventEmitter: EventEmitter2,
+  ) {
+    this.fetchQueue = new Queue(nodeConfig.batchSize * 3);
+    this.processQueue = new AutoQueue(nodeConfig.batchSize * 3);


~~cucurrency is 1 then, I suppose~~

Suggested change

this.processQueue = new AutoQueue(nodeConfig.batchSize * 3);

this.processQueue = new AutoQueue(nodeConfig.batchSize * 3);

if batchSize is overall batchSize, then concurrency will become nodeConfig.batchSize / workers

ianhe8x · 2022-07-07T10:45:06Z

I think round robin isn't as efficient as we dispatch a continuous blocks to a worker.
current:
worker1: 1,3,5
worker2: 2,4,6

better:
worker1: 1,2,3
worker2: 4,5,6

stwiname · 2022-07-11T03:02:45Z

MMR is not currently working with threads: failed to sync poi to mmr Error: Poi block height 1463, Poi mmr 0x3ac9ff27f72360faffb6c63fd64c96d82233bb6811dde95428f91fc5f350fafc not same as filebased mmr: 0xc5d2460186f7233c927e7db2dcc703c0e500b653ca82273b7bfad8045d85a470

… workers

Co-authored-by: Ian He <39037239+ianhe8x@users.noreply.github.com>

* fix set block offset (#1133) * [SKIP CI] Prerelease * while hilarious, fix SupQuery » SubQuery output (#1135) * [SKIP CI] Prerelease * fix warning for fetch specVersion when dictionary undefined, and tidy up logs (#1127) * [SKIP CI] Prerelease * fix subql-query help (#1137) * [SKIP CI] Prerelease * Fix pending data being used when it should be reset with new DS (#1140) * [SKIP CI] Prerelease * Manifestv1 tests (#1114) * added manifest_v1 test manifest tests touchup add loadProjecttemplate test * fix test Co-authored-by: JQQQ <jiqiang90@hotmail.com> * [SKIP CI] Prerelease * draft pr for redecorateRPC (#1111) * wip * handle at early block ,and throw at future block. Add tests Co-authored-by: JQQQ <jiqiang90@hotmail.com> * [SKIP CI] Prerelease * draft PR, dynamic interval (#1134) * init wip tidy up .muln and .toNumber err * move calcInterval to substrate * fix tests Co-authored-by: JQQQ <jiqiang90@hotmail.com> * [SKIP CI] Prerelease * [release] substrate 20220629 (#1146) * Rename deployment command (#1149) * renamed deployment to deploy * tidy up * [SKIP CI] Prerelease * use 0.1.2 common-avalanche (#1151) * [SKIP CI] Prerelease * Fix DI issue with EventEmitter not being resolved (#1154) * [SKIP CI] Prerelease * Try to catch error if create server failed, also give retry build schema (#1153) * Try to catch error if create server failed, also give retry build schema * Throw error * [SKIP CI] Prerelease * Http keepAlive and maxsockets, use http2 (#1150) * [SKIP CI] Prerelease * update common-avalanche,fix missing asset ande options (#1158) * [SKIP CI] Prerelease * Validate dictionary (#1152) * init * validate during init * patch getSpecVersionMap err * fixed tests * tidy up * refactor getspecversion * [SKIP CI] Prerelease * tidy up dependency, fix tests (#1160) * [SKIP CI] Prerelease * [release] 20220705 (#1162) * Add cosmos family to cli init (#1165) * Add cosmos family to cli init * use network families from @subql/common * [SKIP CI] Prerelease * Add tempate for issues and pull requests (#1166) * Add tempate for issues and pull requests * Update issue template * [release] 20220706 (#1168) * Fix trigger name (#1171) * fix oversize notification trigger name * drop * [SKIP CI] Prerelease * [release] 20220707 (#1172) * better error stack (#1174) * better error stack * flyby fix test * [SKIP CI] Prerelease * Local ipfs (#1169) * init * createsnewfile * test written * wip * comments * test written * ipfs issue * clarify test name * fixed tests Co-authored-by: Ben Zhao <ben@Bens-MBP.fritz.box> * [SKIP CI] Prerelease * add dictionary query timeout option (#1177) * add dictionary query timeout option * use nodeConfig * fix tests (#1178) * [SKIP CI] Prerelease * Cli default flag (#1176) * wip * set types * promptWithDefaultValues function * tidy up and include changes on changelog * tidy up * [SKIP CI] Prerelease * update polkadot to 8.12.2 (#1179) * [SKIP CI] Prerelease * fixed deployment bug (#1180) * fixed deployment bug * tidy up * tidy up var names for project command * updated change logs * tidy up, ref comments * tidy up 3 * Update readme Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * [SKIP CI] Prerelease * [release] 20220712 Substrate (#1181) * node-fetch resolution (#1182) * node-fetch resolution * conflict res * [SKIP CI] Prerelease * manual lock dependecies for node-fetch * [SKIP CI] Prerelease * added encodedURI (#1183) * [SKIP CI] Prerelease * resolution node-fetch 2.6.7 for cli * [SKIP CI] Prerelease * node fetch 2.6.7 * manual prerelease * [SKIP CI] Prerelease * add node-fetch 2.6.7 to cli * fix node-fetch 2.6.7 to cli (#1184) * [SKIP CI] Prerelease * [release] 20220713 cli patch (#1185) * [release] 20220713 cli patch * Update CHANGELOG.md * Fix ds template not being found as it was in another tmp directory (#1190) * Fix ds template not being found as it was in another tmp directory * Update changelog * [SKIP CI] Prerelease * [release] 20220715 (#1191) * Query explain in SQL (#1192) * [release] 20220715 * allow query explain to SQL statement * [SKIP CI] Prerelease * Restrict runner (#1195) * semver validate latest or dev * err log * restrict pre-release issue * clear logs * wip * restrict dev, latest * minor fix on deploy test * tidy up * tidy up if statement * [SKIP CI] Prerelease * add block modulo filter (#1196) * add block modulo filter * add getModuloBlocks function * [SKIP CI] Prerelease * fix endpoint bug (#1201) * [SKIP CI] Prerelease * Make handler types generic (#1194) * Make handler types generic * Fix typeo, update types for ds processors * Update changelogs * Improve changelog message * [SKIP CI] Prerelease * Indexer workers (#1103) * POC fetching blocks in workers * Use transferList * WIP worker implementation * Worker threads processing * Clean up * Get threaded indexing working * Improve worker performance, get tests building * Fix build issues and apply dynamic ds dictionary changes * Discard pending data when dynamic ds is created * Fix rebase build issues * Fix lint error * Fix test errors * Improve tests, fix bug with queue capacity * Remove flush on shutdown * Rework queue recursion into while loops, improve error logging * Change aftereach order for fetch service tests * Add some batching/concurrency to queue and use for fetching blocks in workers * Attempt to fix test cleanup, code clean up * Bring back MetaModule * Revert test docker compose * Fix error where _metadata entity not found * Update packages/node/src/indexer/worker/block-dispatcher.service.ts Co-authored-by: Ian He <39037239+ianhe8x@users.noreply.github.com> * Fix query tests * Fix queue capacity bug, improve logging * assign continuous blocks to worker (#1173) * Add different worker dispatch strategies * Port over error handling changes * Logging improvements, fix init poi service with threads * Run MMR service loop on main thread * Various improvements to fetch service tests * fix fetch test * Fix error with poi disabled and worker threads enabled * Fix MMR not updating * Update changelog * Update workers flag description * fix mmr mismatch due to fail to set blockOffset in metadata Co-authored-by: Ian He <39037239+ianhe8x@users.noreply.github.com> Co-authored-by: JQQQ <jiqiang90@hotmail.com> * Fix build issues (#1209) * [SKIP CI] Prerelease * Add reindexing feature (#1208) Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * [SKIP CI] Prerelease * Cli redeploy (#1205) * wip * redeployment * tests for redeploy * fix tests * tidy up test * fix test * refactor, tidyup * changeLog update * changeLog update * [SKIP CI] Prerelease * exit rather than throw error when api connection dropped (#1212) * [SKIP CI] Prerelease * [release] 202207027 Substrate (#1215) * [release] 202207027 Substrate * Update changelog * Update CHANGELOG.md * Update packages/types/CHANGELOG.md Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> * Update packages/cli/CHANGELOG.md Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> * Update packages/types/CHANGELOG.md Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> Co-authored-by: Scott Twiname <skott.twiname@gmail.com> Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> * removed rmraf on .github (#1223) * [SKIP CI] Prerelease * Export TypedEventRecord type (#1225) * [SKIP CI] Prerelease * Fix one off events being missed on startup (#1224) * Fix ready service never being ready Since introducing the project service, the event for ready was emit before the ready service was init and listening for the event. The solution is to check directly with the project service which is considered ready when a schema exists. * Revert ready service, manually init services outside of factories to not miss one time events * Fix up more events/metrics * [SKIP CI] Prerelease * add aggregate flag (#1219) * [SKIP CI] Prerelease * [release] 20220803 subql-node patch (#1229) * [CLI] Add algorand publish support (#1228) * [CLI] Add algorand publish support * Update validator and network types for algorand * Update deps, fix release script when pacakage name doesnt match path * Update readme to refer to other packages * Update README.md Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> * [SKIP CI] Prerelease * Fix dictionary being disabled with modulo filter (#1232) * Fix dictionary being disabled with modulo filter * Update changelog * [SKIP CI] Prerelease * [release] 20220804 (#1231) * Update polkadot 9 (#1234) * Update polkadot api to 9 * ignore init runtime decorate error * Bring back profiling fetchBlocksBatches, remove unnessary await (#1235) * Clean up Co-authored-by: Jay Ji <jiqiang90@hotmail.com> Co-authored-by: Mike Purvis <1042667+mikedotexe@users.noreply.github.com> Co-authored-by: Ben <89335033+bz888@users.noreply.github.com> Co-authored-by: Ben Zhao <ben@Bens-MBP.fritz.box> Co-authored-by: Naveen V <velnaveen99@gmail.com> Co-authored-by: Ian He <39037239+ianhe8x@users.noreply.github.com> Co-authored-by: Samuel Pitko <43212351+spitko@users.noreply.github.com> Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com>

* fix set block offset (#1133) * [SKIP CI] Prerelease * while hilarious, fix SupQuery » SubQuery output (#1135) * [SKIP CI] Prerelease * fix warning for fetch specVersion when dictionary undefined, and tidy up logs (#1127) * [SKIP CI] Prerelease * fix subql-query help (#1137) * [SKIP CI] Prerelease * Fix pending data being used when it should be reset with new DS (#1140) * [SKIP CI] Prerelease * Manifestv1 tests (#1114) * added manifest_v1 test manifest tests touchup add loadProjecttemplate test * fix test Co-authored-by: JQQQ <jiqiang90@hotmail.com> * [SKIP CI] Prerelease * draft pr for redecorateRPC (#1111) * wip * handle at early block ,and throw at future block. Add tests Co-authored-by: JQQQ <jiqiang90@hotmail.com> * [SKIP CI] Prerelease * draft PR, dynamic interval (#1134) * init wip tidy up .muln and .toNumber err * move calcInterval to substrate * fix tests Co-authored-by: JQQQ <jiqiang90@hotmail.com> * [SKIP CI] Prerelease * [release] substrate 20220629 (#1146) * Rename deployment command (#1149) * renamed deployment to deploy * tidy up * [SKIP CI] Prerelease * use 0.1.2 common-avalanche (#1151) * [SKIP CI] Prerelease * Fix DI issue with EventEmitter not being resolved (#1154) * [SKIP CI] Prerelease * Try to catch error if create server failed, also give retry build schema (#1153) * Try to catch error if create server failed, also give retry build schema * Throw error * [SKIP CI] Prerelease * Http keepAlive and maxsockets, use http2 (#1150) * [SKIP CI] Prerelease * update common-avalanche,fix missing asset ande options (#1158) * [SKIP CI] Prerelease * Validate dictionary (#1152) * init * validate during init * patch getSpecVersionMap err * fixed tests * tidy up * refactor getspecversion * [SKIP CI] Prerelease * tidy up dependency, fix tests (#1160) * [SKIP CI] Prerelease * [release] 20220705 (#1162) * Add cosmos family to cli init (#1165) * Add cosmos family to cli init * use network families from @subql/common * [SKIP CI] Prerelease * Add tempate for issues and pull requests (#1166) * Add tempate for issues and pull requests * Update issue template * [release] 20220706 (#1168) * Fix trigger name (#1171) * fix oversize notification trigger name * drop * [SKIP CI] Prerelease * [release] 20220707 (#1172) * better error stack (#1174) * better error stack * flyby fix test * [SKIP CI] Prerelease * Local ipfs (#1169) * init * createsnewfile * test written * wip * comments * test written * ipfs issue * clarify test name * fixed tests Co-authored-by: Ben Zhao <ben@Bens-MBP.fritz.box> * [SKIP CI] Prerelease * add dictionary query timeout option (#1177) * add dictionary query timeout option * use nodeConfig * fix tests (#1178) * [SKIP CI] Prerelease * Cli default flag (#1176) * wip * set types * promptWithDefaultValues function * tidy up and include changes on changelog * tidy up * [SKIP CI] Prerelease * update polkadot to 8.12.2 (#1179) * [SKIP CI] Prerelease * fixed deployment bug (#1180) * fixed deployment bug * tidy up * tidy up var names for project command * updated change logs * tidy up, ref comments * tidy up 3 * Update readme Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * [SKIP CI] Prerelease * [release] 20220712 Substrate (#1181) * node-fetch resolution (#1182) * node-fetch resolution * conflict res * [SKIP CI] Prerelease * manual lock dependecies for node-fetch * [SKIP CI] Prerelease * added encodedURI (#1183) * [SKIP CI] Prerelease * resolution node-fetch 2.6.7 for cli * [SKIP CI] Prerelease * node fetch 2.6.7 * manual prerelease * [SKIP CI] Prerelease * add node-fetch 2.6.7 to cli * fix node-fetch 2.6.7 to cli (#1184) * [SKIP CI] Prerelease * [release] 20220713 cli patch (#1185) * [release] 20220713 cli patch * Update CHANGELOG.md * Fix ds template not being found as it was in another tmp directory (#1190) * Fix ds template not being found as it was in another tmp directory * Update changelog * [SKIP CI] Prerelease * [release] 20220715 (#1191) * Query explain in SQL (#1192) * [release] 20220715 * allow query explain to SQL statement * [SKIP CI] Prerelease * Restrict runner (#1195) * semver validate latest or dev * err log * restrict pre-release issue * clear logs * wip * restrict dev, latest * minor fix on deploy test * tidy up * tidy up if statement * [SKIP CI] Prerelease * add block modulo filter (#1196) * add block modulo filter * add getModuloBlocks function * [SKIP CI] Prerelease * fix endpoint bug (#1201) * [SKIP CI] Prerelease * Make handler types generic (#1194) * Make handler types generic * Fix typeo, update types for ds processors * Update changelogs * Improve changelog message * [SKIP CI] Prerelease * Indexer workers (#1103) * POC fetching blocks in workers * Use transferList * WIP worker implementation * Worker threads processing * Clean up * Get threaded indexing working * Improve worker performance, get tests building * Fix build issues and apply dynamic ds dictionary changes * Discard pending data when dynamic ds is created * Fix rebase build issues * Fix lint error * Fix test errors * Improve tests, fix bug with queue capacity * Remove flush on shutdown * Rework queue recursion into while loops, improve error logging * Change aftereach order for fetch service tests * Add some batching/concurrency to queue and use for fetching blocks in workers * Attempt to fix test cleanup, code clean up * Bring back MetaModule * Revert test docker compose * Fix error where _metadata entity not found * Update packages/node/src/indexer/worker/block-dispatcher.service.ts Co-authored-by: Ian He <39037239+ianhe8x@users.noreply.github.com> * Fix query tests * Fix queue capacity bug, improve logging * assign continuous blocks to worker (#1173) * Add different worker dispatch strategies * Port over error handling changes * Logging improvements, fix init poi service with threads * Run MMR service loop on main thread * Various improvements to fetch service tests * fix fetch test * Fix error with poi disabled and worker threads enabled * Fix MMR not updating * Update changelog * Update workers flag description * fix mmr mismatch due to fail to set blockOffset in metadata Co-authored-by: Ian He <39037239+ianhe8x@users.noreply.github.com> Co-authored-by: JQQQ <jiqiang90@hotmail.com> * Fix build issues (#1209) * [SKIP CI] Prerelease * Add reindexing feature (#1208) Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * [SKIP CI] Prerelease * Cli redeploy (#1205) * wip * redeployment * tests for redeploy * fix tests * tidy up test * fix test * refactor, tidyup * changeLog update * changeLog update * [SKIP CI] Prerelease * exit rather than throw error when api connection dropped (#1212) * [SKIP CI] Prerelease * [release] 202207027 Substrate (#1215) * [release] 202207027 Substrate * Update changelog * Update CHANGELOG.md * Update packages/types/CHANGELOG.md Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> * Update packages/cli/CHANGELOG.md Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> * Update packages/types/CHANGELOG.md Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> Co-authored-by: Scott Twiname <skott.twiname@gmail.com> Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> * removed rmraf on .github (#1223) * [SKIP CI] Prerelease * Export TypedEventRecord type (#1225) * [SKIP CI] Prerelease * Fix one off events being missed on startup (#1224) * Fix ready service never being ready Since introducing the project service, the event for ready was emit before the ready service was init and listening for the event. The solution is to check directly with the project service which is considered ready when a schema exists. * Revert ready service, manually init services outside of factories to not miss one time events * Fix up more events/metrics * [SKIP CI] Prerelease * add aggregate flag (#1219) * [SKIP CI] Prerelease * [release] 20220803 subql-node patch (#1229) * [CLI] Add algorand publish support (#1228) * [CLI] Add algorand publish support * Update validator and network types for algorand * Update deps, fix release script when pacakage name doesnt match path * Update readme to refer to other packages * Update README.md Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> * [SKIP CI] Prerelease * Fix dictionary being disabled with modulo filter (#1232) * Fix dictionary being disabled with modulo filter * Update changelog * [SKIP CI] Prerelease * [release] 20220804 (#1231) * Update polkadot 9 (#1234) * Update polkadot api to 9 * ignore init runtime decorate error * Bring back profiling fetchBlocksBatches, remove unnessary await (#1235) * [SKIP CI] Prerelease * schema migration (#1226) * [SKIP CI] Prerelease * Fix Queue exceeds max size error (#1239) * [SKIP CI] Prerelease * fix config db selection when subscription is enabled (#1241) * [SKIP CI] Prerelease * Fork OrderByAggregatesPlugin and modify to be compatible with historical feature (#1242) * [SKIP CI] Prerelease * Fix block height being applied to all queries (#1238) * [SKIP CI] Prerelease * Fix running custom ds processors in parallel on different data (#1243) * [SKIP CI] Prerelease * [release] 20220811 (#1245) * [release] 20220811 * update changelog * support bulkGet and add bulkUpdate (#1246) * add bulkUpdate * tidy up * support historical * improve bulkUpdate * fix * [SKIP CI] Prerelease * Fix issue with db sync and workers enabled (#1250) * Fix issue with db sync and workers enabled. Improved worker error handling * Remove cpu count workers limit * [SKIP CI] Prerelease * Improve Performance logging (#1244) * profilewrap for handler * wip * tidying up * remove console.log * tidy up * add transactions * Subql init mono (#1236) * draft * Tidy up * tidy up * Fix tests * [SKIP CI] Prerelease * [SKIP CI] Prerelease * Fix bigInt array entity being treated as a big int (#1252) * [SKIP CI] Prerelease * [release] 20220817 (#1254) * [release] 20220817 * Fix typo * Address comments * Update packages/types/CHANGELOG.md Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> * Update README.md * Node core (#1222) * both packages working * working on node core * avalanche with types avalanche * adding provider * provider extends ethersproject provider * Fix build issues * Remove avalanche packages * Add noImplicitAny flag to node-core and fix errors * Update build tooling and changelogs * Move events to node-core * Move store service to node-core * Use logger/profiler from node-core * Move poi service to node core * Move mmr service to node-core * Move over meta controllers/services to node-core * Fix imports * Fix build issues * Fix linter warning * Fix lint issues, move worker builder to node core * Remove BlockedQueue, fix more lint errors * Move some test files * Fix rebase issues, remove avax deps from node core * Fix build issue * Fix some tests * fix processBlockCount metric with workers Co-authored-by: roduquen <duquenne.robin42@gmail.com> Co-authored-by: JQQQ <jiqiang90@hotmail.com> * Add limit and offset options to store.getByField (#1259) * Add limit and offset options to store.getByField * Revert types change * Add delay and don't break loop when process queue is full (#1261) * Update cmd options in subql-node README (#1265) * Update README.md * Update packages/node/README.md * Fix symbol type being possible in string interpolation (#1264) * Fix symbol type being possible in string interpolation (#1266) * [SKIP CI] Prerelease * [release] 20220826 (#1263) * Fix imports not being relative (#1268) * [SKIP CI] Prerelease * [release] node-core 20220826 (#1269) * prerelease with @subql/node-core (#1270) * [SKIP CI] Prerelease * fix node-core import in node (#1272) * [SKIP CI] Prerelease * [release] 20220829 (#1273) * Move auto queue to node core (#1278) * [SKIP CI] Prerelease * update polkadot and types, to support with wasm (#1280) * update polkadot and types, to support with wasm * move logic to ds processor, update change for interface * remove dsAssets * remove dsAssets * update * fix fetch due to type changed (#1282) * fix fetch due to type changed * prerelease for types * [SKIP CI] Prerelease * fix missing await for dictionary (#1283) * [SKIP CI] Prerelease * rollback changes (#1286) * [SKIP CI] Prerelease * Fix tests failing with Cannot read properties of null (reading 'getProcessedBlockCount') (#1285) * Fix tests failing with Cannot read properties of null (reading 'getProcessedBlockCount') * Fix same error in more tests * Fix lint issue * [SKIP CI] Prerelease * fix remote gitpath (#1288) * [SKIP CI] Prerelease * Fix tests (#1297) * fix tests * tidy up * removed v0.0.1 test * [release] 20220913 (#1291) * [release] 20220913 * Updated CHANGELOG.MD Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> * Set useragent headers on polkadot api (#1296) * [SKIP CI] Prerelease * change dictionary to use equalToInsensitive (#1301) * change dictionary to use equalToInsensitive * add comments * [SKIP CI] Prerelease * [release] 20220919 (#1302) * [release] 20220919 * Update packages/node/CHANGELOG.md Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> * update sequelize (#1311) * update sequelize * update sequelize query * [SKIP CI] Prerelease * Subcommands for force-clean and reindex (#1281) * draft pr for documentation * merge * force-clean command working, refactoring needed * refactor force-clean * unsafe bug, in sandboxService, export package.json bug * updated package.json node-core * subql-node starting * refactor * fix up * node test suites still broken * Fix init logger with tests * Use old logger defaults * tests passes * tidy up * builds * Fix exporting logger (#1300) * tidy up, skip db.module tests * fixed db.module * tidy up getExistingProjectSchema * Reindex command (#1307) * init reindex command * add services to module * reindex functioning * add force-clean for issue #1230 * tidy up * tidy up based on comments * remove comments * edit command des * remove reindex from nodeconfig * moved functions into node-core from utils * update version Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * Fixes #1309 (#1314) * fix publish failed due to file object is null (#1316) * [SKIP CI] Prerelease * [release] 20220924 (#1317) * chnage from multi branch to main (#1318) * fix logger undefined (#1315) * [SKIP CI] Prerelease * change max connection to 10 and query service 10000 (#1322) Co-authored-by: Louise wang <louisewang118@.gmail.com> * [SKIP CI] Prerelease * update polkadot api to 9.4.2 (#1325) * functioning hot schema reload (#1321) * [SKIP CI] Prerelease * improve codegen error log (#1324) Co-authored-by: Louise wang <louisewang118@.gmail.com> * change createproject URL , project create issue, regx for project nam… (#1323) * change createproject URL , project create issue, regx for project name and project key * project name can be cap case * code improve by Jay comment Co-authored-by: Louise wang <louisewang118@.gmail.com> * [SKIP CI] Prerelease * manual prerelease (#1326) * [SKIP CI] Prerelease * enable historical by default (#1327) * [SKIP CI] Prerelease * [release] 20220929 (#1328) * [release] 20220929 * update * Update packages/node/CHANGELOG.md Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> * Update packages/query/CHANGELOG.md Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> * Update packages/utils/CHANGELOG.md Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> * Update packages/query/CHANGELOG.md Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> * Update packages/node/CHANGELOG.md Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> * Update packages/node-core/CHANGELOG.md Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> * Update packages/common/CHANGELOG.md Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> Co-authored-by: JQQQ <jiqiang90@hotmail.com> Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> * Update ipfs endpoints, improve error handling (#1337) * [SKIP CI] Prerelease * Fix nodeConfig not being provided to benchmark service (#1338) * [SKIP CI] Prerelease * [release] 20221006 (#1339) * Remove Subqueries table (#1340) * Remove Subqueries table * Fix NOT_SUPPORT function in project * Remove tests relating to old subqueries table * [SKIP CI] Prerelease * [release] 20221010 (#1342) * [release] 20221010 * Update packages/node/CHANGELOG.md Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> * Validate ds processor in worker threads (#1343) * [SKIP CI] Prerelease * Fix publish ipfs 503 (#1331) * add comments * publish ipfs issue * publish ipfs issue * Update packages/cli/src/controller/publish-controller.ts Co-authored-by: bz888 <ziyu.ben158@gmail.com> Co-authored-by: Louise wang <louisewang118@.gmail.com> Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * [SKIP CI] Prerelease * Move common dictionary service to node-core (#1345) * Move common dictionary service to node-core * Apply matcher from cosmos dictionary * Move sandbox code to node-core * Remove testing log * [SKIP CI] Prerelease * [release] 20221011 * Back port changes in algorand sdk to dictionary service (#1346) * [SKIP CI] Prerelease * temp disable hot schema reload (#1349) * temp disable hot schema reload * add CLI flag for hotschema reload * fix * print node version (#1348) * print node version * update logger info * update with node name Co-authored-by: Louise wang <louisewang118@.gmail.com> * [SKIP CI] Prerelease * [SKIP CI] Prerelease * fix output-format flag on node (#1351) * fix typo * update workers and yargs outputFmt * [SKIP CI] Prerelease * Timestamp block filters (#1310) * add block timestamp filter * bug fixes * bug fixes * clean code and write test for timestamp filter * move cron generation to SubqueryProject * fix next timestamp log bug * remove modifiedDataSources * [SKIP CI] Prerelease * Update to latest polkadot api (#1356) * [SKIP CI] Prerelease * add ad flags (#1352) * add ad flags * add interface for all AD flags and add prefix for indexer/query flags * change string of to for * some variable name change and update interface Co-authored-by: Louise wang <louisewang118@.gmail.com> * support version without v (#1354) * support version without v * improvement addv FUNC and solved undefined input Co-authored-by: Louise wang <louisewang118@.gmail.com> * fix deploy-test (#1357) * [SKIP CI] Prerelease * add dedicate data base flag for create project (#1355) * add dedicate data base flag for create project * test with latest createproject api host service and update wording Co-authored-by: Louise wang <louisewang118@.gmail.com> * Optimise modulo filter (#1358) * scale batch size if modulo filter used * add comments * add comments * fix logic runModuloblock * remove comments * remove more comments * add check datasources, check height logic * removed comments * refactor * flatten handler array * Fix template issues (#1363) * Fix the problem of creating multiple templates in the same block with the same template address * refactor: cache ds in same block Co-authored-by: Orion <orion@starfish.finance> * fix deployment and project tests (#1364) * fix deployment and project tests * fix linting error * [SKIP CI] Prerelease * [release] 20221027 (#1365) * fix: incomplete ds data in same block (#1370) Co-authored-by: Orion <orion@starfish.finance> * [SKIP CI] Prerelease * Best Block (#1308) * Draft * update changes * rebase changes * fix * fix (#1329) * disable best blocks for workers until we support it * Rename bestBlock to unfinalizedBlocks, other clean up * Clean up * Further clean up * Dedupe reindex function * Fix tests * Clean up * Update checking finalization to use parent hash * Rename logger * Refactor unfinalized blocks * Use header rather than full block, improve detecting forks * Verify unfinalized blocks when disabled. Use sorted array for storing unfinalized blocks * Clean up logs * Fix not indexing unfinalized blocks right away, exit if historical not enabled Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * [SKIP CI] Prerelease * Fix bugs with unfinalized blocks (#1374) * Fix bugs with unfinalized blocks - Fix using wrong hash for unfinalized blocks - Fix not removing unfinalized blocks when the latest unfinalized block < finalized * Fix tests getting stuck * Fix issue finding where fork occurred * [SKIP CI] Prerelease * [release] 20221028 (#1372) * Disable warning * Update logging Co-authored-by: Jay Ji <jiqiang90@hotmail.com> Co-authored-by: Mike Purvis <1042667+mikedotexe@users.noreply.github.com> Co-authored-by: Ben <89335033+bz888@users.noreply.github.com> Co-authored-by: Ben Zhao <ben@Bens-MBP.fritz.box> Co-authored-by: Naveen V <velnaveen99@gmail.com> Co-authored-by: Ian He <39037239+ianhe8x@users.noreply.github.com> Co-authored-by: Samuel Pitko <43212351+spitko@users.noreply.github.com> Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> Co-authored-by: James Bayly <46693720+jamesbayly@users.noreply.github.com> Co-authored-by: roduquen <duquenne.robin42@gmail.com> Co-authored-by: HAO ZHAI <zhaihaohk@gmail.com> Co-authored-by: bighammerw <92004774+bighammerw@users.noreply.github.com> Co-authored-by: Louise wang <louisewang118@.gmail.com> Co-authored-by: bz888 <ziyu.ben158@gmail.com> Co-authored-by: hariu-starfish <103621490+hariu-starfish@users.noreply.github.com> Co-authored-by: Orion <orion@starfish.finance>

stwiname force-pushed the indexer-workers branch from 0ee5fd4 to 115ea08 Compare June 20, 2022 04:23

stwiname added this to To do in Development Jun 29, 2022

stwiname force-pushed the indexer-workers branch 3 times, most recently from 668de33 to 509ed9c Compare June 29, 2022 04:55

stwiname moved this from To do to In progress in Development Jul 3, 2022

stwiname marked this pull request as ready for review July 4, 2022 03:54

stwiname requested review from jiqiang90 and ianhe8x July 4, 2022 03:54

stwiname force-pushed the indexer-workers branch 2 times, most recently from bac6010 to cb7259e Compare July 5, 2022 00:02

ianhe8x reviewed Jul 5, 2022

View reviewed changes

packages/node/src/indexer/worker/block-dispatcher.service.ts Outdated Show resolved Hide resolved

ianhe8x reviewed Jul 6, 2022

View reviewed changes

ianhe8x reviewed Jul 7, 2022

View reviewed changes

stwiname force-pushed the indexer-workers branch from f4a1003 to a147ab7 Compare July 10, 2022 22:26

stwiname force-pushed the indexer-workers branch from d88a16b to 2ebac7f Compare July 11, 2022 03:15

stwiname added the enhancement New feature or request label Jul 19, 2022

stwiname force-pushed the indexer-workers branch from e6ac2d5 to 43f7ab8 Compare July 20, 2022 23:04

stwiname added 7 commits July 25, 2022 08:16

POC fetching blocks in workers

daa095b

Use transferList

2da2091

WIP worker implementation

dedd0e3

Worker threads processing

70f2a7e

Clean up

139bd24

Get threaded indexing working

f149ace

Improve worker performance, get tests building

32aa255

stwiname and others added 20 commits July 25, 2022 08:21

Add some batching/concurrency to queue and use for fetching blocks in…

ed17f73

… workers

Attempt to fix test cleanup, code clean up

a4c1f81

Bring back MetaModule

1897ccb

Revert test docker compose

7c91204

Fix error where _metadata entity not found

a5b5bf8

Update packages/node/src/indexer/worker/block-dispatcher.service.ts

9603288

Co-authored-by: Ian He <39037239+ianhe8x@users.noreply.github.com>

Fix query tests

a3a4eb1

Fix queue capacity bug, improve logging

8b88a0b

assign continuous blocks to worker (#1173)

1ff2d30

Add different worker dispatch strategies

a3a0c76

Port over error handling changes

18cae4d

Logging improvements, fix init poi service with threads

ace4fa2

Run MMR service loop on main thread

cd60ec3

Various improvements to fetch service tests

1a1e68a

fix fetch test

ddec31c

Fix error with poi disabled and worker threads enabled

6da5590

Fix MMR not updating

f56db62

Update changelog

610dc9b

Update workers flag description

068889c

fix mmr mismatch due to fail to set blockOffset in metadata

61a1212

stwiname force-pushed the indexer-workers branch from 3922170 to 61a1212 Compare July 24, 2022 20:21

jiqiang90 approved these changes Jul 25, 2022

View reviewed changes

stwiname merged commit 158abee into main Jul 25, 2022

Development automation moved this from In progress to Done Jul 25, 2022

stwiname mentioned this pull request Jul 25, 2022

Fix build issues #1209

Merged

deverka added the crucial Change worth spreading the news label Jul 25, 2022

deverka mentioned this pull request Jul 27, 2022

[release] 202207027 Substrate #1215

Merged

stwiname deleted the indexer-workers branch February 13, 2023 23:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Indexer workers #1103

Indexer workers #1103

stwiname commented Jun 10, 2022 •

edited

ianhe8x commented Jul 5, 2022 •

edited

ianhe8x commented Jul 5, 2022

stwiname commented Jul 6, 2022

ianhe8x Jul 6, 2022

ianhe8x Jul 7, 2022 •

edited

ianhe8x Jul 7, 2022

ianhe8x commented Jul 7, 2022

stwiname commented Jul 11, 2022

	this.queue = new AutoQueue(this.numWorkers * workerQueueSize);
	this.queue = new AutoQueue(workerQueueSize);

	this.processQueue = new AutoQueue(nodeConfig.batchSize * 3);
	this.processQueue = new AutoQueue(nodeConfig.batchSize * 3);

Indexer workers #1103

Indexer workers #1103

Conversation

stwiname commented Jun 10, 2022 • edited

ianhe8x commented Jul 5, 2022 • edited

ianhe8x commented Jul 5, 2022

stwiname commented Jul 6, 2022

ianhe8x Jul 6, 2022

Choose a reason for hiding this comment

ianhe8x Jul 7, 2022 • edited

Choose a reason for hiding this comment

ianhe8x Jul 7, 2022

Choose a reason for hiding this comment

ianhe8x commented Jul 7, 2022

stwiname commented Jul 11, 2022

stwiname commented Jun 10, 2022 •

edited

ianhe8x commented Jul 5, 2022 •

edited

ianhe8x Jul 7, 2022 •

edited