Imporve dictionary query #1371

bz888 · 2022-10-28T01:24:48Z

Description

Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

Fixes #1306

Type of change

Please delete options that are not relevant.

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist

I have tested locally
Linked to any relevant issues
I have added tests relevant to my changes
My code is up to date with the base branch

stwiname · 2022-10-28T02:40:09Z

packages/node-core/src/utils/dictionaryEntry.ts

+
+export function buildDictionaryEntryMap(
+  // unsure how to make this more generic
+  dataSources: any[],


Just leave these as SubqlProjectDs[] for now.

stwiname · 2022-10-28T02:44:32Z

packages/node/src/indexer/fetch.service.ts

+    this.buildDictionaryEntriesMap();
+    let checkDictionaryQueryEntries: boolean;
+    this.mappedDictionaryQueryEntries.forEach((value) => {
+      if (value?.length) {


Im not sure this works. We could have dictionary disabled at a later time if a block filter exists

stwiname · 2022-10-28T02:46:45Z

packages/node-core/src/utils/dictionaryEntry.ts

+import {Dictionary} from '../indexer/dictionary.service';
+// import { SubqlProjectDs } from '../../../node/src/configure/SubqueryProject';
+
+export function buildDictionaryEntryMap(


I'd like to see these functions as a class, and we can maintain the mappedDictionaryQueryEntries in the class. It would be good to clean up the args to these functions too

stwiname · 2022-11-01T20:57:39Z

packages/cli/src/controller/init-controller.test.ts

@@ -67,7 +67,7 @@ describe('Cli can create project', () => {
  it('prepare correctly applies project details', async () => {
    const tempPath = await makeTempDir();
    const templates = await fetchTemplates();
-    const template = templates.find(({name}) => name === 'polkadot-starter');
+    const template = templates.find(({name}) => name === 'Polkadot-starter');


https://github.com/subquery/subql/pull/1360/files#r1001253159

stwiname · 2022-11-01T21:04:04Z

packages/node-core/src/indexer/dictionary.service.ts

+    }
+    // sort map from lowest ds.startBlock to highest
+    this.mappedDictionaryQueryEntries = new Map(
+      [...mappedDictionaryQueryEntries.entries()].sort((a, b) => a[0] - b[0])


Might be easier to sort the datasources

packages/node/src/indexer/fetch.service.ts

jiqiang90 · 2022-11-03T01:07:05Z

packages/node/src/indexer/fetch.service.test.ts

@@ -744,4 +738,20 @@ describe('FetchService', () => {
    // Should be called 91151,9170,9180
    expect(getPrefechMetaSpy).toBeCalledTimes(3);
  }, 500000);
+


do we still need this?

jiqiang90 · 2022-11-03T01:07:30Z

packages/node/src/indexer/fetch.service.test.ts

@@ -15,7 +15,7 @@ import {
 import { NodeConfig } from '@subql/node-core';
 import { GraphQLSchema } from 'graphql';
 import { Sequelize } from 'sequelize';
-import { SubqueryProject } from '../configure/SubqueryProject';
+import { SubqlProjectDs, SubqueryProject } from '../configure/SubqueryProject';


jiqiang90 · 2022-11-03T01:07:44Z

packages/node/src/indexer/fetch.service.spec.ts

@@ -570,6 +570,7 @@ describe('FetchService', () => {
      eventEmitter,
      mockProjectService(),
    );
+


jiqiang90 · 2022-11-03T01:09:43Z

packages/node-core/src/indexer/dictionary.service.test.ts

+    for (let i = 0; i < mockDS.length; i++) {
+      dictionaryQueryMap.set(
+        [mockDS[i].startBlock],
+        HAPPY_PATH_CONDITIONS.filter((dictionaryQuery, index) => i >= index)


I know you are trying to mock HAPPY_PATH_CONDITIONS here, but it is not align with mockDs filters, can we make them match? Or just leave a comment here, otherwise people will confused

jiqiang90 · 2022-11-03T02:32:34Z

Note:

To decide whether include ds filter condition in the query, we are using queryEndBlock height here to compare with ds start height.

Use nextBlock (endblock + batch) is not correct, as the queryEndblock is larger, the last block in previous query result might greater than nextBlock, so between 2nd ds start height to the last block in previous query result is not been checked by dictionary.

stwiname · 2022-11-03T02:37:49Z

packages/node-core/src/indexer/dictionary.service.ts

@@ -228,4 +229,43 @@ export class DictionaryService implements OnApplicationShutdown {
    }
    return buildQuery(vars, nodes);
  }
+  buildDictionaryEntryMap(
+    // TODO: generic type
+    dataSources: any[],


We should type this.

* fix: incomplete ds data in same block (subquery#1370) Co-authored-by: Orion <orion@starfish.finance> * [SKIP CI] Prerelease * Best Block (subquery#1308) * Draft * update changes * rebase changes * fix * fix (subquery#1329) * disable best blocks for workers until we support it * Rename bestBlock to unfinalizedBlocks, other clean up * Clean up * Further clean up * Dedupe reindex function * Fix tests * Clean up * Update checking finalization to use parent hash * Rename logger * Refactor unfinalized blocks * Use header rather than full block, improve detecting forks * Verify unfinalized blocks when disabled. Use sorted array for storing unfinalized blocks * Clean up logs * Fix not indexing unfinalized blocks right away, exit if historical not enabled Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * [SKIP CI] Prerelease * Fix bugs with unfinalized blocks (subquery#1374) * Fix bugs with unfinalized blocks - Fix using wrong hash for unfinalized blocks - Fix not removing unfinalized blocks when the latest unfinalized block < finalized * Fix tests getting stuck * Fix issue finding where fork occurred * [SKIP CI] Prerelease * [release] 20221028 (subquery#1372) * hot fix tests (subquery#1360) * update tests: api.service.spec, init-controller.test, publish-controller.spec * update tests following comments * [SKIP CI] Prerelease * add ethereum to CLI and Validator (subquery#1378) * Imporve dictionary query (subquery#1371) * wip * wip * wip * broken wip * wip refactoring * seperate dictionaryQueryEntries * fix logic for setDicitonaryQueryEntries * cleaning up * move dictionaryEntry.ts to node-core * remove comments * create class for dictionaryqueryentries * update tests * move dictionaryQueryService into dicitonaryService * refactor * update logic for useDicitonary * relocated dictionaryQuery funcs * add test for dictquerymap * tests failing on windows? * test getDictionaryQueryEntries * test fixed for getDictioanryQueryEntries * added test for sorting * conflict fixed ? * fix conflict_2 * fix conflict_3 * add delete temp ds records back * clean up with new logic * clean up, add comments for test * fix * update logic * add generic type * [SKIP CI] Prerelease * Enable for better inheritance of generated entity modeld (subquery#1377) * refactor: enable for entity inheritance * refactor: include also filed getters * [SKIP CI] Prerelease * fix comments issue with new package (subquery#1380) * fix comments issue with new package * moved yaml package * [SKIP CI] Prerelease * fix logic with reindex and unfinalized height and dynamic ds (subquery#1382) * fix logic with reindex and unfinalized height * fix * include fix for subquery#1379 * update polkadot to 9.7.1 (subquery#1384) * [release] 20221107 * Fix remove alter table (subquery#1387) * remove migrate alter table * remove * [SKIP CI] Prerelease * [release] 20221108 (subquery#1388) * fix missing sequelize sync (subquery#1389) * [SKIP CI] Prerelease * [release] 20221108 patch (subquery#1390) * reindex bind (subquery#1391) * [SKIP CI] Prerelease * [release] 20221109 (subquery#1393) * Handle fetch errors, then retry (subquery#1386) * add retryOnFail function * add retryOnFail * add test, fix logic * [SKIP CI] Prerelease * fix (subquery#1395) * [SKIP CI] Prerelease * [release] 20221109 node-core (subquery#1394) * Fix tests hanging (subquery#1396) * Fix tests hanging * Update base docker image with newer git version * [SKIP CI] Prerelease * Add distinct query plugin (subquery#1274) * Add distinct query plugin * Clean up log * Fix distinct not being provided to query * Uppercase enum to be consistent with other enums * Update dictionary queries to try distinct argument * [SKIP CI] Prerelease * Add query distinct dependencies (subquery#1398) * fix missing update forked graphile dependencies * tidy up * tidy up * [SKIP CI] Prerelease * Break block dispatcher file up and move common code to base class (subquery#1397) * [SKIP CI] Prerelease * Hot schema trigger (subquery#1401) * implement trigger with notification * working prior clean up * refactor and clean up on async and await * clean up * clear comments * add filter * fix * fix err * [SKIP CI] Prerelease * [release] 20221115 (subquery#1402) * [release] 20221115 * [release] 20221115 * [release] 20221115 * fix hot schema (subquery#1404) * fix and refactor * refactor getTriggers * [SKIP CI] Prerelease * [release] 20221115 (subquery#1408) * [release] 20221115 * [release] 20221115 * fix fetchblock for works (subquery#1410) * [SKIP CI] Prerelease Co-authored-by: hariu-starfish <103621490+hariu-starfish@users.noreply.github.com> Co-authored-by: Orion <orion@starfish.finance> Co-authored-by: Jay Ji <jiqiang90@hotmail.com> Co-authored-by: Ben <89335033+bz888@users.noreply.github.com> Co-authored-by: Naveen V <velnaveen99@gmail.com> Co-authored-by: Filippo <filippo@embriotech.ch>

* fix: incomplete ds data in same block (subquery#1370) Co-authored-by: Orion <orion@starfish.finance> * [SKIP CI] Prerelease * Best Block (subquery#1308) * Draft * update changes * rebase changes * fix * fix (subquery#1329) * disable best blocks for workers until we support it * Rename bestBlock to unfinalizedBlocks, other clean up * Clean up * Further clean up * Dedupe reindex function * Fix tests * Clean up * Update checking finalization to use parent hash * Rename logger * Refactor unfinalized blocks * Use header rather than full block, improve detecting forks * Verify unfinalized blocks when disabled. Use sorted array for storing unfinalized blocks * Clean up logs * Fix not indexing unfinalized blocks right away, exit if historical not enabled Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * [SKIP CI] Prerelease * Fix bugs with unfinalized blocks (subquery#1374) * Fix bugs with unfinalized blocks - Fix using wrong hash for unfinalized blocks - Fix not removing unfinalized blocks when the latest unfinalized block < finalized * Fix tests getting stuck * Fix issue finding where fork occurred * [SKIP CI] Prerelease * [release] 20221028 (subquery#1372) * hot fix tests (subquery#1360) * update tests: api.service.spec, init-controller.test, publish-controller.spec * update tests following comments * [SKIP CI] Prerelease * add ethereum to CLI and Validator (subquery#1378) * Imporve dictionary query (subquery#1371) * wip * wip * wip * broken wip * wip refactoring * seperate dictionaryQueryEntries * fix logic for setDicitonaryQueryEntries * cleaning up * move dictionaryEntry.ts to node-core * remove comments * create class for dictionaryqueryentries * update tests * move dictionaryQueryService into dicitonaryService * refactor * update logic for useDicitonary * relocated dictionaryQuery funcs * add test for dictquerymap * tests failing on windows? * test getDictionaryQueryEntries * test fixed for getDictioanryQueryEntries * added test for sorting * conflict fixed ? * fix conflict_2 * fix conflict_3 * add delete temp ds records back * clean up with new logic * clean up, add comments for test * fix * update logic * add generic type * [SKIP CI] Prerelease * Enable for better inheritance of generated entity modeld (subquery#1377) * refactor: enable for entity inheritance * refactor: include also filed getters * [SKIP CI] Prerelease * fix comments issue with new package (subquery#1380) * fix comments issue with new package * moved yaml package * [SKIP CI] Prerelease * fix logic with reindex and unfinalized height and dynamic ds (subquery#1382) * fix logic with reindex and unfinalized height * fix * include fix for subquery#1379 * update polkadot to 9.7.1 (subquery#1384) * [release] 20221107 * Fix remove alter table (subquery#1387) * remove migrate alter table * remove * [SKIP CI] Prerelease * [release] 20221108 (subquery#1388) * fix missing sequelize sync (subquery#1389) * [SKIP CI] Prerelease * [release] 20221108 patch (subquery#1390) * reindex bind (subquery#1391) * [SKIP CI] Prerelease * [release] 20221109 (subquery#1393) * Handle fetch errors, then retry (subquery#1386) * add retryOnFail function * add retryOnFail * add test, fix logic * [SKIP CI] Prerelease * fix (subquery#1395) * [SKIP CI] Prerelease * [release] 20221109 node-core (subquery#1394) * Fix tests hanging (subquery#1396) * Fix tests hanging * Update base docker image with newer git version * [SKIP CI] Prerelease * Add distinct query plugin (subquery#1274) * Add distinct query plugin * Clean up log * Fix distinct not being provided to query * Uppercase enum to be consistent with other enums * Update dictionary queries to try distinct argument * [SKIP CI] Prerelease * Add query distinct dependencies (subquery#1398) * fix missing update forked graphile dependencies * tidy up * tidy up * [SKIP CI] Prerelease * Break block dispatcher file up and move common code to base class (subquery#1397) * [SKIP CI] Prerelease * Hot schema trigger (subquery#1401) * implement trigger with notification * working prior clean up * refactor and clean up on async and await * clean up * clear comments * add filter * fix * fix err * [SKIP CI] Prerelease * [release] 20221115 (subquery#1402) * [release] 20221115 * [release] 20221115 * [release] 20221115 * fix hot schema (subquery#1404) * fix and refactor * refactor getTriggers * [SKIP CI] Prerelease * [release] 20221115 (subquery#1408) * [release] 20221115 * [release] 20221115 * fix fetchblock for works (subquery#1410) * [SKIP CI] Prerelease * fix row estimate (subquery#1417) * fix row estimate * tidy up * [SKIP CI] Prerelease * Add support to cli/validator for flare (subquery#1416) * [SKIP CI] Prerelease * missing validator missing ipfs chainTypes (subquery#1419) * [SKIP CI] Prerelease * query explain include graphql query (subquery#1426) * [SKIP CI] Prerelease * dictionary auth link integration (subquery#1411) * add required dependancies and add flag for feature * update modules and add logic to dictionary * update dictionary parameters, update tests, remove hardcoded values * simplify auth dictionary handling * add apollo link type, remove un-needed from wrapper * remove dictionary from indexer module * update flag name, add error check * update yargs flag description * Update appolo-links * update @subql/apollo-links version, remove apollo/client version change * fix yarn.lock * add await init to dictionary tests * fix fetch test dictionary provider, update yarn.lock Co-authored-by: Lachlan McCrae <8554309+lachlan1m@users.noreply.github.com> Co-authored-by: HZ <zhaihaohk@gmail.com> * Fix missing speChanged logic with getRuntime (subquery#1421) * init * need fix worker * fix test * tidy up * fix test, fix missing dictionary * inject service to runtime * remove unused code * [SKIP CI] Prerelease * Feat/multi chain indexing (subquery#1375) * [SKIP CI] Prerelease * remove enums (subquery#1427) * [SKIP CI] Prerelease * change to print graphql sql on request (subquery#1428) * [SKIP CI] Prerelease * fix missing util-crptyo in util package (subquery#1429) * fix missing util-crptyo in util package * need prerelease in query service * [SKIP CI] Prerelease * fix import issue in query service (subquery#1430) * [SKIP CI] Prerelease * fix missing query (subquery#1432) * [SKIP CI] Prerelease * change fetch module SubqueryProject => ISubqueryProject (subquery#1433) Co-authored-by: Lachlan McCrae <8554309+lachlan1m@users.noreply.github.com> * [SKIP CI] Prerelease * Update polkadot 9.9.1 (subquery#1434) * update polkadot api, also fix force-clean dependencies issue * tidy up * [SKIP CI] Prerelease * [release] 20221123 (subquery#1431) * [release] 20221123 * Update packages/query/CHANGELOG.md Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * Update packages/node-core/CHANGELOG.md Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * Update packages/node/CHANGELOG.md Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * Update packages/node-core/CHANGELOG.md Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * Update packages/validator/CHANGELOG.md Co-authored-by: Scott Twiname <skott.twiname@gmail.com> Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * Add a way to pass settings to the graphql playground (subquery#1436) * Add a way to pass settings to the graphql playground * Add a changelog entry * [SKIP CI] Prerelease * Fix the changelog entry for the last PR (subquery#1437) * [SKIP CI] Prerelease * fix enum name missing issue (subquery#1441) * fix enum name missing issue * tidy up * [SKIP CI] Prerelease * hash sql queries (subquery#1438) * hashed sql * add comments * add comment for hashName function * update function * update hashName function * [SKIP CI] Prerelease * [release] 20221130 (subquery#1442) * [release] 20221130 * [release] 20221130 * Add --file flag for codegen (subquery#1446) * init * change flag behaviour so -f can overwrite -l * update getManifestPath method, fix typo * update argument description * update variable names * update arg description * add error message on bad --file path Co-authored-by: Lachlan McCrae <8554309+lachlan1m@users.noreply.github.com> * [SKIP CI] Prerelease * add try and catch for hsr (subquery#1449) * [SKIP CI] Prerelease * Bypass blocks (subquery#1435) * wip * manifest error * manfiest passed * added logging for bypassing blocks * sync main, moved bypassBlocks under networks * pass all tests * mock dictioanry case * fix metadata * clean up * allow range * bypass working, needs more testing * wip on bypass with setting new bufferHeight * working * added runtime test * add test for latestBufferHeight (blocked) * removed unneeded test, improve logic * clean up * add comments * add comments for latestBufferHeight reasoning * update logic * refactor * update logic * refactor * [SKIP CI] Prerelease * [release] 20221205 (subquery#1448) * [release] 20221205 * [release] 20221205 * [release] 20221206 * details for bypassBlocks * Remove unused RuntimeService from indexer module, it had missing dependencies (subquery#1453) * [SKIP CI] Prerelease * Fix/subcommands (subquery#1451) * updated forceClean * update reindex * [SKIP CI] Prerelease * Enable env vars parsing at node and query commands (subquery#1452) * [SKIP CI] Prerelease * Workers: Fix SequelizeDatabaseError - tuple concurrently updated (subquery#1458) * add flag for hot-schema reload on node, ensure query is on main thread * remove flags * [SKIP CI] Prerelease * Add start height to project (subquery#1456) * [SKIP CI] Prerelease * Fix poi offset is 0 (subquery#1459) * [SKIP CI] Prerelease * Handle when fields got undefined with historical (subquery#1463) * [SKIP CI] Prerelease * update deploy cli (subquery#1460) * [SKIP CI] Prerelease * exit on fail (subquery#1464) * [SKIP CI] Prerelease * fix parentSpecVersion could be undefined (subquery#1467) * [SKIP CI] Prerelease * [release] 20221219 (subquery#1466) * [release] 20221219 * update * Update packages/node-core/CHANGELOG.md Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * Update packages/node/CHANGELOG.md Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * Update packages/node-core/CHANGELOG.md Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * Update packages/cli/CHANGELOG.md Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * update Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * Fix templates with other sdks, make -f flag work like other commands (subquery#1470) * Fix templates with other sdks, make -f flag work like other commands * Fix for sdks with first class smart contracts * [SKIP CI] Prerelease * Tweak error message for genesis hash (subquery#1471) * [SKIP CI] Prerelease * Fix trigger function (subquery#1469) * fix trigger and function for subscription and hot schema reload * update * [SKIP CI] Prerelease * [release] 20221222 (subquery#1474) * validate dictionary with start height (subquery#1473) * validate dictionary with start height * Update packages/node/src/indexer/fetch.service.ts Co-authored-by: Scott Twiname <skott.twiname@gmail.com> Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * [SKIP CI] Prerelease * add count for entity (subquery#1480) * add count for entity * Update packages/node-core/src/indexer/store.service.ts Co-authored-by: Scott Twiname <skott.twiname@gmail.com> Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * [SKIP CI] Prerelease * Fixing readme doc for @subql/query (subquery#1485) * doc: correcting grammer * fix: typo * [SKIP CI] Prerelease * update polkadot api to 9.11.1 (subquery#1483) * update polkadot api to 9.11.1 * update * [SKIP CI] Prerelease * add connection filters (subquery#1484) * [SKIP CI] Prerelease * update fetch.module dictionary service * add bypassBlocks Co-authored-by: hariu-starfish <103621490+hariu-starfish@users.noreply.github.com> Co-authored-by: Orion <orion@starfish.finance> Co-authored-by: Jay Ji <jiqiang90@hotmail.com> Co-authored-by: Scott Twiname <skott.twiname@gmail.com> Co-authored-by: Naveen V <velnaveen99@gmail.com> Co-authored-by: Filippo <filippo@embriotech.ch> Co-authored-by: Lachlan McCrae <8554309+lamcc21@users.noreply.github.com> Co-authored-by: Lachlan McCrae <8554309+lachlan1m@users.noreply.github.com> Co-authored-by: HZ <zhaihaohk@gmail.com> Co-authored-by: MOZGIII <mike-n@narod.ru> Co-authored-by: James Bayly <46693720+jamesbayly@users.noreply.github.com> Co-authored-by: Mingyang Li <53138432+Mingyang-Li@users.noreply.github.com>

* [SKIP CI] Prerelease * exit rather than throw error when api connection dropped (#1212) * [SKIP CI] Prerelease * [release] 202207027 Substrate (#1215) * [release] 202207027 Substrate * Update changelog * Update CHANGELOG.md * Update packages/types/CHANGELOG.md Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> * Update packages/cli/CHANGELOG.md Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> * Update packages/types/CHANGELOG.md Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> Co-authored-by: Scott Twiname <skott.twiname@gmail.com> Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> * removed rmraf on .github (#1223) * [SKIP CI] Prerelease * Export TypedEventRecord type (#1225) * [SKIP CI] Prerelease * Fix one off events being missed on startup (#1224) * Fix ready service never being ready Since introducing the project service, the event for ready was emit before the ready service was init and listening for the event. The solution is to check directly with the project service which is considered ready when a schema exists. * Revert ready service, manually init services outside of factories to not miss one time events * Fix up more events/metrics * [SKIP CI] Prerelease * add aggregate flag (#1219) * [SKIP CI] Prerelease * [release] 20220803 subql-node patch (#1229) * [CLI] Add algorand publish support (#1228) * [CLI] Add algorand publish support * Update validator and network types for algorand * Update deps, fix release script when pacakage name doesnt match path * Update readme to refer to other packages * Update README.md Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> * [SKIP CI] Prerelease * Fix dictionary being disabled with modulo filter (#1232) * Fix dictionary being disabled with modulo filter * Update changelog * [SKIP CI] Prerelease * [release] 20220804 (#1231) * Update polkadot 9 (#1234) * Update polkadot api to 9 * ignore init runtime decorate error * Bring back profiling fetchBlocksBatches, remove unnessary await (#1235) * [SKIP CI] Prerelease * schema migration (#1226) * [SKIP CI] Prerelease * Fix Queue exceeds max size error (#1239) * [SKIP CI] Prerelease * fix config db selection when subscription is enabled (#1241) * [SKIP CI] Prerelease * Fork OrderByAggregatesPlugin and modify to be compatible with historical feature (#1242) * [SKIP CI] Prerelease * Fix block height being applied to all queries (#1238) * [SKIP CI] Prerelease * Fix running custom ds processors in parallel on different data (#1243) * [SKIP CI] Prerelease * [release] 20220811 (#1245) * [release] 20220811 * update changelog * support bulkGet and add bulkUpdate (#1246) * add bulkUpdate * tidy up * support historical * improve bulkUpdate * fix * [SKIP CI] Prerelease * Fix issue with db sync and workers enabled (#1250) * Fix issue with db sync and workers enabled. Improved worker error handling * Remove cpu count workers limit * [SKIP CI] Prerelease * Improve Performance logging (#1244) * profilewrap for handler * wip * tidying up * remove console.log * tidy up * add transactions * Subql init mono (#1236) * draft * Tidy up * tidy up * Fix tests * [SKIP CI] Prerelease * [SKIP CI] Prerelease * Fix bigInt array entity being treated as a big int (#1252) * [SKIP CI] Prerelease * [release] 20220817 (#1254) * [release] 20220817 * Fix typo * Address comments * Update packages/types/CHANGELOG.md Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> * Update README.md * Node core (#1222) * both packages working * working on node core * avalanche with types avalanche * adding provider * provider extends ethersproject provider * Fix build issues * Remove avalanche packages * Add noImplicitAny flag to node-core and fix errors * Update build tooling and changelogs * Move events to node-core * Move store service to node-core * Use logger/profiler from node-core * Move poi service to node core * Move mmr service to node-core * Move over meta controllers/services to node-core * Fix imports * Fix build issues * Fix linter warning * Fix lint issues, move worker builder to node core * Remove BlockedQueue, fix more lint errors * Move some test files * Fix rebase issues, remove avax deps from node core * Fix build issue * Fix some tests * fix processBlockCount metric with workers Co-authored-by: roduquen <duquenne.robin42@gmail.com> Co-authored-by: JQQQ <jiqiang90@hotmail.com> * Add limit and offset options to store.getByField (#1259) * Add limit and offset options to store.getByField * Revert types change * Add delay and don't break loop when process queue is full (#1261) * Update cmd options in subql-node README (#1265) * Update README.md * Update packages/node/README.md * Fix symbol type being possible in string interpolation (#1264) * Fix symbol type being possible in string interpolation (#1266) * [SKIP CI] Prerelease * [release] 20220826 (#1263) * Fix imports not being relative (#1268) * [SKIP CI] Prerelease * [release] node-core 20220826 (#1269) * prerelease with @subql/node-core (#1270) * [SKIP CI] Prerelease * fix node-core import in node (#1272) * [SKIP CI] Prerelease * [release] 20220829 (#1273) * Move auto queue to node core (#1278) * [SKIP CI] Prerelease * update polkadot and types, to support with wasm (#1280) * update polkadot and types, to support with wasm * move logic to ds processor, update change for interface * remove dsAssets * remove dsAssets * update * fix fetch due to type changed (#1282) * fix fetch due to type changed * prerelease for types * [SKIP CI] Prerelease * fix missing await for dictionary (#1283) * [SKIP CI] Prerelease * rollback changes (#1286) * [SKIP CI] Prerelease * Fix tests failing with Cannot read properties of null (reading 'getProcessedBlockCount') (#1285) * Fix tests failing with Cannot read properties of null (reading 'getProcessedBlockCount') * Fix same error in more tests * Fix lint issue * [SKIP CI] Prerelease * fix remote gitpath (#1288) * [SKIP CI] Prerelease * Fix tests (#1297) * fix tests * tidy up * removed v0.0.1 test * [release] 20220913 (#1291) * [release] 20220913 * Updated CHANGELOG.MD Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> * Set useragent headers on polkadot api (#1296) * [SKIP CI] Prerelease * change dictionary to use equalToInsensitive (#1301) * change dictionary to use equalToInsensitive * add comments * [SKIP CI] Prerelease * [release] 20220919 (#1302) * [release] 20220919 * Update packages/node/CHANGELOG.md Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> * update sequelize (#1311) * update sequelize * update sequelize query * [SKIP CI] Prerelease * Subcommands for force-clean and reindex (#1281) * draft pr for documentation * merge * force-clean command working, refactoring needed * refactor force-clean * unsafe bug, in sandboxService, export package.json bug * updated package.json node-core * subql-node starting * refactor * fix up * node test suites still broken * Fix init logger with tests * Use old logger defaults * tests passes * tidy up * builds * Fix exporting logger (#1300) * tidy up, skip db.module tests * fixed db.module * tidy up getExistingProjectSchema * Reindex command (#1307) * init reindex command * add services to module * reindex functioning * add force-clean for issue #1230 * tidy up * tidy up based on comments * remove comments * edit command des * remove reindex from nodeconfig * moved functions into node-core from utils * update version Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * Fixes #1309 (#1314) * fix publish failed due to file object is null (#1316) * [SKIP CI] Prerelease * [release] 20220924 (#1317) * chnage from multi branch to main (#1318) * fix logger undefined (#1315) * [SKIP CI] Prerelease * change max connection to 10 and query service 10000 (#1322) Co-authored-by: Louise wang <louisewang118@.gmail.com> * [SKIP CI] Prerelease * update polkadot api to 9.4.2 (#1325) * functioning hot schema reload (#1321) * [SKIP CI] Prerelease * improve codegen error log (#1324) Co-authored-by: Louise wang <louisewang118@.gmail.com> * change createproject URL , project create issue, regx for project nam… (#1323) * change createproject URL , project create issue, regx for project name and project key * project name can be cap case * code improve by Jay comment Co-authored-by: Louise wang <louisewang118@.gmail.com> * [SKIP CI] Prerelease * manual prerelease (#1326) * [SKIP CI] Prerelease * enable historical by default (#1327) * [SKIP CI] Prerelease * [release] 20220929 (#1328) * [release] 20220929 * update * Update packages/node/CHANGELOG.md Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> * Update packages/query/CHANGELOG.md Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> * Update packages/utils/CHANGELOG.md Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> * Update packages/query/CHANGELOG.md Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> * Update packages/node/CHANGELOG.md Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> * Update packages/node-core/CHANGELOG.md Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> * Update packages/common/CHANGELOG.md Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> Co-authored-by: JQQQ <jiqiang90@hotmail.com> Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> * Update ipfs endpoints, improve error handling (#1337) * [SKIP CI] Prerelease * Fix nodeConfig not being provided to benchmark service (#1338) * [SKIP CI] Prerelease * [release] 20221006 (#1339) * Remove Subqueries table (#1340) * Remove Subqueries table * Fix NOT_SUPPORT function in project * Remove tests relating to old subqueries table * [SKIP CI] Prerelease * [release] 20221010 (#1342) * [release] 20221010 * Update packages/node/CHANGELOG.md Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> * Validate ds processor in worker threads (#1343) * [SKIP CI] Prerelease * Fix publish ipfs 503 (#1331) * add comments * publish ipfs issue * publish ipfs issue * Update packages/cli/src/controller/publish-controller.ts Co-authored-by: bz888 <ziyu.ben158@gmail.com> Co-authored-by: Louise wang <louisewang118@.gmail.com> Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * [SKIP CI] Prerelease * Move common dictionary service to node-core (#1345) * Move common dictionary service to node-core * Apply matcher from cosmos dictionary * Move sandbox code to node-core * Remove testing log * [SKIP CI] Prerelease * [release] 20221011 * Back port changes in algorand sdk to dictionary service (#1346) * [SKIP CI] Prerelease * temp disable hot schema reload (#1349) * temp disable hot schema reload * add CLI flag for hotschema reload * fix * print node version (#1348) * print node version * update logger info * update with node name Co-authored-by: Louise wang <louisewang118@.gmail.com> * [SKIP CI] Prerelease * [SKIP CI] Prerelease * fix output-format flag on node (#1351) * fix typo * update workers and yargs outputFmt * [SKIP CI] Prerelease * Timestamp block filters (#1310) * add block timestamp filter * bug fixes * bug fixes * clean code and write test for timestamp filter * move cron generation to SubqueryProject * fix next timestamp log bug * remove modifiedDataSources * [SKIP CI] Prerelease * Update to latest polkadot api (#1356) * [SKIP CI] Prerelease * add ad flags (#1352) * add ad flags * add interface for all AD flags and add prefix for indexer/query flags * change string of to for * some variable name change and update interface Co-authored-by: Louise wang <louisewang118@.gmail.com> * support version without v (#1354) * support version without v * improvement addv FUNC and solved undefined input Co-authored-by: Louise wang <louisewang118@.gmail.com> * fix deploy-test (#1357) * [SKIP CI] Prerelease * add dedicate data base flag for create project (#1355) * add dedicate data base flag for create project * test with latest createproject api host service and update wording Co-authored-by: Louise wang <louisewang118@.gmail.com> * Optimise modulo filter (#1358) * scale batch size if modulo filter used * add comments * add comments * fix logic runModuloblock * remove comments * remove more comments * add check datasources, check height logic * removed comments * refactor * flatten handler array * Fix template issues (#1363) * Fix the problem of creating multiple templates in the same block with the same template address * refactor: cache ds in same block Co-authored-by: Orion <orion@starfish.finance> * fix deployment and project tests (#1364) * fix deployment and project tests * fix linting error * [SKIP CI] Prerelease * [release] 20221027 (#1365) * fix: incomplete ds data in same block (#1370) Co-authored-by: Orion <orion@starfish.finance> * [SKIP CI] Prerelease * Best Block (#1308) * Draft * update changes * rebase changes * fix * fix (#1329) * disable best blocks for workers until we support it * Rename bestBlock to unfinalizedBlocks, other clean up * Clean up * Further clean up * Dedupe reindex function * Fix tests * Clean up * Update checking finalization to use parent hash * Rename logger * Refactor unfinalized blocks * Use header rather than full block, improve detecting forks * Verify unfinalized blocks when disabled. Use sorted array for storing unfinalized blocks * Clean up logs * Fix not indexing unfinalized blocks right away, exit if historical not enabled Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * [SKIP CI] Prerelease * Fix bugs with unfinalized blocks (#1374) * Fix bugs with unfinalized blocks - Fix using wrong hash for unfinalized blocks - Fix not removing unfinalized blocks when the latest unfinalized block < finalized * Fix tests getting stuck * Fix issue finding where fork occurred * [SKIP CI] Prerelease * [release] 20221028 (#1372) * hot fix tests (#1360) * update tests: api.service.spec, init-controller.test, publish-controller.spec * update tests following comments * [SKIP CI] Prerelease * add ethereum to CLI and Validator (#1378) * Imporve dictionary query (#1371) * wip * wip * wip * broken wip * wip refactoring * seperate dictionaryQueryEntries * fix logic for setDicitonaryQueryEntries * cleaning up * move dictionaryEntry.ts to node-core * remove comments * create class for dictionaryqueryentries * update tests * move dictionaryQueryService into dicitonaryService * refactor * update logic for useDicitonary * relocated dictionaryQuery funcs * add test for dictquerymap * tests failing on windows? * test getDictionaryQueryEntries * test fixed for getDictioanryQueryEntries * added test for sorting * conflict fixed ? * fix conflict_2 * fix conflict_3 * add delete temp ds records back * clean up with new logic * clean up, add comments for test * fix * update logic * add generic type * [SKIP CI] Prerelease * Enable for better inheritance of generated entity modeld (#1377) * refactor: enable for entity inheritance * refactor: include also filed getters * [SKIP CI] Prerelease * fix comments issue with new package (#1380) * fix comments issue with new package * moved yaml package * [SKIP CI] Prerelease * fix logic with reindex and unfinalized height and dynamic ds (#1382) * fix logic with reindex and unfinalized height * fix * include fix for #1379 * update polkadot to 9.7.1 (#1384) * [release] 20221107 * Fix remove alter table (#1387) * remove migrate alter table * remove * [SKIP CI] Prerelease * [release] 20221108 (#1388) * fix missing sequelize sync (#1389) * [SKIP CI] Prerelease * [release] 20221108 patch (#1390) * reindex bind (#1391) * [SKIP CI] Prerelease * [release] 20221109 (#1393) * Handle fetch errors, then retry (#1386) * add retryOnFail function * add retryOnFail * add test, fix logic * [SKIP CI] Prerelease * fix (#1395) * [SKIP CI] Prerelease * [release] 20221109 node-core (#1394) * Fix tests hanging (#1396) * Fix tests hanging * Update base docker image with newer git version * [SKIP CI] Prerelease * Add distinct query plugin (#1274) * Add distinct query plugin * Clean up log * Fix distinct not being provided to query * Uppercase enum to be consistent with other enums * Update dictionary queries to try distinct argument * [SKIP CI] Prerelease * Add query distinct dependencies (#1398) * fix missing update forked graphile dependencies * tidy up * tidy up * [SKIP CI] Prerelease * Break block dispatcher file up and move common code to base class (#1397) * [SKIP CI] Prerelease * Hot schema trigger (#1401) * implement trigger with notification * working prior clean up * refactor and clean up on async and await * clean up * clear comments * add filter * fix * fix err * [SKIP CI] Prerelease * [release] 20221115 (#1402) * [release] 20221115 * [release] 20221115 * [release] 20221115 * fix hot schema (#1404) * fix and refactor * refactor getTriggers * [SKIP CI] Prerelease * [release] 20221115 (#1408) * [release] 20221115 * [release] 20221115 * fix fetchblock for works (#1410) * [SKIP CI] Prerelease * fix row estimate (#1417) * fix row estimate * tidy up * [SKIP CI] Prerelease * Add support to cli/validator for flare (#1416) * [SKIP CI] Prerelease * missing validator missing ipfs chainTypes (#1419) * [SKIP CI] Prerelease * query explain include graphql query (#1426) * [SKIP CI] Prerelease * dictionary auth link integration (#1411) * add required dependancies and add flag for feature * update modules and add logic to dictionary * update dictionary parameters, update tests, remove hardcoded values * simplify auth dictionary handling * add apollo link type, remove un-needed from wrapper * remove dictionary from indexer module * update flag name, add error check * update yargs flag description * Update appolo-links * update @subql/apollo-links version, remove apollo/client version change * fix yarn.lock * add await init to dictionary tests * fix fetch test dictionary provider, update yarn.lock Co-authored-by: Lachlan McCrae <8554309+lachlan1m@users.noreply.github.com> Co-authored-by: HZ <zhaihaohk@gmail.com> * Fix missing speChanged logic with getRuntime (#1421) * init * need fix worker * fix test * tidy up * fix test, fix missing dictionary * inject service to runtime * remove unused code * [SKIP CI] Prerelease * Feat/multi chain indexing (#1375) * [SKIP CI] Prerelease * remove enums (#1427) * [SKIP CI] Prerelease * change to print graphql sql on request (#1428) * [SKIP CI] Prerelease * fix missing util-crptyo in util package (#1429) * fix missing util-crptyo in util package * need prerelease in query service * [SKIP CI] Prerelease * fix import issue in query service (#1430) * [SKIP CI] Prerelease * fix missing query (#1432) * [SKIP CI] Prerelease * change fetch module SubqueryProject => ISubqueryProject (#1433) Co-authored-by: Lachlan McCrae <8554309+lachlan1m@users.noreply.github.com> * [SKIP CI] Prerelease * Update polkadot 9.9.1 (#1434) * update polkadot api, also fix force-clean dependencies issue * tidy up * [SKIP CI] Prerelease * [release] 20221123 (#1431) * [release] 20221123 * Update packages/query/CHANGELOG.md Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * Update packages/node-core/CHANGELOG.md Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * Update packages/node/CHANGELOG.md Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * Update packages/node-core/CHANGELOG.md Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * Update packages/validator/CHANGELOG.md Co-authored-by: Scott Twiname <skott.twiname@gmail.com> Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * Add a way to pass settings to the graphql playground (#1436) * Add a way to pass settings to the graphql playground * Add a changelog entry * [SKIP CI] Prerelease * Fix the changelog entry for the last PR (#1437) * [SKIP CI] Prerelease * fix enum name missing issue (#1441) * fix enum name missing issue * tidy up * [SKIP CI] Prerelease * hash sql queries (#1438) * hashed sql * add comments * add comment for hashName function * update function * update hashName function * [SKIP CI] Prerelease * [release] 20221130 (#1442) * [release] 20221130 * [release] 20221130 * Add --file flag for codegen (#1446) * init * change flag behaviour so -f can overwrite -l * update getManifestPath method, fix typo * update argument description * update variable names * update arg description * add error message on bad --file path Co-authored-by: Lachlan McCrae <8554309+lachlan1m@users.noreply.github.com> * [SKIP CI] Prerelease * add try and catch for hsr (#1449) * [SKIP CI] Prerelease * Bypass blocks (#1435) * wip * manifest error * manfiest passed * added logging for bypassing blocks * sync main, moved bypassBlocks under networks * pass all tests * mock dictioanry case * fix metadata * clean up * allow range * bypass working, needs more testing * wip on bypass with setting new bufferHeight * working * added runtime test * add test for latestBufferHeight (blocked) * removed unneeded test, improve logic * clean up * add comments * add comments for latestBufferHeight reasoning * update logic * refactor * update logic * refactor * [SKIP CI] Prerelease * [release] 20221205 (#1448) * [release] 20221205 * [release] 20221205 * [release] 20221206 * details for bypassBlocks * Remove unused RuntimeService from indexer module, it had missing dependencies (#1453) * [SKIP CI] Prerelease * Fix/subcommands (#1451) * updated forceClean * update reindex * [SKIP CI] Prerelease * bumped polkadot/utils * Enable env vars parsing at node and query commands (#1452) * [SKIP CI] Prerelease * Workers: Fix SequelizeDatabaseError - tuple concurrently updated (#1458) * add flag for hot-schema reload on node, ensure query is on main thread * remove flags * [SKIP CI] Prerelease * Add start height to project (#1456) * [SKIP CI] Prerelease * Fix poi offset is 0 (#1459) * [SKIP CI] Prerelease Co-authored-by: Jay Ji <jiqiang90@hotmail.com> Co-authored-by: Scott Twiname <skott.twiname@gmail.com> Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> Co-authored-by: James Bayly <46693720+jamesbayly@users.noreply.github.com> Co-authored-by: roduquen <duquenne.robin42@gmail.com> Co-authored-by: HAO ZHAI <zhaihaohk@gmail.com> Co-authored-by: bighammerw <92004774+bighammerw@users.noreply.github.com> Co-authored-by: Louise wang <louisewang118@.gmail.com> Co-authored-by: Naveen V <velnaveen99@gmail.com> Co-authored-by: hariu-starfish <103621490+hariu-starfish@users.noreply.github.com> Co-authored-by: Orion <orion@starfish.finance> Co-authored-by: Filippo <filippo@embriotech.ch> Co-authored-by: Lachlan McCrae <8554309+lamcc21@users.noreply.github.com> Co-authored-by: Lachlan McCrae <8554309+lachlan1m@users.noreply.github.com> Co-authored-by: MOZGIII <mike-n@narod.ru>

* Bring back profiling fetchBlocksBatches, remove unnessary await (#1235) * [SKIP CI] Prerelease * schema migration (#1226) * [SKIP CI] Prerelease * Fix Queue exceeds max size error (#1239) * [SKIP CI] Prerelease * fix config db selection when subscription is enabled (#1241) * [SKIP CI] Prerelease * Fork OrderByAggregatesPlugin and modify to be compatible with historical feature (#1242) * [SKIP CI] Prerelease * Fix block height being applied to all queries (#1238) * [SKIP CI] Prerelease * Fix running custom ds processors in parallel on different data (#1243) * [SKIP CI] Prerelease * [release] 20220811 (#1245) * [release] 20220811 * update changelog * support bulkGet and add bulkUpdate (#1246) * add bulkUpdate * tidy up * support historical * improve bulkUpdate * fix * [SKIP CI] Prerelease * Fix issue with db sync and workers enabled (#1250) * Fix issue with db sync and workers enabled. Improved worker error handling * Remove cpu count workers limit * [SKIP CI] Prerelease * Improve Performance logging (#1244) * profilewrap for handler * wip * tidying up * remove console.log * tidy up * add transactions * Subql init mono (#1236) * draft * Tidy up * tidy up * Fix tests * [SKIP CI] Prerelease * [SKIP CI] Prerelease * Fix bigInt array entity being treated as a big int (#1252) * [SKIP CI] Prerelease * [release] 20220817 (#1254) * [release] 20220817 * Fix typo * Address comments * Update packages/types/CHANGELOG.md Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> * Update README.md * Node core (#1222) * both packages working * working on node core * avalanche with types avalanche * adding provider * provider extends ethersproject provider * Fix build issues * Remove avalanche packages * Add noImplicitAny flag to node-core and fix errors * Update build tooling and changelogs * Move events to node-core * Move store service to node-core * Use logger/profiler from node-core * Move poi service to node core * Move mmr service to node-core * Move over meta controllers/services to node-core * Fix imports * Fix build issues * Fix linter warning * Fix lint issues, move worker builder to node core * Remove BlockedQueue, fix more lint errors * Move some test files * Fix rebase issues, remove avax deps from node core * Fix build issue * Fix some tests * fix processBlockCount metric with workers Co-authored-by: roduquen <duquenne.robin42@gmail.com> Co-authored-by: JQQQ <jiqiang90@hotmail.com> * Add limit and offset options to store.getByField (#1259) * Add limit and offset options to store.getByField * Revert types change * Add delay and don't break loop when process queue is full (#1261) * Update cmd options in subql-node README (#1265) * Update README.md * Update packages/node/README.md * Fix symbol type being possible in string interpolation (#1264) * Fix symbol type being possible in string interpolation (#1266) * [SKIP CI] Prerelease * [release] 20220826 (#1263) * Fix imports not being relative (#1268) * [SKIP CI] Prerelease * [release] node-core 20220826 (#1269) * prerelease with @subql/node-core (#1270) * [SKIP CI] Prerelease * fix node-core import in node (#1272) * [SKIP CI] Prerelease * [release] 20220829 (#1273) * Move auto queue to node core (#1278) * [SKIP CI] Prerelease * update polkadot and types, to support with wasm (#1280) * update polkadot and types, to support with wasm * move logic to ds processor, update change for interface * remove dsAssets * remove dsAssets * update * fix fetch due to type changed (#1282) * fix fetch due to type changed * prerelease for types * [SKIP CI] Prerelease * fix missing await for dictionary (#1283) * [SKIP CI] Prerelease * rollback changes (#1286) * [SKIP CI] Prerelease * Fix tests failing with Cannot read properties of null (reading 'getProcessedBlockCount') (#1285) * Fix tests failing with Cannot read properties of null (reading 'getProcessedBlockCount') * Fix same error in more tests * Fix lint issue * [SKIP CI] Prerelease * fix remote gitpath (#1288) * [SKIP CI] Prerelease * Fix tests (#1297) * fix tests * tidy up * removed v0.0.1 test * [release] 20220913 (#1291) * [release] 20220913 * Updated CHANGELOG.MD Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> * Set useragent headers on polkadot api (#1296) * [SKIP CI] Prerelease * change dictionary to use equalToInsensitive (#1301) * change dictionary to use equalToInsensitive * add comments * [SKIP CI] Prerelease * [release] 20220919 (#1302) * [release] 20220919 * Update packages/node/CHANGELOG.md Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> * update sequelize (#1311) * update sequelize * update sequelize query * [SKIP CI] Prerelease * Subcommands for force-clean and reindex (#1281) * draft pr for documentation * merge * force-clean command working, refactoring needed * refactor force-clean * unsafe bug, in sandboxService, export package.json bug * updated package.json node-core * subql-node starting * refactor * fix up * node test suites still broken * Fix init logger with tests * Use old logger defaults * tests passes * tidy up * builds * Fix exporting logger (#1300) * tidy up, skip db.module tests * fixed db.module * tidy up getExistingProjectSchema * Reindex command (#1307) * init reindex command * add services to module * reindex functioning * add force-clean for issue #1230 * tidy up * tidy up based on comments * remove comments * edit command des * remove reindex from nodeconfig * moved functions into node-core from utils * update version Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * Fixes #1309 (#1314) * fix publish failed due to file object is null (#1316) * [SKIP CI] Prerelease * [release] 20220924 (#1317) * chnage from multi branch to main (#1318) * fix logger undefined (#1315) * [SKIP CI] Prerelease * change max connection to 10 and query service 10000 (#1322) Co-authored-by: Louise wang <louisewang118@.gmail.com> * [SKIP CI] Prerelease * update polkadot api to 9.4.2 (#1325) * functioning hot schema reload (#1321) * [SKIP CI] Prerelease * improve codegen error log (#1324) Co-authored-by: Louise wang <louisewang118@.gmail.com> * change createproject URL , project create issue, regx for project nam… (#1323) * change createproject URL , project create issue, regx for project name and project key * project name can be cap case * code improve by Jay comment Co-authored-by: Louise wang <louisewang118@.gmail.com> * [SKIP CI] Prerelease * manual prerelease (#1326) * [SKIP CI] Prerelease * enable historical by default (#1327) * [SKIP CI] Prerelease * [release] 20220929 (#1328) * [release] 20220929 * update * Update packages/node/CHANGELOG.md Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> * Update packages/query/CHANGELOG.md Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> * Update packages/utils/CHANGELOG.md Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> * Update packages/query/CHANGELOG.md Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> * Update packages/node/CHANGELOG.md Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> * Update packages/node-core/CHANGELOG.md Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> * Update packages/common/CHANGELOG.md Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> Co-authored-by: JQQQ <jiqiang90@hotmail.com> Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> * Update ipfs endpoints, improve error handling (#1337) * [SKIP CI] Prerelease * Fix nodeConfig not being provided to benchmark service (#1338) * [SKIP CI] Prerelease * [release] 20221006 (#1339) * Remove Subqueries table (#1340) * Remove Subqueries table * Fix NOT_SUPPORT function in project * Remove tests relating to old subqueries table * [SKIP CI] Prerelease * [release] 20221010 (#1342) * [release] 20221010 * Update packages/node/CHANGELOG.md Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> * Validate ds processor in worker threads (#1343) * [SKIP CI] Prerelease * Fix publish ipfs 503 (#1331) * add comments * publish ipfs issue * publish ipfs issue * Update packages/cli/src/controller/publish-controller.ts Co-authored-by: bz888 <ziyu.ben158@gmail.com> Co-authored-by: Louise wang <louisewang118@.gmail.com> Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * [SKIP CI] Prerelease * Move common dictionary service to node-core (#1345) * Move common dictionary service to node-core * Apply matcher from cosmos dictionary * Move sandbox code to node-core * Remove testing log * [SKIP CI] Prerelease * [release] 20221011 * Back port changes in algorand sdk to dictionary service (#1346) * [SKIP CI] Prerelease * temp disable hot schema reload (#1349) * temp disable hot schema reload * add CLI flag for hotschema reload * fix * print node version (#1348) * print node version * update logger info * update with node name Co-authored-by: Louise wang <louisewang118@.gmail.com> * [SKIP CI] Prerelease * [SKIP CI] Prerelease * fix output-format flag on node (#1351) * fix typo * update workers and yargs outputFmt * [SKIP CI] Prerelease * Timestamp block filters (#1310) * add block timestamp filter * bug fixes * bug fixes * clean code and write test for timestamp filter * move cron generation to SubqueryProject * fix next timestamp log bug * remove modifiedDataSources * [SKIP CI] Prerelease * Update to latest polkadot api (#1356) * [SKIP CI] Prerelease * add ad flags (#1352) * add ad flags * add interface for all AD flags and add prefix for indexer/query flags * change string of to for * some variable name change and update interface Co-authored-by: Louise wang <louisewang118@.gmail.com> * support version without v (#1354) * support version without v * improvement addv FUNC and solved undefined input Co-authored-by: Louise wang <louisewang118@.gmail.com> * fix deploy-test (#1357) * [SKIP CI] Prerelease * add dedicate data base flag for create project (#1355) * add dedicate data base flag for create project * test with latest createproject api host service and update wording Co-authored-by: Louise wang <louisewang118@.gmail.com> * Optimise modulo filter (#1358) * scale batch size if modulo filter used * add comments * add comments * fix logic runModuloblock * remove comments * remove more comments * add check datasources, check height logic * removed comments * refactor * flatten handler array * Fix template issues (#1363) * Fix the problem of creating multiple templates in the same block with the same template address * refactor: cache ds in same block Co-authored-by: Orion <orion@starfish.finance> * fix deployment and project tests (#1364) * fix deployment and project tests * fix linting error * [SKIP CI] Prerelease * [release] 20221027 (#1365) * fix: incomplete ds data in same block (#1370) Co-authored-by: Orion <orion@starfish.finance> * [SKIP CI] Prerelease * Best Block (#1308) * Draft * update changes * rebase changes * fix * fix (#1329) * disable best blocks for workers until we support it * Rename bestBlock to unfinalizedBlocks, other clean up * Clean up * Further clean up * Dedupe reindex function * Fix tests * Clean up * Update checking finalization to use parent hash * Rename logger * Refactor unfinalized blocks * Use header rather than full block, improve detecting forks * Verify unfinalized blocks when disabled. Use sorted array for storing unfinalized blocks * Clean up logs * Fix not indexing unfinalized blocks right away, exit if historical not enabled Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * [SKIP CI] Prerelease * Fix bugs with unfinalized blocks (#1374) * Fix bugs with unfinalized blocks - Fix using wrong hash for unfinalized blocks - Fix not removing unfinalized blocks when the latest unfinalized block < finalized * Fix tests getting stuck * Fix issue finding where fork occurred * [SKIP CI] Prerelease * [release] 20221028 (#1372) * hot fix tests (#1360) * update tests: api.service.spec, init-controller.test, publish-controller.spec * update tests following comments * [SKIP CI] Prerelease * add ethereum to CLI and Validator (#1378) * Imporve dictionary query (#1371) * wip * wip * wip * broken wip * wip refactoring * seperate dictionaryQueryEntries * fix logic for setDicitonaryQueryEntries * cleaning up * move dictionaryEntry.ts to node-core * remove comments * create class for dictionaryqueryentries * update tests * move dictionaryQueryService into dicitonaryService * refactor * update logic for useDicitonary * relocated dictionaryQuery funcs * add test for dictquerymap * tests failing on windows? * test getDictionaryQueryEntries * test fixed for getDictioanryQueryEntries * added test for sorting * conflict fixed ? * fix conflict_2 * fix conflict_3 * add delete temp ds records back * clean up with new logic * clean up, add comments for test * fix * update logic * add generic type * [SKIP CI] Prerelease * Enable for better inheritance of generated entity modeld (#1377) * refactor: enable for entity inheritance * refactor: include also filed getters * [SKIP CI] Prerelease * fix comments issue with new package (#1380) * fix comments issue with new package * moved yaml package * [SKIP CI] Prerelease * fix logic with reindex and unfinalized height and dynamic ds (#1382) * fix logic with reindex and unfinalized height * fix * include fix for #1379 * update polkadot to 9.7.1 (#1384) * [release] 20221107 * Fix remove alter table (#1387) * remove migrate alter table * remove * [SKIP CI] Prerelease * [release] 20221108 (#1388) * fix missing sequelize sync (#1389) * [SKIP CI] Prerelease * [release] 20221108 patch (#1390) * reindex bind (#1391) * [SKIP CI] Prerelease * [release] 20221109 (#1393) * Handle fetch errors, then retry (#1386) * add retryOnFail function * add retryOnFail * add test, fix logic * [SKIP CI] Prerelease * fix (#1395) * [SKIP CI] Prerelease * [release] 20221109 node-core (#1394) * Fix tests hanging (#1396) * Fix tests hanging * Update base docker image with newer git version * [SKIP CI] Prerelease * Add distinct query plugin (#1274) * Add distinct query plugin * Clean up log * Fix distinct not being provided to query * Uppercase enum to be consistent with other enums * Update dictionary queries to try distinct argument * [SKIP CI] Prerelease * Add query distinct dependencies (#1398) * fix missing update forked graphile dependencies * tidy up * tidy up * [SKIP CI] Prerelease * Break block dispatcher file up and move common code to base class (#1397) * [SKIP CI] Prerelease * Hot schema trigger (#1401) * implement trigger with notification * working prior clean up * refactor and clean up on async and await * clean up * clear comments * add filter * fix * fix err * [SKIP CI] Prerelease * [release] 20221115 (#1402) * [release] 20221115 * [release] 20221115 * [release] 20221115 * fix hot schema (#1404) * fix and refactor * refactor getTriggers * [SKIP CI] Prerelease * [release] 20221115 (#1408) * [release] 20221115 * [release] 20221115 * fix fetchblock for works (#1410) * [SKIP CI] Prerelease * fix row estimate (#1417) * fix row estimate * tidy up * [SKIP CI] Prerelease * Add support to cli/validator for flare (#1416) * [SKIP CI] Prerelease * missing validator missing ipfs chainTypes (#1419) * [SKIP CI] Prerelease * query explain include graphql query (#1426) * [SKIP CI] Prerelease * dictionary auth link integration (#1411) * add required dependancies and add flag for feature * update modules and add logic to dictionary * update dictionary parameters, update tests, remove hardcoded values * simplify auth dictionary handling * add apollo link type, remove un-needed from wrapper * remove dictionary from indexer module * update flag name, add error check * update yargs flag description * Update appolo-links * update @subql/apollo-links version, remove apollo/client version change * fix yarn.lock * add await init to dictionary tests * fix fetch test dictionary provider, update yarn.lock Co-authored-by: Lachlan McCrae <8554309+lachlan1m@users.noreply.github.com> Co-authored-by: HZ <zhaihaohk@gmail.com> * Fix missing speChanged logic with getRuntime (#1421) * init * need fix worker * fix test * tidy up * fix test, fix missing dictionary * inject service to runtime * remove unused code * [SKIP CI] Prerelease * Feat/multi chain indexing (#1375) * [SKIP CI] Prerelease * remove enums (#1427) * [SKIP CI] Prerelease * change to print graphql sql on request (#1428) * [SKIP CI] Prerelease * fix missing util-crptyo in util package (#1429) * fix missing util-crptyo in util package * need prerelease in query service * [SKIP CI] Prerelease * fix import issue in query service (#1430) * [SKIP CI] Prerelease * fix missing query (#1432) * [SKIP CI] Prerelease * change fetch module SubqueryProject => ISubqueryProject (#1433) Co-authored-by: Lachlan McCrae <8554309+lachlan1m@users.noreply.github.com> * [SKIP CI] Prerelease * Update polkadot 9.9.1 (#1434) * update polkadot api, also fix force-clean dependencies issue * tidy up * [SKIP CI] Prerelease * [release] 20221123 (#1431) * [release] 20221123 * Update packages/query/CHANGELOG.md Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * Update packages/node-core/CHANGELOG.md Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * Update packages/node/CHANGELOG.md Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * Update packages/node-core/CHANGELOG.md Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * Update packages/validator/CHANGELOG.md Co-authored-by: Scott Twiname <skott.twiname@gmail.com> Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * Add a way to pass settings to the graphql playground (#1436) * Add a way to pass settings to the graphql playground * Add a changelog entry * [SKIP CI] Prerelease * Fix the changelog entry for the last PR (#1437) * [SKIP CI] Prerelease * fix enum name missing issue (#1441) * fix enum name missing issue * tidy up * [SKIP CI] Prerelease * hash sql queries (#1438) * hashed sql * add comments * add comment for hashName function * update function * update hashName function * [SKIP CI] Prerelease * [release] 20221130 (#1442) * [release] 20221130 * [release] 20221130 * Add --file flag for codegen (#1446) * init * change flag behaviour so -f can overwrite -l * update getManifestPath method, fix typo * update argument description * update variable names * update arg description * add error message on bad --file path Co-authored-by: Lachlan McCrae <8554309+lachlan1m@users.noreply.github.com> * [SKIP CI] Prerelease * add try and catch for hsr (#1449) * [SKIP CI] Prerelease * Bypass blocks (#1435) * wip * manifest error * manfiest passed * added logging for bypassing blocks * sync main, moved bypassBlocks under networks * pass all tests * mock dictioanry case * fix metadata * clean up * allow range * bypass working, needs more testing * wip on bypass with setting new bufferHeight * working * added runtime test * add test for latestBufferHeight (blocked) * removed unneeded test, improve logic * clean up * add comments * add comments for latestBufferHeight reasoning * update logic * refactor * update logic * refactor * [SKIP CI] Prerelease * [release] 20221205 (#1448) * [release] 20221205 * [release] 20221205 * [release] 20221206 * details for bypassBlocks * Remove unused RuntimeService from indexer module, it had missing dependencies (#1453) * [SKIP CI] Prerelease * Fix/subcommands (#1451) * updated forceClean * update reindex * [SKIP CI] Prerelease * Enable env vars parsing at node and query commands (#1452) * [SKIP CI] Prerelease * Workers: Fix SequelizeDatabaseError - tuple concurrently updated (#1458) * add flag for hot-schema reload on node, ensure query is on main thread * remove flags * [SKIP CI] Prerelease * Add start height to project (#1456) * [SKIP CI] Prerelease * Fix poi offset is 0 (#1459) * [SKIP CI] Prerelease * Handle when fields got undefined with historical (#1463) * [SKIP CI] Prerelease * update deploy cli (#1460) * [SKIP CI] Prerelease * exit on fail (#1464) * [SKIP CI] Prerelease * fix parentSpecVersion could be undefined (#1467) * [SKIP CI] Prerelease * [release] 20221219 (#1466) * [release] 20221219 * update * Update packages/node-core/CHANGELOG.md Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * Update packages/node/CHANGELOG.md Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * Update packages/node-core/CHANGELOG.md Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * Update packages/cli/CHANGELOG.md Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * update Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * Fix templates with other sdks, make -f flag work like other commands (#1470) * Fix templates with other sdks, make -f flag work like other commands * Fix for sdks with first class smart contracts * [SKIP CI] Prerelease * Tweak error message for genesis hash (#1471) * [SKIP CI] Prerelease * Fix trigger function (#1469) * fix trigger and function for subscription and hot schema reload * update * [SKIP CI] Prerelease * [release] 20221222 (#1474) * validate dictionary with start height (#1473) * validate dictionary with start height * Update packages/node/src/indexer/fetch.service.ts Co-authored-by: Scott Twiname <skott.twiname@gmail.com> Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * [SKIP CI] Prerelease * add count for entity (#1480) * add count for entity * Update packages/node-core/src/indexer/store.service.ts Co-authored-by: Scott Twiname <skott.twiname@gmail.com> Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * [SKIP CI] Prerelease * add dictionary validation expections Co-authored-by: Scott Twiname <skott.twiname@gmail.com> Co-authored-by: Jay Ji <jiqiang90@hotmail.com> Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> Co-authored-by: James Bayly <46693720+jamesbayly@users.noreply.github.com> Co-authored-by: roduquen <duquenne.robin42@gmail.com> Co-authored-by: HAO ZHAI <zhaihaohk@gmail.com> Co-authored-by: bighammerw <92004774+bighammerw@users.noreply.github.com> Co-authored-by: Louise wang <louisewang118@.gmail.com> Co-authored-by: Naveen V <velnaveen99@gmail.com> Co-authored-by: hariu-starfish <103621490+hariu-starfish@users.noreply.github.com> Co-authored-by: Orion <orion@starfish.finance> Co-authored-by: Filippo <filippo@embriotech.ch> Co-authored-by: Lachlan McCrae <8554309+lamcc21@users.noreply.github.com> Co-authored-by: Lachlan McCrae <8554309+lachlan1m@users.noreply.github.com> Co-authored-by: MOZGIII <mike-n@narod.ru>

* [SKIP CI] Prerelease * manual prerelease (#1326) * [SKIP CI] Prerelease * enable historical by default (#1327) * [SKIP CI] Prerelease * [release] 20220929 (#1328) * [release] 20220929 * update * Update packages/node/CHANGELOG.md Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> * Update packages/query/CHANGELOG.md Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> * Update packages/utils/CHANGELOG.md Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> * Update packages/query/CHANGELOG.md Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> * Update packages/node/CHANGELOG.md Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> * Update packages/node-core/CHANGELOG.md Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> * Update packages/common/CHANGELOG.md Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> Co-authored-by: JQQQ <jiqiang90@hotmail.com> Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> * Update ipfs endpoints, improve error handling (#1337) * [SKIP CI] Prerelease * Fix nodeConfig not being provided to benchmark service (#1338) * [SKIP CI] Prerelease * [release] 20221006 (#1339) * Remove Subqueries table (#1340) * Remove Subqueries table * Fix NOT_SUPPORT function in project * Remove tests relating to old subqueries table * [SKIP CI] Prerelease * [release] 20221010 (#1342) * [release] 20221010 * Update packages/node/CHANGELOG.md Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> * Validate ds processor in worker threads (#1343) * [SKIP CI] Prerelease * Fix publish ipfs 503 (#1331) * add comments * publish ipfs issue * publish ipfs issue * Update packages/cli/src/controller/publish-controller.ts Co-authored-by: bz888 <ziyu.ben158@gmail.com> Co-authored-by: Louise wang <louisewang118@.gmail.com> Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * [SKIP CI] Prerelease * Move common dictionary service to node-core (#1345) * Move common dictionary service to node-core * Apply matcher from cosmos dictionary * Move sandbox code to node-core * Remove testing log * [SKIP CI] Prerelease * [release] 20221011 * Back port changes in algorand sdk to dictionary service (#1346) * [SKIP CI] Prerelease * temp disable hot schema reload (#1349) * temp disable hot schema reload * add CLI flag for hotschema reload * fix * print node version (#1348) * print node version * update logger info * update with node name Co-authored-by: Louise wang <louisewang118@.gmail.com> * [SKIP CI] Prerelease * [SKIP CI] Prerelease * fix output-format flag on node (#1351) * fix typo * update workers and yargs outputFmt * [SKIP CI] Prerelease * Timestamp block filters (#1310) * add block timestamp filter * bug fixes * bug fixes * clean code and write test for timestamp filter * move cron generation to SubqueryProject * fix next timestamp log bug * remove modifiedDataSources * [SKIP CI] Prerelease * Update to latest polkadot api (#1356) * [SKIP CI] Prerelease * add ad flags (#1352) * add ad flags * add interface for all AD flags and add prefix for indexer/query flags * change string of to for * some variable name change and update interface Co-authored-by: Louise wang <louisewang118@.gmail.com> * support version without v (#1354) * support version without v * improvement addv FUNC and solved undefined input Co-authored-by: Louise wang <louisewang118@.gmail.com> * fix deploy-test (#1357) * [SKIP CI] Prerelease * add dedicate data base flag for create project (#1355) * add dedicate data base flag for create project * test with latest createproject api host service and update wording Co-authored-by: Louise wang <louisewang118@.gmail.com> * Optimise modulo filter (#1358) * scale batch size if modulo filter used * add comments * add comments * fix logic runModuloblock * remove comments * remove more comments * add check datasources, check height logic * removed comments * refactor * flatten handler array * Fix template issues (#1363) * Fix the problem of creating multiple templates in the same block with the same template address * refactor: cache ds in same block Co-authored-by: Orion <orion@starfish.finance> * fix deployment and project tests (#1364) * fix deployment and project tests * fix linting error * [SKIP CI] Prerelease * [release] 20221027 (#1365) * fix: incomplete ds data in same block (#1370) Co-authored-by: Orion <orion@starfish.finance> * [SKIP CI] Prerelease * Best Block (#1308) * Draft * update changes * rebase changes * fix * fix (#1329) * disable best blocks for workers until we support it * Rename bestBlock to unfinalizedBlocks, other clean up * Clean up * Further clean up * Dedupe reindex function * Fix tests * Clean up * Update checking finalization to use parent hash * Rename logger * Refactor unfinalized blocks * Use header rather than full block, improve detecting forks * Verify unfinalized blocks when disabled. Use sorted array for storing unfinalized blocks * Clean up logs * Fix not indexing unfinalized blocks right away, exit if historical not enabled Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * [SKIP CI] Prerelease * Fix bugs with unfinalized blocks (#1374) * Fix bugs with unfinalized blocks - Fix using wrong hash for unfinalized blocks - Fix not removing unfinalized blocks when the latest unfinalized block < finalized * Fix tests getting stuck * Fix issue finding where fork occurred * [SKIP CI] Prerelease * [release] 20221028 (#1372) * hot fix tests (#1360) * update tests: api.service.spec, init-controller.test, publish-controller.spec * update tests following comments * [SKIP CI] Prerelease * add ethereum to CLI and Validator (#1378) * Imporve dictionary query (#1371) * wip * wip * wip * broken wip * wip refactoring * seperate dictionaryQueryEntries * fix logic for setDicitonaryQueryEntries * cleaning up * move dictionaryEntry.ts to node-core * remove comments * create class for dictionaryqueryentries * update tests * move dictionaryQueryService into dicitonaryService * refactor * update logic for useDicitonary * relocated dictionaryQuery funcs * add test for dictquerymap * tests failing on windows? * test getDictionaryQueryEntries * test fixed for getDictioanryQueryEntries * added test for sorting * conflict fixed ? * fix conflict_2 * fix conflict_3 * add delete temp ds records back * clean up with new logic * clean up, add comments for test * fix * update logic * add generic type * [SKIP CI] Prerelease * Enable for better inheritance of generated entity modeld (#1377) * refactor: enable for entity inheritance * refactor: include also filed getters * [SKIP CI] Prerelease * fix comments issue with new package (#1380) * fix comments issue with new package * moved yaml package * [SKIP CI] Prerelease * fix logic with reindex and unfinalized height and dynamic ds (#1382) * fix logic with reindex and unfinalized height * fix * include fix for #1379 * update polkadot to 9.7.1 (#1384) * [release] 20221107 * Fix remove alter table (#1387) * remove migrate alter table * remove * [SKIP CI] Prerelease * [release] 20221108 (#1388) * fix missing sequelize sync (#1389) * [SKIP CI] Prerelease * [release] 20221108 patch (#1390) * reindex bind (#1391) * [SKIP CI] Prerelease * [release] 20221109 (#1393) * Handle fetch errors, then retry (#1386) * add retryOnFail function * add retryOnFail * add test, fix logic * [SKIP CI] Prerelease * fix (#1395) * [SKIP CI] Prerelease * [release] 20221109 node-core (#1394) * Fix tests hanging (#1396) * Fix tests hanging * Update base docker image with newer git version * [SKIP CI] Prerelease * Add distinct query plugin (#1274) * Add distinct query plugin * Clean up log * Fix distinct not being provided to query * Uppercase enum to be consistent with other enums * Update dictionary queries to try distinct argument * [SKIP CI] Prerelease * Add query distinct dependencies (#1398) * fix missing update forked graphile dependencies * tidy up * tidy up * [SKIP CI] Prerelease * Break block dispatcher file up and move common code to base class (#1397) * [SKIP CI] Prerelease * Hot schema trigger (#1401) * implement trigger with notification * working prior clean up * refactor and clean up on async and await * clean up * clear comments * add filter * fix * fix err * [SKIP CI] Prerelease * [release] 20221115 (#1402) * [release] 20221115 * [release] 20221115 * [release] 20221115 * fix hot schema (#1404) * fix and refactor * refactor getTriggers * [SKIP CI] Prerelease * [release] 20221115 (#1408) * [release] 20221115 * [release] 20221115 * fix fetchblock for works (#1410) * [SKIP CI] Prerelease * fix row estimate (#1417) * fix row estimate * tidy up * [SKIP CI] Prerelease * Add support to cli/validator for flare (#1416) * [SKIP CI] Prerelease * missing validator missing ipfs chainTypes (#1419) * [SKIP CI] Prerelease * query explain include graphql query (#1426) * [SKIP CI] Prerelease * dictionary auth link integration (#1411) * add required dependancies and add flag for feature * update modules and add logic to dictionary * update dictionary parameters, update tests, remove hardcoded values * simplify auth dictionary handling * add apollo link type, remove un-needed from wrapper * remove dictionary from indexer module * update flag name, add error check * update yargs flag description * Update appolo-links * update @subql/apollo-links version, remove apollo/client version change * fix yarn.lock * add await init to dictionary tests * fix fetch test dictionary provider, update yarn.lock Co-authored-by: Lachlan McCrae <8554309+lachlan1m@users.noreply.github.com> Co-authored-by: HZ <zhaihaohk@gmail.com> * Fix missing speChanged logic with getRuntime (#1421) * init * need fix worker * fix test * tidy up * fix test, fix missing dictionary * inject service to runtime * remove unused code * [SKIP CI] Prerelease * Feat/multi chain indexing (#1375) * [SKIP CI] Prerelease * remove enums (#1427) * [SKIP CI] Prerelease * change to print graphql sql on request (#1428) * [SKIP CI] Prerelease * fix missing util-crptyo in util package (#1429) * fix missing util-crptyo in util package * need prerelease in query service * [SKIP CI] Prerelease * fix import issue in query service (#1430) * [SKIP CI] Prerelease * fix missing query (#1432) * [SKIP CI] Prerelease * change fetch module SubqueryProject => ISubqueryProject (#1433) Co-authored-by: Lachlan McCrae <8554309+lachlan1m@users.noreply.github.com> * [SKIP CI] Prerelease * Update polkadot 9.9.1 (#1434) * update polkadot api, also fix force-clean dependencies issue * tidy up * [SKIP CI] Prerelease * [release] 20221123 (#1431) * [release] 20221123 * Update packages/query/CHANGELOG.md Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * Update packages/node-core/CHANGELOG.md Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * Update packages/node/CHANGELOG.md Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * Update packages/node-core/CHANGELOG.md Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * Update packages/validator/CHANGELOG.md Co-authored-by: Scott Twiname <skott.twiname@gmail.com> Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * Add a way to pass settings to the graphql playground (#1436) * Add a way to pass settings to the graphql playground * Add a changelog entry * [SKIP CI] Prerelease * Fix the changelog entry for the last PR (#1437) * [SKIP CI] Prerelease * fix enum name missing issue (#1441) * fix enum name missing issue * tidy up * [SKIP CI] Prerelease * hash sql queries (#1438) * hashed sql * add comments * add comment for hashName function * update function * update hashName function * [SKIP CI] Prerelease * [release] 20221130 (#1442) * [release] 20221130 * [release] 20221130 * Add --file flag for codegen (#1446) * init * change flag behaviour so -f can overwrite -l * update getManifestPath method, fix typo * update argument description * update variable names * update arg description * add error message on bad --file path Co-authored-by: Lachlan McCrae <8554309+lachlan1m@users.noreply.github.com> * [SKIP CI] Prerelease * add try and catch for hsr (#1449) * [SKIP CI] Prerelease * Bypass blocks (#1435) * wip * manifest error * manfiest passed * added logging for bypassing blocks * sync main, moved bypassBlocks under networks * pass all tests * mock dictioanry case * fix metadata * clean up * allow range * bypass working, needs more testing * wip on bypass with setting new bufferHeight * working * added runtime test * add test for latestBufferHeight (blocked) * removed unneeded test, improve logic * clean up * add comments * add comments for latestBufferHeight reasoning * update logic * refactor * update logic * refactor * [SKIP CI] Prerelease * [release] 20221205 (#1448) * [release] 20221205 * [release] 20221205 * [release] 20221206 * details for bypassBlocks * Remove unused RuntimeService from indexer module, it had missing dependencies (#1453) * [SKIP CI] Prerelease * Fix/subcommands (#1451) * updated forceClean * update reindex * [SKIP CI] Prerelease * Enable env vars parsing at node and query commands (#1452) * [SKIP CI] Prerelease * Workers: Fix SequelizeDatabaseError - tuple concurrently updated (#1458) * add flag for hot-schema reload on node, ensure query is on main thread * remove flags * [SKIP CI] Prerelease * Add start height to project (#1456) * [SKIP CI] Prerelease * Fix poi offset is 0 (#1459) * [SKIP CI] Prerelease * Handle when fields got undefined with historical (#1463) * [SKIP CI] Prerelease * update deploy cli (#1460) * [SKIP CI] Prerelease * exit on fail (#1464) * [SKIP CI] Prerelease * fix parentSpecVersion could be undefined (#1467) * [SKIP CI] Prerelease * [release] 20221219 (#1466) * [release] 20221219 * update * Update packages/node-core/CHANGELOG.md Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * Update packages/node/CHANGELOG.md Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * Update packages/node-core/CHANGELOG.md Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * Update packages/cli/CHANGELOG.md Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * update Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * Fix templates with other sdks, make -f flag work like other commands (#1470) * Fix templates with other sdks, make -f flag work like other commands * Fix for sdks with first class smart contracts * [SKIP CI] Prerelease * Tweak error message for genesis hash (#1471) * [SKIP CI] Prerelease * Fix trigger function (#1469) * fix trigger and function for subscription and hot schema reload * update * [SKIP CI] Prerelease * [release] 20221222 (#1474) * validate dictionary with start height (#1473) * validate dictionary with start height * Update packages/node/src/indexer/fetch.service.ts Co-authored-by: Scott Twiname <skott.twiname@gmail.com> Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * [SKIP CI] Prerelease * add count for entity (#1480) * add count for entity * Update packages/node-core/src/indexer/store.service.ts Co-authored-by: Scott Twiname <skott.twiname@gmail.com> Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * [SKIP CI] Prerelease * Fixing readme doc for @subql/query (#1485) * doc: correcting grammer * fix: typo * [SKIP CI] Prerelease * update polkadot api to 9.11.1 (#1483) * update polkadot api to 9.11.1 * update * [SKIP CI] Prerelease * add connection filters (#1484) * [SKIP CI] Prerelease * fix workers missing set last buffered height from empty dictionary (#1492) * [SKIP CI] Prerelease * Fix custom metadata keys not being applied (#1496) * add block hash validation after fetch (#1494) * add block hash validation after fetch * add tests * skip fetch test * Update log * [SKIP CI] Prerelease * [release] 20230123 (#1497) * Add support for near projects to the CLI (#1498) * Add support for near projects to the CLI * Update common near version * Update near packages * [SKIP CI] Prerelease * Arranges commands in alphabetical order (#1495) * [SKIP CI] Prerelease * user process.env.USERPROFILE for windows platform (#1502) * [SKIP CI] Prerelease * add blockTime to NodeConfig (#1501) * [SKIP CI] Prerelease * [release] 20230131 (#1503) * Worker use runtime service (#1491) * draft * update * tidy up * update * working worker runtimes * into base class * clean up * improve base on review * [SKIP CI] Prerelease * Check queue for lower blocks before processing fetched blocks (#1509) * [SKIP CI] Prerelease * Support array type in dictionary queries (#1510) * [SKIP CI] Prerelease * add flag query-limit (#1514) * [SKIP CI] Prerelease * Fix error with tempDsRecords being undefined (#1516) * Fix error with tempDsRecords being undefined * Fix schema not being used for metadata, sync dynamic-ds across workers * Be smarter about reloading dynamic ds in workers * [SKIP CI] Prerelease * Update postgraphile enable table partitions (#1520) * [SKIP CI] Prerelease * `dictionary-optimisation` flag (#1519) * add flag, to orderby blockheight instead of pk * add comment * [SKIP CI] Prerelease * [WIP] Feature/adding postgres ssl connection option (#1513) * enable ssl option postgres * Check DB_SSL Env Value , default is false * adding ENV to pass in the Path to certificates * adding SSL option for query pacakge. * use args to pass the postgres ssl informations. * load file content moved to comman package. * Update packages/node-core/src/configure/NodeConfig.ts --------- Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * [SKIP CI] Prerelease * Support Cockroach db (#1521) * support cockroach db * Update comment * [SKIP CI] Prerelease * ensure pgpool connection work with ssl (#1525) * [SKIP CI] Prerelease * Fix issue when use cockroach query service not support pgPartition (#1526) * [SKIP CI] Prerelease * [release] 20230221 (#1528) * fix issue store getByField limit could potentially excess config limit (#1529) * fix issue store getByField limit could potentially excess config limit * Update packages/node-core/src/indexer/store.service.ts Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * typo --------- Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * [SKIP CI] Prerelease * Abi codegen (#1532) * draft * fix abi name, add support for transaction * Update packages/cli/src/template/abi-interface.ts.ejs Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * fix * fix dependencies --------- Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * [SKIP CI] Prerelease * Move enum under schema (#1527) * need fix query get enum * fix enum query, clean all enums * add logic to handle original enum type * fix check exclude constrain check missing schema and table (#1537) * [SKIP CI] Prerelease * manual prerelease node for node-core update (#1538) * [SKIP CI] Prerelease * resolution for sequelize and prerelease node (#1539) * [SKIP CI] Prerelease * fixed the version of sequelize and prerelease node (#1540) * [SKIP CI] Prerelease * improve force clean with deprecated enum (#1541) * [SKIP CI] Prerelease * Remove exclude constraint (#1543) * deprecate exclude contraint * remove unused * [SKIP CI] Prerelease * [release] 20230306 (#1544) * [release] 20230307 (#1545) * bump yarn.lock, update imports (#1546) * [SKIP CI] Prerelease * [release] 20230307 (#1547) * increase wsProvider timeout (#1550) * increase wsProvider timeout * remove .mmr dir * [SKIP CI] Prerelease * Use performance.now instead of date for profiler. (#1549) * add SmartBatchService and BlockSizeBuffer (#1506) * add SmartBatchService and BlockSizeBuffer * check memory usage before fetching a batch * use v8 heap usage statistics * set block to null after indexing * extend smart batch service to workers * make smart batch size injectible * rebase * bug fixes * code cleanup * assign workers based on free memory * fix liniting * cleanup code * cleanup logs * clear blocksizebuffer inside smartbatchservice --------- Co-authored-by: Scott Twiname <skott.twiname@gmail.com> * Fix build issue with profiler * [SKIP CI] Prerelease * fix abi codegen runtime validation (#1552) * [SKIP CI] Prerelease * [release] 20230309 Cli (#1553) * Fix enum under schema not being escaped (#1555) * [SKIP CI] Prerelease * rename `--sponsored-dictionary` to `--dictionary-resolver` (#1559) * renamed flag * add missing tpyo * [SKIP CI] Prerelease * code cleanup * code cleanup --------- Co-authored-by: bighammerw <92004774+bighammerw@users.noreply.github.com> Co-authored-by: Jay Ji <jiqiang90@hotmail.com> Co-authored-by: Ben <89335033+bz888@users.noreply.github.com> Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com> Co-authored-by: Scott Twiname <skott.twiname@gmail.com> Co-authored-by: bz888 <ziyu.ben158@gmail.com> Co-authored-by: Louise wang <louisewang118@.gmail.com> Co-authored-by: hariu-starfish <103621490+hariu-starfish@users.noreply.github.com> Co-authored-by: Orion <orion@starfish.finance> Co-authored-by: Filippo <filippo@embriotech.ch> Co-authored-by: Lachlan McCrae <8554309+lamcc21@users.noreply.github.com> Co-authored-by: Lachlan McCrae <8554309+lachlan1m@users.noreply.github.com> Co-authored-by: HZ <zhaihaohk@gmail.com> Co-authored-by: MOZGIII <mike-n@narod.ru> Co-authored-by: James Bayly <46693720+jamesbayly@users.noreply.github.com> Co-authored-by: Mingyang Li <53138432+Mingyang-Li@users.noreply.github.com> Co-authored-by: seandotau <seandotau@gmail.com> Co-authored-by: Darwin Subramaniam <darwinsubramaniam@googlemail.com>

bz888 requested a review from stwiname October 28, 2022 01:56

stwiname reviewed Oct 28, 2022

View reviewed changes

stwiname reviewed Nov 1, 2022

View reviewed changes

bz888 added 21 commits November 2, 2022 10:46

wip

b6c180e

wip

6bf4027

wip

f1535fd

broken wip

54a4ffa

wip refactoring

cd2c53a

seperate dictionaryQueryEntries

d1d594b

fix logic for setDicitonaryQueryEntries

8a4fe75

cleaning up

f4584e6

move dictionaryEntry.ts to node-core

329f927

remove comments

e981d33

create class for dictionaryqueryentries

187891e

update tests

2370d10

move dictionaryQueryService into dicitonaryService

2ee8c05

refactor

87f5a0a

update logic for useDicitonary

190599a

relocated dictionaryQuery funcs

2d6d77f

add test for dictquerymap

6af7d8b

tests failing on windows?

72e3fc1

test getDictionaryQueryEntries

11c0bf3

test fixed for getDictioanryQueryEntries

c463769

added test for sorting

25698a7

bz888 force-pushed the imporve-dictionary-query branch from 86f3522 to d0235df Compare November 1, 2022 22:04

jiqiang90 marked this pull request as ready for review November 1, 2022 22:13

bz888 force-pushed the imporve-dictionary-query branch 2 times, most recently from af4021e to 25698a7 Compare November 1, 2022 22:59

bz888 added 2 commits November 2, 2022 12:23

conflict fixed ?

9e35a32

fix conflict_2

fb92b82

bz888 added 3 commits November 2, 2022 12:33

fix conflict_3

c154b94

add delete temp ds records back

0e5a3c2

clean up with new logic

624f54d

jiqiang90 reviewed Nov 3, 2022

View reviewed changes

bz888 added 3 commits November 3, 2022 14:19

clean up, add comments for test

04c9ac0

fix

4223671

update logic

7c2f35e

jiqiang90 approved these changes Nov 3, 2022

View reviewed changes

stwiname reviewed Nov 3, 2022

View reviewed changes

add generic type

d0f5d52

stwiname merged commit db52b2c into main Nov 3, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Imporve dictionary query #1371

Imporve dictionary query #1371

bz888 commented Oct 28, 2022

stwiname Oct 28, 2022

stwiname Oct 28, 2022

stwiname Oct 28, 2022

stwiname Nov 1, 2022

stwiname Nov 1, 2022

jiqiang90 Nov 3, 2022

jiqiang90 Nov 3, 2022

jiqiang90 Nov 3, 2022

jiqiang90 Nov 3, 2022

jiqiang90 commented Nov 3, 2022 •

edited

Loading

stwiname Nov 3, 2022

Imporve dictionary query #1371

Imporve dictionary query #1371

Conversation

bz888 commented Oct 28, 2022

Description

Type of change

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jiqiang90 commented Nov 3, 2022 • edited Loading

Choose a reason for hiding this comment

jiqiang90 commented Nov 3, 2022 •

edited

Loading