Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix dependencies issue with subql-publish , validate #1546

Merged
merged 1 commit into from
Mar 7, 2023
Merged

Fix dependencies issue with subql-publish , validate #1546

merged 1 commit into from
Mar 7, 2023

Conversation

bz888
Copy link
Contributor

@bz888 bz888 commented Mar 7, 2023

Description

Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

Fixes #1534 #1533 #1535

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)

Checklist

  • I have tested locally
  • I have performed a self review of my changes
  • Linked to any relevant issues
  • Any dependent changes have been merged and published in downstream modules
  • My code is up to date with the base branch

@jiqiang90 jiqiang90 merged commit f9f039e into main Mar 7, 2023
stwiname added a commit to subquery/subql-ethereum that referenced this pull request Apr 25, 2023
@stwiname @filo87
@bz888 @jiqiang90 @lamcc21 @mzxyz @MOZGIII @jamesbayly @Mingyang-Li @seandotau @guplersaxanoid @darwinsubramaniam
Sync 2.0 (#62)
48ed63c 
* [SKIP CI] Prerelease

* fix comments issue with new package (subquery#1380)

* fix comments issue with new package

* moved yaml package

* [SKIP CI] Prerelease

* fix logic with reindex and unfinalized height and dynamic ds (subquery#1382)

* fix logic with reindex and unfinalized height

* fix

* include fix for subquery#1379

* update polkadot to 9.7.1 (subquery#1384)

* [release] 20221107

* Fix remove alter table (subquery#1387)

* remove migrate alter table

* remove

* [SKIP CI] Prerelease

* [release] 20221108 (subquery#1388)

* fix missing sequelize sync (subquery#1389)

* [SKIP CI] Prerelease

* [release] 20221108 patch (subquery#1390)

* reindex bind (subquery#1391)

* [SKIP CI] Prerelease

* [release] 20221109 (subquery#1393)

* Handle fetch errors, then retry (subquery#1386)

* add retryOnFail function

* add retryOnFail

* add test, fix logic

* [SKIP CI] Prerelease

* fix (subquery#1395)

* [SKIP CI] Prerelease

* [release] 20221109 node-core (subquery#1394)

* Fix tests hanging (subquery#1396)

* Fix tests hanging

* Update base docker image with newer git version

* [SKIP CI] Prerelease

* Add distinct query plugin (subquery#1274)

* Add distinct query plugin

* Clean up log

* Fix distinct not being provided to query

* Uppercase enum to be consistent with other enums

* Update dictionary queries to try distinct argument

* [SKIP CI] Prerelease

* Add query distinct dependencies (subquery#1398)

* fix missing update forked graphile dependencies

* tidy up

* tidy up

* [SKIP CI] Prerelease

* Break block dispatcher file up and move common code to base class (subquery#1397)

* [SKIP CI] Prerelease

* Hot schema trigger (subquery#1401)

* implement trigger with notification

* working prior clean up

* refactor and clean up on async and await

* clean up

* clear comments

* add filter

* fix

* fix err

* [SKIP CI] Prerelease

* [release] 20221115 (subquery#1402)

* [release] 20221115

* [release] 20221115

* [release] 20221115

* fix hot schema (subquery#1404)

* fix and refactor

* refactor getTriggers

* [SKIP CI] Prerelease

* [release] 20221115 (subquery#1408)

* [release] 20221115

* [release] 20221115

* fix fetchblock for works (subquery#1410)

* [SKIP CI] Prerelease

* fix row estimate (subquery#1417)

* fix row estimate

* tidy up

* [SKIP CI] Prerelease

* Add support to cli/validator for flare (subquery#1416)

* [SKIP CI] Prerelease

* missing validator missing ipfs chainTypes (subquery#1419)

* [SKIP CI] Prerelease

* query explain include graphql query (subquery#1426)

* [SKIP CI] Prerelease

* dictionary auth link integration (subquery#1411)

* add required dependancies and add flag for feature

* update modules and add logic to dictionary

* update dictionary parameters, update tests, remove hardcoded values

* simplify auth dictionary handling

* add apollo link type, remove un-needed from wrapper

* remove dictionary from indexer module

* update flag name, add error check

* update yargs flag description

* Update appolo-links

* update @subql/apollo-links version, remove apollo/client version change

* fix yarn.lock

* add await init to dictionary tests

* fix fetch test dictionary provider, update yarn.lock

Co-authored-by: Lachlan McCrae <8554309+lachlan1m@users.noreply.github.com>
Co-authored-by: HZ <zhaihaohk@gmail.com>

* Fix missing speChanged logic with getRuntime (subquery#1421)

* init

* need fix worker

* fix test

* tidy up

* fix test, fix missing dictionary

* inject service to runtime

* remove unused code

* [SKIP CI] Prerelease

* Feat/multi chain indexing (subquery#1375)

* [SKIP CI] Prerelease

* remove enums (subquery#1427)

* [SKIP CI] Prerelease

* change to print graphql sql on request (subquery#1428)

* [SKIP CI] Prerelease

* fix missing util-crptyo in util package (subquery#1429)

* fix missing util-crptyo in util package

* need prerelease in query service

* [SKIP CI] Prerelease

* fix import issue in query service (subquery#1430)

* [SKIP CI] Prerelease

* fix missing query (subquery#1432)

* [SKIP CI] Prerelease

* change fetch module SubqueryProject => ISubqueryProject (subquery#1433)

Co-authored-by: Lachlan McCrae <8554309+lachlan1m@users.noreply.github.com>

* [SKIP CI] Prerelease

* Update polkadot 9.9.1 (subquery#1434)

* update polkadot api, also fix force-clean dependencies issue

* tidy up

* [SKIP CI] Prerelease

* [release] 20221123 (subquery#1431)

* [release] 20221123

* Update packages/query/CHANGELOG.md

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Update packages/node-core/CHANGELOG.md

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Update packages/node/CHANGELOG.md

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Update packages/node-core/CHANGELOG.md

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Update packages/validator/CHANGELOG.md

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Add a way to pass settings to the graphql playground (subquery#1436)

* Add a way to pass settings to the graphql playground

* Add a changelog entry

* [SKIP CI] Prerelease

* Fix the changelog entry for the last PR (subquery#1437)

* [SKIP CI] Prerelease

* fix enum name missing issue (subquery#1441)

* fix enum name missing issue

* tidy up

* [SKIP CI] Prerelease

* hash sql queries (subquery#1438)

* hashed sql

* add comments

* add comment for hashName function

* update function

* update hashName function

* [SKIP CI] Prerelease

* [release] 20221130 (subquery#1442)

* [release] 20221130

* [release] 20221130

* Add --file flag for codegen (subquery#1446)

* init

* change flag behaviour so -f can overwrite -l

* update getManifestPath method, fix typo

* update argument description

* update variable names

* update arg description

* add error message on bad --file path

Co-authored-by: Lachlan McCrae <8554309+lachlan1m@users.noreply.github.com>

* [SKIP CI] Prerelease

* add try and catch for hsr (subquery#1449)

* [SKIP CI] Prerelease

* Bypass blocks (subquery#1435)

* wip

* manifest error

* manfiest passed

* added logging for bypassing blocks

* sync main, moved bypassBlocks under networks

* pass all tests

* mock dictioanry case

* fix metadata

* clean up

* allow range

* bypass working, needs more testing

* wip on bypass with setting new bufferHeight

* working

* added runtime test

* add test for latestBufferHeight (blocked)

* removed unneeded test, improve logic

* clean up

* add comments

* add comments for latestBufferHeight reasoning

* update logic

* refactor

* update logic

* refactor

* [SKIP CI] Prerelease

* [release] 20221205 (subquery#1448)

* [release] 20221205

* [release] 20221205

* [release] 20221206

* details for bypassBlocks

* Remove unused RuntimeService from indexer module, it had missing dependencies (subquery#1453)

* [SKIP CI] Prerelease

* Fix/subcommands (subquery#1451)

* updated forceClean

* update reindex

* [SKIP CI] Prerelease

* Enable env vars parsing at node and query commands (subquery#1452)

* [SKIP CI] Prerelease

* Workers: Fix SequelizeDatabaseError - tuple concurrently updated (subquery#1458)

* add flag for hot-schema reload on node, ensure query is on main thread

* remove flags

* [SKIP CI] Prerelease

* Add start height to project (subquery#1456)

* [SKIP CI] Prerelease

* Fix poi offset is 0 (subquery#1459)

* [SKIP CI] Prerelease

* Handle when fields got undefined with historical (subquery#1463)

* [SKIP CI] Prerelease

* update deploy cli (subquery#1460)

* [SKIP CI] Prerelease

* exit on fail (subquery#1464)

* [SKIP CI] Prerelease

* fix parentSpecVersion could be undefined (subquery#1467)

* [SKIP CI] Prerelease

* [release] 20221219 (subquery#1466)

* [release] 20221219

* update

* Update packages/node-core/CHANGELOG.md

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Update packages/node/CHANGELOG.md

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Update packages/node-core/CHANGELOG.md

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Update packages/cli/CHANGELOG.md

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* update

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Fix templates with other sdks, make -f flag work like other commands (subquery#1470)

* Fix templates with other sdks, make -f flag work like other commands

* Fix for sdks with first class smart contracts

* [SKIP CI] Prerelease

* Tweak error message for genesis hash (subquery#1471)

* [SKIP CI] Prerelease

* Fix trigger function (subquery#1469)

* fix trigger and function for subscription and hot schema reload

* update

* [SKIP CI] Prerelease

* [release] 20221222 (subquery#1474)

* validate dictionary with start height (subquery#1473)

* validate dictionary with start height

* Update packages/node/src/indexer/fetch.service.ts

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* add count for entity (subquery#1480)

* add count for entity

* Update packages/node-core/src/indexer/store.service.ts

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* Fixing readme doc for @subql/query (subquery#1485)

* doc: correcting grammer

* fix: typo

* [SKIP CI] Prerelease

* update polkadot api to 9.11.1 (subquery#1483)

* update polkadot api to 9.11.1

* update

* [SKIP CI] Prerelease

* add connection filters (subquery#1484)

* [SKIP CI] Prerelease

* fix workers missing set last buffered height from empty dictionary (subquery#1492)

* [SKIP CI] Prerelease

* Fix custom metadata keys not being applied (subquery#1496)

* add block hash validation after fetch (subquery#1494)

* add block hash validation after fetch

* add tests

* skip fetch test

* Update log

* [SKIP CI] Prerelease

* [release] 20230123 (subquery#1497)

* Add support for near projects to the CLI (subquery#1498)

* Add support for near projects to the CLI

* Update common near version

* Update near packages

* [SKIP CI] Prerelease

* Arranges commands in alphabetical order (subquery#1495)

* [SKIP CI] Prerelease

* user process.env.USERPROFILE for windows platform (subquery#1502)

* [SKIP CI] Prerelease

* add blockTime to NodeConfig (subquery#1501)

* [SKIP CI] Prerelease

* [release] 20230131 (subquery#1503)

* Worker use runtime service (subquery#1491)

* draft

* update

* tidy up

* update

* working worker runtimes

* into base class

* clean up

* improve base on review

* [SKIP CI] Prerelease

* Check queue for lower blocks before processing fetched blocks (subquery#1509)

* [SKIP CI] Prerelease

* Support array type in dictionary queries (subquery#1510)

* [SKIP CI] Prerelease

* add flag query-limit (subquery#1514)

* [SKIP CI] Prerelease

* Fix error with tempDsRecords being undefined (subquery#1516)

* Fix error with tempDsRecords being undefined

* Fix schema not being used for metadata, sync dynamic-ds across workers

* Be smarter about reloading dynamic ds in workers

* [SKIP CI] Prerelease

* Update postgraphile enable table partitions (subquery#1520)

* [SKIP CI] Prerelease

* `dictionary-optimisation` flag  (subquery#1519)

* add flag, to orderby blockheight instead of pk

* add comment

* [SKIP CI] Prerelease

* [WIP] Feature/adding postgres ssl connection option (subquery#1513)

* enable ssl option postgres

* Check DB_SSL Env Value , default is false

* adding ENV to pass in the Path to certificates

* adding SSL option for query pacakge.

* use args to pass the postgres ssl informations.

* load file content moved to comman package.

* Update packages/node-core/src/configure/NodeConfig.ts

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* Support Cockroach db (subquery#1521)

* support cockroach db

* Update comment

* [SKIP CI] Prerelease

* ensure pgpool connection work with ssl (subquery#1525)

* [SKIP CI] Prerelease

* Fix issue when use cockroach query service not support pgPartition (subquery#1526)

* [SKIP CI] Prerelease

* [release] 20230221 (subquery#1528)

* fix issue store getByField limit could potentially excess config limit (subquery#1529)

* fix issue store getByField limit could potentially excess config limit

* Update packages/node-core/src/indexer/store.service.ts

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* typo

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* Abi codegen (subquery#1532)

* draft

* fix abi name, add support for transaction

* Update packages/cli/src/template/abi-interface.ts.ejs

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* fix

* fix dependencies

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* Move enum under schema (subquery#1527)

* need fix query get enum

* fix enum query, clean all enums

* add logic to handle original enum type

* fix check exclude constrain check missing schema and table (subquery#1537)

* [SKIP CI] Prerelease

* manual prerelease node for node-core update (subquery#1538)

* [SKIP CI] Prerelease

* resolution for sequelize and prerelease node (subquery#1539)

* [SKIP CI] Prerelease

* fixed the version of sequelize and prerelease node (subquery#1540)

* [SKIP CI] Prerelease

* improve force clean with deprecated enum (subquery#1541)

* [SKIP CI] Prerelease

* Remove exclude constraint (subquery#1543)

* deprecate exclude contraint

* remove unused

* [SKIP CI] Prerelease

* [release] 20230306 (subquery#1544)

* [release] 20230307 (subquery#1545)

* bump yarn.lock, update imports (subquery#1546)

* [SKIP CI] Prerelease

* [release] 20230307 (subquery#1547)

* increase wsProvider timeout (subquery#1550)

* increase wsProvider timeout

* remove .mmr dir

* [SKIP CI] Prerelease

* Use performance.now instead of date for profiler. (subquery#1549)

* add SmartBatchService and BlockSizeBuffer (subquery#1506)

* add SmartBatchService and BlockSizeBuffer

* check memory usage before fetching a batch

* use v8 heap usage statistics

* set block to null after indexing

* extend smart batch service to workers

* make smart batch size injectible

* rebase

* bug fixes

* code cleanup

* assign workers based on free memory

* fix liniting

* cleanup code

* cleanup logs

* clear blocksizebuffer inside smartbatchservice

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Fix build issue with profiler

* [SKIP CI] Prerelease

* fix abi codegen runtime validation (subquery#1552)

* [SKIP CI] Prerelease

* [release] 20230309 Cli (subquery#1553)

* Fix enum under schema not being escaped (subquery#1555)

* [SKIP CI] Prerelease

* rename `--sponsored-dictionary` to `--dictionary-resolver` (subquery#1559)

* renamed flag

* add missing tpyo

* [SKIP CI] Prerelease

* handle bigint in json.stringify (subquery#1562)

* [SKIP CI] Prerelease

* add validation for `codegen`, grapql conflicts (subquery#1564)

* add test for codegen with reserved keys

* wip validateEntityName fn

* update test, update logc

* remove logs

* [SKIP CI] Prerelease

* Fix codegen when abi is not valid (subquery#1570)

* Fix codegen when abi is not valid

* Update packages/cli/src/controller/codegen-controller.ts

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* Fix block offset been reset to 1 (subquery#1571)

* [SKIP CI] Prerelease

* remove blocking in process queueing (subquery#1572)

* remove blocking in process queueing

* redefine locking pattern

* fix fetchedBlocks status

* [SKIP CI] Prerelease

* improve api error handling (subquery#1576)

* [SKIP CI] Prerelease

* generate sourcemap for projects (subquery#1569)

* improve error handling

* use inline-source-map

* revert api error handling

* beautify minified output from webpack

* code cleanup

* move terser plugin dependency to cli

* log warnings for unresolved stack trace

* get entry file from datasource

* alter convertStack prototype

* add ES2020 to lib compiler option

* update polkadot api to 10.1.4 (subquery#1580)

* update polkadot api to 10.1.4

* update rest of dep

* [release] 20230329 (subquery#1587)

* Multiple endpoints (subquery#1551)

* allow multiple endpoints

* add response time based load balancing

* handle endpoint disconnects

* move fetchBlockBatches to apiService

* cleanup code

* move ConnectionPoolService to node-core

* fix order of module injection in indexermodule

* remove manually disconnecting api

* move api creation code to ApiConnection class

* code cleanup

* handle reconnection error

* change networkEndpoints to networkEndpoint

* change to use yarn build (subquery#1589)

* test prerelease build (subquery#1590)

* [SKIP CI] Prerelease

* [release] 20230330 (subquery#1592)

* Upgrade `apollo-links` (subquery#1593)

* [SKIP CI] Prerelease

* improve blockSize function (subquery#1602)

* improve blockSize function

* add return type to blockSize

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* [release] 20230405 (subquery#1603)

* Add evmChainId to metadata (subquery#1607)

* [SKIP CI] Prerelease

* [release] 20230411 query (subquery#1608)

* Update progress string (subquery#1612)

* [SKIP CI] Prerelease

* Update the jsonField directive to have an option for disabling the index (subquery#1613)

* Update the jsonField directive to have an option for disabling the index

* Clean up

* [SKIP CI] Prerelease

* [release] 20230414 utils (subquery#1615)

* Fix workers fetching blocks out of order (subquery#1616)

* Fix workers fetching blocks out of order

* Remove logging endpoints with api keys

* [SKIP CI] Prerelease

* [release] 20230417 (subquery#1617)

* DB cache and various other improvements (subquery#1561)

* store cache

* improve on flush cache

* draft storeCache service

* update

* minor change

* Updates

* add historical data to cache (subquery#1557)

* add __block_range to creation

* add historical record to cache

* correct range

* Complete cached store methods (subquery#1563)

* add __block_range to creation

add historical record to cache

correct range

cache store methods

start add tests

* fix set

* update

* WIP cache with workers support

* Fix up dynamic ds service, clean up

* Update store serialisation over workers

* Improve unfinalized blocks to work without tx

* Move all chain agnostic code from block dispatchers to node core

* Add comments about worker store conversion

* Fix rebase build issues

* Implement cache for metadata entity (subquery#1565)

* Implement cache for metadata entity

* Clean up and exclude unfinalized blocks keys from metadata cache

* Clean up

* flush with cache stored records size (subquery#1566)

* flush with cache stored records size

* update

* Lfu get cache (subquery#1568)

* still need fix get/count logic

fix method for LFU getCache

* Update tests

* Fix and more tests

* Update base on review

* Fix

* Run all store operations through cache (subquery#1567)

* Run all store operations through hash

* Log when cache flushed, flush cache on application shutdown

* Cache fixes

* Fix removeCache not being init

* POI cache fixes

* Fixes

* Fix caches being cleared after db operation and potential cache changes have happened

* Fix issues with injected store

* Clean up

* Fix tests from rebase

* Fix get cache not being used

* Rename function

* Fix reindex issues

* Fix casing of import

* Clean up, log if finalized is enabled

* parent 8b0b558 (subquery#1574)

author Scott Twiname <skott.twiname@gmail.com> 1679452749 +1300
committer JQQQ <jiqiang90@hotmail.com> 1679461182 +1300

Clean up, log if finalized is enabled

Fix rebase build issues

parent 5e4ea5d
author Scott Twiname <skott.twiname@gmail.com> 1679013784 +1300
committer JQQQ <jiqiang90@hotmail.com> 1679460224 +1300

parent 5e4ea5d
author Scott Twiname <skott.twiname@gmail.com> 1679013784 +1300
committer JQQQ <jiqiang90@hotmail.com> 1679460210 +1300

parent 5e4ea5d
author Scott Twiname <skott.twiname@gmail.com> 1679013784 +1300
committer JQQQ <jiqiang90@hotmail.com> 1679460149 +1300

parent 5e4ea5d
author Scott Twiname <skott.twiname@gmail.com> 1679013784 +1300
committer JQQQ <jiqiang90@hotmail.com> 1679460101 +1300

parent 5e4ea5d
author Scott Twiname <skott.twiname@gmail.com> 1679013784 +1300
committer JQQQ <jiqiang90@hotmail.com> 1679460038 +1300

Run all store operations through hash

Log when cache flushed, flush cache on application shutdown

Fix removeCache not being init

Fixes

Fix issues with injected store

Clean up

Fix removeCache not being init

Fixes

Fix issues with injected store

Clean up

Fix tests from rebase

Fix get cache not being used

tidy up

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Update metadata when dictionary skips large number of blocks (subquery#1577)

* Update metadata when dictionary skips large number of blocks

* Bring back option to log sql

* Dynamic cache threshold and metrics (subquery#1578)

* Bring back fetch service tests

* Fix build issues

* Tidy up

* Fix workers historical disabled (subquery#1579)

* Fix imports in node core

* improve close record sql, and add flag max size for getCache (subquery#1583)

* improve close record sql, and add flag max size for getCache

* Fix

* escape id string

* Flush data at specific block heights (subquery#1582)

* Flush data at specific block heights

* Add test

* Fix value fromBelowHeight being empty

* Fix mergedRecords potentially being empty

* Fix flushableRecordCounter

* Change log to debug

* Fix dictionary validation

* fix reindex storeCache missing EventEmitterModule (subquery#1585)

* Make flushing cache configurable (subquery#1586)

* Add an index to Id field on entities with historical (subquery#1588)

* Fix tests

* Fix possible infinite loop with cache and fix test (subquery#1591)

* Fix possible infinite loop with cache and fix test

* Add some comments

* Fix more tests

---------

Co-authored-by: JQQQ <jiqiang90@hotmail.com>

* Connect historical option up to cache model option (subquery#1596)

* Only run delete query if there is data to delete

* Add support for non-historical indexing with cache (subquery#1597)

* Add support for non-historical indexing with cache

* Fix api test

* remove constraintDeferrableQuery from hasOne

* fix createUniqueIndexQuery

---------

Co-authored-by: JQQQ <jiqiang90@hotmail.com>

* Cache and worker related fixes (subquery#1598)

* fix index name too long issue (subquery#1599)

* hash the indexes, except pk indexes

* handle existed indexes

* update

* Remove store count function (subquery#1600)

* support cockroach db with non-historical (subquery#1604)

* support cockroach db with non-historical

* Add more comments

* move fetchBlocks to node-core (subquery#1611)

* move fetchBlocks to node-core

* code cleanup

* code cleanup

* remove extra args from fetchBlock

* Fix making data serializable for store bulkCreate and bulkUpdate (subquery#1622)

* Improve big int error in workers (subquery#1619)

* improve error handling in workers

* Make bigint stringifiable in worker and main threads

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* fix type error (subquery#1623)

* fix type error

* cast type

* Flush store cache in operational order (subquery#1614)

* Init

* better log message

* tidy up

* Tidy up and test

* Update packages/node/src/yargs.ts

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Update packages/node-core/src/indexer/storeCache/storeCache.service.ts

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* add benchmarking

* update benchmarking

* Tidy up

* test

* remove flag, use last flushed index, more tidy up

* make operationIndex required

* fix test

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* update default values

* fix missing indexes on cockroach db (subquery#1624)

* fix getLatest() could be undefined

* Fix StoreOperation not pushing enums

---------

Co-authored-by: JQQQ <jiqiang90@hotmail.com>
Co-authored-by: Naveen V <velnaveen99@gmail.com>

* [SKIP CI] Prerelease

* Testing Framework (subquery#1584)

* implement runTest

* check handler kind for running test

* implement testing service

* run tests in sandbox

* build test files through subql build

* run handlers in sandbox

* cleanup sandbox script and improve logging

* index data with `IndexerManager.indexData`

* use indexBlock() instead of indexData() for tests

* add getter _name to entities during codegen

* code cleanup

* improve logging

* improve logging

* bug fixes

* code cleanup

* improve logging

* indexBlock: get datasource from arguments

* move testingService to node-core

* code cleanup

* update ci

* code cleanup

* cleanup

* cleanup

* cleanup

* handle 0 test cases

* prevent calling getGqlType for 'contains' matcher (subquery#1626)

* prevent calling getGqlType for 'contains' matcher

* get gqlType with ternary condition

* remove indexer module from testing module (subquery#1628)

* remove indexer module from testing module

* cleanup

* cleanup

* Enable strict ts setting for node core (subquery#1625)

* WIP enable strict ts setting for node core

* fix undefined in historicalMarkPreviousHeightRecordsBatch

* typo

* Update typescript, fix most strict related errors

* Strict changes for testing framework

* Runtime fixes

* change log level on cache poi

---------

Co-authored-by: JQQQ <jiqiang90@hotmail.com>

* fix prerelease (subquery#1629)

* [SKIP CI] Prerelease

* [SKIPCI] fix testing publish order (subquery#1630)

* fix missing @subql/testing in node-core (subquery#1631)

* [SKIP CI] Prerelease

* manual prerelease for node-core fix (subquery#1632)

* [SKIP CI] Prerelease

* Improve yargs descriptions (subquery#1633)

* [SKIP CI] Prerelease

* Support tests from ./test and ./tests dirs, fix store blockheight issue

* Remove log

* [SKIP CI] Prerelease

* [release] 2.0.0 (subquery#1627)

* release-2.0.0

* update

* update node changelog

* remove typescript from query dependency

* remove typescript from node dependency

* [SKIPCI] fix release

* [release] 2.0.0 republish (subquery#1635)

* Update common, tidy up

---------

Co-authored-by: Filippo <filippo@embriotech.ch>
Co-authored-by: Ben <89335033+bz888@users.noreply.github.com>
Co-authored-by: Jay Ji <jiqiang90@hotmail.com>
Co-authored-by: Lachlan McCrae <8554309+lamcc21@users.noreply.github.com>
Co-authored-by: Lachlan McCrae <8554309+lachlan1m@users.noreply.github.com>
Co-authored-by: HZ <zhaihaohk@gmail.com>
Co-authored-by: MOZGIII <mike-n@narod.ru>
Co-authored-by: James Bayly <46693720+jamesbayly@users.noreply.github.com>
Co-authored-by: Mingyang Li <53138432+Mingyang-Li@users.noreply.github.com>
Co-authored-by: seandotau <seandotau@gmail.com>
Co-authored-by: Naveen V <velnaveen99@gmail.com>
Co-authored-by: Darwin Subramaniam <darwinsubramaniam@googlemail.com>
stwiname added a commit that referenced this pull request Apr 30, 2023
@guplersaxanoid @jiqiang90
@bz888 @stwiname
Rebase 20230313 (#100)
735fa85 
* Move enum under schema (#1527)

* need fix query get enum

* fix enum query, clean all enums

* add logic to handle original enum type

* fix check exclude constrain check missing schema and table (#1537)

* [SKIP CI] Prerelease

* manual prerelease node for node-core update (#1538)

* [SKIP CI] Prerelease

* resolution for sequelize and prerelease node (#1539)

* [SKIP CI] Prerelease

* fixed the version of sequelize and prerelease node (#1540)

* [SKIP CI] Prerelease

* improve force clean with deprecated enum (#1541)

* [SKIP CI] Prerelease

* Remove exclude constraint (#1543)

* deprecate exclude contraint

* remove unused

* [SKIP CI] Prerelease

* [release] 20230306 (#1544)

* [release] 20230307 (#1545)

* bump yarn.lock, update imports (#1546)

* [SKIP CI] Prerelease

* [release] 20230307 (#1547)

* increase wsProvider timeout (#1550)

* increase wsProvider timeout

* remove .mmr dir

* [SKIP CI] Prerelease

* Use performance.now instead of date for profiler. (#1549)

* add SmartBatchService and BlockSizeBuffer (#1506)

* add SmartBatchService and BlockSizeBuffer

* check memory usage before fetching a batch

* use v8 heap usage statistics

* set block to null after indexing

* extend smart batch service to workers

* make smart batch size injectible

* rebase

* bug fixes

* code cleanup

* assign workers based on free memory

* fix liniting

* cleanup code

* cleanup logs

* clear blocksizebuffer inside smartbatchservice

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Fix build issue with profiler

* [SKIP CI] Prerelease

* fix abi codegen runtime validation (#1552)

* [SKIP CI] Prerelease

* [release] 20230309 Cli (#1553)

* Fix enum under schema not being escaped (#1555)

* [SKIP CI] Prerelease

---------

Co-authored-by: Jay Ji <jiqiang90@hotmail.com>
Co-authored-by: Ben <89335033+bz888@users.noreply.github.com>
Co-authored-by: Scott Twiname <skott.twiname@gmail.com>
stwiname added a commit that referenced this pull request Apr 30, 2023
@guplersaxanoid @jiqiang90
@bz888 @stwiname @mzxyz @jamesbayly
Rebase with main SDK (#109)
503d744 
* Move enum under schema (#1527)

* need fix query get enum

* fix enum query, clean all enums

* add logic to handle original enum type

* fix check exclude constrain check missing schema and table (#1537)

* [SKIP CI] Prerelease

* manual prerelease node for node-core update (#1538)

* [SKIP CI] Prerelease

* resolution for sequelize and prerelease node (#1539)

* [SKIP CI] Prerelease

* fixed the version of sequelize and prerelease node (#1540)

* [SKIP CI] Prerelease

* improve force clean with deprecated enum (#1541)

* [SKIP CI] Prerelease

* Remove exclude constraint (#1543)

* deprecate exclude contraint

* remove unused

* [SKIP CI] Prerelease

* [release] 20230306 (#1544)

* [release] 20230307 (#1545)

* bump yarn.lock, update imports (#1546)

* [SKIP CI] Prerelease

* [release] 20230307 (#1547)

* increase wsProvider timeout (#1550)

* increase wsProvider timeout

* remove .mmr dir

* [SKIP CI] Prerelease

* Use performance.now instead of date for profiler. (#1549)

* add SmartBatchService and BlockSizeBuffer (#1506)

* add SmartBatchService and BlockSizeBuffer

* check memory usage before fetching a batch

* use v8 heap usage statistics

* set block to null after indexing

* extend smart batch service to workers

* make smart batch size injectible

* rebase

* bug fixes

* code cleanup

* assign workers based on free memory

* fix liniting

* cleanup code

* cleanup logs

* clear blocksizebuffer inside smartbatchservice

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Fix build issue with profiler

* [SKIP CI] Prerelease

* fix abi codegen runtime validation (#1552)

* [SKIP CI] Prerelease

* [release] 20230309 Cli (#1553)

* Fix enum under schema not being escaped (#1555)

* [SKIP CI] Prerelease

* rename `--sponsored-dictionary` to `--dictionary-resolver` (#1559)

* renamed flag

* add missing tpyo

* [SKIP CI] Prerelease

* handle bigint in json.stringify (#1562)

* [SKIP CI] Prerelease

* add validation for `codegen`, grapql conflicts (#1564)

* add test for codegen with reserved keys

* wip validateEntityName fn

* update test, update logc

* remove logs

* [SKIP CI] Prerelease

* Fix codegen when abi is not valid (#1570)

* Fix codegen when abi is not valid

* Update packages/cli/src/controller/codegen-controller.ts

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* Fix block offset been reset to 1 (#1571)

* [SKIP CI] Prerelease

* remove blocking in process queueing (#1572)

* remove blocking in process queueing

* redefine locking pattern

* fix fetchedBlocks status

* [SKIP CI] Prerelease

* improve api error handling (#1576)

* [SKIP CI] Prerelease

* generate sourcemap for projects (#1569)

* improve error handling

* use inline-source-map

* revert api error handling

* beautify minified output from webpack

* code cleanup

* move terser plugin dependency to cli

* log warnings for unresolved stack trace

* get entry file from datasource

* alter convertStack prototype

* add ES2020 to lib compiler option

* update polkadot api to 10.1.4 (#1580)

* update polkadot api to 10.1.4

* update rest of dep

* [release] 20230329 (#1587)

* Multiple endpoints (#1551)

* allow multiple endpoints

* add response time based load balancing

* handle endpoint disconnects

* move fetchBlockBatches to apiService

* cleanup code

* move ConnectionPoolService to node-core

* fix order of module injection in indexermodule

* remove manually disconnecting api

* move api creation code to ApiConnection class

* code cleanup

* handle reconnection error

* change networkEndpoints to networkEndpoint

* change to use yarn build (#1589)

* test prerelease build (#1590)

* [SKIP CI] Prerelease

* [release] 20230330 (#1592)

* Upgrade `apollo-links` (#1593)

* [SKIP CI] Prerelease

* improve blockSize function (#1602)

* improve blockSize function

* add return type to blockSize

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* [release] 20230405 (#1603)

* Add evmChainId to metadata (#1607)

* [SKIP CI] Prerelease

* [release] 20230411 query (#1608)

* Update progress string (#1612)

* [SKIP CI] Prerelease

* Update the jsonField directive to have an option for disabling the index (#1613)

* Update the jsonField directive to have an option for disabling the index

* Clean up

* [SKIP CI] Prerelease

* [release] 20230414 utils (#1615)

* Fix workers fetching blocks out of order (#1616)

* Fix workers fetching blocks out of order

* Remove logging endpoints with api keys

* [SKIP CI] Prerelease

* [release] 20230417 (#1617)

* DB cache and various other improvements (#1561)

* store cache

* improve on flush cache

* draft storeCache service

* update

* minor change

* Updates

* add historical data to cache (#1557)

* add __block_range to creation

* add historical record to cache

* correct range

* Complete cached store methods (#1563)

* add __block_range to creation

add historical record to cache

correct range

cache store methods

start add tests

* fix set

* update

* WIP cache with workers support

* Fix up dynamic ds service, clean up

* Update store serialisation over workers

* Improve unfinalized blocks to work without tx

* Move all chain agnostic code from block dispatchers to node core

* Add comments about worker store conversion

* Fix rebase build issues

* Implement cache for metadata entity (#1565)

* Implement cache for metadata entity

* Clean up and exclude unfinalized blocks keys from metadata cache

* Clean up

* flush with cache stored records size (#1566)

* flush with cache stored records size

* update

* Lfu get cache (#1568)

* still need fix get/count logic

fix method for LFU getCache

* Update tests

* Fix and more tests

* Update base on review

* Fix

* Run all store operations through cache (#1567)

* Run all store operations through hash

* Log when cache flushed, flush cache on application shutdown

* Cache fixes

* Fix removeCache not being init

* POI cache fixes

* Fixes

* Fix caches being cleared after db operation and potential cache changes have happened

* Fix issues with injected store

* Clean up

* Fix tests from rebase

* Fix get cache not being used

* Rename function

* Fix reindex issues

* Fix casing of import

* Clean up, log if finalized is enabled

* parent 8b0b558 (#1574)

author Scott Twiname <skott.twiname@gmail.com> 1679452749 +1300
committer JQQQ <jiqiang90@hotmail.com> 1679461182 +1300

Clean up, log if finalized is enabled

Fix rebase build issues

parent 5e4ea5d
author Scott Twiname <skott.twiname@gmail.com> 1679013784 +1300
committer JQQQ <jiqiang90@hotmail.com> 1679460224 +1300

parent 5e4ea5d
author Scott Twiname <skott.twiname@gmail.com> 1679013784 +1300
committer JQQQ <jiqiang90@hotmail.com> 1679460210 +1300

parent 5e4ea5d
author Scott Twiname <skott.twiname@gmail.com> 1679013784 +1300
committer JQQQ <jiqiang90@hotmail.com> 1679460149 +1300

parent 5e4ea5d
author Scott Twiname <skott.twiname@gmail.com> 1679013784 +1300
committer JQQQ <jiqiang90@hotmail.com> 1679460101 +1300

parent 5e4ea5d
author Scott Twiname <skott.twiname@gmail.com> 1679013784 +1300
committer JQQQ <jiqiang90@hotmail.com> 1679460038 +1300

Run all store operations through hash

Log when cache flushed, flush cache on application shutdown

Fix removeCache not being init

Fixes

Fix issues with injected store

Clean up

Fix removeCache not being init

Fixes

Fix issues with injected store

Clean up

Fix tests from rebase

Fix get cache not being used

tidy up

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Update metadata when dictionary skips large number of blocks (#1577)

* Update metadata when dictionary skips large number of blocks

* Bring back option to log sql

* Dynamic cache threshold and metrics (#1578)

* Bring back fetch service tests

* Fix build issues

* Tidy up

* Fix workers historical disabled (#1579)

* Fix imports in node core

* improve close record sql, and add flag max size for getCache (#1583)

* improve close record sql, and add flag max size for getCache

* Fix

* escape id string

* Flush data at specific block heights (#1582)

* Flush data at specific block heights

* Add test

* Fix value fromBelowHeight being empty

* Fix mergedRecords potentially being empty

* Fix flushableRecordCounter

* Change log to debug

* Fix dictionary validation

* fix reindex storeCache missing EventEmitterModule (#1585)

* Make flushing cache configurable (#1586)

* Add an index to Id field on entities with historical (#1588)

* Fix tests

* Fix possible infinite loop with cache and fix test (#1591)

* Fix possible infinite loop with cache and fix test

* Add some comments

* Fix more tests

---------

Co-authored-by: JQQQ <jiqiang90@hotmail.com>

* Connect historical option up to cache model option (#1596)

* Only run delete query if there is data to delete

* Add support for non-historical indexing with cache (#1597)

* Add support for non-historical indexing with cache

* Fix api test

* remove constraintDeferrableQuery from hasOne

* fix createUniqueIndexQuery

---------

Co-authored-by: JQQQ <jiqiang90@hotmail.com>

* Cache and worker related fixes (#1598)

* fix index name too long issue (#1599)

* hash the indexes, except pk indexes

* handle existed indexes

* update

* Remove store count function (#1600)

* support cockroach db with non-historical (#1604)

* support cockroach db with non-historical

* Add more comments

* move fetchBlocks to node-core (#1611)

* move fetchBlocks to node-core

* code cleanup

* code cleanup

* remove extra args from fetchBlock

* Fix making data serializable for store bulkCreate and bulkUpdate (#1622)

* Improve big int error in workers (#1619)

* improve error handling in workers

* Make bigint stringifiable in worker and main threads

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* fix type error (#1623)

* fix type error

* cast type

* Flush store cache in operational order (#1614)

* Init

* better log message

* tidy up

* Tidy up and test

* Update packages/node/src/yargs.ts

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Update packages/node-core/src/indexer/storeCache/storeCache.service.ts

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* add benchmarking

* update benchmarking

* Tidy up

* test

* remove flag, use last flushed index, more tidy up

* make operationIndex required

* fix test

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* update default values

* fix missing indexes on cockroach db (#1624)

* fix getLatest() could be undefined

* Fix StoreOperation not pushing enums

---------

Co-authored-by: JQQQ <jiqiang90@hotmail.com>
Co-authored-by: Naveen V <velnaveen99@gmail.com>

* [SKIP CI] Prerelease

* Testing Framework (#1584)

* implement runTest

* check handler kind for running test

* implement testing service

* run tests in sandbox

* build test files through subql build

* run handlers in sandbox

* cleanup sandbox script and improve logging

* index data with `IndexerManager.indexData`

* use indexBlock() instead of indexData() for tests

* add getter _name to entities during codegen

* code cleanup

* improve logging

* improve logging

* bug fixes

* code cleanup

* improve logging

* indexBlock: get datasource from arguments

* move testingService to node-core

* code cleanup

* update ci

* code cleanup

* cleanup

* cleanup

* cleanup

* handle 0 test cases

* prevent calling getGqlType for 'contains' matcher (#1626)

* prevent calling getGqlType for 'contains' matcher

* get gqlType with ternary condition

* remove indexer module from testing module (#1628)

* remove indexer module from testing module

* cleanup

* cleanup

* Enable strict ts setting for node core (#1625)

* WIP enable strict ts setting for node core

* fix undefined in historicalMarkPreviousHeightRecordsBatch

* typo

* Update typescript, fix most strict related errors

* Strict changes for testing framework

* Runtime fixes

* change log level on cache poi

---------

Co-authored-by: JQQQ <jiqiang90@hotmail.com>

* fix prerelease (#1629)

* [SKIP CI] Prerelease

* [SKIPCI] fix testing publish order (#1630)

* fix missing @subql/testing in node-core (#1631)

* [SKIP CI] Prerelease

* manual prerelease for node-core fix (#1632)

* [SKIP CI] Prerelease

* Improve yargs descriptions (#1633)

* [SKIP CI] Prerelease

* Support tests from ./test and ./tests dirs, fix store blockheight issue

* Remove log

* [SKIP CI] Prerelease

* [release] 2.0.0 (#1627)

* release-2.0.0

* update

* update node changelog

* remove typescript from query dependency

* remove typescript from node dependency

* [SKIPCI] fix release

* [release] 2.0.0 republish (#1635)

* Fix ApiService abstract class, improve getting all DS (#1638)

* Fix ApiService abstract class, improve getting all DS

* Revert api import and minor test fixes

* [SKIP CI] Prerelease

* fix assertion error and pk index type (#1641)

* [SKIP CI] Prerelease

* update node-core

* Tidy up dependencies

* Remove unnecessary fiiles

* Fix worker DI issue

---------

Co-authored-by: Jay Ji <jiqiang90@hotmail.com>
Co-authored-by: Ben <89335033+bz888@users.noreply.github.com>
Co-authored-by: Scott Twiname <skott.twiname@gmail.com>
Co-authored-by: HZHAIX <zhaihaohk@gmail.com>
Co-authored-by: James Bayly <46693720+jamesbayly@users.noreply.github.com>
jiqiang90 added a commit to subquery/subql-ethereum that referenced this pull request May 11, 2023
@stwiname @bz888
@jiqiang90 @lamcc21 @MOZGIII @jamesbayly @Mingyang-Li @seandotau @guplersaxanoid @darwinsubramaniam @mzxyz
Sync 20230511 eth (#70)
f3d1c8d 
* remove enums (subquery#1427)

* [SKIP CI] Prerelease

* change to print graphql sql on request (subquery#1428)

* [SKIP CI] Prerelease

* fix missing util-crptyo in util package (subquery#1429)

* fix missing util-crptyo in util package

* need prerelease in query service

* [SKIP CI] Prerelease

* fix import issue in query service (subquery#1430)

* [SKIP CI] Prerelease

* fix missing query (subquery#1432)

* [SKIP CI] Prerelease

* change fetch module SubqueryProject => ISubqueryProject (subquery#1433)

Co-authored-by: Lachlan McCrae <8554309+lachlan1m@users.noreply.github.com>

* [SKIP CI] Prerelease

* Update polkadot 9.9.1 (subquery#1434)

* update polkadot api, also fix force-clean dependencies issue

* tidy up

* [SKIP CI] Prerelease

* [release] 20221123 (subquery#1431)

* [release] 20221123

* Update packages/query/CHANGELOG.md

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Update packages/node-core/CHANGELOG.md

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Update packages/node/CHANGELOG.md

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Update packages/node-core/CHANGELOG.md

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Update packages/validator/CHANGELOG.md

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Add a way to pass settings to the graphql playground (subquery#1436)

* Add a way to pass settings to the graphql playground

* Add a changelog entry

* [SKIP CI] Prerelease

* Fix the changelog entry for the last PR (subquery#1437)

* [SKIP CI] Prerelease

* fix enum name missing issue (subquery#1441)

* fix enum name missing issue

* tidy up

* [SKIP CI] Prerelease

* hash sql queries (subquery#1438)

* hashed sql

* add comments

* add comment for hashName function

* update function

* update hashName function

* [SKIP CI] Prerelease

* [release] 20221130 (subquery#1442)

* [release] 20221130

* [release] 20221130

* Add --file flag for codegen (subquery#1446)

* init

* change flag behaviour so -f can overwrite -l

* update getManifestPath method, fix typo

* update argument description

* update variable names

* update arg description

* add error message on bad --file path

Co-authored-by: Lachlan McCrae <8554309+lachlan1m@users.noreply.github.com>

* [SKIP CI] Prerelease

* add try and catch for hsr (subquery#1449)

* [SKIP CI] Prerelease

* Bypass blocks (subquery#1435)

* wip

* manifest error

* manfiest passed

* added logging for bypassing blocks

* sync main, moved bypassBlocks under networks

* pass all tests

* mock dictioanry case

* fix metadata

* clean up

* allow range

* bypass working, needs more testing

* wip on bypass with setting new bufferHeight

* working

* added runtime test

* add test for latestBufferHeight (blocked)

* removed unneeded test, improve logic

* clean up

* add comments

* add comments for latestBufferHeight reasoning

* update logic

* refactor

* update logic

* refactor

* [SKIP CI] Prerelease

* [release] 20221205 (subquery#1448)

* [release] 20221205

* [release] 20221205

* [release] 20221206

* details for bypassBlocks

* Remove unused RuntimeService from indexer module, it had missing dependencies (subquery#1453)

* [SKIP CI] Prerelease

* Fix/subcommands (subquery#1451)

* updated forceClean

* update reindex

* [SKIP CI] Prerelease

* Enable env vars parsing at node and query commands (subquery#1452)

* [SKIP CI] Prerelease

* Workers: Fix SequelizeDatabaseError - tuple concurrently updated (subquery#1458)

* add flag for hot-schema reload on node, ensure query is on main thread

* remove flags

* [SKIP CI] Prerelease

* Add start height to project (subquery#1456)

* [SKIP CI] Prerelease

* Fix poi offset is 0 (subquery#1459)

* [SKIP CI] Prerelease

* Handle when fields got undefined with historical (subquery#1463)

* [SKIP CI] Prerelease

* update deploy cli (subquery#1460)

* [SKIP CI] Prerelease

* exit on fail (subquery#1464)

* [SKIP CI] Prerelease

* fix parentSpecVersion could be undefined (subquery#1467)

* [SKIP CI] Prerelease

* [release] 20221219 (subquery#1466)

* [release] 20221219

* update

* Update packages/node-core/CHANGELOG.md

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Update packages/node/CHANGELOG.md

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Update packages/node-core/CHANGELOG.md

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Update packages/cli/CHANGELOG.md

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* update

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Fix templates with other sdks, make -f flag work like other commands (subquery#1470)

* Fix templates with other sdks, make -f flag work like other commands

* Fix for sdks with first class smart contracts

* [SKIP CI] Prerelease

* Tweak error message for genesis hash (subquery#1471)

* [SKIP CI] Prerelease

* Fix trigger function (subquery#1469)

* fix trigger and function for subscription and hot schema reload

* update

* [SKIP CI] Prerelease

* [release] 20221222 (subquery#1474)

* validate dictionary with start height (subquery#1473)

* validate dictionary with start height

* Update packages/node/src/indexer/fetch.service.ts

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* add count for entity (subquery#1480)

* add count for entity

* Update packages/node-core/src/indexer/store.service.ts

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* Fixing readme doc for @subql/query (subquery#1485)

* doc: correcting grammer

* fix: typo

* [SKIP CI] Prerelease

* update polkadot api to 9.11.1 (subquery#1483)

* update polkadot api to 9.11.1

* update

* [SKIP CI] Prerelease

* add connection filters (subquery#1484)

* [SKIP CI] Prerelease

* fix workers missing set last buffered height from empty dictionary (subquery#1492)

* [SKIP CI] Prerelease

* Fix custom metadata keys not being applied (subquery#1496)

* add block hash validation after fetch (subquery#1494)

* add block hash validation after fetch

* add tests

* skip fetch test

* Update log

* [SKIP CI] Prerelease

* [release] 20230123 (subquery#1497)

* Add support for near projects to the CLI (subquery#1498)

* Add support for near projects to the CLI

* Update common near version

* Update near packages

* [SKIP CI] Prerelease

* Arranges commands in alphabetical order (subquery#1495)

* [SKIP CI] Prerelease

* user process.env.USERPROFILE for windows platform (subquery#1502)

* [SKIP CI] Prerelease

* add blockTime to NodeConfig (subquery#1501)

* [SKIP CI] Prerelease

* [release] 20230131 (subquery#1503)

* Worker use runtime service (subquery#1491)

* draft

* update

* tidy up

* update

* working worker runtimes

* into base class

* clean up

* improve base on review

* [SKIP CI] Prerelease

* Check queue for lower blocks before processing fetched blocks (subquery#1509)

* [SKIP CI] Prerelease

* Support array type in dictionary queries (subquery#1510)

* [SKIP CI] Prerelease

* add flag query-limit (subquery#1514)

* [SKIP CI] Prerelease

* Fix error with tempDsRecords being undefined (subquery#1516)

* Fix error with tempDsRecords being undefined

* Fix schema not being used for metadata, sync dynamic-ds across workers

* Be smarter about reloading dynamic ds in workers

* [SKIP CI] Prerelease

* Update postgraphile enable table partitions (subquery#1520)

* [SKIP CI] Prerelease

* `dictionary-optimisation` flag  (subquery#1519)

* add flag, to orderby blockheight instead of pk

* add comment

* [SKIP CI] Prerelease

* [WIP] Feature/adding postgres ssl connection option (subquery#1513)

* enable ssl option postgres

* Check DB_SSL Env Value , default is false

* adding ENV to pass in the Path to certificates

* adding SSL option for query pacakge.

* use args to pass the postgres ssl informations.

* load file content moved to comman package.

* Update packages/node-core/src/configure/NodeConfig.ts

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* Support Cockroach db (subquery#1521)

* support cockroach db

* Update comment

* [SKIP CI] Prerelease

* ensure pgpool connection work with ssl (subquery#1525)

* [SKIP CI] Prerelease

* Fix issue when use cockroach query service not support pgPartition (subquery#1526)

* [SKIP CI] Prerelease

* [release] 20230221 (subquery#1528)

* fix issue store getByField limit could potentially excess config limit (subquery#1529)

* fix issue store getByField limit could potentially excess config limit

* Update packages/node-core/src/indexer/store.service.ts

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* typo

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* Abi codegen (subquery#1532)

* draft

* fix abi name, add support for transaction

* Update packages/cli/src/template/abi-interface.ts.ejs

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* fix

* fix dependencies

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* Move enum under schema (subquery#1527)

* need fix query get enum

* fix enum query, clean all enums

* add logic to handle original enum type

* fix check exclude constrain check missing schema and table (subquery#1537)

* [SKIP CI] Prerelease

* manual prerelease node for node-core update (subquery#1538)

* [SKIP CI] Prerelease

* resolution for sequelize and prerelease node (subquery#1539)

* [SKIP CI] Prerelease

* fixed the version of sequelize and prerelease node (subquery#1540)

* [SKIP CI] Prerelease

* improve force clean with deprecated enum (subquery#1541)

* [SKIP CI] Prerelease

* Remove exclude constraint (subquery#1543)

* deprecate exclude contraint

* remove unused

* [SKIP CI] Prerelease

* [release] 20230306 (subquery#1544)

* [release] 20230307 (subquery#1545)

* bump yarn.lock, update imports (subquery#1546)

* [SKIP CI] Prerelease

* [release] 20230307 (subquery#1547)

* increase wsProvider timeout (subquery#1550)

* increase wsProvider timeout

* remove .mmr dir

* [SKIP CI] Prerelease

* Use performance.now instead of date for profiler. (subquery#1549)

* add SmartBatchService and BlockSizeBuffer (subquery#1506)

* add SmartBatchService and BlockSizeBuffer

* check memory usage before fetching a batch

* use v8 heap usage statistics

* set block to null after indexing

* extend smart batch service to workers

* make smart batch size injectible

* rebase

* bug fixes

* code cleanup

* assign workers based on free memory

* fix liniting

* cleanup code

* cleanup logs

* clear blocksizebuffer inside smartbatchservice

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Fix build issue with profiler

* [SKIP CI] Prerelease

* fix abi codegen runtime validation (subquery#1552)

* [SKIP CI] Prerelease

* [release] 20230309 Cli (subquery#1553)

* Fix enum under schema not being escaped (subquery#1555)

* [SKIP CI] Prerelease

* rename `--sponsored-dictionary` to `--dictionary-resolver` (subquery#1559)

* renamed flag

* add missing tpyo

* [SKIP CI] Prerelease

* handle bigint in json.stringify (subquery#1562)

* [SKIP CI] Prerelease

* add validation for `codegen`, grapql conflicts (subquery#1564)

* add test for codegen with reserved keys

* wip validateEntityName fn

* update test, update logc

* remove logs

* [SKIP CI] Prerelease

* Fix codegen when abi is not valid (subquery#1570)

* Fix codegen when abi is not valid

* Update packages/cli/src/controller/codegen-controller.ts

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* Fix block offset been reset to 1 (subquery#1571)

* [SKIP CI] Prerelease

* remove blocking in process queueing (subquery#1572)

* remove blocking in process queueing

* redefine locking pattern

* fix fetchedBlocks status

* [SKIP CI] Prerelease

* improve api error handling (subquery#1576)

* [SKIP CI] Prerelease

* generate sourcemap for projects (subquery#1569)

* improve error handling

* use inline-source-map

* revert api error handling

* beautify minified output from webpack

* code cleanup

* move terser plugin dependency to cli

* log warnings for unresolved stack trace

* get entry file from datasource

* alter convertStack prototype

* add ES2020 to lib compiler option

* update polkadot api to 10.1.4 (subquery#1580)

* update polkadot api to 10.1.4

* update rest of dep

* [release] 20230329 (subquery#1587)

* Multiple endpoints (subquery#1551)

* allow multiple endpoints

* add response time based load balancing

* handle endpoint disconnects

* move fetchBlockBatches to apiService

* cleanup code

* move ConnectionPoolService to node-core

* fix order of module injection in indexermodule

* remove manually disconnecting api

* move api creation code to ApiConnection class

* code cleanup

* handle reconnection error

* change networkEndpoints to networkEndpoint

* change to use yarn build (subquery#1589)

* test prerelease build (subquery#1590)

* [SKIP CI] Prerelease

* [release] 20230330 (subquery#1592)

* Upgrade `apollo-links` (subquery#1593)

* [SKIP CI] Prerelease

* improve blockSize function (subquery#1602)

* improve blockSize function

* add return type to blockSize

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* [release] 20230405 (subquery#1603)

* Add evmChainId to metadata (subquery#1607)

* [SKIP CI] Prerelease

* [release] 20230411 query (subquery#1608)

* Update progress string (subquery#1612)

* [SKIP CI] Prerelease

* Update the jsonField directive to have an option for disabling the index (subquery#1613)

* Update the jsonField directive to have an option for disabling the index

* Clean up

* [SKIP CI] Prerelease

* [release] 20230414 utils (subquery#1615)

* Fix workers fetching blocks out of order (subquery#1616)

* Fix workers fetching blocks out of order

* Remove logging endpoints with api keys

* [SKIP CI] Prerelease

* [release] 20230417 (subquery#1617)

* DB cache and various other improvements (subquery#1561)

* store cache

* improve on flush cache

* draft storeCache service

* update

* minor change

* Updates

* add historical data to cache (subquery#1557)

* add __block_range to creation

* add historical record to cache

* correct range

* Complete cached store methods (subquery#1563)

* add __block_range to creation

add historical record to cache

correct range

cache store methods

start add tests

* fix set

* update

* WIP cache with workers support

* Fix up dynamic ds service, clean up

* Update store serialisation over workers

* Improve unfinalized blocks to work without tx

* Move all chain agnostic code from block dispatchers to node core

* Add comments about worker store conversion

* Fix rebase build issues

* Implement cache for metadata entity (subquery#1565)

* Implement cache for metadata entity

* Clean up and exclude unfinalized blocks keys from metadata cache

* Clean up

* flush with cache stored records size (subquery#1566)

* flush with cache stored records size

* update

* Lfu get cache (subquery#1568)

* still need fix get/count logic

fix method for LFU getCache

* Update tests

* Fix and more tests

* Update base on review

* Fix

* Run all store operations through cache (subquery#1567)

* Run all store operations through hash

* Log when cache flushed, flush cache on application shutdown

* Cache fixes

* Fix removeCache not being init

* POI cache fixes

* Fixes

* Fix caches being cleared after db operation and potential cache changes have happened

* Fix issues with injected store

* Clean up

* Fix tests from rebase

* Fix get cache not being used

* Rename function

* Fix reindex issues

* Fix casing of import

* Clean up, log if finalized is enabled

* parent 8b0b558 (subquery#1574)

author Scott Twiname <skott.twiname@gmail.com> 1679452749 +1300
committer JQQQ <jiqiang90@hotmail.com> 1679461182 +1300

Clean up, log if finalized is enabled

Fix rebase build issues

parent 5e4ea5d
author Scott Twiname <skott.twiname@gmail.com> 1679013784 +1300
committer JQQQ <jiqiang90@hotmail.com> 1679460224 +1300

parent 5e4ea5d
author Scott Twiname <skott.twiname@gmail.com> 1679013784 +1300
committer JQQQ <jiqiang90@hotmail.com> 1679460210 +1300

parent 5e4ea5d
author Scott Twiname <skott.twiname@gmail.com> 1679013784 +1300
committer JQQQ <jiqiang90@hotmail.com> 1679460149 +1300

parent 5e4ea5d
author Scott Twiname <skott.twiname@gmail.com> 1679013784 +1300
committer JQQQ <jiqiang90@hotmail.com> 1679460101 +1300

parent 5e4ea5d
author Scott Twiname <skott.twiname@gmail.com> 1679013784 +1300
committer JQQQ <jiqiang90@hotmail.com> 1679460038 +1300

Run all store operations through hash

Log when cache flushed, flush cache on application shutdown

Fix removeCache not being init

Fixes

Fix issues with injected store

Clean up

Fix removeCache not being init

Fixes

Fix issues with injected store

Clean up

Fix tests from rebase

Fix get cache not being used

tidy up

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Update metadata when dictionary skips large number of blocks (subquery#1577)

* Update metadata when dictionary skips large number of blocks

* Bring back option to log sql

* Dynamic cache threshold and metrics (subquery#1578)

* Bring back fetch service tests

* Fix build issues

* Tidy up

* Fix workers historical disabled (subquery#1579)

* Fix imports in node core

* improve close record sql, and add flag max size for getCache (subquery#1583)

* improve close record sql, and add flag max size for getCache

* Fix

* escape id string

* Flush data at specific block heights (subquery#1582)

* Flush data at specific block heights

* Add test

* Fix value fromBelowHeight being empty

* Fix mergedRecords potentially being empty

* Fix flushableRecordCounter

* Change log to debug

* Fix dictionary validation

* fix reindex storeCache missing EventEmitterModule (subquery#1585)

* Make flushing cache configurable (subquery#1586)

* Add an index to Id field on entities with historical (subquery#1588)

* Fix tests

* Fix possible infinite loop with cache and fix test (subquery#1591)

* Fix possible infinite loop with cache and fix test

* Add some comments

* Fix more tests

---------

Co-authored-by: JQQQ <jiqiang90@hotmail.com>

* Connect historical option up to cache model option (subquery#1596)

* Only run delete query if there is data to delete

* Add support for non-historical indexing with cache (subquery#1597)

* Add support for non-historical indexing with cache

* Fix api test

* remove constraintDeferrableQuery from hasOne

* fix createUniqueIndexQuery

---------

Co-authored-by: JQQQ <jiqiang90@hotmail.com>

* Cache and worker related fixes (subquery#1598)

* fix index name too long issue (subquery#1599)

* hash the indexes, except pk indexes

* handle existed indexes

* update

* Remove store count function (subquery#1600)

* support cockroach db with non-historical (subquery#1604)

* support cockroach db with non-historical

* Add more comments

* move fetchBlocks to node-core (subquery#1611)

* move fetchBlocks to node-core

* code cleanup

* code cleanup

* remove extra args from fetchBlock

* Fix making data serializable for store bulkCreate and bulkUpdate (subquery#1622)

* Improve big int error in workers (subquery#1619)

* improve error handling in workers

* Make bigint stringifiable in worker and main threads

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* fix type error (subquery#1623)

* fix type error

* cast type

* Flush store cache in operational order (subquery#1614)

* Init

* better log message

* tidy up

* Tidy up and test

* Update packages/node/src/yargs.ts

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Update packages/node-core/src/indexer/storeCache/storeCache.service.ts

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* add benchmarking

* update benchmarking

* Tidy up

* test

* remove flag, use last flushed index, more tidy up

* make operationIndex required

* fix test

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* update default values

* fix missing indexes on cockroach db (subquery#1624)

* fix getLatest() could be undefined

* Fix StoreOperation not pushing enums

---------

Co-authored-by: JQQQ <jiqiang90@hotmail.com>
Co-authored-by: Naveen V <velnaveen99@gmail.com>

* [SKIP CI] Prerelease

* Testing Framework (subquery#1584)

* implement runTest

* check handler kind for running test

* implement testing service

* run tests in sandbox

* build test files through subql build

* run handlers in sandbox

* cleanup sandbox script and improve logging

* index data with `IndexerManager.indexData`

* use indexBlock() instead of indexData() for tests

* add getter _name to entities during codegen

* code cleanup

* improve logging

* improve logging

* bug fixes

* code cleanup

* improve logging

* indexBlock: get datasource from arguments

* move testingService to node-core

* code cleanup

* update ci

* code cleanup

* cleanup

* cleanup

* cleanup

* handle 0 test cases

* prevent calling getGqlType for 'contains' matcher (subquery#1626)

* prevent calling getGqlType for 'contains' matcher

* get gqlType with ternary condition

* remove indexer module from testing module (subquery#1628)

* remove indexer module from testing module

* cleanup

* cleanup

* Enable strict ts setting for node core (subquery#1625)

* WIP enable strict ts setting for node core

* fix undefined in historicalMarkPreviousHeightRecordsBatch

* typo

* Update typescript, fix most strict related errors

* Strict changes for testing framework

* Runtime fixes

* change log level on cache poi

---------

Co-authored-by: JQQQ <jiqiang90@hotmail.com>

* fix prerelease (subquery#1629)

* [SKIP CI] Prerelease

* [SKIPCI] fix testing publish order (subquery#1630)

* fix missing @subql/testing in node-core (subquery#1631)

* [SKIP CI] Prerelease

* manual prerelease for node-core fix (subquery#1632)

* [SKIP CI] Prerelease

* Improve yargs descriptions (subquery#1633)

* [SKIP CI] Prerelease

* Support tests from ./test and ./tests dirs, fix store blockheight issue

* Remove log

* [SKIP CI] Prerelease

* [release] 2.0.0 (subquery#1627)

* release-2.0.0

* update

* update node changelog

* remove typescript from query dependency

* remove typescript from node dependency

* [SKIPCI] fix release

* [release] 2.0.0 republish (subquery#1635)

* Fix ApiService abstract class, improve getting all DS (subquery#1638)

* Fix ApiService abstract class, improve getting all DS

* Revert api import and minor test fixes

* [SKIP CI] Prerelease

* fix assertion error and pk index type (subquery#1641)

* [SKIP CI] Prerelease

* fix workers crashes error (subquery#1642)

* try to fix workers

* Fix dependency injection

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* WIP fix test part 1 (subquery#1640)

* fix test part 1

* Update packages/node/src/indexer/fetch.service.ts

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* fix tests 20230426

* remove unused code

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* fix missing esModuleInterop (subquery#1644)

* [SKIP CI] Prerelease

* Force flush cache with reindex command (subquery#1645)

* [SKIP CI] Prerelease

* [release] 20230427 (subquery#1646)

* [release] 20230427

* improve log

* Fix api not being defined, fix not using filter on datasources (subquery#1647)

* [SKIP CI] Prerelease

* deprecate local mode (subquery#1648)

* [SKIP CI] Prerelease

* [release] 20230427 patch (subquery#1649)

* Add missing node-fetch dependency to lock down version (subquery#1651)

* [SKIP CI] Prerelease

* postgres based mmr DB (subquery#1618)

* postgres based mmr DB

* add mmr db store option to cli

* add mmr migrate

* bug fixes

* alter file based names

* make migration bidirectional

* cleanup and improve logging

* remove localMode and add mmr-path to args

* remove check for localMode in createProjectSchema

* improve error handling

* improve error handling for postgres mmr table

* improve error handling for mmr indexer

* Fix test, create db table same as other tables, tidy up

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* Extract ProjectService, FetchService and IndexerManager to base classes (subquery#1643)

* Move bulk of project service logic to node core

* Allow overriding buildDictQueryFragment

* Improve visibility of dependencies on project service

* Extract base class from fetch service

* Fix error message

* Fix tests

* Create indexer manager base class

* [SKIP CI] Prerelease

* Move polkadot util imports to utils package, fix api type (subquery#1653)

* Move polkadot util imports to utils package, fix api type

* remove duplicate log

---------

Co-authored-by: JQQQ <jiqiang90@hotmail.com>

* [SKIP CI] Prerelease

* manifest node with options (subquery#1652)

* manifest node with options

* fix

* fix self import

* remove

* Use single version of class-* packages, extract base node-runner

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* POI cache fixes (subquery#1660)

* Add tests for cache poi and fix related issuess

* Add mutex around cachePoi

* [SKIP CI] Prerelease

* Add bulkRemove function to store (subquery#1666)

* [SKIP CI] Prerelease

* fix postMmrDb.get and mmrMigrate node sort (subquery#1669)

* [SKIP CI] Prerelease

* Fix indexed column check with historical (subquery#1667)

* [SKIP CI] Prerelease

* Regenerate mmr (subquery#1664)

* preparation

Unsafe mode and draft

bulkUpsert with poi mmr

move service to node-core

* tidy up, fix tests

* Fix gap mmr node missing issue, poiCache use plainPoiModel

* [SKIP CI] Prerelease

* Node core ds processor (subquery#1658)

* Move ds processor to node core

* Clean up deps

* Move forceClean and meta to node-core

* Move configure logic to node core

* Move unfinalzied blocks to node-core

* Add missing logger

* Fix tests

* [SKIP CI] Prerelease

* Add interval to flush the store cache (subquery#1670)

* Add interval to flush the store cache

* Fix infinite loop in nodeConfig

* [SKIP CI] Prerelease

* Remove support for manifest version <1.0.0 (subquery#1659)

* Clean up deps

* Move forceClean and meta to node-core

* Remove support for manifest version <1.0.0

* Fix build issues

* Move sandbox back from node-core

* Move gauge providers into node core

* Tidy up indexer module

* Move FileType to core

* Remove tests for unsupported manifest versions

* [SKIP CI] Prerelease

* Fixes and tidy up

* Fixes

* Fix mmrRegen module missing scheduler

* Fix subcommands module missing scheduler

* Update deps, add chainId to sandbox

---------

Co-authored-by: Ben <89335033+bz888@users.noreply.github.com>
Co-authored-by: Jay Ji <jiqiang90@hotmail.com>
Co-authored-by: Lachlan McCrae <8554309+lamcc21@users.noreply.github.com>
Co-authored-by: Lachlan McCrae <8554309+lachlan1m@users.noreply.github.com>
Co-authored-by: MOZGIII <mike-n@narod.ru>
Co-authored-by: James Bayly <46693720+jamesbayly@users.noreply.github.com>
Co-authored-by: Mingyang Li <53138432+Mingyang-Li@users.noreply.github.com>
Co-authored-by: seandotau <seandotau@gmail.com>
Co-authored-by: Naveen V <velnaveen99@gmail.com>
Co-authored-by: Darwin Subramaniam <darwinsubramaniam@googlemail.com>
Co-authored-by: HZHAIX <zhaihaohk@gmail.com>
@stwiname stwiname deleted the fix/subql-publish branch May 15, 2023 22:53
stwiname added a commit that referenced this pull request May 23, 2023
@stwiname @jiqiang90
@bz888 @guplersaxanoid @mzxyz @jamesbayly
Sync 20230512 cosmos (#119)
3d1bcf4 
* Move enum under schema (#1527)

* need fix query get enum

* fix enum query, clean all enums

* add logic to handle original enum type

* fix check exclude constrain check missing schema and table (#1537)

* [SKIP CI] Prerelease

* manual prerelease node for node-core update (#1538)

* [SKIP CI] Prerelease

* resolution for sequelize and prerelease node (#1539)

* [SKIP CI] Prerelease

* fixed the version of sequelize and prerelease node (#1540)

* [SKIP CI] Prerelease

* improve force clean with deprecated enum (#1541)

* [SKIP CI] Prerelease

* Remove exclude constraint (#1543)

* deprecate exclude contraint

* remove unused

* [SKIP CI] Prerelease

* [release] 20230306 (#1544)

* [release] 20230307 (#1545)

* bump yarn.lock, update imports (#1546)

* [SKIP CI] Prerelease

* [release] 20230307 (#1547)

* increase wsProvider timeout (#1550)

* increase wsProvider timeout

* remove .mmr dir

* [SKIP CI] Prerelease

* Use performance.now instead of date for profiler. (#1549)

* add SmartBatchService and BlockSizeBuffer (#1506)

* add SmartBatchService and BlockSizeBuffer

* check memory usage before fetching a batch

* use v8 heap usage statistics

* set block to null after indexing

* extend smart batch service to workers

* make smart batch size injectible

* rebase

* bug fixes

* code cleanup

* assign workers based on free memory

* fix liniting

* cleanup code

* cleanup logs

* clear blocksizebuffer inside smartbatchservice

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Fix build issue with profiler

* [SKIP CI] Prerelease

* fix abi codegen runtime validation (#1552)

* [SKIP CI] Prerelease

* [release] 20230309 Cli (#1553)

* Fix enum under schema not being escaped (#1555)

* [SKIP CI] Prerelease

* rename `--sponsored-dictionary` to `--dictionary-resolver` (#1559)

* renamed flag

* add missing tpyo

* [SKIP CI] Prerelease

* handle bigint in json.stringify (#1562)

* [SKIP CI] Prerelease

* add validation for `codegen`, grapql conflicts (#1564)

* add test for codegen with reserved keys

* wip validateEntityName fn

* update test, update logc

* remove logs

* [SKIP CI] Prerelease

* Fix codegen when abi is not valid (#1570)

* Fix codegen when abi is not valid

* Update packages/cli/src/controller/codegen-controller.ts

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* Fix block offset been reset to 1 (#1571)

* [SKIP CI] Prerelease

* remove blocking in process queueing (#1572)

* remove blocking in process queueing

* redefine locking pattern

* fix fetchedBlocks status

* [SKIP CI] Prerelease

* improve api error handling (#1576)

* [SKIP CI] Prerelease

* generate sourcemap for projects (#1569)

* improve error handling

* use inline-source-map

* revert api error handling

* beautify minified output from webpack

* code cleanup

* move terser plugin dependency to cli

* log warnings for unresolved stack trace

* get entry file from datasource

* alter convertStack prototype

* add ES2020 to lib compiler option

* update polkadot api to 10.1.4 (#1580)

* update polkadot api to 10.1.4

* update rest of dep

* [release] 20230329 (#1587)

* Multiple endpoints (#1551)

* allow multiple endpoints

* add response time based load balancing

* handle endpoint disconnects

* move fetchBlockBatches to apiService

* cleanup code

* move ConnectionPoolService to node-core

* fix order of module injection in indexermodule

* remove manually disconnecting api

* move api creation code to ApiConnection class

* code cleanup

* handle reconnection error

* change networkEndpoints to networkEndpoint

* change to use yarn build (#1589)

* test prerelease build (#1590)

* [SKIP CI] Prerelease

* [release] 20230330 (#1592)

* Upgrade `apollo-links` (#1593)

* [SKIP CI] Prerelease

* improve blockSize function (#1602)

* improve blockSize function

* add return type to blockSize

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* [release] 20230405 (#1603)

* Add evmChainId to metadata (#1607)

* [SKIP CI] Prerelease

* [release] 20230411 query (#1608)

* Update progress string (#1612)

* [SKIP CI] Prerelease

* Update the jsonField directive to have an option for disabling the index (#1613)

* Update the jsonField directive to have an option for disabling the index

* Clean up

* [SKIP CI] Prerelease

* [release] 20230414 utils (#1615)

* Fix workers fetching blocks out of order (#1616)

* Fix workers fetching blocks out of order

* Remove logging endpoints with api keys

* [SKIP CI] Prerelease

* [release] 20230417 (#1617)

* DB cache and various other improvements (#1561)

* store cache

* improve on flush cache

* draft storeCache service

* update

* minor change

* Updates

* add historical data to cache (#1557)

* add __block_range to creation

* add historical record to cache

* correct range

* Complete cached store methods (#1563)

* add __block_range to creation

add historical record to cache

correct range

cache store methods

start add tests

* fix set

* update

* WIP cache with workers support

* Fix up dynamic ds service, clean up

* Update store serialisation over workers

* Improve unfinalized blocks to work without tx

* Move all chain agnostic code from block dispatchers to node core

* Add comments about worker store conversion

* Fix rebase build issues

* Implement cache for metadata entity (#1565)

* Implement cache for metadata entity

* Clean up and exclude unfinalized blocks keys from metadata cache

* Clean up

* flush with cache stored records size (#1566)

* flush with cache stored records size

* update

* Lfu get cache (#1568)

* still need fix get/count logic

fix method for LFU getCache

* Update tests

* Fix and more tests

* Update base on review

* Fix

* Run all store operations through cache (#1567)

* Run all store operations through hash

* Log when cache flushed, flush cache on application shutdown

* Cache fixes

* Fix removeCache not being init

* POI cache fixes

* Fixes

* Fix caches being cleared after db operation and potential cache changes have happened

* Fix issues with injected store

* Clean up

* Fix tests from rebase

* Fix get cache not being used

* Rename function

* Fix reindex issues

* Fix casing of import

* Clean up, log if finalized is enabled

* parent 8b0b558 (#1574)

author Scott Twiname <skott.twiname@gmail.com> 1679452749 +1300
committer JQQQ <jiqiang90@hotmail.com> 1679461182 +1300

Clean up, log if finalized is enabled

Fix rebase build issues

parent 5e4ea5d
author Scott Twiname <skott.twiname@gmail.com> 1679013784 +1300
committer JQQQ <jiqiang90@hotmail.com> 1679460224 +1300

parent 5e4ea5d
author Scott Twiname <skott.twiname@gmail.com> 1679013784 +1300
committer JQQQ <jiqiang90@hotmail.com> 1679460210 +1300

parent 5e4ea5d
author Scott Twiname <skott.twiname@gmail.com> 1679013784 +1300
committer JQQQ <jiqiang90@hotmail.com> 1679460149 +1300

parent 5e4ea5d
author Scott Twiname <skott.twiname@gmail.com> 1679013784 +1300
committer JQQQ <jiqiang90@hotmail.com> 1679460101 +1300

parent 5e4ea5d
author Scott Twiname <skott.twiname@gmail.com> 1679013784 +1300
committer JQQQ <jiqiang90@hotmail.com> 1679460038 +1300

Run all store operations through hash

Log when cache flushed, flush cache on application shutdown

Fix removeCache not being init

Fixes

Fix issues with injected store

Clean up

Fix removeCache not being init

Fixes

Fix issues with injected store

Clean up

Fix tests from rebase

Fix get cache not being used

tidy up

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Update metadata when dictionary skips large number of blocks (#1577)

* Update metadata when dictionary skips large number of blocks

* Bring back option to log sql

* Dynamic cache threshold and metrics (#1578)

* Bring back fetch service tests

* Fix build issues

* Tidy up

* Fix workers historical disabled (#1579)

* Fix imports in node core

* improve close record sql, and add flag max size for getCache (#1583)

* improve close record sql, and add flag max size for getCache

* Fix

* escape id string

* Flush data at specific block heights (#1582)

* Flush data at specific block heights

* Add test

* Fix value fromBelowHeight being empty

* Fix mergedRecords potentially being empty

* Fix flushableRecordCounter

* Change log to debug

* Fix dictionary validation

* fix reindex storeCache missing EventEmitterModule (#1585)

* Make flushing cache configurable (#1586)

* Add an index to Id field on entities with historical (#1588)

* Fix tests

* Fix possible infinite loop with cache and fix test (#1591)

* Fix possible infinite loop with cache and fix test

* Add some comments

* Fix more tests

---------

Co-authored-by: JQQQ <jiqiang90@hotmail.com>

* Connect historical option up to cache model option (#1596)

* Only run delete query if there is data to delete

* Add support for non-historical indexing with cache (#1597)

* Add support for non-historical indexing with cache

* Fix api test

* remove constraintDeferrableQuery from hasOne

* fix createUniqueIndexQuery

---------

Co-authored-by: JQQQ <jiqiang90@hotmail.com>

* Cache and worker related fixes (#1598)

* fix index name too long issue (#1599)

* hash the indexes, except pk indexes

* handle existed indexes

* update

* Remove store count function (#1600)

* support cockroach db with non-historical (#1604)

* support cockroach db with non-historical

* Add more comments

* move fetchBlocks to node-core (#1611)

* move fetchBlocks to node-core

* code cleanup

* code cleanup

* remove extra args from fetchBlock

* Fix making data serializable for store bulkCreate and bulkUpdate (#1622)

* Improve big int error in workers (#1619)

* improve error handling in workers

* Make bigint stringifiable in worker and main threads

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* fix type error (#1623)

* fix type error

* cast type

* Flush store cache in operational order (#1614)

* Init

* better log message

* tidy up

* Tidy up and test

* Update packages/node/src/yargs.ts

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Update packages/node-core/src/indexer/storeCache/storeCache.service.ts

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* add benchmarking

* update benchmarking

* Tidy up

* test

* remove flag, use last flushed index, more tidy up

* make operationIndex required

* fix test

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* update default values

* fix missing indexes on cockroach db (#1624)

* fix getLatest() could be undefined

* Fix StoreOperation not pushing enums

---------

Co-authored-by: JQQQ <jiqiang90@hotmail.com>
Co-authored-by: Naveen V <velnaveen99@gmail.com>

* [SKIP CI] Prerelease

* Testing Framework (#1584)

* implement runTest

* check handler kind for running test

* implement testing service

* run tests in sandbox

* build test files through subql build

* run handlers in sandbox

* cleanup sandbox script and improve logging

* index data with `IndexerManager.indexData`

* use indexBlock() instead of indexData() for tests

* add getter _name to entities during codegen

* code cleanup

* improve logging

* improve logging

* bug fixes

* code cleanup

* improve logging

* indexBlock: get datasource from arguments

* move testingService to node-core

* code cleanup

* update ci

* code cleanup

* cleanup

* cleanup

* cleanup

* handle 0 test cases

* prevent calling getGqlType for 'contains' matcher (#1626)

* prevent calling getGqlType for 'contains' matcher

* get gqlType with ternary condition

* remove indexer module from testing module (#1628)

* remove indexer module from testing module

* cleanup

* cleanup

* Enable strict ts setting for node core (#1625)

* WIP enable strict ts setting for node core

* fix undefined in historicalMarkPreviousHeightRecordsBatch

* typo

* Update typescript, fix most strict related errors

* Strict changes for testing framework

* Runtime fixes

* change log level on cache poi

---------

Co-authored-by: JQQQ <jiqiang90@hotmail.com>

* fix prerelease (#1629)

* [SKIP CI] Prerelease

* [SKIPCI] fix testing publish order (#1630)

* fix missing @subql/testing in node-core (#1631)

* [SKIP CI] Prerelease

* manual prerelease for node-core fix (#1632)

* [SKIP CI] Prerelease

* Improve yargs descriptions (#1633)

* [SKIP CI] Prerelease

* Support tests from ./test and ./tests dirs, fix store blockheight issue

* Remove log

* [SKIP CI] Prerelease

* [release] 2.0.0 (#1627)

* release-2.0.0

* update

* update node changelog

* remove typescript from query dependency

* remove typescript from node dependency

* [SKIPCI] fix release

* [release] 2.0.0 republish (#1635)

* Fix ApiService abstract class, improve getting all DS (#1638)

* Fix ApiService abstract class, improve getting all DS

* Revert api import and minor test fixes

* [SKIP CI] Prerelease

* fix assertion error and pk index type (#1641)

* [SKIP CI] Prerelease

* fix workers crashes error (#1642)

* try to fix workers

* Fix dependency injection

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* WIP fix test part 1 (#1640)

* fix test part 1

* Update packages/node/src/indexer/fetch.service.ts

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* fix tests 20230426

* remove unused code

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* fix missing esModuleInterop (#1644)

* [SKIP CI] Prerelease

* Force flush cache with reindex command (#1645)

* [SKIP CI] Prerelease

* [release] 20230427 (#1646)

* [release] 20230427

* improve log

* Fix api not being defined, fix not using filter on datasources (#1647)

* [SKIP CI] Prerelease

* deprecate local mode (#1648)

* [SKIP CI] Prerelease

* [release] 20230427 patch (#1649)

* Add missing node-fetch dependency to lock down version (#1651)

* [SKIP CI] Prerelease

* postgres based mmr DB (#1618)

* postgres based mmr DB

* add mmr db store option to cli

* add mmr migrate

* bug fixes

* alter file based names

* make migration bidirectional

* cleanup and improve logging

* remove localMode and add mmr-path to args

* remove check for localMode in createProjectSchema

* improve error handling

* improve error handling for postgres mmr table

* improve error handling for mmr indexer

* Fix test, create db table same as other tables, tidy up

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* Extract ProjectService, FetchService and IndexerManager to base classes (#1643)

* Move bulk of project service logic to node core

* Allow overriding buildDictQueryFragment

* Improve visibility of dependencies on project service

* Extract base class from fetch service

* Fix error message

* Fix tests

* Create indexer manager base class

* [SKIP CI] Prerelease

* Move polkadot util imports to utils package, fix api type (#1653)

* Move polkadot util imports to utils package, fix api type

* remove duplicate log

---------

Co-authored-by: JQQQ <jiqiang90@hotmail.com>

* [SKIP CI] Prerelease

* manifest node with options (#1652)

* manifest node with options

* fix

* fix self import

* remove

* Use single version of class-* packages, extract base node-runner

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* POI cache fixes (#1660)

* Add tests for cache poi and fix related issuess

* Add mutex around cachePoi

* [SKIP CI] Prerelease

* Add bulkRemove function to store (#1666)

* [SKIP CI] Prerelease

* fix postMmrDb.get and mmrMigrate node sort (#1669)

* [SKIP CI] Prerelease

* Fix indexed column check with historical (#1667)

* [SKIP CI] Prerelease

* Regenerate mmr (#1664)

* preparation

Unsafe mode and draft

bulkUpsert with poi mmr

move service to node-core

* tidy up, fix tests

* Fix gap mmr node missing issue, poiCache use plainPoiModel

* [SKIP CI] Prerelease

* Node core ds processor (#1658)

* Move ds processor to node core

* Clean up deps

* Move forceClean and meta to node-core

* Move configure logic to node core

* Move unfinalzied blocks to node-core

* Add missing logger

* Fix tests

* [SKIP CI] Prerelease

* Add interval to flush the store cache (#1670)

* Add interval to flush the store cache

* Fix infinite loop in nodeConfig

* [SKIP CI] Prerelease

* Remove support for manifest version <1.0.0 (#1659)

* Clean up deps

* Move forceClean and meta to node-core

* Remove support for manifest version <1.0.0

* Fix build issues

* Move sandbox back from node-core

* Move gauge providers into node core

* Tidy up indexer module

* Move FileType to core

* Remove tests for unsupported manifest versions

* [SKIP CI] Prerelease

* Fix Cache missing remove (#1675)

* Fix missing removeCache from flush

* tidy up

* [SKIP CI] Prerelease

* [release] 20230510 (#1673)

* [release] 20230510

* Update package.json

* Update node-core changelog

* Fix missing scheduler subcommands (#1677)

* Fix mmrRegen module missing scheduler

* Fix subcommands module missing scheduler

* [SKIP CI] Prerelease

* Extract dictionary meta validation so it can be overridden (#1679)

* Extract dictionary meta validation so it can be overridden

* Update type

* [SKIP CI] Prerelease

* [release] 20230511 (#1678)

* [release] 20230511

* Update node-core

* [release] 20230511 types (#1680)

* Fix metadata check, allow base indexer manager to parse abis with ethereum (#1682)

* [SKIP CI] Prerelease

* Move validate function to common (#1683)

* Move validate function to common

* Make prepareFilteredData async

* [SKIP CI] Prerelease

* Inject the chain id into sandboxes (#1684)

* [SKIP CI] Prerelease

* [release] 20230511 v2 (#1685)

* Fix dictionary-resolver arg type

* Add case for cosmos transactions

* Add back pg dependency

---------

Co-authored-by: Jay Ji <jiqiang90@hotmail.com>
Co-authored-by: Ben <89335033+bz888@users.noreply.github.com>
Co-authored-by: Naveen V <velnaveen99@gmail.com>
Co-authored-by: HZHAIX <zhaihaohk@gmail.com>
Co-authored-by: James Bayly <46693720+jamesbayly@users.noreply.github.com>
stwiname added a commit that referenced this pull request Jun 2, 2023
@guplersaxanoid @bighammerw
@jiqiang90 @bz888 @deverka @stwiname @hariu-starfish @Orion-starfish @filo87 @lamcc21 @mzxyz @MOZGIII @jamesbayly @Mingyang-Li @seandotau @darwinsubramaniam
Rebase 20230313 (#37)
10cd128 
* [SKIP CI] Prerelease

* manual prerelease (#1326)

* [SKIP CI] Prerelease

* enable historical by default (#1327)

* [SKIP CI] Prerelease

* [release] 20220929 (#1328)

* [release] 20220929

* update

* Update packages/node/CHANGELOG.md

Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com>

* Update packages/query/CHANGELOG.md

Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com>

* Update packages/utils/CHANGELOG.md

Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com>

* Update packages/query/CHANGELOG.md

Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com>

* Update packages/node/CHANGELOG.md

Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com>

* Update packages/node-core/CHANGELOG.md

Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com>

* Update packages/common/CHANGELOG.md

Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com>

Co-authored-by: JQQQ <jiqiang90@hotmail.com>
Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com>

* Update ipfs endpoints, improve error handling (#1337)

* [SKIP CI] Prerelease

* Fix nodeConfig not being provided to benchmark service (#1338)

* [SKIP CI] Prerelease

* [release] 20221006 (#1339)

* Remove Subqueries table (#1340)

* Remove Subqueries table

* Fix NOT_SUPPORT function in project

* Remove tests relating to old subqueries table

* [SKIP CI] Prerelease

* [release] 20221010 (#1342)

* [release] 20221010

* Update packages/node/CHANGELOG.md

Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com>

Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com>

* Validate ds processor in worker threads (#1343)

* [SKIP CI] Prerelease

* Fix publish ipfs 503 (#1331)

* add comments

* publish ipfs issue

* publish ipfs issue

* Update packages/cli/src/controller/publish-controller.ts

Co-authored-by: bz888 <ziyu.ben158@gmail.com>
Co-authored-by: Louise wang <louisewang118@.gmail.com>
Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* Move common dictionary service to node-core (#1345)

* Move common dictionary service to node-core

* Apply matcher from cosmos dictionary

* Move sandbox code to node-core

* Remove testing log

* [SKIP CI] Prerelease

* [release] 20221011

* Back port changes in algorand sdk to dictionary service (#1346)

* [SKIP CI] Prerelease

* temp disable hot schema reload (#1349)

* temp disable hot schema reload

* add CLI flag for hotschema reload

* fix

* print node version (#1348)

* print node version

* update logger info

* update with node name

Co-authored-by: Louise wang <louisewang118@.gmail.com>

* [SKIP CI] Prerelease

* [SKIP CI] Prerelease

* fix output-format flag on node (#1351)

* fix typo

* update workers and yargs outputFmt

* [SKIP CI] Prerelease

* Timestamp block filters (#1310)

* add block timestamp filter

* bug fixes

* bug fixes

* clean code and write test for timestamp filter

* move cron generation to SubqueryProject

* fix next timestamp log bug

* remove modifiedDataSources

* [SKIP CI] Prerelease

* Update to latest polkadot api (#1356)

* [SKIP CI] Prerelease

* add ad flags (#1352)

* add ad flags

* add interface for all AD flags and add prefix for indexer/query flags

* change string of to for

* some variable name change and update interface

Co-authored-by: Louise wang <louisewang118@.gmail.com>

* support version without v (#1354)

* support version without v

* improvement addv FUNC and solved undefined input

Co-authored-by: Louise wang <louisewang118@.gmail.com>

* fix deploy-test (#1357)

* [SKIP CI] Prerelease

* add dedicate data base flag for create project (#1355)

* add dedicate data base flag for create project

* test with latest createproject api host service and update wording

Co-authored-by: Louise wang <louisewang118@.gmail.com>

* Optimise modulo filter (#1358)

* scale batch size if modulo filter used

* add comments

* add comments

* fix logic runModuloblock

* remove comments

* remove more comments

* add check datasources, check height logic

* removed comments

* refactor

* flatten handler array

* Fix template issues (#1363)

* Fix the problem of creating multiple templates in the same block with the same template address

* refactor: cache ds in same block

Co-authored-by: Orion <orion@starfish.finance>

* fix deployment and project tests (#1364)

* fix deployment and project tests

* fix linting error

* [SKIP CI] Prerelease

* [release] 20221027 (#1365)

* fix: incomplete ds data in same block (#1370)

Co-authored-by: Orion <orion@starfish.finance>

* [SKIP CI] Prerelease

* Best Block (#1308)

* Draft

* update changes

* rebase changes

* fix

* fix (#1329)

* disable best blocks for workers until we support it

* Rename bestBlock to unfinalizedBlocks, other clean up

* Clean up

* Further clean up

* Dedupe reindex function

* Fix tests

* Clean up

* Update checking finalization to use parent hash

* Rename logger

* Refactor unfinalized blocks

* Use header rather than full block, improve detecting forks

* Verify unfinalized blocks when disabled.

Use sorted array for storing unfinalized blocks

* Clean up logs

* Fix not indexing unfinalized blocks right away, exit if historical not enabled

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* Fix bugs with unfinalized blocks (#1374)

* Fix bugs with unfinalized blocks

- Fix using wrong hash for unfinalized blocks
- Fix not removing unfinalized blocks when the latest unfinalized block < finalized

* Fix tests getting stuck

* Fix issue finding where fork occurred

* [SKIP CI] Prerelease

* [release] 20221028 (#1372)

* hot fix tests (#1360)

* update tests: api.service.spec, init-controller.test, publish-controller.spec

* update tests following comments

* [SKIP CI] Prerelease

* add ethereum to CLI and Validator (#1378)

* Imporve dictionary query (#1371)

* wip

* wip

* wip

* broken wip

* wip refactoring

* seperate dictionaryQueryEntries

* fix logic for setDicitonaryQueryEntries

* cleaning up

* move dictionaryEntry.ts to node-core

* remove comments

* create class for dictionaryqueryentries

* update tests

* move dictionaryQueryService into dicitonaryService

* refactor

* update logic for useDicitonary

* relocated dictionaryQuery funcs

* add test for dictquerymap

* tests failing on windows?

* test getDictionaryQueryEntries

* test fixed for getDictioanryQueryEntries

* added test for sorting

* conflict fixed ?

* fix conflict_2

* fix conflict_3

* add delete temp ds records back

* clean up with new logic

* clean up, add comments for test

* fix

* update logic

* add generic type

* [SKIP CI] Prerelease

* Enable for better inheritance of generated entity modeld (#1377)

* refactor: enable for entity inheritance

* refactor: include also filed getters

* [SKIP CI] Prerelease

* fix comments issue with new package (#1380)

* fix comments issue with new package

* moved yaml package

* [SKIP CI] Prerelease

* fix logic with reindex and unfinalized height and dynamic ds (#1382)

* fix logic with reindex and unfinalized height

* fix

* include fix for #1379

* update polkadot to 9.7.1 (#1384)

* [release] 20221107

* Fix remove alter table (#1387)

* remove migrate alter table

* remove

* [SKIP CI] Prerelease

* [release] 20221108 (#1388)

* fix missing sequelize sync (#1389)

* [SKIP CI] Prerelease

* [release] 20221108 patch (#1390)

* reindex bind (#1391)

* [SKIP CI] Prerelease

* [release] 20221109 (#1393)

* Handle fetch errors, then retry (#1386)

* add retryOnFail function

* add retryOnFail

* add test, fix logic

* [SKIP CI] Prerelease

* fix (#1395)

* [SKIP CI] Prerelease

* [release] 20221109 node-core (#1394)

* Fix tests hanging (#1396)

* Fix tests hanging

* Update base docker image with newer git version

* [SKIP CI] Prerelease

* Add distinct query plugin (#1274)

* Add distinct query plugin

* Clean up log

* Fix distinct not being provided to query

* Uppercase enum to be consistent with other enums

* Update dictionary queries to try distinct argument

* [SKIP CI] Prerelease

* Add query distinct dependencies (#1398)

* fix missing update forked graphile dependencies

* tidy up

* tidy up

* [SKIP CI] Prerelease

* Break block dispatcher file up and move common code to base class (#1397)

* [SKIP CI] Prerelease

* Hot schema trigger (#1401)

* implement trigger with notification

* working prior clean up

* refactor and clean up on async and await

* clean up

* clear comments

* add filter

* fix

* fix err

* [SKIP CI] Prerelease

* [release] 20221115 (#1402)

* [release] 20221115

* [release] 20221115

* [release] 20221115

* fix hot schema (#1404)

* fix and refactor

* refactor getTriggers

* [SKIP CI] Prerelease

* [release] 20221115 (#1408)

* [release] 20221115

* [release] 20221115

* fix fetchblock for works (#1410)

* [SKIP CI] Prerelease

* fix row estimate (#1417)

* fix row estimate

* tidy up

* [SKIP CI] Prerelease

* Add support to cli/validator for flare (#1416)

* [SKIP CI] Prerelease

* missing validator missing ipfs chainTypes (#1419)

* [SKIP CI] Prerelease

* query explain include graphql query (#1426)

* [SKIP CI] Prerelease

* dictionary auth link integration (#1411)

* add required dependancies and add flag for feature

* update modules and add logic to dictionary

* update dictionary parameters, update tests, remove hardcoded values

* simplify auth dictionary handling

* add apollo link type, remove un-needed from wrapper

* remove dictionary from indexer module

* update flag name, add error check

* update yargs flag description

* Update appolo-links

* update @subql/apollo-links version, remove apollo/client version change

* fix yarn.lock

* add await init to dictionary tests

* fix fetch test dictionary provider, update yarn.lock

Co-authored-by: Lachlan McCrae <8554309+lachlan1m@users.noreply.github.com>
Co-authored-by: HZ <zhaihaohk@gmail.com>

* Fix missing speChanged logic with getRuntime (#1421)

* init

* need fix worker

* fix test

* tidy up

* fix test, fix missing dictionary

* inject service to runtime

* remove unused code

* [SKIP CI] Prerelease

* Feat/multi chain indexing (#1375)

* [SKIP CI] Prerelease

* remove enums (#1427)

* [SKIP CI] Prerelease

* change to print graphql sql on request (#1428)

* [SKIP CI] Prerelease

* fix missing util-crptyo in util package (#1429)

* fix missing util-crptyo in util package

* need prerelease in query service

* [SKIP CI] Prerelease

* fix import issue in query service (#1430)

* [SKIP CI] Prerelease

* fix missing query (#1432)

* [SKIP CI] Prerelease

* change fetch module SubqueryProject => ISubqueryProject (#1433)

Co-authored-by: Lachlan McCrae <8554309+lachlan1m@users.noreply.github.com>

* [SKIP CI] Prerelease

* Update polkadot 9.9.1 (#1434)

* update polkadot api, also fix force-clean dependencies issue

* tidy up

* [SKIP CI] Prerelease

* [release] 20221123 (#1431)

* [release] 20221123

* Update packages/query/CHANGELOG.md

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Update packages/node-core/CHANGELOG.md

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Update packages/node/CHANGELOG.md

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Update packages/node-core/CHANGELOG.md

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Update packages/validator/CHANGELOG.md

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Add a way to pass settings to the graphql playground (#1436)

* Add a way to pass settings to the graphql playground

* Add a changelog entry

* [SKIP CI] Prerelease

* Fix the changelog entry for the last PR (#1437)

* [SKIP CI] Prerelease

* fix enum name missing issue (#1441)

* fix enum name missing issue

* tidy up

* [SKIP CI] Prerelease

* hash sql queries (#1438)

* hashed sql

* add comments

* add comment for hashName function

* update function

* update hashName function

* [SKIP CI] Prerelease

* [release] 20221130 (#1442)

* [release] 20221130

* [release] 20221130

* Add --file flag for codegen (#1446)

* init

* change flag behaviour so -f can overwrite -l

* update getManifestPath method, fix typo

* update argument description

* update variable names

* update arg description

* add error message on bad --file path

Co-authored-by: Lachlan McCrae <8554309+lachlan1m@users.noreply.github.com>

* [SKIP CI] Prerelease

* add try and catch for hsr (#1449)

* [SKIP CI] Prerelease

* Bypass blocks (#1435)

* wip

* manifest error

* manfiest passed

* added logging for bypassing blocks

* sync main, moved bypassBlocks under networks

* pass all tests

* mock dictioanry case

* fix metadata

* clean up

* allow range

* bypass working, needs more testing

* wip on bypass with setting new bufferHeight

* working

* added runtime test

* add test for latestBufferHeight (blocked)

* removed unneeded test, improve logic

* clean up

* add comments

* add comments for latestBufferHeight reasoning

* update logic

* refactor

* update logic

* refactor

* [SKIP CI] Prerelease

* [release] 20221205 (#1448)

* [release] 20221205

* [release] 20221205

* [release] 20221206

* details for bypassBlocks

* Remove unused RuntimeService from indexer module, it had missing dependencies (#1453)

* [SKIP CI] Prerelease

* Fix/subcommands (#1451)

* updated forceClean

* update reindex

* [SKIP CI] Prerelease

* Enable env vars parsing at node and query commands (#1452)

* [SKIP CI] Prerelease

* Workers: Fix SequelizeDatabaseError - tuple concurrently updated (#1458)

* add flag for hot-schema reload on node, ensure query is on main thread

* remove flags

* [SKIP CI] Prerelease

* Add start height to project (#1456)

* [SKIP CI] Prerelease

* Fix poi offset is 0 (#1459)

* [SKIP CI] Prerelease

* Handle when fields got undefined with historical (#1463)

* [SKIP CI] Prerelease

* update deploy cli (#1460)

* [SKIP CI] Prerelease

* exit on fail (#1464)

* [SKIP CI] Prerelease

* fix parentSpecVersion could be undefined (#1467)

* [SKIP CI] Prerelease

* [release] 20221219 (#1466)

* [release] 20221219

* update

* Update packages/node-core/CHANGELOG.md

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Update packages/node/CHANGELOG.md

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Update packages/node-core/CHANGELOG.md

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Update packages/cli/CHANGELOG.md

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* update

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Fix templates with other sdks, make -f flag work like other commands (#1470)

* Fix templates with other sdks, make -f flag work like other commands

* Fix for sdks with first class smart contracts

* [SKIP CI] Prerelease

* Tweak error message for genesis hash (#1471)

* [SKIP CI] Prerelease

* Fix trigger function (#1469)

* fix trigger and function for subscription and hot schema reload

* update

* [SKIP CI] Prerelease

* [release] 20221222 (#1474)

* validate dictionary with start height (#1473)

* validate dictionary with start height

* Update packages/node/src/indexer/fetch.service.ts

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* add count for entity (#1480)

* add count for entity

* Update packages/node-core/src/indexer/store.service.ts

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* Fixing readme doc for @subql/query (#1485)

* doc: correcting grammer

* fix: typo

* [SKIP CI] Prerelease

* update polkadot api to 9.11.1 (#1483)

* update polkadot api to 9.11.1

* update

* [SKIP CI] Prerelease

* add connection filters (#1484)

* [SKIP CI] Prerelease

* fix workers missing set last buffered height from empty dictionary (#1492)

* [SKIP CI] Prerelease

* Fix custom metadata keys not being applied (#1496)

* add block hash validation after fetch (#1494)

* add block hash validation after fetch

* add tests

* skip fetch test

* Update log

* [SKIP CI] Prerelease

* [release] 20230123 (#1497)

* Add support for near projects to the CLI (#1498)

* Add support for near projects to the CLI

* Update common near version

* Update near packages

* [SKIP CI] Prerelease

* Arranges commands in alphabetical order (#1495)

* [SKIP CI] Prerelease

* user process.env.USERPROFILE for windows platform (#1502)

* [SKIP CI] Prerelease

* add blockTime to NodeConfig (#1501)

* [SKIP CI] Prerelease

* [release] 20230131 (#1503)

* Worker use runtime service (#1491)

* draft

* update

* tidy up

* update

* working worker runtimes

* into base class

* clean up

* improve base on review

* [SKIP CI] Prerelease

* Check queue for lower blocks before processing fetched blocks (#1509)

* [SKIP CI] Prerelease

* Support array type in dictionary queries (#1510)

* [SKIP CI] Prerelease

* add flag query-limit (#1514)

* [SKIP CI] Prerelease

* Fix error with tempDsRecords being undefined (#1516)

* Fix error with tempDsRecords being undefined

* Fix schema not being used for metadata, sync dynamic-ds across workers

* Be smarter about reloading dynamic ds in workers

* [SKIP CI] Prerelease

* Update postgraphile enable table partitions (#1520)

* [SKIP CI] Prerelease

* `dictionary-optimisation` flag  (#1519)

* add flag, to orderby blockheight instead of pk

* add comment

* [SKIP CI] Prerelease

* [WIP] Feature/adding postgres ssl connection option (#1513)

* enable ssl option postgres

* Check DB_SSL Env Value , default is false

* adding ENV to pass in the Path to certificates

* adding SSL option for query pacakge.

* use args to pass the postgres ssl informations.

* load file content moved to comman package.

* Update packages/node-core/src/configure/NodeConfig.ts

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* Support Cockroach db (#1521)

* support cockroach db

* Update comment

* [SKIP CI] Prerelease

* ensure pgpool connection work with ssl (#1525)

* [SKIP CI] Prerelease

* Fix issue when use cockroach query service not support pgPartition (#1526)

* [SKIP CI] Prerelease

* [release] 20230221 (#1528)

* fix issue store getByField limit could potentially excess config limit (#1529)

* fix issue store getByField limit could potentially excess config limit

* Update packages/node-core/src/indexer/store.service.ts

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* typo

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* Abi codegen (#1532)

* draft

* fix abi name, add support for transaction

* Update packages/cli/src/template/abi-interface.ts.ejs

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* fix

* fix dependencies

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* Move enum under schema (#1527)

* need fix query get enum

* fix enum query, clean all enums

* add logic to handle original enum type

* fix check exclude constrain check missing schema and table (#1537)

* [SKIP CI] Prerelease

* manual prerelease node for node-core update (#1538)

* [SKIP CI] Prerelease

* resolution for sequelize and prerelease node (#1539)

* [SKIP CI] Prerelease

* fixed the version of sequelize and prerelease node (#1540)

* [SKIP CI] Prerelease

* improve force clean with deprecated enum (#1541)

* [SKIP CI] Prerelease

* Remove exclude constraint (#1543)

* deprecate exclude contraint

* remove unused

* [SKIP CI] Prerelease

* [release] 20230306 (#1544)

* [release] 20230307 (#1545)

* bump yarn.lock, update imports (#1546)

* [SKIP CI] Prerelease

* [release] 20230307 (#1547)

* increase wsProvider timeout (#1550)

* increase wsProvider timeout

* remove .mmr dir

* [SKIP CI] Prerelease

* Use performance.now instead of date for profiler. (#1549)

* add SmartBatchService and BlockSizeBuffer (#1506)

* add SmartBatchService and BlockSizeBuffer

* check memory usage before fetching a batch

* use v8 heap usage statistics

* set block to null after indexing

* extend smart batch service to workers

* make smart batch size injectible

* rebase

* bug fixes

* code cleanup

* assign workers based on free memory

* fix liniting

* cleanup code

* cleanup logs

* clear blocksizebuffer inside smartbatchservice

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Fix build issue with profiler

* [SKIP CI] Prerelease

* fix abi codegen runtime validation (#1552)

* [SKIP CI] Prerelease

* [release] 20230309 Cli (#1553)

* Fix enum under schema not being escaped (#1555)

* [SKIP CI] Prerelease

* rename `--sponsored-dictionary` to `--dictionary-resolver` (#1559)

* renamed flag

* add missing tpyo

* [SKIP CI] Prerelease

* code cleanup

* code cleanup

---------

Co-authored-by: bighammerw <92004774+bighammerw@users.noreply.github.com>
Co-authored-by: Jay Ji <jiqiang90@hotmail.com>
Co-authored-by: Ben <89335033+bz888@users.noreply.github.com>
Co-authored-by: Marta Adamczyk <marta.kowalska.987@gmail.com>
Co-authored-by: Scott Twiname <skott.twiname@gmail.com>
Co-authored-by: bz888 <ziyu.ben158@gmail.com>
Co-authored-by: Louise wang <louisewang118@.gmail.com>
Co-authored-by: hariu-starfish <103621490+hariu-starfish@users.noreply.github.com>
Co-authored-by: Orion <orion@starfish.finance>
Co-authored-by: Filippo <filippo@embriotech.ch>
Co-authored-by: Lachlan McCrae <8554309+lamcc21@users.noreply.github.com>
Co-authored-by: Lachlan McCrae <8554309+lachlan1m@users.noreply.github.com>
Co-authored-by: HZ <zhaihaohk@gmail.com>
Co-authored-by: MOZGIII <mike-n@narod.ru>
Co-authored-by: James Bayly <46693720+jamesbayly@users.noreply.github.com>
Co-authored-by: Mingyang Li <53138432+Mingyang-Li@users.noreply.github.com>
Co-authored-by: seandotau <seandotau@gmail.com>
Co-authored-by: Darwin Subramaniam <darwinsubramaniam@googlemail.com>
stwiname added a commit that referenced this pull request Jun 2, 2023
@stwiname @jiqiang90
@bz888 @lamcc21 @mzxyz @MOZGIII @jamesbayly @Mingyang-Li @seandotau @guplersaxanoid @darwinsubramaniam
Sync 2.0 algo (#39)
1d5bb5b 
* [release] 20221109 (#1393)

* Handle fetch errors, then retry (#1386)

* add retryOnFail function

* add retryOnFail

* add test, fix logic

* [SKIP CI] Prerelease

* fix (#1395)

* [SKIP CI] Prerelease

* [release] 20221109 node-core (#1394)

* Fix tests hanging (#1396)

* Fix tests hanging

* Update base docker image with newer git version

* [SKIP CI] Prerelease

* Add distinct query plugin (#1274)

* Add distinct query plugin

* Clean up log

* Fix distinct not being provided to query

* Uppercase enum to be consistent with other enums

* Update dictionary queries to try distinct argument

* [SKIP CI] Prerelease

* Add query distinct dependencies (#1398)

* fix missing update forked graphile dependencies

* tidy up

* tidy up

* [SKIP CI] Prerelease

* Break block dispatcher file up and move common code to base class (#1397)

* [SKIP CI] Prerelease

* Hot schema trigger (#1401)

* implement trigger with notification

* working prior clean up

* refactor and clean up on async and await

* clean up

* clear comments

* add filter

* fix

* fix err

* [SKIP CI] Prerelease

* [release] 20221115 (#1402)

* [release] 20221115

* [release] 20221115

* [release] 20221115

* fix hot schema (#1404)

* fix and refactor

* refactor getTriggers

* [SKIP CI] Prerelease

* [release] 20221115 (#1408)

* [release] 20221115

* [release] 20221115

* fix fetchblock for works (#1410)

* [SKIP CI] Prerelease

* fix row estimate (#1417)

* fix row estimate

* tidy up

* [SKIP CI] Prerelease

* Add support to cli/validator for flare (#1416)

* [SKIP CI] Prerelease

* missing validator missing ipfs chainTypes (#1419)

* [SKIP CI] Prerelease

* query explain include graphql query (#1426)

* [SKIP CI] Prerelease

* dictionary auth link integration (#1411)

* add required dependancies and add flag for feature

* update modules and add logic to dictionary

* update dictionary parameters, update tests, remove hardcoded values

* simplify auth dictionary handling

* add apollo link type, remove un-needed from wrapper

* remove dictionary from indexer module

* update flag name, add error check

* update yargs flag description

* Update appolo-links

* update @subql/apollo-links version, remove apollo/client version change

* fix yarn.lock

* add await init to dictionary tests

* fix fetch test dictionary provider, update yarn.lock

Co-authored-by: Lachlan McCrae <8554309+lachlan1m@users.noreply.github.com>
Co-authored-by: HZ <zhaihaohk@gmail.com>

* Fix missing speChanged logic with getRuntime (#1421)

* init

* need fix worker

* fix test

* tidy up

* fix test, fix missing dictionary

* inject service to runtime

* remove unused code

* [SKIP CI] Prerelease

* Feat/multi chain indexing (#1375)

* [SKIP CI] Prerelease

* remove enums (#1427)

* [SKIP CI] Prerelease

* change to print graphql sql on request (#1428)

* [SKIP CI] Prerelease

* fix missing util-crptyo in util package (#1429)

* fix missing util-crptyo in util package

* need prerelease in query service

* [SKIP CI] Prerelease

* fix import issue in query service (#1430)

* [SKIP CI] Prerelease

* fix missing query (#1432)

* [SKIP CI] Prerelease

* change fetch module SubqueryProject => ISubqueryProject (#1433)

Co-authored-by: Lachlan McCrae <8554309+lachlan1m@users.noreply.github.com>

* [SKIP CI] Prerelease

* Update polkadot 9.9.1 (#1434)

* update polkadot api, also fix force-clean dependencies issue

* tidy up

* [SKIP CI] Prerelease

* [release] 20221123 (#1431)

* [release] 20221123

* Update packages/query/CHANGELOG.md

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Update packages/node-core/CHANGELOG.md

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Update packages/node/CHANGELOG.md

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Update packages/node-core/CHANGELOG.md

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Update packages/validator/CHANGELOG.md

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Add a way to pass settings to the graphql playground (#1436)

* Add a way to pass settings to the graphql playground

* Add a changelog entry

* [SKIP CI] Prerelease

* Fix the changelog entry for the last PR (#1437)

* [SKIP CI] Prerelease

* fix enum name missing issue (#1441)

* fix enum name missing issue

* tidy up

* [SKIP CI] Prerelease

* hash sql queries (#1438)

* hashed sql

* add comments

* add comment for hashName function

* update function

* update hashName function

* [SKIP CI] Prerelease

* [release] 20221130 (#1442)

* [release] 20221130

* [release] 20221130

* Add --file flag for codegen (#1446)

* init

* change flag behaviour so -f can overwrite -l

* update getManifestPath method, fix typo

* update argument description

* update variable names

* update arg description

* add error message on bad --file path

Co-authored-by: Lachlan McCrae <8554309+lachlan1m@users.noreply.github.com>

* [SKIP CI] Prerelease

* add try and catch for hsr (#1449)

* [SKIP CI] Prerelease

* Bypass blocks (#1435)

* wip

* manifest error

* manfiest passed

* added logging for bypassing blocks

* sync main, moved bypassBlocks under networks

* pass all tests

* mock dictioanry case

* fix metadata

* clean up

* allow range

* bypass working, needs more testing

* wip on bypass with setting new bufferHeight

* working

* added runtime test

* add test for latestBufferHeight (blocked)

* removed unneeded test, improve logic

* clean up

* add comments

* add comments for latestBufferHeight reasoning

* update logic

* refactor

* update logic

* refactor

* [SKIP CI] Prerelease

* [release] 20221205 (#1448)

* [release] 20221205

* [release] 20221205

* [release] 20221206

* details for bypassBlocks

* Remove unused RuntimeService from indexer module, it had missing dependencies (#1453)

* [SKIP CI] Prerelease

* Fix/subcommands (#1451)

* updated forceClean

* update reindex

* [SKIP CI] Prerelease

* Enable env vars parsing at node and query commands (#1452)

* [SKIP CI] Prerelease

* Workers: Fix SequelizeDatabaseError - tuple concurrently updated (#1458)

* add flag for hot-schema reload on node, ensure query is on main thread

* remove flags

* [SKIP CI] Prerelease

* Add start height to project (#1456)

* [SKIP CI] Prerelease

* Fix poi offset is 0 (#1459)

* [SKIP CI] Prerelease

* Handle when fields got undefined with historical (#1463)

* [SKIP CI] Prerelease

* update deploy cli (#1460)

* [SKIP CI] Prerelease

* exit on fail (#1464)

* [SKIP CI] Prerelease

* fix parentSpecVersion could be undefined (#1467)

* [SKIP CI] Prerelease

* [release] 20221219 (#1466)

* [release] 20221219

* update

* Update packages/node-core/CHANGELOG.md

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Update packages/node/CHANGELOG.md

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Update packages/node-core/CHANGELOG.md

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Update packages/cli/CHANGELOG.md

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* update

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Fix templates with other sdks, make -f flag work like other commands (#1470)

* Fix templates with other sdks, make -f flag work like other commands

* Fix for sdks with first class smart contracts

* [SKIP CI] Prerelease

* Tweak error message for genesis hash (#1471)

* [SKIP CI] Prerelease

* Fix trigger function (#1469)

* fix trigger and function for subscription and hot schema reload

* update

* [SKIP CI] Prerelease

* [release] 20221222 (#1474)

* validate dictionary with start height (#1473)

* validate dictionary with start height

* Update packages/node/src/indexer/fetch.service.ts

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* add count for entity (#1480)

* add count for entity

* Update packages/node-core/src/indexer/store.service.ts

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* Fixing readme doc for @subql/query (#1485)

* doc: correcting grammer

* fix: typo

* [SKIP CI] Prerelease

* update polkadot api to 9.11.1 (#1483)

* update polkadot api to 9.11.1

* update

* [SKIP CI] Prerelease

* add connection filters (#1484)

* [SKIP CI] Prerelease

* fix workers missing set last buffered height from empty dictionary (#1492)

* [SKIP CI] Prerelease

* Fix custom metadata keys not being applied (#1496)

* add block hash validation after fetch (#1494)

* add block hash validation after fetch

* add tests

* skip fetch test

* Update log

* [SKIP CI] Prerelease

* [release] 20230123 (#1497)

* Add support for near projects to the CLI (#1498)

* Add support for near projects to the CLI

* Update common near version

* Update near packages

* [SKIP CI] Prerelease

* Arranges commands in alphabetical order (#1495)

* [SKIP CI] Prerelease

* user process.env.USERPROFILE for windows platform (#1502)

* [SKIP CI] Prerelease

* add blockTime to NodeConfig (#1501)

* [SKIP CI] Prerelease

* [release] 20230131 (#1503)

* Worker use runtime service (#1491)

* draft

* update

* tidy up

* update

* working worker runtimes

* into base class

* clean up

* improve base on review

* [SKIP CI] Prerelease

* Check queue for lower blocks before processing fetched blocks (#1509)

* [SKIP CI] Prerelease

* Support array type in dictionary queries (#1510)

* [SKIP CI] Prerelease

* add flag query-limit (#1514)

* [SKIP CI] Prerelease

* Fix error with tempDsRecords being undefined (#1516)

* Fix error with tempDsRecords being undefined

* Fix schema not being used for metadata, sync dynamic-ds across workers

* Be smarter about reloading dynamic ds in workers

* [SKIP CI] Prerelease

* Update postgraphile enable table partitions (#1520)

* [SKIP CI] Prerelease

* `dictionary-optimisation` flag  (#1519)

* add flag, to orderby blockheight instead of pk

* add comment

* [SKIP CI] Prerelease

* [WIP] Feature/adding postgres ssl connection option (#1513)

* enable ssl option postgres

* Check DB_SSL Env Value , default is false

* adding ENV to pass in the Path to certificates

* adding SSL option for query pacakge.

* use args to pass the postgres ssl informations.

* load file content moved to comman package.

* Update packages/node-core/src/configure/NodeConfig.ts

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* Support Cockroach db (#1521)

* support cockroach db

* Update comment

* [SKIP CI] Prerelease

* ensure pgpool connection work with ssl (#1525)

* [SKIP CI] Prerelease

* Fix issue when use cockroach query service not support pgPartition (#1526)

* [SKIP CI] Prerelease

* [release] 20230221 (#1528)

* fix issue store getByField limit could potentially excess config limit (#1529)

* fix issue store getByField limit could potentially excess config limit

* Update packages/node-core/src/indexer/store.service.ts

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* typo

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* Abi codegen (#1532)

* draft

* fix abi name, add support for transaction

* Update packages/cli/src/template/abi-interface.ts.ejs

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* fix

* fix dependencies

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* Move enum under schema (#1527)

* need fix query get enum

* fix enum query, clean all enums

* add logic to handle original enum type

* fix check exclude constrain check missing schema and table (#1537)

* [SKIP CI] Prerelease

* manual prerelease node for node-core update (#1538)

* [SKIP CI] Prerelease

* resolution for sequelize and prerelease node (#1539)

* [SKIP CI] Prerelease

* fixed the version of sequelize and prerelease node (#1540)

* [SKIP CI] Prerelease

* improve force clean with deprecated enum (#1541)

* [SKIP CI] Prerelease

* Remove exclude constraint (#1543)

* deprecate exclude contraint

* remove unused

* [SKIP CI] Prerelease

* [release] 20230306 (#1544)

* [release] 20230307 (#1545)

* bump yarn.lock, update imports (#1546)

* [SKIP CI] Prerelease

* [release] 20230307 (#1547)

* increase wsProvider timeout (#1550)

* increase wsProvider timeout

* remove .mmr dir

* [SKIP CI] Prerelease

* Use performance.now instead of date for profiler. (#1549)

* add SmartBatchService and BlockSizeBuffer (#1506)

* add SmartBatchService and BlockSizeBuffer

* check memory usage before fetching a batch

* use v8 heap usage statistics

* set block to null after indexing

* extend smart batch service to workers

* make smart batch size injectible

* rebase

* bug fixes

* code cleanup

* assign workers based on free memory

* fix liniting

* cleanup code

* cleanup logs

* clear blocksizebuffer inside smartbatchservice

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Fix build issue with profiler

* [SKIP CI] Prerelease

* fix abi codegen runtime validation (#1552)

* [SKIP CI] Prerelease

* [release] 20230309 Cli (#1553)

* Fix enum under schema not being escaped (#1555)

* [SKIP CI] Prerelease

* rename `--sponsored-dictionary` to `--dictionary-resolver` (#1559)

* renamed flag

* add missing tpyo

* [SKIP CI] Prerelease

* handle bigint in json.stringify (#1562)

* [SKIP CI] Prerelease

* add validation for `codegen`, grapql conflicts (#1564)

* add test for codegen with reserved keys

* wip validateEntityName fn

* update test, update logc

* remove logs

* [SKIP CI] Prerelease

* Fix codegen when abi is not valid (#1570)

* Fix codegen when abi is not valid

* Update packages/cli/src/controller/codegen-controller.ts

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* Fix block offset been reset to 1 (#1571)

* [SKIP CI] Prerelease

* remove blocking in process queueing (#1572)

* remove blocking in process queueing

* redefine locking pattern

* fix fetchedBlocks status

* [SKIP CI] Prerelease

* improve api error handling (#1576)

* [SKIP CI] Prerelease

* generate sourcemap for projects (#1569)

* improve error handling

* use inline-source-map

* revert api error handling

* beautify minified output from webpack

* code cleanup

* move terser plugin dependency to cli

* log warnings for unresolved stack trace

* get entry file from datasource

* alter convertStack prototype

* add ES2020 to lib compiler option

* update polkadot api to 10.1.4 (#1580)

* update polkadot api to 10.1.4

* update rest of dep

* [release] 20230329 (#1587)

* Multiple endpoints (#1551)

* allow multiple endpoints

* add response time based load balancing

* handle endpoint disconnects

* move fetchBlockBatches to apiService

* cleanup code

* move ConnectionPoolService to node-core

* fix order of module injection in indexermodule

* remove manually disconnecting api

* move api creation code to ApiConnection class

* code cleanup

* handle reconnection error

* change networkEndpoints to networkEndpoint

* change to use yarn build (#1589)

* test prerelease build (#1590)

* [SKIP CI] Prerelease

* [release] 20230330 (#1592)

* Upgrade `apollo-links` (#1593)

* [SKIP CI] Prerelease

* improve blockSize function (#1602)

* improve blockSize function

* add return type to blockSize

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* [release] 20230405 (#1603)

* Add evmChainId to metadata (#1607)

* [SKIP CI] Prerelease

* [release] 20230411 query (#1608)

* Update progress string (#1612)

* [SKIP CI] Prerelease

* Update the jsonField directive to have an option for disabling the index (#1613)

* Update the jsonField directive to have an option for disabling the index

* Clean up

* [SKIP CI] Prerelease

* [release] 20230414 utils (#1615)

* Fix workers fetching blocks out of order (#1616)

* Fix workers fetching blocks out of order

* Remove logging endpoints with api keys

* [SKIP CI] Prerelease

* [release] 20230417 (#1617)

* DB cache and various other improvements (#1561)

* store cache

* improve on flush cache

* draft storeCache service

* update

* minor change

* Updates

* add historical data to cache (#1557)

* add __block_range to creation

* add historical record to cache

* correct range

* Complete cached store methods (#1563)

* add __block_range to creation

add historical record to cache

correct range

cache store methods

start add tests

* fix set

* update

* WIP cache with workers support

* Fix up dynamic ds service, clean up

* Update store serialisation over workers

* Improve unfinalized blocks to work without tx

* Move all chain agnostic code from block dispatchers to node core

* Add comments about worker store conversion

* Fix rebase build issues

* Implement cache for metadata entity (#1565)

* Implement cache for metadata entity

* Clean up and exclude unfinalized blocks keys from metadata cache

* Clean up

* flush with cache stored records size (#1566)

* flush with cache stored records size

* update

* Lfu get cache (#1568)

* still need fix get/count logic

fix method for LFU getCache

* Update tests

* Fix and more tests

* Update base on review

* Fix

* Run all store operations through cache (#1567)

* Run all store operations through hash

* Log when cache flushed, flush cache on application shutdown

* Cache fixes

* Fix removeCache not being init

* POI cache fixes

* Fixes

* Fix caches being cleared after db operation and potential cache changes have happened

* Fix issues with injected store

* Clean up

* Fix tests from rebase

* Fix get cache not being used

* Rename function

* Fix reindex issues

* Fix casing of import

* Clean up, log if finalized is enabled

* parent 8b0b558 (#1574)

author Scott Twiname <skott.twiname@gmail.com> 1679452749 +1300
committer JQQQ <jiqiang90@hotmail.com> 1679461182 +1300

Clean up, log if finalized is enabled

Fix rebase build issues

parent 5e4ea5d
author Scott Twiname <skott.twiname@gmail.com> 1679013784 +1300
committer JQQQ <jiqiang90@hotmail.com> 1679460224 +1300

parent 5e4ea5d
author Scott Twiname <skott.twiname@gmail.com> 1679013784 +1300
committer JQQQ <jiqiang90@hotmail.com> 1679460210 +1300

parent 5e4ea5d
author Scott Twiname <skott.twiname@gmail.com> 1679013784 +1300
committer JQQQ <jiqiang90@hotmail.com> 1679460149 +1300

parent 5e4ea5d
author Scott Twiname <skott.twiname@gmail.com> 1679013784 +1300
committer JQQQ <jiqiang90@hotmail.com> 1679460101 +1300

parent 5e4ea5d
author Scott Twiname <skott.twiname@gmail.com> 1679013784 +1300
committer JQQQ <jiqiang90@hotmail.com> 1679460038 +1300

Run all store operations through hash

Log when cache flushed, flush cache on application shutdown

Fix removeCache not being init

Fixes

Fix issues with injected store

Clean up

Fix removeCache not being init

Fixes

Fix issues with injected store

Clean up

Fix tests from rebase

Fix get cache not being used

tidy up

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Update metadata when dictionary skips large number of blocks (#1577)

* Update metadata when dictionary skips large number of blocks

* Bring back option to log sql

* Dynamic cache threshold and metrics (#1578)

* Bring back fetch service tests

* Fix build issues

* Tidy up

* Fix workers historical disabled (#1579)

* Fix imports in node core

* improve close record sql, and add flag max size for getCache (#1583)

* improve close record sql, and add flag max size for getCache

* Fix

* escape id string

* Flush data at specific block heights (#1582)

* Flush data at specific block heights

* Add test

* Fix value fromBelowHeight being empty

* Fix mergedRecords potentially being empty

* Fix flushableRecordCounter

* Change log to debug

* Fix dictionary validation

* fix reindex storeCache missing EventEmitterModule (#1585)

* Make flushing cache configurable (#1586)

* Add an index to Id field on entities with historical (#1588)

* Fix tests

* Fix possible infinite loop with cache and fix test (#1591)

* Fix possible infinite loop with cache and fix test

* Add some comments

* Fix more tests

---------

Co-authored-by: JQQQ <jiqiang90@hotmail.com>

* Connect historical option up to cache model option (#1596)

* Only run delete query if there is data to delete

* Add support for non-historical indexing with cache (#1597)

* Add support for non-historical indexing with cache

* Fix api test

* remove constraintDeferrableQuery from hasOne

* fix createUniqueIndexQuery

---------

Co-authored-by: JQQQ <jiqiang90@hotmail.com>

* Cache and worker related fixes (#1598)

* fix index name too long issue (#1599)

* hash the indexes, except pk indexes

* handle existed indexes

* update

* Remove store count function (#1600)

* support cockroach db with non-historical (#1604)

* support cockroach db with non-historical

* Add more comments

* move fetchBlocks to node-core (#1611)

* move fetchBlocks to node-core

* code cleanup

* code cleanup

* remove extra args from fetchBlock

* Fix making data serializable for store bulkCreate and bulkUpdate (#1622)

* Improve big int error in workers (#1619)

* improve error handling in workers

* Make bigint stringifiable in worker and main threads

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* fix type error (#1623)

* fix type error

* cast type

* Flush store cache in operational order (#1614)

* Init

* better log message

* tidy up

* Tidy up and test

* Update packages/node/src/yargs.ts

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Update packages/node-core/src/indexer/storeCache/storeCache.service.ts

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* add benchmarking

* update benchmarking

* Tidy up

* test

* remove flag, use last flushed index, more tidy up

* make operationIndex required

* fix test

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* update default values

* fix missing indexes on cockroach db (#1624)

* fix getLatest() could be undefined

* Fix StoreOperation not pushing enums

---------

Co-authored-by: JQQQ <jiqiang90@hotmail.com>
Co-authored-by: Naveen V <velnaveen99@gmail.com>

* [SKIP CI] Prerelease

* Testing Framework (#1584)

* implement runTest

* check handler kind for running test

* implement testing service

* run tests in sandbox

* build test files through subql build

* run handlers in sandbox

* cleanup sandbox script and improve logging

* index data with `IndexerManager.indexData`

* use indexBlock() instead of indexData() for tests

* add getter _name to entities during codegen

* code cleanup

* improve logging

* improve logging

* bug fixes

* code cleanup

* improve logging

* indexBlock: get datasource from arguments

* move testingService to node-core

* code cleanup

* update ci

* code cleanup

* cleanup

* cleanup

* cleanup

* handle 0 test cases

* prevent calling getGqlType for 'contains' matcher (#1626)

* prevent calling getGqlType for 'contains' matcher

* get gqlType with ternary condition

* remove indexer module from testing module (#1628)

* remove indexer module from testing module

* cleanup

* cleanup

* Enable strict ts setting for node core (#1625)

* WIP enable strict ts setting for node core

* fix undefined in historicalMarkPreviousHeightRecordsBatch

* typo

* Update typescript, fix most strict related errors

* Strict changes for testing framework

* Runtime fixes

* change log level on cache poi

---------

Co-authored-by: JQQQ <jiqiang90@hotmail.com>

* fix prerelease (#1629)

* [SKIP CI] Prerelease

* [SKIPCI] fix testing publish order (#1630)

* fix missing @subql/testing in node-core (#1631)

* [SKIP CI] Prerelease

* manual prerelease for node-core fix (#1632)

* [SKIP CI] Prerelease

* Improve yargs descriptions (#1633)

* [SKIP CI] Prerelease

* Support tests from ./test and ./tests dirs, fix store blockheight issue

* Remove log

* [SKIP CI] Prerelease

* [release] 2.0.0 (#1627)

* release-2.0.0

* update

* update node changelog

* remove typescript from query dependency

* remove typescript from node dependency

* [SKIPCI] fix release

* [release] 2.0.0 republish (#1635)

* Fix ApiService abstract class, improve getting all DS (#1638)

* Fix ApiService abstract class, improve getting all DS

* Revert api import and minor test fixes

* [SKIP CI] Prerelease

* fix assertion error and pk index type (#1641)

* [SKIP CI] Prerelease

* fix workers crashes error (#1642)

* try to fix workers

* Fix dependency injection

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* WIP fix test part 1 (#1640)

* fix test part 1

* Update packages/node/src/indexer/fetch.service.ts

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* fix tests 20230426

* remove unused code

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* Clean up actions

* Update algosdk

* Fix tests

* Fix test chainId

* Remove local mode

* Remove unnecessary override of base method

* Remove irrelevant comment

---------

Co-authored-by: Jay Ji <jiqiang90@hotmail.com>
Co-authored-by: Ben <89335033+bz888@users.noreply.github.com>
Co-authored-by: Lachlan McCrae <8554309+lamcc21@users.noreply.github.com>
Co-authored-by: Lachlan McCrae <8554309+lachlan1m@users.noreply.github.com>
Co-authored-by: HZ <zhaihaohk@gmail.com>
Co-authored-by: MOZGIII <mike-n@narod.ru>
Co-authored-by: James Bayly <46693720+jamesbayly@users.noreply.github.com>
Co-authored-by: Mingyang Li <53138432+Mingyang-Li@users.noreply.github.com>
Co-authored-by: seandotau <seandotau@gmail.com>
Co-authored-by: Naveen V <velnaveen99@gmail.com>
Co-authored-by: Darwin Subramaniam <darwinsubramaniam@googlemail.com>
stwiname added a commit that referenced this pull request Jun 2, 2023
@guplersaxanoid @jiqiang90
@stwiname @MOZGIII @bz888 @lamcc21 @jamesbayly @Mingyang-Li @seandotau @darwinsubramaniam @mzxyz
Sync 20230512 algorand (#41)
40837c9 
* [release] 20221123 (#1431)

* [release] 20221123

* Update packages/query/CHANGELOG.md

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Update packages/node-core/CHANGELOG.md

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Update packages/node/CHANGELOG.md

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Update packages/node-core/CHANGELOG.md

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Update packages/validator/CHANGELOG.md

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Add a way to pass settings to the graphql playground (#1436)

* Add a way to pass settings to the graphql playground

* Add a changelog entry

* [SKIP CI] Prerelease

* Fix the changelog entry for the last PR (#1437)

* [SKIP CI] Prerelease

* fix enum name missing issue (#1441)

* fix enum name missing issue

* tidy up

* [SKIP CI] Prerelease

* hash sql queries (#1438)

* hashed sql

* add comments

* add comment for hashName function

* update function

* update hashName function

* [SKIP CI] Prerelease

* [release] 20221130 (#1442)

* [release] 20221130

* [release] 20221130

* Add --file flag for codegen (#1446)

* init

* change flag behaviour so -f can overwrite -l

* update getManifestPath method, fix typo

* update argument description

* update variable names

* update arg description

* add error message on bad --file path

Co-authored-by: Lachlan McCrae <8554309+lachlan1m@users.noreply.github.com>

* [SKIP CI] Prerelease

* add try and catch for hsr (#1449)

* [SKIP CI] Prerelease

* Bypass blocks (#1435)

* wip

* manifest error

* manfiest passed

* added logging for bypassing blocks

* sync main, moved bypassBlocks under networks

* pass all tests

* mock dictioanry case

* fix metadata

* clean up

* allow range

* bypass working, needs more testing

* wip on bypass with setting new bufferHeight

* working

* added runtime test

* add test for latestBufferHeight (blocked)

* removed unneeded test, improve logic

* clean up

* add comments

* add comments for latestBufferHeight reasoning

* update logic

* refactor

* update logic

* refactor

* [SKIP CI] Prerelease

* [release] 20221205 (#1448)

* [release] 20221205

* [release] 20221205

* [release] 20221206

* details for bypassBlocks

* Remove unused RuntimeService from indexer module, it had missing dependencies (#1453)

* [SKIP CI] Prerelease

* Fix/subcommands (#1451)

* updated forceClean

* update reindex

* [SKIP CI] Prerelease

* Enable env vars parsing at node and query commands (#1452)

* [SKIP CI] Prerelease

* Workers: Fix SequelizeDatabaseError - tuple concurrently updated (#1458)

* add flag for hot-schema reload on node, ensure query is on main thread

* remove flags

* [SKIP CI] Prerelease

* Add start height to project (#1456)

* [SKIP CI] Prerelease

* Fix poi offset is 0 (#1459)

* [SKIP CI] Prerelease

* Handle when fields got undefined with historical (#1463)

* [SKIP CI] Prerelease

* update deploy cli (#1460)

* [SKIP CI] Prerelease

* exit on fail (#1464)

* [SKIP CI] Prerelease

* fix parentSpecVersion could be undefined (#1467)

* [SKIP CI] Prerelease

* [release] 20221219 (#1466)

* [release] 20221219

* update

* Update packages/node-core/CHANGELOG.md

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Update packages/node/CHANGELOG.md

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Update packages/node-core/CHANGELOG.md

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Update packages/cli/CHANGELOG.md

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* update

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Fix templates with other sdks, make -f flag work like other commands (#1470)

* Fix templates with other sdks, make -f flag work like other commands

* Fix for sdks with first class smart contracts

* [SKIP CI] Prerelease

* Tweak error message for genesis hash (#1471)

* [SKIP CI] Prerelease

* Fix trigger function (#1469)

* fix trigger and function for subscription and hot schema reload

* update

* [SKIP CI] Prerelease

* [release] 20221222 (#1474)

* validate dictionary with start height (#1473)

* validate dictionary with start height

* Update packages/node/src/indexer/fetch.service.ts

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* add count for entity (#1480)

* add count for entity

* Update packages/node-core/src/indexer/store.service.ts

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* Fixing readme doc for @subql/query (#1485)

* doc: correcting grammer

* fix: typo

* [SKIP CI] Prerelease

* update polkadot api to 9.11.1 (#1483)

* update polkadot api to 9.11.1

* update

* [SKIP CI] Prerelease

* add connection filters (#1484)

* [SKIP CI] Prerelease

* fix workers missing set last buffered height from empty dictionary (#1492)

* [SKIP CI] Prerelease

* Fix custom metadata keys not being applied (#1496)

* add block hash validation after fetch (#1494)

* add block hash validation after fetch

* add tests

* skip fetch test

* Update log

* [SKIP CI] Prerelease

* [release] 20230123 (#1497)

* Add support for near projects to the CLI (#1498)

* Add support for near projects to the CLI

* Update common near version

* Update near packages

* [SKIP CI] Prerelease

* Arranges commands in alphabetical order (#1495)

* [SKIP CI] Prerelease

* user process.env.USERPROFILE for windows platform (#1502)

* [SKIP CI] Prerelease

* add blockTime to NodeConfig (#1501)

* [SKIP CI] Prerelease

* [release] 20230131 (#1503)

* Worker use runtime service (#1491)

* draft

* update

* tidy up

* update

* working worker runtimes

* into base class

* clean up

* improve base on review

* [SKIP CI] Prerelease

* Check queue for lower blocks before processing fetched blocks (#1509)

* [SKIP CI] Prerelease

* Support array type in dictionary queries (#1510)

* [SKIP CI] Prerelease

* add flag query-limit (#1514)

* [SKIP CI] Prerelease

* Fix error with tempDsRecords being undefined (#1516)

* Fix error with tempDsRecords being undefined

* Fix schema not being used for metadata, sync dynamic-ds across workers

* Be smarter about reloading dynamic ds in workers

* [SKIP CI] Prerelease

* Update postgraphile enable table partitions (#1520)

* [SKIP CI] Prerelease

* `dictionary-optimisation` flag  (#1519)

* add flag, to orderby blockheight instead of pk

* add comment

* [SKIP CI] Prerelease

* [WIP] Feature/adding postgres ssl connection option (#1513)

* enable ssl option postgres

* Check DB_SSL Env Value , default is false

* adding ENV to pass in the Path to certificates

* adding SSL option for query pacakge.

* use args to pass the postgres ssl informations.

* load file content moved to comman package.

* Update packages/node-core/src/configure/NodeConfig.ts

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* Support Cockroach db (#1521)

* support cockroach db

* Update comment

* [SKIP CI] Prerelease

* ensure pgpool connection work with ssl (#1525)

* [SKIP CI] Prerelease

* Fix issue when use cockroach query service not support pgPartition (#1526)

* [SKIP CI] Prerelease

* [release] 20230221 (#1528)

* fix issue store getByField limit could potentially excess config limit (#1529)

* fix issue store getByField limit could potentially excess config limit

* Update packages/node-core/src/indexer/store.service.ts

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* typo

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* Abi codegen (#1532)

* draft

* fix abi name, add support for transaction

* Update packages/cli/src/template/abi-interface.ts.ejs

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* fix

* fix dependencies

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* Move enum under schema (#1527)

* need fix query get enum

* fix enum query, clean all enums

* add logic to handle original enum type

* fix check exclude constrain check missing schema and table (#1537)

* [SKIP CI] Prerelease

* manual prerelease node for node-core update (#1538)

* [SKIP CI] Prerelease

* resolution for sequelize and prerelease node (#1539)

* [SKIP CI] Prerelease

* fixed the version of sequelize and prerelease node (#1540)

* [SKIP CI] Prerelease

* improve force clean with deprecated enum (#1541)

* [SKIP CI] Prerelease

* Remove exclude constraint (#1543)

* deprecate exclude contraint

* remove unused

* [SKIP CI] Prerelease

* [release] 20230306 (#1544)

* [release] 20230307 (#1545)

* bump yarn.lock, update imports (#1546)

* [SKIP CI] Prerelease

* [release] 20230307 (#1547)

* increase wsProvider timeout (#1550)

* increase wsProvider timeout

* remove .mmr dir

* [SKIP CI] Prerelease

* Use performance.now instead of date for profiler. (#1549)

* add SmartBatchService and BlockSizeBuffer (#1506)

* add SmartBatchService and BlockSizeBuffer

* check memory usage before fetching a batch

* use v8 heap usage statistics

* set block to null after indexing

* extend smart batch service to workers

* make smart batch size injectible

* rebase

* bug fixes

* code cleanup

* assign workers based on free memory

* fix liniting

* cleanup code

* cleanup logs

* clear blocksizebuffer inside smartbatchservice

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Fix build issue with profiler

* [SKIP CI] Prerelease

* fix abi codegen runtime validation (#1552)

* [SKIP CI] Prerelease

* [release] 20230309 Cli (#1553)

* Fix enum under schema not being escaped (#1555)

* [SKIP CI] Prerelease

* rename `--sponsored-dictionary` to `--dictionary-resolver` (#1559)

* renamed flag

* add missing tpyo

* [SKIP CI] Prerelease

* handle bigint in json.stringify (#1562)

* [SKIP CI] Prerelease

* add validation for `codegen`, grapql conflicts (#1564)

* add test for codegen with reserved keys

* wip validateEntityName fn

* update test, update logc

* remove logs

* [SKIP CI] Prerelease

* Fix codegen when abi is not valid (#1570)

* Fix codegen when abi is not valid

* Update packages/cli/src/controller/codegen-controller.ts

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* Fix block offset been reset to 1 (#1571)

* [SKIP CI] Prerelease

* remove blocking in process queueing (#1572)

* remove blocking in process queueing

* redefine locking pattern

* fix fetchedBlocks status

* [SKIP CI] Prerelease

* improve api error handling (#1576)

* [SKIP CI] Prerelease

* generate sourcemap for projects (#1569)

* improve error handling

* use inline-source-map

* revert api error handling

* beautify minified output from webpack

* code cleanup

* move terser plugin dependency to cli

* log warnings for unresolved stack trace

* get entry file from datasource

* alter convertStack prototype

* add ES2020 to lib compiler option

* update polkadot api to 10.1.4 (#1580)

* update polkadot api to 10.1.4

* update rest of dep

* [release] 20230329 (#1587)

* Multiple endpoints (#1551)

* allow multiple endpoints

* add response time based load balancing

* handle endpoint disconnects

* move fetchBlockBatches to apiService

* cleanup code

* move ConnectionPoolService to node-core

* fix order of module injection in indexermodule

* remove manually disconnecting api

* move api creation code to ApiConnection class

* code cleanup

* handle reconnection error

* change networkEndpoints to networkEndpoint

* change to use yarn build (#1589)

* test prerelease build (#1590)

* [SKIP CI] Prerelease

* [release] 20230330 (#1592)

* Upgrade `apollo-links` (#1593)

* [SKIP CI] Prerelease

* improve blockSize function (#1602)

* improve blockSize function

* add return type to blockSize

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* [release] 20230405 (#1603)

* Add evmChainId to metadata (#1607)

* [SKIP CI] Prerelease

* [release] 20230411 query (#1608)

* Update progress string (#1612)

* [SKIP CI] Prerelease

* Update the jsonField directive to have an option for disabling the index (#1613)

* Update the jsonField directive to have an option for disabling the index

* Clean up

* [SKIP CI] Prerelease

* [release] 20230414 utils (#1615)

* Fix workers fetching blocks out of order (#1616)

* Fix workers fetching blocks out of order

* Remove logging endpoints with api keys

* [SKIP CI] Prerelease

* [release] 20230417 (#1617)

* DB cache and various other improvements (#1561)

* store cache

* improve on flush cache

* draft storeCache service

* update

* minor change

* Updates

* add historical data to cache (#1557)

* add __block_range to creation

* add historical record to cache

* correct range

* Complete cached store methods (#1563)

* add __block_range to creation

add historical record to cache

correct range

cache store methods

start add tests

* fix set

* update

* WIP cache with workers support

* Fix up dynamic ds service, clean up

* Update store serialisation over workers

* Improve unfinalized blocks to work without tx

* Move all chain agnostic code from block dispatchers to node core

* Add comments about worker store conversion

* Fix rebase build issues

* Implement cache for metadata entity (#1565)

* Implement cache for metadata entity

* Clean up and exclude unfinalized blocks keys from metadata cache

* Clean up

* flush with cache stored records size (#1566)

* flush with cache stored records size

* update

* Lfu get cache (#1568)

* still need fix get/count logic

fix method for LFU getCache

* Update tests

* Fix and more tests

* Update base on review

* Fix

* Run all store operations through cache (#1567)

* Run all store operations through hash

* Log when cache flushed, flush cache on application shutdown

* Cache fixes

* Fix removeCache not being init

* POI cache fixes

* Fixes

* Fix caches being cleared after db operation and potential cache changes have happened

* Fix issues with injected store

* Clean up

* Fix tests from rebase

* Fix get cache not being used

* Rename function

* Fix reindex issues

* Fix casing of import

* Clean up, log if finalized is enabled

* parent 8b0b558 (#1574)

author Scott Twiname <skott.twiname@gmail.com> 1679452749 +1300
committer JQQQ <jiqiang90@hotmail.com> 1679461182 +1300

Clean up, log if finalized is enabled

Fix rebase build issues

parent 5e4ea5d
author Scott Twiname <skott.twiname@gmail.com> 1679013784 +1300
committer JQQQ <jiqiang90@hotmail.com> 1679460224 +1300

parent 5e4ea5d
author Scott Twiname <skott.twiname@gmail.com> 1679013784 +1300
committer JQQQ <jiqiang90@hotmail.com> 1679460210 +1300

parent 5e4ea5d
author Scott Twiname <skott.twiname@gmail.com> 1679013784 +1300
committer JQQQ <jiqiang90@hotmail.com> 1679460149 +1300

parent 5e4ea5d
author Scott Twiname <skott.twiname@gmail.com> 1679013784 +1300
committer JQQQ <jiqiang90@hotmail.com> 1679460101 +1300

parent 5e4ea5d
author Scott Twiname <skott.twiname@gmail.com> 1679013784 +1300
committer JQQQ <jiqiang90@hotmail.com> 1679460038 +1300

Run all store operations through hash

Log when cache flushed, flush cache on application shutdown

Fix removeCache not being init

Fixes

Fix issues with injected store

Clean up

Fix removeCache not being init

Fixes

Fix issues with injected store

Clean up

Fix tests from rebase

Fix get cache not being used

tidy up

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Update metadata when dictionary skips large number of blocks (#1577)

* Update metadata when dictionary skips large number of blocks

* Bring back option to log sql

* Dynamic cache threshold and metrics (#1578)

* Bring back fetch service tests

* Fix build issues

* Tidy up

* Fix workers historical disabled (#1579)

* Fix imports in node core

* improve close record sql, and add flag max size for getCache (#1583)

* improve close record sql, and add flag max size for getCache

* Fix

* escape id string

* Flush data at specific block heights (#1582)

* Flush data at specific block heights

* Add test

* Fix value fromBelowHeight being empty

* Fix mergedRecords potentially being empty

* Fix flushableRecordCounter

* Change log to debug

* Fix dictionary validation

* fix reindex storeCache missing EventEmitterModule (#1585)

* Make flushing cache configurable (#1586)

* Add an index to Id field on entities with historical (#1588)

* Fix tests

* Fix possible infinite loop with cache and fix test (#1591)

* Fix possible infinite loop with cache and fix test

* Add some comments

* Fix more tests

---------

Co-authored-by: JQQQ <jiqiang90@hotmail.com>

* Connect historical option up to cache model option (#1596)

* Only run delete query if there is data to delete

* Add support for non-historical indexing with cache (#1597)

* Add support for non-historical indexing with cache

* Fix api test

* remove constraintDeferrableQuery from hasOne

* fix createUniqueIndexQuery

---------

Co-authored-by: JQQQ <jiqiang90@hotmail.com>

* Cache and worker related fixes (#1598)

* fix index name too long issue (#1599)

* hash the indexes, except pk indexes

* handle existed indexes

* update

* Remove store count function (#1600)

* support cockroach db with non-historical (#1604)

* support cockroach db with non-historical

* Add more comments

* move fetchBlocks to node-core (#1611)

* move fetchBlocks to node-core

* code cleanup

* code cleanup

* remove extra args from fetchBlock

* Fix making data serializable for store bulkCreate and bulkUpdate (#1622)

* Improve big int error in workers (#1619)

* improve error handling in workers

* Make bigint stringifiable in worker and main threads

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* fix type error (#1623)

* fix type error

* cast type

* Flush store cache in operational order (#1614)

* Init

* better log message

* tidy up

* Tidy up and test

* Update packages/node/src/yargs.ts

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Update packages/node-core/src/indexer/storeCache/storeCache.service.ts

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* add benchmarking

* update benchmarking

* Tidy up

* test

* remove flag, use last flushed index, more tidy up

* make operationIndex required

* fix test

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* update default values

* fix missing indexes on cockroach db (#1624)

* fix getLatest() could be undefined

* Fix StoreOperation not pushing enums

---------

Co-authored-by: JQQQ <jiqiang90@hotmail.com>
Co-authored-by: Naveen V <velnaveen99@gmail.com>

* [SKIP CI] Prerelease

* Testing Framework (#1584)

* implement runTest

* check handler kind for running test

* implement testing service

* run tests in sandbox

* build test files through subql build

* run handlers in sandbox

* cleanup sandbox script and improve logging

* index data with `IndexerManager.indexData`

* use indexBlock() instead of indexData() for tests

* add getter _name to entities during codegen

* code cleanup

* improve logging

* improve logging

* bug fixes

* code cleanup

* improve logging

* indexBlock: get datasource from arguments

* move testingService to node-core

* code cleanup

* update ci

* code cleanup

* cleanup

* cleanup

* cleanup

* handle 0 test cases

* prevent calling getGqlType for 'contains' matcher (#1626)

* prevent calling getGqlType for 'contains' matcher

* get gqlType with ternary condition

* remove indexer module from testing module (#1628)

* remove indexer module from testing module

* cleanup

* cleanup

* Enable strict ts setting for node core (#1625)

* WIP enable strict ts setting for node core

* fix undefined in historicalMarkPreviousHeightRecordsBatch

* typo

* Update typescript, fix most strict related errors

* Strict changes for testing framework

* Runtime fixes

* change log level on cache poi

---------

Co-authored-by: JQQQ <jiqiang90@hotmail.com>

* fix prerelease (#1629)

* [SKIP CI] Prerelease

* [SKIPCI] fix testing publish order (#1630)

* fix missing @subql/testing in node-core (#1631)

* [SKIP CI] Prerelease

* manual prerelease for node-core fix (#1632)

* [SKIP CI] Prerelease

* Improve yargs descriptions (#1633)

* [SKIP CI] Prerelease

* Support tests from ./test and ./tests dirs, fix store blockheight issue

* Remove log

* [SKIP CI] Prerelease

* [release] 2.0.0 (#1627)

* release-2.0.0

* update

* update node changelog

* remove typescript from query dependency

* remove typescript from node dependency

* [SKIPCI] fix release

* [release] 2.0.0 republish (#1635)

* Fix ApiService abstract class, improve getting all DS (#1638)

* Fix ApiService abstract class, improve getting all DS

* Revert api import and minor test fixes

* [SKIP CI] Prerelease

* fix assertion error and pk index type (#1641)

* [SKIP CI] Prerelease

* fix workers crashes error (#1642)

* try to fix workers

* Fix dependency injection

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* WIP fix test part 1 (#1640)

* fix test part 1

* Update packages/node/src/indexer/fetch.service.ts

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* fix tests 20230426

* remove unused code

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* fix missing esModuleInterop (#1644)

* [SKIP CI] Prerelease

* Force flush cache with reindex command (#1645)

* [SKIP CI] Prerelease

* [release] 20230427 (#1646)

* [release] 20230427

* improve log

* Fix api not being defined, fix not using filter on datasources (#1647)

* [SKIP CI] Prerelease

* deprecate local mode (#1648)

* [SKIP CI] Prerelease

* [release] 20230427 patch (#1649)

* Add missing node-fetch dependency to lock down version (#1651)

* [SKIP CI] Prerelease

* postgres based mmr DB (#1618)

* postgres based mmr DB

* add mmr db store option to cli

* add mmr migrate

* bug fixes

* alter file based names

* make migration bidirectional

* cleanup and improve logging

* remove localMode and add mmr-path to args

* remove check for localMode in createProjectSchema

* improve error handling

* improve error handling for postgres mmr table

* improve error handling for mmr indexer

* Fix test, create db table same as other tables, tidy up

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* Extract ProjectService, FetchService and IndexerManager to base classes (#1643)

* Move bulk of project service logic to node core

* Allow overriding buildDictQueryFragment

* Improve visibility of dependencies on project service

* Extract base class from fetch service

* Fix error message

* Fix tests

* Create indexer manager base class

* [SKIP CI] Prerelease

* Move polkadot util imports to utils package, fix api type (#1653)

* Move polkadot util imports to utils package, fix api type

* remove duplicate log

---------

Co-authored-by: JQQQ <jiqiang90@hotmail.com>

* [SKIP CI] Prerelease

* manifest node with options (#1652)

* manifest node with options

* fix

* fix self import

* remove

* Use single version of class-* packages, extract base node-runner

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* POI cache fixes (#1660)

* Add tests for cache poi and fix related issuess

* Add mutex around cachePoi

* [SKIP CI] Prerelease

* Add bulkRemove function to store (#1666)

* [SKIP CI] Prerelease

* fix postMmrDb.get and mmrMigrate node sort (#1669)

* [SKIP CI] Prerelease

* Fix indexed column check with historical (#1667)

* [SKIP CI] Prerelease

* Regenerate mmr (#1664)

* preparation

Unsafe mode and draft

bulkUpsert with poi mmr

move service to node-core

* tidy up, fix tests

* Fix gap mmr node missing issue, poiCache use plainPoiModel

* [SKIP CI] Prerelease

* Node core ds processor (#1658)

* Move ds processor to node core

* Clean up deps

* Move forceClean and meta to node-core

* Move configure logic to node core

* Move unfinalzied blocks to node-core

* Add missing logger

* Fix tests

* [SKIP CI] Prerelease

* Add interval to flush the store cache (#1670)

* Add interval to flush the store cache

* Fix infinite loop in nodeConfig

* [SKIP CI] Prerelease

* Remove support for manifest version <1.0.0 (#1659)

* Clean up deps

* Move forceClean and meta to node-core

* Remove support for manifest version <1.0.0

* Fix build issues

* Move sandbox back from node-core

* Move gauge providers into node core

* Tidy up indexer module

* Move FileType to core

* Remove tests for unsupported manifest versions

* [SKIP CI] Prerelease

* Fix Cache missing remove (#1675)

* Fix missing removeCache from flush

* tidy up

* [SKIP CI] Prerelease

* [release] 20230510 (#1673)

* [release] 20230510

* Update package.json

* Update node-core changelog

* Fix missing scheduler subcommands (#1677)

* Fix mmrRegen module missing scheduler

* Fix subcommands module missing scheduler

* [SKIP CI] Prerelease

* Extract dictionary meta validation so it can be overridden (#1679)

* Extract dictionary meta validation so it can be overridden

* Update type

* [SKIP CI] Prerelease

* [release] 20230511 (#1678)

* [release] 20230511

* Update node-core

* [release] 20230511 types (#1680)

* Fix metadata check, allow base indexer manager to parse abis with ethereum (#1682)

* [SKIP CI] Prerelease

* Move validate function to common (#1683)

* Move validate function to common

* Make prepareFilteredData async

* [SKIP CI] Prerelease

* Inject the chain id into sandboxes (#1684)

* [SKIP CI] Prerelease

* [release] 20230511 v2 (#1685)

* Fix flush init before metadata repo set (#1688)

* [SKIP CI] Prerelease

* [release] 20230512 (#1690)

* cleanup

---------

Co-authored-by: Jay Ji <jiqiang90@hotmail.com>
Co-authored-by: Scott Twiname <skott.twiname@gmail.com>
Co-authored-by: MOZGIII <mike-n@narod.ru>
Co-authored-by: Ben <89335033+bz888@users.noreply.github.com>
Co-authored-by: Lachlan McCrae <8554309+lamcc21@users.noreply.github.com>
Co-authored-by: Lachlan McCrae <8554309+lachlan1m@users.noreply.github.com>
Co-authored-by: James Bayly <46693720+jamesbayly@users.noreply.github.com>
Co-authored-by: Mingyang Li <53138432+Mingyang-Li@users.noreply.github.com>
Co-authored-by: seandotau <seandotau@gmail.com>
Co-authored-by: Darwin Subramaniam <darwinsubramaniam@googlemail.com>
Co-authored-by: HZHAIX <zhaihaohk@gmail.com>
stwiname added a commit that referenced this pull request Jun 27, 2023
@guplersaxanoid @stwiname
@seandotau @jiqiang90 @bz888 @darwinsubramaniam
Rebase 20230313 (#22)
5cd75d0 
* Add support for near projects to the CLI (#1498)

* Add support for near projects to the CLI

* Update common near version

* Update near packages

* [SKIP CI] Prerelease

* Arranges commands in alphabetical order (#1495)

* [SKIP CI] Prerelease

* user process.env.USERPROFILE for windows platform (#1502)

* [SKIP CI] Prerelease

* add blockTime to NodeConfig (#1501)

* [SKIP CI] Prerelease

* [release] 20230131 (#1503)

* Worker use runtime service (#1491)

* draft

* update

* tidy up

* update

* working worker runtimes

* into base class

* clean up

* improve base on review

* [SKIP CI] Prerelease

* Check queue for lower blocks before processing fetched blocks (#1509)

* [SKIP CI] Prerelease

* Support array type in dictionary queries (#1510)

* [SKIP CI] Prerelease

* add flag query-limit (#1514)

* [SKIP CI] Prerelease

* Fix error with tempDsRecords being undefined (#1516)

* Fix error with tempDsRecords being undefined

* Fix schema not being used for metadata, sync dynamic-ds across workers

* Be smarter about reloading dynamic ds in workers

* [SKIP CI] Prerelease

* Update postgraphile enable table partitions (#1520)

* [SKIP CI] Prerelease

* `dictionary-optimisation` flag  (#1519)

* add flag, to orderby blockheight instead of pk

* add comment

* [SKIP CI] Prerelease

* [WIP] Feature/adding postgres ssl connection option (#1513)

* enable ssl option postgres

* Check DB_SSL Env Value , default is false

* adding ENV to pass in the Path to certificates

* adding SSL option for query pacakge.

* use args to pass the postgres ssl informations.

* load file content moved to comman package.

* Update packages/node-core/src/configure/NodeConfig.ts

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* Support Cockroach db (#1521)

* support cockroach db

* Update comment

* [SKIP CI] Prerelease

* ensure pgpool connection work with ssl (#1525)

* [SKIP CI] Prerelease

* Fix issue when use cockroach query service not support pgPartition (#1526)

* [SKIP CI] Prerelease

* [release] 20230221 (#1528)

* fix issue store getByField limit could potentially excess config limit (#1529)

* fix issue store getByField limit could potentially excess config limit

* Update packages/node-core/src/indexer/store.service.ts

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* typo

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* Abi codegen (#1532)

* draft

* fix abi name, add support for transaction

* Update packages/cli/src/template/abi-interface.ts.ejs

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* fix

* fix dependencies

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* Move enum under schema (#1527)

* need fix query get enum

* fix enum query, clean all enums

* add logic to handle original enum type

* fix check exclude constrain check missing schema and table (#1537)

* [SKIP CI] Prerelease

* manual prerelease node for node-core update (#1538)

* [SKIP CI] Prerelease

* resolution for sequelize and prerelease node (#1539)

* [SKIP CI] Prerelease

* fixed the version of sequelize and prerelease node (#1540)

* [SKIP CI] Prerelease

* improve force clean with deprecated enum (#1541)

* [SKIP CI] Prerelease

* Remove exclude constraint (#1543)

* deprecate exclude contraint

* remove unused

* [SKIP CI] Prerelease

* [release] 20230306 (#1544)

* [release] 20230307 (#1545)

* bump yarn.lock, update imports (#1546)

* [SKIP CI] Prerelease

* [release] 20230307 (#1547)

* increase wsProvider timeout (#1550)

* increase wsProvider timeout

* remove .mmr dir

* [SKIP CI] Prerelease

* Use performance.now instead of date for profiler. (#1549)

* add SmartBatchService and BlockSizeBuffer (#1506)

* add SmartBatchService and BlockSizeBuffer

* check memory usage before fetching a batch

* use v8 heap usage statistics

* set block to null after indexing

* extend smart batch service to workers

* make smart batch size injectible

* rebase

* bug fixes

* code cleanup

* assign workers based on free memory

* fix liniting

* cleanup code

* cleanup logs

* clear blocksizebuffer inside smartbatchservice

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Fix build issue with profiler

* [SKIP CI] Prerelease

* fix abi codegen runtime validation (#1552)

* [SKIP CI] Prerelease

* [release] 20230309 Cli (#1553)

* Fix enum under schema not being escaped (#1555)

* [SKIP CI] Prerelease

* rename `--sponsored-dictionary` to `--dictionary-resolver` (#1559)

* renamed flag

* add missing tpyo

* [SKIP CI] Prerelease

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>
Co-authored-by: seandotau <seandotau@gmail.com>
Co-authored-by: Jay Ji <jiqiang90@hotmail.com>
Co-authored-by: Ben <89335033+bz888@users.noreply.github.com>
Co-authored-by: Darwin Subramaniam <darwinsubramaniam@googlemail.com>
stwiname added a commit that referenced this pull request Jun 27, 2023
@guplersaxanoid @stwiname
@seandotau @jiqiang90 @bz888 @darwinsubramaniam @mzxyz @jamesbayly
Rebase 2.0 - Near (#30)
e91da2d 
* Add support for near projects to the CLI (#1498)

* Add support for near projects to the CLI

* Update common near version

* Update near packages

* [SKIP CI] Prerelease

* Arranges commands in alphabetical order (#1495)

* [SKIP CI] Prerelease

* user process.env.USERPROFILE for windows platform (#1502)

* [SKIP CI] Prerelease

* add blockTime to NodeConfig (#1501)

* [SKIP CI] Prerelease

* [release] 20230131 (#1503)

* Worker use runtime service (#1491)

* draft

* update

* tidy up

* update

* working worker runtimes

* into base class

* clean up

* improve base on review

* [SKIP CI] Prerelease

* Check queue for lower blocks before processing fetched blocks (#1509)

* [SKIP CI] Prerelease

* Support array type in dictionary queries (#1510)

* [SKIP CI] Prerelease

* add flag query-limit (#1514)

* [SKIP CI] Prerelease

* Fix error with tempDsRecords being undefined (#1516)

* Fix error with tempDsRecords being undefined

* Fix schema not being used for metadata, sync dynamic-ds across workers

* Be smarter about reloading dynamic ds in workers

* [SKIP CI] Prerelease

* Update postgraphile enable table partitions (#1520)

* [SKIP CI] Prerelease

* `dictionary-optimisation` flag  (#1519)

* add flag, to orderby blockheight instead of pk

* add comment

* [SKIP CI] Prerelease

* [WIP] Feature/adding postgres ssl connection option (#1513)

* enable ssl option postgres

* Check DB_SSL Env Value , default is false

* adding ENV to pass in the Path to certificates

* adding SSL option for query pacakge.

* use args to pass the postgres ssl informations.

* load file content moved to comman package.

* Update packages/node-core/src/configure/NodeConfig.ts

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* Support Cockroach db (#1521)

* support cockroach db

* Update comment

* [SKIP CI] Prerelease

* ensure pgpool connection work with ssl (#1525)

* [SKIP CI] Prerelease

* Fix issue when use cockroach query service not support pgPartition (#1526)

* [SKIP CI] Prerelease

* [release] 20230221 (#1528)

* fix issue store getByField limit could potentially excess config limit (#1529)

* fix issue store getByField limit could potentially excess config limit

* Update packages/node-core/src/indexer/store.service.ts

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* typo

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* Abi codegen (#1532)

* draft

* fix abi name, add support for transaction

* Update packages/cli/src/template/abi-interface.ts.ejs

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* fix

* fix dependencies

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* Move enum under schema (#1527)

* need fix query get enum

* fix enum query, clean all enums

* add logic to handle original enum type

* fix check exclude constrain check missing schema and table (#1537)

* [SKIP CI] Prerelease

* manual prerelease node for node-core update (#1538)

* [SKIP CI] Prerelease

* resolution for sequelize and prerelease node (#1539)

* [SKIP CI] Prerelease

* fixed the version of sequelize and prerelease node (#1540)

* [SKIP CI] Prerelease

* improve force clean with deprecated enum (#1541)

* [SKIP CI] Prerelease

* Remove exclude constraint (#1543)

* deprecate exclude contraint

* remove unused

* [SKIP CI] Prerelease

* [release] 20230306 (#1544)

* [release] 20230307 (#1545)

* bump yarn.lock, update imports (#1546)

* [SKIP CI] Prerelease

* [release] 20230307 (#1547)

* increase wsProvider timeout (#1550)

* increase wsProvider timeout

* remove .mmr dir

* [SKIP CI] Prerelease

* Use performance.now instead of date for profiler. (#1549)

* add SmartBatchService and BlockSizeBuffer (#1506)

* add SmartBatchService and BlockSizeBuffer

* check memory usage before fetching a batch

* use v8 heap usage statistics

* set block to null after indexing

* extend smart batch service to workers

* make smart batch size injectible

* rebase

* bug fixes

* code cleanup

* assign workers based on free memory

* fix liniting

* cleanup code

* cleanup logs

* clear blocksizebuffer inside smartbatchservice

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Fix build issue with profiler

* [SKIP CI] Prerelease

* fix abi codegen runtime validation (#1552)

* [SKIP CI] Prerelease

* [release] 20230309 Cli (#1553)

* Fix enum under schema not being escaped (#1555)

* [SKIP CI] Prerelease

* rename `--sponsored-dictionary` to `--dictionary-resolver` (#1559)

* renamed flag

* add missing tpyo

* [SKIP CI] Prerelease

* handle bigint in json.stringify (#1562)

* [SKIP CI] Prerelease

* add validation for `codegen`, grapql conflicts (#1564)

* add test for codegen with reserved keys

* wip validateEntityName fn

* update test, update logc

* remove logs

* [SKIP CI] Prerelease

* Fix codegen when abi is not valid (#1570)

* Fix codegen when abi is not valid

* Update packages/cli/src/controller/codegen-controller.ts

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* Fix block offset been reset to 1 (#1571)

* [SKIP CI] Prerelease

* remove blocking in process queueing (#1572)

* remove blocking in process queueing

* redefine locking pattern

* fix fetchedBlocks status

* [SKIP CI] Prerelease

* improve api error handling (#1576)

* [SKIP CI] Prerelease

* generate sourcemap for projects (#1569)

* improve error handling

* use inline-source-map

* revert api error handling

* beautify minified output from webpack

* code cleanup

* move terser plugin dependency to cli

* log warnings for unresolved stack trace

* get entry file from datasource

* alter convertStack prototype

* add ES2020 to lib compiler option

* update polkadot api to 10.1.4 (#1580)

* update polkadot api to 10.1.4

* update rest of dep

* [release] 20230329 (#1587)

* Multiple endpoints (#1551)

* allow multiple endpoints

* add response time based load balancing

* handle endpoint disconnects

* move fetchBlockBatches to apiService

* cleanup code

* move ConnectionPoolService to node-core

* fix order of module injection in indexermodule

* remove manually disconnecting api

* move api creation code to ApiConnection class

* code cleanup

* handle reconnection error

* change networkEndpoints to networkEndpoint

* change to use yarn build (#1589)

* test prerelease build (#1590)

* [SKIP CI] Prerelease

* [release] 20230330 (#1592)

* Upgrade `apollo-links` (#1593)

* [SKIP CI] Prerelease

* improve blockSize function (#1602)

* improve blockSize function

* add return type to blockSize

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* [release] 20230405 (#1603)

* Add evmChainId to metadata (#1607)

* [SKIP CI] Prerelease

* [release] 20230411 query (#1608)

* Update progress string (#1612)

* [SKIP CI] Prerelease

* Update the jsonField directive to have an option for disabling the index (#1613)

* Update the jsonField directive to have an option for disabling the index

* Clean up

* [SKIP CI] Prerelease

* [release] 20230414 utils (#1615)

* Fix workers fetching blocks out of order (#1616)

* Fix workers fetching blocks out of order

* Remove logging endpoints with api keys

* [SKIP CI] Prerelease

* [release] 20230417 (#1617)

* DB cache and various other improvements (#1561)

* store cache

* improve on flush cache

* draft storeCache service

* update

* minor change

* Updates

* add historical data to cache (#1557)

* add __block_range to creation

* add historical record to cache

* correct range

* Complete cached store methods (#1563)

* add __block_range to creation

add historical record to cache

correct range

cache store methods

start add tests

* fix set

* update

* WIP cache with workers support

* Fix up dynamic ds service, clean up

* Update store serialisation over workers

* Improve unfinalized blocks to work without tx

* Move all chain agnostic code from block dispatchers to node core

* Add comments about worker store conversion

* Fix rebase build issues

* Implement cache for metadata entity (#1565)

* Implement cache for metadata entity

* Clean up and exclude unfinalized blocks keys from metadata cache

* Clean up

* flush with cache stored records size (#1566)

* flush with cache stored records size

* update

* Lfu get cache (#1568)

* still need fix get/count logic

fix method for LFU getCache

* Update tests

* Fix and more tests

* Update base on review

* Fix

* Run all store operations through cache (#1567)

* Run all store operations through hash

* Log when cache flushed, flush cache on application shutdown

* Cache fixes

* Fix removeCache not being init

* POI cache fixes

* Fixes

* Fix caches being cleared after db operation and potential cache changes have happened

* Fix issues with injected store

* Clean up

* Fix tests from rebase

* Fix get cache not being used

* Rename function

* Fix reindex issues

* Fix casing of import

* Clean up, log if finalized is enabled

* parent 8b0b558 (#1574)

author Scott Twiname <skott.twiname@gmail.com> 1679452749 +1300
committer JQQQ <jiqiang90@hotmail.com> 1679461182 +1300

Clean up, log if finalized is enabled

Fix rebase build issues

parent 5e4ea5d
author Scott Twiname <skott.twiname@gmail.com> 1679013784 +1300
committer JQQQ <jiqiang90@hotmail.com> 1679460224 +1300

parent 5e4ea5d
author Scott Twiname <skott.twiname@gmail.com> 1679013784 +1300
committer JQQQ <jiqiang90@hotmail.com> 1679460210 +1300

parent 5e4ea5d
author Scott Twiname <skott.twiname@gmail.com> 1679013784 +1300
committer JQQQ <jiqiang90@hotmail.com> 1679460149 +1300

parent 5e4ea5d
author Scott Twiname <skott.twiname@gmail.com> 1679013784 +1300
committer JQQQ <jiqiang90@hotmail.com> 1679460101 +1300

parent 5e4ea5d
author Scott Twiname <skott.twiname@gmail.com> 1679013784 +1300
committer JQQQ <jiqiang90@hotmail.com> 1679460038 +1300

Run all store operations through hash

Log when cache flushed, flush cache on application shutdown

Fix removeCache not being init

Fixes

Fix issues with injected store

Clean up

Fix removeCache not being init

Fixes

Fix issues with injected store

Clean up

Fix tests from rebase

Fix get cache not being used

tidy up

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Update metadata when dictionary skips large number of blocks (#1577)

* Update metadata when dictionary skips large number of blocks

* Bring back option to log sql

* Dynamic cache threshold and metrics (#1578)

* Bring back fetch service tests

* Fix build issues

* Tidy up

* Fix workers historical disabled (#1579)

* Fix imports in node core

* improve close record sql, and add flag max size for getCache (#1583)

* improve close record sql, and add flag max size for getCache

* Fix

* escape id string

* Flush data at specific block heights (#1582)

* Flush data at specific block heights

* Add test

* Fix value fromBelowHeight being empty

* Fix mergedRecords potentially being empty

* Fix flushableRecordCounter

* Change log to debug

* Fix dictionary validation

* fix reindex storeCache missing EventEmitterModule (#1585)

* Make flushing cache configurable (#1586)

* Add an index to Id field on entities with historical (#1588)

* Fix tests

* Fix possible infinite loop with cache and fix test (#1591)

* Fix possible infinite loop with cache and fix test

* Add some comments

* Fix more tests

---------

Co-authored-by: JQQQ <jiqiang90@hotmail.com>

* Connect historical option up to cache model option (#1596)

* Only run delete query if there is data to delete

* Add support for non-historical indexing with cache (#1597)

* Add support for non-historical indexing with cache

* Fix api test

* remove constraintDeferrableQuery from hasOne

* fix createUniqueIndexQuery

---------

Co-authored-by: JQQQ <jiqiang90@hotmail.com>

* Cache and worker related fixes (#1598)

* fix index name too long issue (#1599)

* hash the indexes, except pk indexes

* handle existed indexes

* update

* Remove store count function (#1600)

* support cockroach db with non-historical (#1604)

* support cockroach db with non-historical

* Add more comments

* move fetchBlocks to node-core (#1611)

* move fetchBlocks to node-core

* code cleanup

* code cleanup

* remove extra args from fetchBlock

* Fix making data serializable for store bulkCreate and bulkUpdate (#1622)

* Improve big int error in workers (#1619)

* improve error handling in workers

* Make bigint stringifiable in worker and main threads

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* fix type error (#1623)

* fix type error

* cast type

* Flush store cache in operational order (#1614)

* Init

* better log message

* tidy up

* Tidy up and test

* Update packages/node/src/yargs.ts

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Update packages/node-core/src/indexer/storeCache/storeCache.service.ts

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* add benchmarking

* update benchmarking

* Tidy up

* test

* remove flag, use last flushed index, more tidy up

* make operationIndex required

* fix test

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* update default values

* fix missing indexes on cockroach db (#1624)

* fix getLatest() could be undefined

* Fix StoreOperation not pushing enums

---------

Co-authored-by: JQQQ <jiqiang90@hotmail.com>
Co-authored-by: Naveen V <velnaveen99@gmail.com>

* [SKIP CI] Prerelease

* Testing Framework (#1584)

* implement runTest

* check handler kind for running test

* implement testing service

* run tests in sandbox

* build test files through subql build

* run handlers in sandbox

* cleanup sandbox script and improve logging

* index data with `IndexerManager.indexData`

* use indexBlock() instead of indexData() for tests

* add getter _name to entities during codegen

* code cleanup

* improve logging

* improve logging

* bug fixes

* code cleanup

* improve logging

* indexBlock: get datasource from arguments

* move testingService to node-core

* code cleanup

* update ci

* code cleanup

* cleanup

* cleanup

* cleanup

* handle 0 test cases

* prevent calling getGqlType for 'contains' matcher (#1626)

* prevent calling getGqlType for 'contains' matcher

* get gqlType with ternary condition

* remove indexer module from testing module (#1628)

* remove indexer module from testing module

* cleanup

* cleanup

* Enable strict ts setting for node core (#1625)

* WIP enable strict ts setting for node core

* fix undefined in historicalMarkPreviousHeightRecordsBatch

* typo

* Update typescript, fix most strict related errors

* Strict changes for testing framework

* Runtime fixes

* change log level on cache poi

---------

Co-authored-by: JQQQ <jiqiang90@hotmail.com>

* fix prerelease (#1629)

* [SKIP CI] Prerelease

* [SKIPCI] fix testing publish order (#1630)

* fix missing @subql/testing in node-core (#1631)

* [SKIP CI] Prerelease

* manual prerelease for node-core fix (#1632)

* [SKIP CI] Prerelease

* Improve yargs descriptions (#1633)

* [SKIP CI] Prerelease

* Support tests from ./test and ./tests dirs, fix store blockheight issue

* Remove log

* [SKIP CI] Prerelease

* [release] 2.0.0 (#1627)

* release-2.0.0

* update

* update node changelog

* remove typescript from query dependency

* remove typescript from node dependency

* [SKIPCI] fix release

* [release] 2.0.0 republish (#1635)

* Fix ApiService abstract class, improve getting all DS (#1638)

* Fix ApiService abstract class, improve getting all DS

* Revert api import and minor test fixes

* [SKIP CI] Prerelease

* fix assertion error and pk index type (#1641)

* [SKIP CI] Prerelease

* fix workers crashes error (#1642)

* try to fix workers

* Fix dependency injection

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* WIP fix test part 1 (#1640)

* fix test part 1

* Update packages/node/src/indexer/fetch.service.ts

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* fix tests 20230426

* remove unused code

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* fix missing esModuleInterop (#1644)

* [SKIP CI] Prerelease

* Force flush cache with reindex command (#1645)

* [SKIP CI] Prerelease

* [release] 20230427 (#1646)

* [release] 20230427

* improve log

* Fix api not being defined, fix not using filter on datasources (#1647)

* [SKIP CI] Prerelease

* deprecate local mode (#1648)

* [SKIP CI] Prerelease

* [release] 20230427 patch (#1649)

* cleanup

* Update common package

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>
Co-authored-by: seandotau <seandotau@gmail.com>
Co-authored-by: Jay Ji <jiqiang90@hotmail.com>
Co-authored-by: Ben <89335033+bz888@users.noreply.github.com>
Co-authored-by: Darwin Subramaniam <darwinsubramaniam@googlemail.com>
Co-authored-by: HZHAIX <zhaihaohk@gmail.com>
Co-authored-by: James Bayly <46693720+jamesbayly@users.noreply.github.com>
stwiname added a commit that referenced this pull request Jun 27, 2023
@guplersaxanoid @stwiname
@seandotau @jiqiang90 @bz888 @darwinsubramaniam @mzxyz @jamesbayly
Sync 20230515 (#32)
ea885b5 
* Add support for near projects to the CLI (#1498)

* Add support for near projects to the CLI

* Update common near version

* Update near packages

* [SKIP CI] Prerelease

* Arranges commands in alphabetical order (#1495)

* [SKIP CI] Prerelease

* user process.env.USERPROFILE for windows platform (#1502)

* [SKIP CI] Prerelease

* add blockTime to NodeConfig (#1501)

* [SKIP CI] Prerelease

* [release] 20230131 (#1503)

* Worker use runtime service (#1491)

* draft

* update

* tidy up

* update

* working worker runtimes

* into base class

* clean up

* improve base on review

* [SKIP CI] Prerelease

* Check queue for lower blocks before processing fetched blocks (#1509)

* [SKIP CI] Prerelease

* Support array type in dictionary queries (#1510)

* [SKIP CI] Prerelease

* add flag query-limit (#1514)

* [SKIP CI] Prerelease

* Fix error with tempDsRecords being undefined (#1516)

* Fix error with tempDsRecords being undefined

* Fix schema not being used for metadata, sync dynamic-ds across workers

* Be smarter about reloading dynamic ds in workers

* [SKIP CI] Prerelease

* Update postgraphile enable table partitions (#1520)

* [SKIP CI] Prerelease

* `dictionary-optimisation` flag  (#1519)

* add flag, to orderby blockheight instead of pk

* add comment

* [SKIP CI] Prerelease

* [WIP] Feature/adding postgres ssl connection option (#1513)

* enable ssl option postgres

* Check DB_SSL Env Value , default is false

* adding ENV to pass in the Path to certificates

* adding SSL option for query pacakge.

* use args to pass the postgres ssl informations.

* load file content moved to comman package.

* Update packages/node-core/src/configure/NodeConfig.ts

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* Support Cockroach db (#1521)

* support cockroach db

* Update comment

* [SKIP CI] Prerelease

* ensure pgpool connection work with ssl (#1525)

* [SKIP CI] Prerelease

* Fix issue when use cockroach query service not support pgPartition (#1526)

* [SKIP CI] Prerelease

* [release] 20230221 (#1528)

* fix issue store getByField limit could potentially excess config limit (#1529)

* fix issue store getByField limit could potentially excess config limit

* Update packages/node-core/src/indexer/store.service.ts

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* typo

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* Abi codegen (#1532)

* draft

* fix abi name, add support for transaction

* Update packages/cli/src/template/abi-interface.ts.ejs

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* fix

* fix dependencies

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* Move enum under schema (#1527)

* need fix query get enum

* fix enum query, clean all enums

* add logic to handle original enum type

* fix check exclude constrain check missing schema and table (#1537)

* [SKIP CI] Prerelease

* manual prerelease node for node-core update (#1538)

* [SKIP CI] Prerelease

* resolution for sequelize and prerelease node (#1539)

* [SKIP CI] Prerelease

* fixed the version of sequelize and prerelease node (#1540)

* [SKIP CI] Prerelease

* improve force clean with deprecated enum (#1541)

* [SKIP CI] Prerelease

* Remove exclude constraint (#1543)

* deprecate exclude contraint

* remove unused

* [SKIP CI] Prerelease

* [release] 20230306 (#1544)

* [release] 20230307 (#1545)

* bump yarn.lock, update imports (#1546)

* [SKIP CI] Prerelease

* [release] 20230307 (#1547)

* increase wsProvider timeout (#1550)

* increase wsProvider timeout

* remove .mmr dir

* [SKIP CI] Prerelease

* Use performance.now instead of date for profiler. (#1549)

* add SmartBatchService and BlockSizeBuffer (#1506)

* add SmartBatchService and BlockSizeBuffer

* check memory usage before fetching a batch

* use v8 heap usage statistics

* set block to null after indexing

* extend smart batch service to workers

* make smart batch size injectible

* rebase

* bug fixes

* code cleanup

* assign workers based on free memory

* fix liniting

* cleanup code

* cleanup logs

* clear blocksizebuffer inside smartbatchservice

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Fix build issue with profiler

* [SKIP CI] Prerelease

* fix abi codegen runtime validation (#1552)

* [SKIP CI] Prerelease

* [release] 20230309 Cli (#1553)

* Fix enum under schema not being escaped (#1555)

* [SKIP CI] Prerelease

* rename `--sponsored-dictionary` to `--dictionary-resolver` (#1559)

* renamed flag

* add missing tpyo

* [SKIP CI] Prerelease

* handle bigint in json.stringify (#1562)

* [SKIP CI] Prerelease

* add validation for `codegen`, grapql conflicts (#1564)

* add test for codegen with reserved keys

* wip validateEntityName fn

* update test, update logc

* remove logs

* [SKIP CI] Prerelease

* Fix codegen when abi is not valid (#1570)

* Fix codegen when abi is not valid

* Update packages/cli/src/controller/codegen-controller.ts

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* Fix block offset been reset to 1 (#1571)

* [SKIP CI] Prerelease

* remove blocking in process queueing (#1572)

* remove blocking in process queueing

* redefine locking pattern

* fix fetchedBlocks status

* [SKIP CI] Prerelease

* improve api error handling (#1576)

* [SKIP CI] Prerelease

* generate sourcemap for projects (#1569)

* improve error handling

* use inline-source-map

* revert api error handling

* beautify minified output from webpack

* code cleanup

* move terser plugin dependency to cli

* log warnings for unresolved stack trace

* get entry file from datasource

* alter convertStack prototype

* add ES2020 to lib compiler option

* update polkadot api to 10.1.4 (#1580)

* update polkadot api to 10.1.4

* update rest of dep

* [release] 20230329 (#1587)

* Multiple endpoints (#1551)

* allow multiple endpoints

* add response time based load balancing

* handle endpoint disconnects

* move fetchBlockBatches to apiService

* cleanup code

* move ConnectionPoolService to node-core

* fix order of module injection in indexermodule

* remove manually disconnecting api

* move api creation code to ApiConnection class

* code cleanup

* handle reconnection error

* change networkEndpoints to networkEndpoint

* change to use yarn build (#1589)

* test prerelease build (#1590)

* [SKIP CI] Prerelease

* [release] 20230330 (#1592)

* Upgrade `apollo-links` (#1593)

* [SKIP CI] Prerelease

* improve blockSize function (#1602)

* improve blockSize function

* add return type to blockSize

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* [release] 20230405 (#1603)

* Add evmChainId to metadata (#1607)

* [SKIP CI] Prerelease

* [release] 20230411 query (#1608)

* Update progress string (#1612)

* [SKIP CI] Prerelease

* Update the jsonField directive to have an option for disabling the index (#1613)

* Update the jsonField directive to have an option for disabling the index

* Clean up

* [SKIP CI] Prerelease

* [release] 20230414 utils (#1615)

* Fix workers fetching blocks out of order (#1616)

* Fix workers fetching blocks out of order

* Remove logging endpoints with api keys

* [SKIP CI] Prerelease

* [release] 20230417 (#1617)

* DB cache and various other improvements (#1561)

* store cache

* improve on flush cache

* draft storeCache service

* update

* minor change

* Updates

* add historical data to cache (#1557)

* add __block_range to creation

* add historical record to cache

* correct range

* Complete cached store methods (#1563)

* add __block_range to creation

add historical record to cache

correct range

cache store methods

start add tests

* fix set

* update

* WIP cache with workers support

* Fix up dynamic ds service, clean up

* Update store serialisation over workers

* Improve unfinalized blocks to work without tx

* Move all chain agnostic code from block dispatchers to node core

* Add comments about worker store conversion

* Fix rebase build issues

* Implement cache for metadata entity (#1565)

* Implement cache for metadata entity

* Clean up and exclude unfinalized blocks keys from metadata cache

* Clean up

* flush with cache stored records size (#1566)

* flush with cache stored records size

* update

* Lfu get cache (#1568)

* still need fix get/count logic

fix method for LFU getCache

* Update tests

* Fix and more tests

* Update base on review

* Fix

* Run all store operations through cache (#1567)

* Run all store operations through hash

* Log when cache flushed, flush cache on application shutdown

* Cache fixes

* Fix removeCache not being init

* POI cache fixes

* Fixes

* Fix caches being cleared after db operation and potential cache changes have happened

* Fix issues with injected store

* Clean up

* Fix tests from rebase

* Fix get cache not being used

* Rename function

* Fix reindex issues

* Fix casing of import

* Clean up, log if finalized is enabled

* parent 8b0b558 (#1574)

author Scott Twiname <skott.twiname@gmail.com> 1679452749 +1300
committer JQQQ <jiqiang90@hotmail.com> 1679461182 +1300

Clean up, log if finalized is enabled

Fix rebase build issues

parent 5e4ea5d
author Scott Twiname <skott.twiname@gmail.com> 1679013784 +1300
committer JQQQ <jiqiang90@hotmail.com> 1679460224 +1300

parent 5e4ea5d
author Scott Twiname <skott.twiname@gmail.com> 1679013784 +1300
committer JQQQ <jiqiang90@hotmail.com> 1679460210 +1300

parent 5e4ea5d
author Scott Twiname <skott.twiname@gmail.com> 1679013784 +1300
committer JQQQ <jiqiang90@hotmail.com> 1679460149 +1300

parent 5e4ea5d
author Scott Twiname <skott.twiname@gmail.com> 1679013784 +1300
committer JQQQ <jiqiang90@hotmail.com> 1679460101 +1300

parent 5e4ea5d
author Scott Twiname <skott.twiname@gmail.com> 1679013784 +1300
committer JQQQ <jiqiang90@hotmail.com> 1679460038 +1300

Run all store operations through hash

Log when cache flushed, flush cache on application shutdown

Fix removeCache not being init

Fixes

Fix issues with injected store

Clean up

Fix removeCache not being init

Fixes

Fix issues with injected store

Clean up

Fix tests from rebase

Fix get cache not being used

tidy up

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Update metadata when dictionary skips large number of blocks (#1577)

* Update metadata when dictionary skips large number of blocks

* Bring back option to log sql

* Dynamic cache threshold and metrics (#1578)

* Bring back fetch service tests

* Fix build issues

* Tidy up

* Fix workers historical disabled (#1579)

* Fix imports in node core

* improve close record sql, and add flag max size for getCache (#1583)

* improve close record sql, and add flag max size for getCache

* Fix

* escape id string

* Flush data at specific block heights (#1582)

* Flush data at specific block heights

* Add test

* Fix value fromBelowHeight being empty

* Fix mergedRecords potentially being empty

* Fix flushableRecordCounter

* Change log to debug

* Fix dictionary validation

* fix reindex storeCache missing EventEmitterModule (#1585)

* Make flushing cache configurable (#1586)

* Add an index to Id field on entities with historical (#1588)

* Fix tests

* Fix possible infinite loop with cache and fix test (#1591)

* Fix possible infinite loop with cache and fix test

* Add some comments

* Fix more tests

---------

Co-authored-by: JQQQ <jiqiang90@hotmail.com>

* Connect historical option up to cache model option (#1596)

* Only run delete query if there is data to delete

* Add support for non-historical indexing with cache (#1597)

* Add support for non-historical indexing with cache

* Fix api test

* remove constraintDeferrableQuery from hasOne

* fix createUniqueIndexQuery

---------

Co-authored-by: JQQQ <jiqiang90@hotmail.com>

* Cache and worker related fixes (#1598)

* fix index name too long issue (#1599)

* hash the indexes, except pk indexes

* handle existed indexes

* update

* Remove store count function (#1600)

* support cockroach db with non-historical (#1604)

* support cockroach db with non-historical

* Add more comments

* move fetchBlocks to node-core (#1611)

* move fetchBlocks to node-core

* code cleanup

* code cleanup

* remove extra args from fetchBlock

* Fix making data serializable for store bulkCreate and bulkUpdate (#1622)

* Improve big int error in workers (#1619)

* improve error handling in workers

* Make bigint stringifiable in worker and main threads

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* fix type error (#1623)

* fix type error

* cast type

* Flush store cache in operational order (#1614)

* Init

* better log message

* tidy up

* Tidy up and test

* Update packages/node/src/yargs.ts

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* Update packages/node-core/src/indexer/storeCache/storeCache.service.ts

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* add benchmarking

* update benchmarking

* Tidy up

* test

* remove flag, use last flushed index, more tidy up

* make operationIndex required

* fix test

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* update default values

* fix missing indexes on cockroach db (#1624)

* fix getLatest() could be undefined

* Fix StoreOperation not pushing enums

---------

Co-authored-by: JQQQ <jiqiang90@hotmail.com>
Co-authored-by: Naveen V <velnaveen99@gmail.com>

* [SKIP CI] Prerelease

* Testing Framework (#1584)

* implement runTest

* check handler kind for running test

* implement testing service

* run tests in sandbox

* build test files through subql build

* run handlers in sandbox

* cleanup sandbox script and improve logging

* index data with `IndexerManager.indexData`

* use indexBlock() instead of indexData() for tests

* add getter _name to entities during codegen

* code cleanup

* improve logging

* improve logging

* bug fixes

* code cleanup

* improve logging

* indexBlock: get datasource from arguments

* move testingService to node-core

* code cleanup

* update ci

* code cleanup

* cleanup

* cleanup

* cleanup

* handle 0 test cases

* prevent calling getGqlType for 'contains' matcher (#1626)

* prevent calling getGqlType for 'contains' matcher

* get gqlType with ternary condition

* remove indexer module from testing module (#1628)

* remove indexer module from testing module

* cleanup

* cleanup

* Enable strict ts setting for node core (#1625)

* WIP enable strict ts setting for node core

* fix undefined in historicalMarkPreviousHeightRecordsBatch

* typo

* Update typescript, fix most strict related errors

* Strict changes for testing framework

* Runtime fixes

* change log level on cache poi

---------

Co-authored-by: JQQQ <jiqiang90@hotmail.com>

* fix prerelease (#1629)

* [SKIP CI] Prerelease

* [SKIPCI] fix testing publish order (#1630)

* fix missing @subql/testing in node-core (#1631)

* [SKIP CI] Prerelease

* manual prerelease for node-core fix (#1632)

* [SKIP CI] Prerelease

* Improve yargs descriptions (#1633)

* [SKIP CI] Prerelease

* Support tests from ./test and ./tests dirs, fix store blockheight issue

* Remove log

* [SKIP CI] Prerelease

* [release] 2.0.0 (#1627)

* release-2.0.0

* update

* update node changelog

* remove typescript from query dependency

* remove typescript from node dependency

* [SKIPCI] fix release

* [release] 2.0.0 republish (#1635)

* Fix ApiService abstract class, improve getting all DS (#1638)

* Fix ApiService abstract class, improve getting all DS

* Revert api import and minor test fixes

* [SKIP CI] Prerelease

* fix assertion error and pk index type (#1641)

* [SKIP CI] Prerelease

* fix workers crashes error (#1642)

* try to fix workers

* Fix dependency injection

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* WIP fix test part 1 (#1640)

* fix test part 1

* Update packages/node/src/indexer/fetch.service.ts

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* fix tests 20230426

* remove unused code

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* fix missing esModuleInterop (#1644)

* [SKIP CI] Prerelease

* Force flush cache with reindex command (#1645)

* [SKIP CI] Prerelease

* [release] 20230427 (#1646)

* [release] 20230427

* improve log

* Fix api not being defined, fix not using filter on datasources (#1647)

* [SKIP CI] Prerelease

* deprecate local mode (#1648)

* [SKIP CI] Prerelease

* [release] 20230427 patch (#1649)

* Add missing node-fetch dependency to lock down version (#1651)

* [SKIP CI] Prerelease

* postgres based mmr DB (#1618)

* postgres based mmr DB

* add mmr db store option to cli

* add mmr migrate

* bug fixes

* alter file based names

* make migration bidirectional

* cleanup and improve logging

* remove localMode and add mmr-path to args

* remove check for localMode in createProjectSchema

* improve error handling

* improve error handling for postgres mmr table

* improve error handling for mmr indexer

* Fix test, create db table same as other tables, tidy up

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* Extract ProjectService, FetchService and IndexerManager to base classes (#1643)

* Move bulk of project service logic to node core

* Allow overriding buildDictQueryFragment

* Improve visibility of dependencies on project service

* Extract base class from fetch service

* Fix error message

* Fix tests

* Create indexer manager base class

* [SKIP CI] Prerelease

* Move polkadot util imports to utils package, fix api type (#1653)

* Move polkadot util imports to utils package, fix api type

* remove duplicate log

---------

Co-authored-by: JQQQ <jiqiang90@hotmail.com>

* [SKIP CI] Prerelease

* manifest node with options (#1652)

* manifest node with options

* fix

* fix self import

* remove

* Use single version of class-* packages, extract base node-runner

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>

* [SKIP CI] Prerelease

* POI cache fixes (#1660)

* Add tests for cache poi and fix related issuess

* Add mutex around cachePoi

* [SKIP CI] Prerelease

* Add bulkRemove function to store (#1666)

* [SKIP CI] Prerelease

* fix postMmrDb.get and mmrMigrate node sort (#1669)

* [SKIP CI] Prerelease

* Fix indexed column check with historical (#1667)

* [SKIP CI] Prerelease

* Regenerate mmr (#1664)

* preparation

Unsafe mode and draft

bulkUpsert with poi mmr

move service to node-core

* tidy up, fix tests

* Fix gap mmr node missing issue, poiCache use plainPoiModel

* [SKIP CI] Prerelease

* Node core ds processor (#1658)

* Move ds processor to node core

* Clean up deps

* Move forceClean and meta to node-core

* Move configure logic to node core

* Move unfinalzied blocks to node-core

* Add missing logger

* Fix tests

* [SKIP CI] Prerelease

* Add interval to flush the store cache (#1670)

* Add interval to flush the store cache

* Fix infinite loop in nodeConfig

* [SKIP CI] Prerelease

* Remove support for manifest version <1.0.0 (#1659)

* Clean up deps

* Move forceClean and meta to node-core

* Remove support for manifest version <1.0.0

* Fix build issues

* Move sandbox back from node-core

* Move gauge providers into node core

* Tidy up indexer module

* Move FileType to core

* Remove tests for unsupported manifest versions

* [SKIP CI] Prerelease

* Fix Cache missing remove (#1675)

* Fix missing removeCache from flush

* tidy up

* [SKIP CI] Prerelease

* [release] 20230510 (#1673)

* [release] 20230510

* Update package.json

* Update node-core changelog

* Fix missing scheduler subcommands (#1677)

* Fix mmrRegen module missing scheduler

* Fix subcommands module missing scheduler

* [SKIP CI] Prerelease

* Extract dictionary meta validation so it can be overridden (#1679)

* Extract dictionary meta validation so it can be overridden

* Update type

* [SKIP CI] Prerelease

* [release] 20230511 (#1678)

* [release] 20230511

* Update node-core

* [release] 20230511 types (#1680)

* Fix metadata check, allow base indexer manager to parse abis with ethereum (#1682)

* [SKIP CI] Prerelease

* Move validate function to common (#1683)

* Move validate function to common

* Make prepareFilteredData async

* [SKIP CI] Prerelease

* Inject the chain id into sandboxes (#1684)

* [SKIP CI] Prerelease

* [release] 20230511 v2 (#1685)

* Fix flush init before metadata repo set (#1688)

* [SKIP CI] Prerelease

* [release] 20230512 (#1690)

* fix dictionary resolver type (#1692)

* [SKIP CI] Prerelease

* check undefined when flushing without historical (#1698)

* update fromBelowHeight function

* records check

* [SKIP CI] Prerelease

* Fix initial dictionary check before lastBufferedHeight is set (#1700)

* [SKIP CI] Prerelease

* Add DS to FilterTypeMap, remove redundant code (#1703)

* [SKIP CI] Prerelease

* cleanup

---------

Co-authored-by: Scott Twiname <skott.twiname@gmail.com>
Co-authored-by: seandotau <seandotau@gmail.com>
Co-authored-by: Jay Ji <jiqiang90@hotmail.com>
Co-authored-by: Ben <89335033+bz888@users.noreply.github.com>
Co-authored-by: Darwin Subramaniam <darwinsubramaniam@googlemail.com>
Co-authored-by: HZHAIX <zhaihaohk@gmail.com>
Co-authored-by: James Bayly <46693720+jamesbayly@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jiqiang90 jiqiang90 jiqiang90 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

subql publish command error
2 participants
@bz888 @jiqiang90