Unable to deploy project version through CLI deployment:deploy with disabled historical data

[filo87]

Prerequisites

• Are you running the latest version(s)?
• Did you check the debugging guide?
• Did you check the FAQs and Discussions?
• Are you reporting to the correct repository?
• Did you search for an existing issue or pull request?

Description

By default subql deployment:deploy also when omitting the --indexerHistoricalData flag deploys a project version and enables historical data. Projects which are incompatible with historicalData, should be able to disable the feature also via CLI and not only through the UI.

Details

When deploying our project with the command:

IPFSCID=$(npx subql publish -o -f .)
sleep 5
npx subql deployment:deploy \
--org="$SUBQL_PROJ_ORG" \
--endpoint="$SUBQL_ENDPOINT" \
--dict=' ' \
--projectName="$SUBQL_PROJ_NAME" \
--ipfsCID="$IPFSCID" \
--type=primary \
--indexerVersion="v1.15.1" \
--queryVersion="v1.9.1"

The deployment successfully runs but the indexer lands in an unhealthy state.

Our data model is NOT compatible with historical data, as we implemented a custom logic for state tracking and snapshotting. For this reason we need to disable it, in order for the indexer to work correctly. This is currently not possible through the CLI but only with the UI.

When the exact same project is deployed through the UI with the same CID, historical data IS disabled and the indexer works fine:

Steps to Reproduce

1. Deploy a project with the following options:

npx subql deployment:deploy \
--org="$SUBQL_PROJ_ORG" \
--endpoint="$SUBQL_ENDPOINT" \
--dict=' ' \
--projectName="$SUBQL_PROJ_NAME" \
--ipfsCID="$IPFSCID" \
--type=primary \
--indexerVersion="v1.15.1" \
--queryVersion="v1.9.1"

2. Observe historicalData feature enabled by default

Any other information

Deployment Error Info

{"level":"info","timestamp":"2022-12-08T10:56:45.076Z","pid":7,"hostname":"subquery-16824-indexer-deployment-845d6d6486-kcgjr","category":"subql-node","message":"Current @subql/node version is 1.15.1"}
{"level":"info","timestamp":"2022-12-08T10:56:46.822Z","pid":7,"hostname":"subquery-16824-indexer-deployment-845d6d6486-kcgjr","category":"indexer","message":"indexer manager start"}
{"level":"info","timestamp":"2022-12-08T10:56:51.311Z","pid":7,"hostname":"subquery-16824-indexer-deployment-845d6d6486-kcgjr","category":"store","message":"Historical state is enabled"}
{"level":"error","timestamp":"2022-12-08T10:56:51.425Z","pid":7,"hostname":"subquery-16824-indexer-deployment-845d6d6486-kcgjr","category":"store","payload":{"type":"error","name":"SequelizeUnknownConstraintError","message":"Unknown constraint error","stack":"Error\n    at Query.run (/usr/local/lib/node_modules/@subql/node/node_modules/sequelize/lib/dialects/postgres/query.js:50:25)\n    at /usr/local/lib/node_modules/@subql/node/node_modules/sequelize/lib/sequelize.js:314:28\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)\n    at async PostgresQueryInterface.addIndex (/usr/local/lib/node_modules/@subql/node/node_modules/sequelize/lib/dialects/abstract/query-interface.js:250:12)\n    at async Function.sync (/usr/local/lib/node_modules/@subql/node/node_modules/sequelize/lib/model.js:999:7)\n    at async Sequelize.sync (/usr/local/lib/node_modules/@subql/node/node_modules/sequelize/lib/sequelize.js:376:9)\n    at async StoreService.syncSchema (/usr/local/lib/node_modules/@subql/node/node_modules/@subql/node-core/dist/indexer/store.service.js:203:9)\n    at async StoreService.init (/usr/local/lib/node_modules/@subql/node/node_modules/@subql/node-core/dist/indexer/store.service.js:45:13)\n    at async initDbSchema (/usr/local/lib/node_modules/@subql/node/dist/utils/project.js:238:5)\n    at async ProjectService.initDbSchema (/usr/local/lib/node_modules/@subql/node/dist/indexer/project.service.js:137:9)"},"message":"Having a problem when syncing schema"}

[jamesbayly]

Please add the following command to your deployment script --indexerHistoricalData=false

npx subql deployment:deploy \
--org="$SUBQL_PROJ_ORG" \
--endpoint="$SUBQL_ENDPOINT" \
--dict=' ' \
--projectName="$SUBQL_PROJ_NAME" \
--ipfsCID="$IPFSCID" \
--type=primary \
--indexerVersion="v1.15.1" \
--indexerHistoricalData=false \
--queryVersion="v1.9.1"

[filo87]

Hey @jamesbayly thanks for the hint. I tried, but looks like the flag --indexerHistoricalData=false with the false argument gets rejected.

[jamesbayly]

@bz888

[filo87]

Maybe here:

subql/packages/cli/src/commands/deployment/deploy.ts

Lines 85 to 98 in cbe2187

	const queryAD = {
	unsafe: flags.queryUnsafe,
	subscription: flags.querySubscription,
	queryTimeout: flags.queryTimeout,
	maxConnection: flags.queryMaxConnection,
	Aggregate: flags.queryAggregate,
	};
	const indexerAD = {
	unsafe: flags.indexerUnsafe,
	batchSize: flags.indexerBatchSize,
	subscription: flags.indexerSubscription,
	historicalData: flags.indexerHistoricalData,
	workers: flags.indexerWorkers,
	};

providing some defaults fallback boolean values might help:

    const indexerAD = {
      unsafe: flags.indexerUnsafe || false,
      batchSize: flags.indexerBatchSize || 30,
      subscription: flags.indexerSubscription || false ,
      historicalData: flags.indexerHistoricalData || false,
      workers: flags.indexerWorkers || 1,
    };

i suspect that, if no --indexerHistoricalData is set in the CLI, then in indexerAD the key historicalData is set to undefined and the parameter might not be included in the subsequent API call.

[filo87]

or a another approach is to set default value for the Flag according to oclif:

subql/packages/cli/src/commands/deployment/deploy.ts

Line 40 in cbe2187

indexerHistoricalData: Flags.boolean({description: 'Enable Historical Data', required: false}),

to

indexerHistoricalData: Flags.boolean({description: 'Enable Historical Data', required: false, default: false}),