Skip to content

Commit

Permalink
Merge branch 'master' into chore/lw-6514-hardware-trezor-package
Browse files Browse the repository at this point in the history
  • Loading branch information
DominikGuzei committed May 30, 2023
2 parents 73f7d28 + c30ea8e commit 3f0b512
Show file tree
Hide file tree
Showing 39 changed files with 707 additions and 115 deletions.
2 changes: 2 additions & 0 deletions Dockerfile
Expand Up @@ -40,6 +40,8 @@ COPY --from=cardano-services-production-deps /app/packages/projection-typeorm/no
COPY --from=cardano-services-builder /app/scripts /app/scripts
COPY --from=cardano-services-builder /app/packages/cardano-services/dist /app/packages/cardano-services/dist
COPY --from=cardano-services-builder /app/packages/cardano-services/package.json /app/packages/cardano-services/package.json
COPY --from=cardano-services-builder /app/packages/cardano-services-client/dist /app/packages/cardano-services-client/dist
COPY --from=cardano-services-builder /app/packages/cardano-services-client/package.json /app/packages/cardano-services-client/package.json
COPY --from=cardano-services-builder /app/packages/core/dist /app/packages/core/dist
COPY --from=cardano-services-builder /app/packages/core/package.json /app/packages/core/package.json
COPY --from=cardano-services-builder /app/packages/crypto/dist /app/packages/crypto/dist
Expand Down
5 changes: 4 additions & 1 deletion compose/common.yml
Expand Up @@ -60,7 +60,8 @@ services:
condition: service_healthy
environment:
- LOGGER_MIN_SEVERITY=${LOGGER_MIN_SEVERITY:-info}
- QUEUES=${QUEUES:-pool-metadata}
- QUEUES=${QUEUES:-pool-metadata,pool-metrics}
- STAKE_POOL_PROVIDER_URL=http://provider-server:3000/stake-pool
healthcheck:
test: ['CMD-SHELL', 'curl -s --fail http://localhost:3003/health']
interval: 10s
Expand Down Expand Up @@ -112,6 +113,8 @@ services:
- DROP_SCHEMA=${DROP_PROJECTOR_SCHEMA:-false}
- LOGGER_MIN_SEVERITY=${LOGGER_MIN_SEVERITY:-info}
- OGMIOS_URL=ws://cardano-node-ogmios:1337
- PROJECTION_NAMES=${PROJECTION_NAMES:-stake-pool,stake-pool-metadata-job,stake-pool-metrics-job,utxo}
- SYNCHRONIZE=${SYNCHRONIZE:-true}
healthcheck:
test:
['CMD-SHELL', 'test `curl -fs http://localhost:3002/health | jq -r ".services[0].projectedTip.blockNo"` -gt 1']
Expand Down
2 changes: 1 addition & 1 deletion compose/projector/init.sh
Expand Up @@ -26,7 +26,7 @@ _term() {
trap _term SIGTERM

cd /app/packages/cardano-services
node dist/cjs/cli.js start-projector stake-pool,stake-pool-metadata-job &
node dist/cjs/cli.js start-projector &

CHILD=$!
wait "$CHILD"
1 change: 1 addition & 0 deletions packages/cardano-services/.env.test
@@ -1,4 +1,5 @@
POSTGRES_CONNECTION_STRING_DB_SYNC=postgresql://postgres:doNoUseThisSecret!@127.0.0.1:5433/localnetwork
POSTGRES_CONNECTION_STRING_STAKE_POOL=postgresql://postgres:doNoUseThisSecret!@127.0.0.1:5433/stake_pool
CARDANO_NODE_CONFIG_PATH=./config/network/testnet/cardano-node/config.json
DB_CACHE_TTL=120
EPOCH_POLL_INTERVAL=10000
Expand Down
54 changes: 47 additions & 7 deletions packages/cardano-services/README.md
Expand Up @@ -139,17 +139,57 @@ RABBITMQ_SRV_SERVICE_NAME=some-domain-for-rabbitmq \
stake-pool,stake-pool-metadata-job
```

## Production

The _Docker images_ produced by the SDK and the _docker compose infrastructures_ (_mainnet_, _preprod_ and _local-network_) it includes are ready to be used in
production environment.

**Note:** the _docker compose infrastructures_ included in the SDK are mainly used for development purposes: to use
them in production environments, the projector service(s) must be instructed to run the _migration scripts_ rather than
to use the `synchronize` development option from **TypeORM**. This can be achieved through environment variables:

```
SYNCHRONIZE=false yarn preprod:up
```

## Development

To speed up the development process, developers can ignore the migrations while developing or debugging changes.
This freedom is granted by the `synchronize` development option from **TypeORM**, which is enabled by default in the
_docker compose infrastructures_ included in the SDK.

### Generating Projector Schema Migrations

1. Start local projection dependencies (e.g. `yarn preprod:up cardano-node-ogmios postgres`).
2. Initialize base schema by running `start-projector` with **all** projections enabled. _Hint: you can use `git` to checkout a previous commit if you already updated entities and skipped this step_
3. Create new entities (or update existing ones).
4. Include any new entities in `entities` object at `src/Projection/prepareTypeormProjection.ts`.
5. Run `yarn generate-migration`
6. Inspect the generated migration and add a static `entity` property. _Hint: you may need to split generated migration into multiple migrations manually if it affects more than 1 entity_
7. Export the new migration(s) from `migrations` array at `src/Projections/migrations/index.ts`
In order to grant to the projection service the ability to choose which projections it needs to activate, **the
migrations must be scoped to a single model**: if a single change has impact on more models, one migration for each
impacted model must be generated.

_Hint:_ isolating all the changes to each model in distinct commits can be so helpful for this target!

For each migration, once the change is _finalized_ (all the new entities are added to the `entities` object at
`src/Projection/prepareTypeormProjection.ts`, apart from last minor refinements the PR is approved, etc...), the
relative migration can be generated following these steps.

_Hint:_ if previous hint was followed, to checkout each commit which requires a migration to produce a _fixup_ commit
for each of them can be an easy way to iterate over all the impacted models.

1. Start a new fresh database with `DROP_PROJECTOR_SCHEMA=true SYNCHRONIZE=false yarn preprod up`
- **Note:** do not override `PROJECTION_NAMES` since in this scope all the projections must be loaded
- **Note:** this will not apply the current changes to the models into the database schema, so the currently
developed feature may not work properly, this is not relevant for the target of creating the migration
- `DROP_PROJECTOR_SCHEMA=true` is used to let the projection service to create the database schema from scratch
- with `SYNCHRONIZE=false` the projection service runs all migrations rather than reflecting the changes to the
models on the schema (through the `synchronize` development option from **TypeORM**)
2. Run `yarn generate-migration` to produce a new migration in
`src/Projections/migrations` directory
- this compares the database schema against all the models (repeat: only one of them should be changed) and
generates the required migration
3. Inspect the generated migration
- **Check:** if the migration has impact on more than one table, the changes was not isolated per model!
(the change must be reworked)
4. Add the `static entity` property to the migration `class` (to see other migrations for reference)
5. Rename the newly generated migration file and `class` giving them mnemonic names
6. Export the new migration from `migrations` array at `src/Projections/migrations/index.ts`

## Tests

Expand Down
2 changes: 1 addition & 1 deletion packages/cardano-services/package.json
Expand Up @@ -70,7 +70,6 @@
},
"devDependencies": {
"@cardano-ogmios/client": "5.6.0",
"@cardano-sdk/cardano-services-client": "workspace:~",
"@cardano-sdk/util-dev": "workspace:~",
"@types/amqplib": "^0.8.2",
"@types/bunyan": "^1.8.8",
Expand Down Expand Up @@ -100,6 +99,7 @@
},
"dependencies": {
"@blockfrost/blockfrost-js": "^5.2.0",
"@cardano-sdk/cardano-services-client": "workspace:~",
"@cardano-sdk/core": "workspace:~",
"@cardano-sdk/crypto": "workspace:~",
"@cardano-sdk/ogmios": "workspace:~",
Expand Down
17 changes: 17 additions & 0 deletions packages/cardano-services/src/PgBoss/index.ts
@@ -0,0 +1,17 @@
import { PgBossQueue, WorkerHandlerFactory } from './types';
import { STAKE_POOL_METADATA_QUEUE, STAKE_POOL_METRICS_UPDATE } from '@cardano-sdk/projection-typeorm';
import { stakePoolMetadataHandlerFactory } from './stakePoolMetadataHandler';
import { stakePoolMetricsHandlerFactory } from './stakePoolMetricsHandler';

export * from './stakePoolMetadataHandler';
export * from './stakePoolMetricsHandler';
export * from './types';
export * from './util';

/**
* Defines the handler for each pg-boss queue
*/
export const queueHandlers: Record<PgBossQueue, WorkerHandlerFactory> = {
[STAKE_POOL_METADATA_QUEUE]: stakePoolMetadataHandlerFactory,
[STAKE_POOL_METRICS_UPDATE]: stakePoolMetricsHandlerFactory
};
11 changes: 5 additions & 6 deletions packages/cardano-services/src/PgBoss/stakePoolMetadataHandler.ts
@@ -1,12 +1,10 @@
import { Cardano } from '@cardano-sdk/core';
import { DataSource, MoreThan } from 'typeorm';
import { Hash32ByteBase16 } from '@cardano-sdk/crypto';
import { PoolMetadataEntity, PoolRegistrationEntity, StakePoolMetadataTask } from '@cardano-sdk/projection-typeorm';
import { PoolMetadataEntity, PoolRegistrationEntity, StakePoolMetadataJob } from '@cardano-sdk/projection-typeorm';
import { WorkerHandlerFactory } from './types';
import { createHttpStakePoolMetadataService } from '../StakePool';

const isErrorWithConstraint = (error: unknown): error is Error & { constraint: unknown } =>
error instanceof Error && 'constraint' in error;
import { isErrorWithConstraint } from './util';

export const isUpdateOutdated = async (dataSource: DataSource, poolId: Cardano.PoolId, poolRegistrationId: string) => {
const repos = dataSource.getRepository(PoolRegistrationEntity);
Expand Down Expand Up @@ -48,10 +46,11 @@ export const savePoolMetadata = async (args: SavePoolMetadataArguments) => {
}
};

export const stakePoolMetadataHandlerFactory: WorkerHandlerFactory = (dataSource, logger) => {
export const stakePoolMetadataHandlerFactory: WorkerHandlerFactory = (options) => {
const { dataSource, logger } = options;
const service = createHttpStakePoolMetadataService(logger);

return async (task: StakePoolMetadataTask) => {
return async (task: StakePoolMetadataJob) => {
const { metadataJson, poolId, poolRegistrationId } = task;
const { hash, url } = metadataJson;

Expand Down
79 changes: 79 additions & 0 deletions packages/cardano-services/src/PgBoss/stakePoolMetricsHandler.ts
@@ -0,0 +1,79 @@
import { Cardano, StakePoolProvider } from '@cardano-sdk/core';
import { CurrentPoolMetricsEntity, StakePoolEntity, StakePoolMetricsUpdateJob } from '@cardano-sdk/projection-typeorm';
import { DataSource } from 'typeorm';
import { Logger } from 'ts-log';
import { WorkerHandlerFactory } from './types';
import { isErrorWithConstraint } from './util';
import { stakePoolHttpProvider } from '@cardano-sdk/cardano-services-client';

interface RefreshPoolMetricsOptions {
dataSource: DataSource;
id: Cardano.PoolId;
logger: Logger;
provider: StakePoolProvider;
slot: Cardano.Slot;
}

export const savePoolMetrics = async (options: RefreshPoolMetricsOptions & { metrics: Cardano.StakePoolMetrics }) => {
const { dataSource, id, metrics, slot } = options;
const repos = dataSource.getRepository(CurrentPoolMetricsEntity);
const entity = {
activeSize: metrics.size.active,
activeStake: metrics.stake.active,
apy: metrics.apy || 0,
id,
liveDelegators: metrics.delegators,
livePledge: metrics.livePledge,
liveSaturation: metrics.saturation,
liveSize: metrics.size.live,
liveStake: metrics.stake.live,
mintedBlocks: metrics.blocksCreated,
slot,
stakePool: { id }
};

try {
await repos.upsert(entity, ['stakePool']);
} catch (error) {
// If no poolRegistration record is present, it was rolled back: do nothing
if (isErrorWithConstraint(error) && error.constraint === 'FK_current_pool_metrics_stake_pool_id') return;

throw error;
}
};

export const refreshPoolMetrics = async (options: RefreshPoolMetricsOptions) => {
const { id, logger, provider } = options;

logger.info(`Refreshing metrics for stake pool ${id}`);

try {
const { pageResults, totalResultCount } = await provider.queryStakePools({
filters: { identifier: { values: [{ id }] } },
pagination: { limit: 1, startAt: 0 }
});

if (totalResultCount === 0) return logger.warn(`No data fetched for stake pool ${id}`);

const { metrics } = pageResults[0];

await savePoolMetrics({ ...options, metrics });
} catch (error) {
logger.error(`Error while refreshing metrics for stake pool ${id}`, error);
}
};

export const stakePoolMetricsHandlerFactory: WorkerHandlerFactory = (options) => {
const { dataSource, logger, stakePoolProviderUrl } = options;
const provider = stakePoolHttpProvider({ baseUrl: stakePoolProviderUrl, logger });

return async (data: StakePoolMetricsUpdateJob) => {
const { slot } = data;

logger.info('Starting stake pools metrics job');

const pools = await dataSource.getRepository(StakePoolEntity).find({ select: { id: true } });

for (const { id } of pools) await refreshPoolMetrics({ dataSource, id: id!, logger, provider, slot });
};
};
14 changes: 11 additions & 3 deletions packages/cardano-services/src/PgBoss/types.ts
@@ -1,11 +1,19 @@
import { DataSource } from 'typeorm';
import { Logger } from 'ts-log';
import { STAKE_POOL_METADATA_QUEUE } from '@cardano-sdk/projection-typeorm';
import { Pool } from 'pg';
import { STAKE_POOL_METADATA_QUEUE, STAKE_POOL_METRICS_UPDATE } from '@cardano-sdk/projection-typeorm';

export const workerQueues = [STAKE_POOL_METADATA_QUEUE] as const;
export const workerQueues = [STAKE_POOL_METADATA_QUEUE, STAKE_POOL_METRICS_UPDATE] as const;

export type PgBossQueue = typeof workerQueues[number];

export interface WorkerHandlerFactoryOptions {
dataSource: DataSource;
db: Pool;
logger: Logger;
stakePoolProviderUrl: string;
}

// eslint-disable-next-line @typescript-eslint/no-explicit-any
export type WorkerHandler = (data: any) => Promise<void>;
export type WorkerHandlerFactory = (dataSource: DataSource, logger: Logger) => WorkerHandler;
export type WorkerHandlerFactory = (options: WorkerHandlerFactoryOptions) => WorkerHandler;
12 changes: 3 additions & 9 deletions packages/cardano-services/src/PgBoss/util.ts
@@ -1,12 +1,6 @@
import { PgBossQueue, WorkerHandlerFactory, workerQueues } from './types';
import { STAKE_POOL_METADATA_QUEUE } from '@cardano-sdk/projection-typeorm';
import { stakePoolMetadataHandlerFactory } from './stakePoolMetadataHandler';
import { PgBossQueue, workerQueues } from './types';

export const isValidQueue = (queue: string): queue is PgBossQueue => workerQueues.includes(queue as PgBossQueue);

/**
* Defines the handler for each pg-boss queue
*/
export const queueHandlers: Record<PgBossQueue, WorkerHandlerFactory> = {
[STAKE_POOL_METADATA_QUEUE]: stakePoolMetadataHandlerFactory
};
export const isErrorWithConstraint = (error: unknown): error is Error & { constraint: unknown } =>
error instanceof Error && 'constraint' in error;
Expand Up @@ -3,6 +3,7 @@ import { HttpServer } from '../../Http/HttpServer';
import { Logger } from 'ts-log';
import { MissingProgramOption } from '../errors';
import { PgBossHttpService, PgBossServiceConfig, PgBossServiceDependencies } from '../services/pgboss';
import { STAKE_POOL_METRICS_UPDATE } from '@cardano-sdk/projection-typeorm';
import { SrvRecord } from 'dns';
import { createDnsResolver } from '../utils';
import { createLogger } from 'bunyan';
Expand All @@ -14,7 +15,8 @@ export const PG_BOSS_WORKER_API_URL_DEFAULT = new URL('http://localhost:3003');

export enum PgBossWorkerOptionDescriptions {
ParallelJobs = 'Parallel jobs to run',
Queues = 'Comma separated queue names'
Queues = 'Comma separated queue names',
StakePoolProviderUrl = 'Stake pool provider URL'
}

export type PgBossWorkerArgs = CommonProgramOptions &
Expand Down Expand Up @@ -59,5 +61,8 @@ export const loadPgBossWorker = async (args: PgBossWorkerArgs, deps: LoadPgBossW

if (!db) throw new MissingProgramOption(pgBossWorker, PostgresOptionDescriptions.ConnectionString);

if (args.queues.includes(STAKE_POOL_METRICS_UPDATE) && !args.stakePoolProviderUrl)
throw new MissingProgramOption(STAKE_POOL_METRICS_UPDATE, PgBossWorkerOptionDescriptions.StakePoolProviderUrl);

return new PgBossWorkerHttpServer(args, { connectionConfig$, db, logger });
};
19 changes: 10 additions & 9 deletions packages/cardano-services/src/Program/programs/projector.ts
Expand Up @@ -3,9 +3,9 @@ import { CommonProgramOptions, OgmiosProgramOptions, PosgresProgramOptions } fro
import { DnsResolver, createDnsResolver } from '../utils';
import { HttpServer, HttpServerConfig } from '../../Http';
import { Logger } from 'ts-log';
import { ProjectionHttpService, ProjectionName, createTypeormProjection } from '../../Projection';
import { ProjectionHttpService, ProjectionName, createTypeormProjection, storeOperators } from '../../Projection';
import { SrvRecord } from 'dns';
import { TypeormStabilityWindowBuffer } from '@cardano-sdk/projection-typeorm';
import { TypeormStabilityWindowBuffer, createStorePoolMetricsUpdateJob } from '@cardano-sdk/projection-typeorm';
import { URL } from 'url';
import { UnknownServiceName } from '../errors';
import { createLogger } from 'bunyan';
Expand All @@ -16,9 +16,11 @@ export const PROJECTOR_API_URL_DEFAULT = new URL('http://localhost:3002');
export type ProjectorArgs = CommonProgramOptions &
PosgresProgramOptions<'StakePool'> &
OgmiosProgramOptions & {
projectionNames: ProjectionName[];
dropSchema: boolean;
dryRun: boolean;
poolsMetricsInterval: number;
projectionNames: ProjectionName[];
synchronize: boolean;
};
export interface LoadProjectorDependencies {
dnsResolver?: (serviceName: string) => Promise<SrvRecord>;
Expand All @@ -33,28 +35,27 @@ interface ProjectionMapFactoryOptions {

const createProjectionHttpService = async (options: ProjectionMapFactoryOptions) => {
const { args, dnsResolver, logger } = options;
storeOperators.storePoolMetricsUpdateJob = createStorePoolMetricsUpdateJob(args.poolsMetricsInterval)();
const cardanoNode = getOgmiosObservableCardanoNode(dnsResolver, logger, {
ogmiosSrvServiceName: args.ogmiosSrvServiceName,
ogmiosUrl: args.ogmiosUrl
});
const connectionConfig$ = getConnectionConfig(dnsResolver, 'projector', args);
const buffer = new TypeormStabilityWindowBuffer({ logger });
const { dropSchema, dryRun, projectionNames, synchronize } = args;
const projection$ = createTypeormProjection({
buffer,
connectionConfig$,
devOptions: args.dropSchema ? { dropSchema: true, synchronize: true } : undefined,
devOptions: { dropSchema, synchronize },
logger,
projectionSource$: Bootstrap.fromCardanoNode({
buffer,
cardanoNode,
logger
}),
projections: args.projectionNames
projections: projectionNames
});
return new ProjectionHttpService(
{ dryRun: args.dryRun, projection$, projectionNames: args.projectionNames },
{ logger }
);
return new ProjectionHttpService({ dryRun, projection$, projectionNames }, { logger });
};

export const loadProjector = async (args: ProjectorArgs, deps: LoadProjectorDependencies = {}): Promise<HttpServer> => {
Expand Down
6 changes: 5 additions & 1 deletion packages/cardano-services/src/Program/programs/types.ts
Expand Up @@ -27,9 +27,13 @@ export enum ServiceNames {
Rewards = 'rewards'
}

export const POOLS_METRICS_INTERVAL_DEFAULT = 1000;

export enum ProjectorOptionDescriptions {
DropSchema = 'Drop and recreate database schema to project from origin',
DryRun = 'Initialize the projection, but do not start it'
DryRun = 'Initialize the projection, but do not start it',
PoolsMetricsInterval = 'Interval between two stake pools metrics jobs in number of blocks',
Synchronize = 'Synchronize the schema from the models'
}

export enum ProviderServerOptionDescriptions {
Expand Down

0 comments on commit 3f0b512

Please sign in to comment.