Search

[rainydio]

Search

I'm replacing search implementations within core-database. There are many parallels between 2.x version and the one being made.

Caution: Some terms were reused, but now have different meaning. This may cause confusion.

Service layer

Instead of directly referencing TransactionRepository, API controllers will inject TransactionsDatabaseService that is referencing TransactionRepository.

Currently repositories do not have access to IoC container. Relatively high level search function requires access to wallet repository. TransactionRepository injects wallet repository reference in a way that it should not.

// service-provider.ts

transactionRepository.getWalletRepository = () => {
  // Inversify isn't responsible for the instance creation so we can't inject.
  return this.app.getTagged(
    Container.Identifiers.WalletRepository,
    "state",
    "blockchain"
  );
};

// repositories/transaction-repository.ts

// Workaround to include transactions (e.g. type 2) where the recipient_id is missing in the database
const recipientWallet: Contracts.State.Wallet = this.getWalletRepository().findByAddress(
  first.value as string
);

TransactionsDatabaseService

New TransactionsDatabaseService has benefits of dependency injection and sits in between API controllers and TransactionRepository.

It has similar responsibilities to TransactionsBusinessRepository in 2.x. For now it will include only one general search method that combines all use cases.

class TransactionsDatabaseService {
  public search(
    query: SearchQuery<Transaction, TransactionCriteria>
  ): SearchResult<Transaction>;
}

SearchQuery is built from request by API layer:

type SearchQuery<TEntity, TCriteria> = {
  pagination?: {
    offset: number;
    limit: number;
  };

  ordering?: {
    entity: new () => TEntity;
    property: keyof TEntity;
  }[];

  criteria: TCriteria;
};

Apart from pagination and ordering fields it also includes generic criteria field. Notably it has different meaning than SearchCriteria used in 2.x. Few examples of criteria object:

{
  typeGroup: Enums.TransactionTypeGroup.Core,
  type: Enums.TransactionType.Transfer,
  recipientId: "DPNQjSnZg1SkJr5ZaBw9QGiYeuFDV7mDnh"
};

{
  senderPublicKey: "03f87e8d4ecac9f7a167b95c43acf7a8ec87619f9dac241fb58894d96c572bc760",
  nonce: [{ from: 1, to: 10 }, 15, { from: 20 }],
}

Properties are combined using AND expression. Each property value may be an array. Array elements are combined using OR expression.

Search layer

The key building block is Filter interface and its single function:

interface Filter<TCriteria> {
  getExpression(criteria: TCriteria): Promise<Expression>;
}

Filter

Filter converts criteria into expression:

• Criteria is complex object describing filter parameters in business-level terms.
• Expression is complex object describing query WHERE expression to be executed by TransactionRepository.

Expression

Expression object role is similar to SearchCriteria in 2.x. It describes WHERE expression structure. There are several types of expressions: void, equal, between, greaterThanEqual, lessThanEqual, like, contains, and, or. The last two allows composition that was missing in 2.x.

Query builder provided by TypeORM isn't really suited for dynamically building complex queries. I found it easier to compose my own pieces and convert them to sql string and parameters object.

Criteria

Criteria object role is similar to SearchQuery in 2.x. It describes filter parameters in simple terms. Notably TransactionCriteria type isn't really defined anywhere in code. Instead it is inferred from filter composition.

Filter composition

It's easier to understand composition starting from simple filters and how they can be combined.

EntityEnumPropertyFilter

Simplest filter that only produces equal expression.

class EntityEnumPropertyFilter<TEntity, TProperty extends keyof TEntity>
  implements Filter<TEntity[TProperty]> {
  private readonly entity: new () => TEntity;
  private readonly property: TProperty;

  public constructor(entity: new () => TEntity, property: TProperty) {
    this.entity = entity;
    this.property = property;
  }

  public getExpression(criteria: TEntity[TProperty]): Promise<EqualExpression> {
    return new EqualExpression(this.entity, this.property, criteria);
  }
}

// for example:
//  TEntity = Transaction
//  TProperty = "type"
//  TEntity[TProperty] => Transaction["type"] => Enums.TransactionType

const transactionTypeFilter = new EntityEnumPropertyFilter(Transaction, "type");

const typeIsTransferExpression = await transactionTypeFilter.getExpression(
  Enums.Transaction.Transfer
);

Type of criteria is taken from entity property type.

OptionalOrFilter

Simplest composition filter adds optional OR criteria for existing filter:

class OptionalOrFilter<TCriteria> implements Filter<TCriteria> {
  private readonly filter: Filter<TCriteria>;

  public constructor(filter: Filter<TCriteria>) {
    this.filter = filter;
  }

  public async getExpression(
    criteria: TCriteria | TCriteria[]
  ): Promise<Expression> {
    if (Array.isArray(criteria)) {
      const expressions = await Promise.all(
        criteria.map(c => this.filter.getExpression(c))
      );
      return new OrExpression(expressions);
    } else {
      return this.filter(criteria);
    }
  }
}

const transactionTypeFilter = new EntityEnumPropertyFilter(Transaction, "type");
const transactionTypeOrFilter = new OptionalOrFilter(transactionTypeFilter);

const typeIsTransferExpression = await transactionTypeOrFilter.getExpression(
  Enums.Transaction.Transfer
);
const typeIsTransferOrMultiPaymentExpression = await transactionTypeOrFilter.getExpression(
  [Enums.Transaction.Transfer, Enums.Transaction.MultiPayment]
);

Variants of providing criteria as single object, or array of objects are allowed.

Other simple filters

There are also:

• FilterFn to create filter instance from function.
• AndFilter to AND expressions by criteria of filter1 applied also to filter2.
• PropertiesAndFilter to compose filters by properties creating combined criteria type.

Business-level filters:

These filters are injectable, and may inject services or other business-level filters.

• TransactionFilter injected by TransactionsDatabaseService.
• TransactionWalletFilter produces expression filtering by recipient or sender public key (used by /wallets/{id}/transactions).
• TransactionSenderIdFilter injects WalletRepository to resolve sender public key out of address.

In the end TransactionsDatabaseService converts TransactionCriteria into Expression using TransactionFilter and calls TransactionRepository to execute query using expression provided.

Aftewards

There are couple of features that may be added later. API validation schemas can be built by filters. Or criteria candidate object can be built from any and errors thrown if it can't be.

Type information may be exported and used by REST client.

[rainydio]

#3644 merged, so this is done

[kristjank]

👏