Feature: Implementing pagination strategy and data source driver interface

[kokokuo]

Description

This PR includes some components for two parts:

1. Support setting pagination parameter to provide mapping query parameter in API

Implementing pagination strategy abstract class and its sub-classes

Add PaginationStragtegy abstract class and support OffsetBasedStragtegy, CursorBasedStragtegy, KeysetBasedStragtegy sub-clasesses for transforming the koa ctx request to pagination format.

2. Add data source driver interface

Add IDataSource interface for implementation in the future, the return format currently returns objects temporarily.

How To Test / Expected Results

For the test result, please see the below test cases that passed the unit test:

Commit Message

• faf6f08 - feat(serve): add interface of data source, pagination, and pass to the data query builder
  • add pagination field in APISchema.
  • add PaginationStrategy interface and offset, cursor, keyset based strategy class.
  • add PaginationTransformer to do transfrom from koa ctx .
  • add data source interface IDataSource.
  • update test cases of each SQL clause method of DataQueryBuilder.

[oscar60310]

Other part LGTM

[oscar60310]

Just want to confirm: Would we call templateEngine.execute(name, req, pagination) here?

[kokokuo]

Thanks for reviewing, yes, the original is called templateEngine.render, right?

[oscar60310]

Shall we move "paginate transforms" to "route-component" level? I guess you use header here in order because of using the same transformer in both restful and graphql routes, but indicating pagination in headers is strange.

We use in params or arguments more often:
/users?offset=100&limit=10

users(offset: 100, limit: 10) {
  id
}

[kokokuo]

Thanks for reviewing and finding the weird part, it should pass by query string would be correct.

[kokokuo]

It has been fixed and move the "paginate transforms" to "route-component" level, please check it.

[oscar60310]

1. We won't always use the key "createAt" to paginate, I think we can let users definition their keys in schema file.
2. I don't know the difference between keyset and cursor strategy. IMO, they both use a "key" to select limited data below or above the row and return a page. Can we merge these two strategies?

[kokokuo]

Thanks for reviewing and suggesting.

After discussing with Ivan, we will keep the keyset strategy now for the second part, because not sure how the Driver section handles pagination.

For the first part - define the key name in the schema to define what key used in keyset pagination has been added in KeysetBaseStrategy, please check it.

[oscar60310]

Wrong message here~

[kokokuo]

Thanks for reviewing and finding wrong message, it has been fixed.

[oscar60310]

We're still checking the "header", it should be query too :D

[kokokuo]

Yes, thanks founding the missing part, it has been fix~

[oscar60310]

It has been fixed and move the "paginate transforms" to "route-component" level, please check it.

Nice, but I think we should do some further action.

Shall we move "paginate transforms" to "route-component" level? I guess you use header here in order because of using the same transformer in both restful and graphql routes, but indicating pagination in headers is strange.

We use in params or arguments more often: /users?offset=100&limit=10

users(offset: 100, limit: 10) {
  id
}

I meant we might need to move the transform code into routers, because the pagination arguments come from different ways.

restfulRoute.ts

 protected async handleRequest(
    ctx: KoaRouterContext,
    reqParams: RequestParameters,
  ) {
   // handle user?offset=10&limit=10
    const pagination = someFunctionToGetPaginationViaQueryParameters(ctx);
    // TODO: implement query by dataset
    ctx.response.body = {
      reqParams,
      pagination,
    };
    return;
  }

graphQLRoute.ts

 protected async handleRequest(
    ctx: KoaRouterContext,
    reqParams: RequestParameters,
  ) { 
    // handle user(offset: 10, limit; 10) { }
    const pagination = someFunctionToGetPaginationViaRequestBody(ctx);
    // TODO: implement query by dataset
    ctx.response.body = {
      reqParams,
      pagination,
    };
    return;
  }

[kokokuo]

Hi, @oscar60310, thanks for catching the graphql and restful handle request content, not a consistent part!

Currently, I have refactored for separating the request and pagination transformer to a prepare method, and made graphql* and restful route implemented prepare method to handle request and pagination transformer

However, currently, the graphql route is not implemented the prepare method because we have not yet developed graphql part ).

restfulRoute.ts:

public async respond(ctx: KoaRouterContext) {
    const transformed = await this.prepare(ctx);
    await this.handle(transformed);
   // TODO: get template engine handled result and return response by checking API schema
    ctx.response.body = {
      ...transformed,
    };
  }

  protected async prepare(ctx: KoaRouterContext) {
    const reqParams = await this.reqTransformer.transform(ctx, this.apiSchema);
    await this.reqValidator.validate(reqParams, this.apiSchema);
    const pagination = await this.paginationTransformer.transform(
      ctx,
      this.apiSchema
    );
    return {
      reqParams,
      pagination,
    };
  }

grapqlRoute.ts:

 protected async prepare(ctx: KoaRouterContext) {
    /**
     * TODO: the graphql need to transform from body.
     * Therefore, current request and pagination transformer not suitable (need to provide another graphql transform method or class)
     */

    return {
      reqParams: {},
    };
  }

  public async respond(ctx: KoaRouterContext) {
    const transformed = await this.prepare(ctx);
    await this.handle(transformed);
   // TODO: get template engine handled result and return response by checking API schema
    return transformed;
  }