Feature: Implementing built-in middleware and middleware loader

[kokokuo]

Description

This PR makes 4 things:

1. Provider logger through getLogger

Using the tslog to implement getLogger method and get logger by each package name LoggingScope enum. We use the tslog is that the plugin could help us to log the data to different places by attachTransport to implement code easily, so we could integrate to grafana, also the tslog could update setting after the logger generated not like log4js.

// scopeName including `CORE`, `BUILD`, `SERVE`, `AUDIT`
const logger = getLogger({ scopeName: 'AUDIT' });
logger.info('....')

2. refactor the validator loader supporting the dynamic module import

After the last meeting suggestion, change the original lookup of each file way to import module (seeking the index.ts default) way and load validators.

Here is the built-in validators:

// module folder
- data-type-validators/
   - index.ts
   - IntegerValidator.ts
   - StringValidator.ts

// The index.ts code
import { DateTypeValidator } from './dateTypeValidator';
import { IntegerTypeValidator } from './integerTypeValidator';
import { StringTypeValidator } from './stringTypeValidator';
import { UUIDTypeValidator } from './uuidTypeValidator';

export default [
  DateTypeValidator,
  IntegerTypeValidator,
  StringTypeValidator,
  UUIDTypeValidator,
];

Below is the custom validators by user-defined:

// folder in module user customized
- extension/
   - index.ts
   - validators/
      - index.ts
      - customValidator.ts
- package.json

// The extension/index.ts
import validatorClasses from './validators'

export default = {
   validators: validatorClasses,
}

// The extension/validators/index.ts
import { CustomValidator } from './customValidator'
export default [CustomValidator]

3. add built-in middleware and middleware loader which supports the dynamic module import

Support 3 components in the built-in middleware currently:

• CORS ( through koa-cors )
• Rate Limit ( through koa2-ratelimit )
• Request Id ( Generate Request Id )
• Audit Logging ( Log the request and response, if set dispayRequestId, it will get the request-id from the above Request Id middleware.

Below is the built-in module:

// module folder
- built-in-middlewares/
   - index.ts
   - corsMiddleware.ts
   - requestIdMiddleware.ts
   - auditLogMiddleware.ts

// The index.ts in built-in-middlewares/
import { CorsMiddleware } from './corsMiddleware';
import { RequestIdMiddleware } from './requestIdMiddleware';
import { AuditLoggingMiddleware } from './auditLogMiddleware';

export default [CorsMiddleware, RequestIdMiddleware, AuditLoggingMiddleware];

And below is the custom middleware by user-defined:

// folder in module user customized
- extension/
   - index.ts
   - middlewares/
      - index.ts
      - customMiddleware.ts
   - validators/
      - index.ts
      - customValidator.ts
- package.json

// The extension/index.ts
import middlewareClasses from './middlewares'
import validatorClasses from './validators'

export default = {
   middlewares: middlewareClasses,
   validators: validatorClasses,
}

// The extension/middlewares/index.ts
import { CustomMiddleware } from './customMiddleware'
export default [CustomMiddleware]

// The extension/validators/index.ts
import { CustomValidator } from './customValidator'
export default [CustomValidator]

How To Test / Expected Results

For the test result, please see the below test cases that passed the unit test:

The core package

The serve package

Commit Message

• ef09460 - feat(core,serve): add logger, refactor schema type, add pagination tests.

  • add tslog package and create getLogger by LoggerFactory.
  • refactor PaginationMode, FieldInType, FieldDataType value to lower case.
  • fix transform method in PaginationTransformer.

• e81916e - feat(core): refactor validator loader for supporting load module by reading index file through read folder.

  • add ModuleLoader for loader extended.
  • fix getLogger to support update options for the existing logger in the map.
  • refactor DateTypeValidator, IntegerTypeValidator, StringTypeValidator, UUIDTypeValidator for removing default.
  • add index.ts in data-type-validators for export default.
  • refactor validator test cases.

• 096f2a5 - feat(serve): add built-in middleware, and loader

  • add built-in middlewares: CORSMiddleware, RequestIdMiddleware, AuditLoggingMiddleware
  • add a middleware loader to load the middleware module.
  • add middleware-related test cases.

• 7951238 - feat(core, serve): refactor validator and middleware loader, add middleware in server.

  • refactor ModuleLoader class to defaultImport function.
  • rename "Type" to ClassType.
  • fix checking options in middleware.
  • set middleware and add test cases in vulcan app server.

• b1e883b - feat(serve): add rate limit middleware, refactor middleware config structure.

  • add BuiltInMiddleware for built in middleware extend.
  • support RateLimitMiddleware.
  • refactor MiddlewareConfig for enabling or disabling middleware.

• 70c1b13 - fix(core): restore upper case for enum type in the artifact, add ignore tag to prevent detecting class .ts file coverage in test/

  • restore upper case for enum type in artifact, including PaginationMode, FieldInType, FieldDataType.
  • add ignore tag in IPTypeValidator class in test/ to prevent detecting unnecessary code.

• 1d011d6 - fix(serve): fix for returning next() directly, converting to lowercase to get request-id, and using the run method for async local storage.

  • make next() return directly when enabled is false in middleware.
  • fix for converting to lowercase to get request-id in RequestdMiddleware and setup default value if name or field does not exist.
  • fix to use run method for async local storage to prevent using experimental method.
  • move the loadBuiltIn method to loader.spec.ts for only the test used.

[oscar60310]

Other part LGTM.

[oscar60310]

NIT: Maybe we can do an early return here to avoid some codes being executed accidentally.

Suggested change

	if (!this.enabled) await next();
	if (!this.enabled) return next();

[kokokuo]

Thanks for reviewing and suggesting, it has been changed to return 😄 .

[oscar60310]

I remember node http server “lower casing” the headers, but I'm not sure :D
https://medium.com/@andrelimamail/http-node-server-lower-casing-headers-365764218527

Suggested change

	? (request.header[name] as string)
	? (request.header[name.toLowerCase()] as string)

[kokokuo]

Thanks for reviewing and suggesting, it has been changed to return 😄 .

[oscar60310]

Nice to set requestId for synchronous executions, but because it is still Experimental feature, can we use .run function instead?
https://nodejs.org/api/async_context.html#asynclocalstorageenterwithstore

[kokokuo]

Thanks for reviewing and suggesting, it has been changed to return 😄 .

[oscar60310]

We've just created some middleware to normalize these values XD
https://github.com/Canner/vulcan/pull/16/files#diff-b597fbd22f8a9d7fcff5148d1a0ab79ba84c58844bd02b00c66d36c99b5b974cR23

I can update my code if you decided to use lower cases.

[kokokuo]

Thanks for reviewing and pointing, I change back to uppercase, so you don't need to update 😃

[oscar60310]

There was no difference between these functions here, may I ask why we changed it?

[kokokuo]

Thanks for reviewing and pointing, I just want to use join more easily understand, because resolve is used to like cd, but in this case, join could do the same thing by joining the path, and no need to think the cd path in mind and check.

[oscar60310]

NIT: The response body of our API server might be huge, we can let users to set what data they want to record in config in the future.

[kokokuo]

Thanks for reviewing and suggesting, currently I add the TODO comment to mention me to enhance the part after the feature almost ready in our phase 1 plan 😃

[oscar60310]

NIT
We used dynamic import in the middleware loader:

export const loadBuiltIn = async () => {
  // built-in middleware folder
  const builtInFolder = path.join(__dirname, 'built-in-middlewares');
  // read built-in middlewares in index.ts, the content is an array middleware class
  return (
    (await defaultImport<ClassType<BaseRouteMiddleware>[]>(builtInFolder)) || []
  );
};

It sometimes means we don't want to load the code unit we need them, re-exporting them here causes these codes to be loaded when the application starts, we might remove the export here to fix the issue.

[kokokuo]

Thanks for reviewing and pointing, and you are right, it will be used if someone calls it when the application has started and use it again, so I move the loadBuiltIn to loader.spec.ts because it was originally used for test only 😃 .

If I misunderstand your point and suggestion, please let me know~

[oscar60310]

Could you add a line at the top of this file /* istanbul ignore file */ to exclude this file from coverage calculation?

[kokokuo]

Thanks for reviewing and suggesting, I have added the ignore line on the class, and it has disappeared in coverage, thanks!

[kokokuo]

Hi @oscar60310, I have fixed it, please check it, thank you for your reviewing ~