New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migration from LoopBack 3.x #1849

Open
bajtos opened this Issue Oct 15, 2018 · 5 comments

Comments

Projects
None yet
5 participants
@bajtos
Copy link
Member

bajtos commented Oct 15, 2018

When LoopBack 4 gets more mature and reaches feature parity with LoopBack 3, we should write a migration guide and automated tooling to help existing LoopBack 3 users to upgrade their projects to LoopBack 4.

Spike tasks: #485

@bajtos bajtos added the epic label Oct 15, 2018

@bajtos

This comment has been minimized.

Copy link
Member

bajtos commented Oct 15, 2018

Cross posting my earlier thoughts - please note some parts are outdated and need tweaks to make them compatible with the latest LB4 design.


#485 (comment)

Here are my thoughts on the migration story.

At high-level, LoopBack 3.x applications consist of three big "parts"

  • Persistence layer (this includes talking to backend services like SOAP/REST)
  • Outwards facing REST API
  • App-wide setup - Express middleware, boot scripts, etc.

In the persistence layer, users can contribute the following artifacts:

  1. Definitions of Model data types (properties, validations)
  2. Definition of data sources
  3. Configuration of models (which datasource are they attached to)
  4. Operation hooks

At the public API side, users can define:

  1. Which built-in methods should be exposed (think of disableRemoteMethodByName)
  2. Custom remote methods
  3. before/after/afterError hooks at application-level
  4. before/after/afterError hooks at model-level
  5. before/after/afterError hooks at model method level

LoopBack Next was intentionally designed to allow users to choose their ORM/persistence solution, and our initial version of @loopback/repository is based on juggler 3.x. That makes it possible for users to reuse their existing model definitions, migrating their application incrementally.

The proposal

Here is my proposal:

  • Keep the existing model definitions in .json and .js files, datasources configuration, etc. (But consider moving them around to get a saner file structure - this should be done by our migration tool we need to build.)
  • Write a small amount of code to bind LB 3.x Models with LB Next Context, e.g. for each {Model} create models.{Model} binding.
  • This allows us to reuse model definitions, datasources & connectors, operation hooks.
  • The biggest difference is in the REST API. Fortunately, we have tools to inspect all remote methods and generate Swagger specification for them.
  • For each public model, use introspection and swagger generator to code-generate a controller class implementing the same public REST API as the original LB 3.x model. This is a tool we need to build.
  • Each controller method should be a simple wrapper calling the original model method.

A simplified example:

@api({basePath: '/products'})
export class ProductController {
  constructor(
    @inject('models.Product') private Product: Entity
  ){}

  @get('/')
  @param.query.object('where')
  // TODO describe response type
  find(where: Object): Product[] {
    return await this.Product.find(where);
  }

  @get('/custom')
  // ...
  customMethod(arg: number): Object {
    return await this.Product.customMethod(arg);
  }

  @patch('/{id}')
  // ...
  prototypeUpdateAttributes(id: number, data: Partial<Model>) {
    const instance = await this.Model.sharedCtor(id);
    return await instance.updateAttributes(data);
  }

  // ...
}

Remoting hooks

Let's talk about remoting hooks now and how to translate them to LB Next:

  1. Application-level hooks should be invoked as part of the Sequence.
  2. For model-level hooks, I am proposing the convention that a controller class can provide before, after and afterError methods. These methods can be either invoked explicitly from each method (which is brittle), or we need to improve our framework to invoke them automatically as part of invokeMethod.
  3. For method-level hooks, the code should go directly into the controller method itself.
@api({basePath: '/products'})
export class ProductController {
  constructor(
    @inject('models.Product') private Product: Entity
  ){}

  before(route: Route, args: OperationArgs) {
    console.log('Invoking remote method %s with arguments %s', 
      route.describe(); JSON.stringify(args));
  }

  @get('/')
  @param.query.object('where')
  find(where: Object): Product[] {
    console.log('Here we can modify `where` filter like we were in a before-remote hook');

    return await this.Product.find(where);
  }

  // etc.
}

I don't think we can automate this migration step, because remoting hooks expect to receive a strong-remoting context that we don't have in LoopBack next. A good documentation with helpful examples is needed.

Alternatively, we can let the controller to provide invokeMethod where it's up to users to define whatever phases they need, so that they are not constrained by three phases before/after/afterError. (However, they can implement invokeMethod calling before/after/afterError if they like it.)

class MyController {
  invokeMethod(route: Route, args: OperationArgs) {
    console.log('Invoking remote method %s with arguments %s', 
      route.describe(); JSON.stringify(args));
    return await this[route.methodName](...args);
  }
}

// alternate invokeMethod implementation supporting before/after/afterError hooks
// this can be a Controller Mixin provided by an extension
  invokeMethod(route: Route, args: OperationArgs) {
    if (this.before)
      await this.before(route, args);

    try {
      const result = await this[route.methodName](...args);
      if (this.after)
        result = await this.after(route, args, result);
      return result;
    } catch(err) {
     if (this.afterError)
       const handled = await this.afterError(route, args, error);
       if (handled) return;
    }
    throw err;
  }

Summary:

  • A tool for shuffling files around to get a sane structure for LB3 + LB4 files
  • A tool for generating controllers from remoting metadata
  • Runtime support for controller hooks (before, after, afterError)
  • Instructions on how to migrate remoting hooks

Remaining things to figure out:

  • how to migrate middleware into Sequence
  • how to migrate boot scripts
  • where to keep Swagger definitions for Models so that we can reference them in controller specs

Possibly out of scope:

  • how to migrate ACLs that we don't support yet
  • how to migrate built-in models like User, AccessToken, etc.
  • change-tracking, replication and offline-sync
  • push notifications
  • storage component

Next level

  • Migrate LB 3.x .json and .js files to use @loopback/repository. The generated controllers should use typed Repositories instead of juggler 3.x based Model classes.

Justification

You may ask why to code-generate all those controller files? In my experience, one of the most frequently sought feature of LB 3.x is the ability to control and customize the public API. Users start with a wide built-in CRUD API provided by the LoopBack, but soon enough then need to lock the API down, expose only subset of endpoints, and customize how exactly these endpoints work.

In LoopBack 3.x, this was a complex process that required hooks at many different levels (middleware/strong-remoting phases, remoting hooks, operation hooks) and unintuitive API like disableRemoteMethodByName.

In my proposal, customization of the REST API is as easy as editing the generated code.

  • Need to hide a built-in (generated) endpoint? Just delete the controller method!
  • Need to modify the request arguments before executing the underlying CRUD method? Add that code into the controller method.
  • Need to add more parameters to a built-in (generated) endpoint? Just modify the controller method.
  • Need to post-process the value returned from the model? Add the transformation after you awaited the result of the model method.

#485 (comment)

Remaining things to figure out:

  • how to migrate middleware into Sequence
  • where to keep Swagger definitions for Models so that we can reference them in controller specs

Possibly out of scope:

  • how to migrate boot scripts
  • how to migrate ACLs that we don't support yet
  • how to migrate built-in models like User, AccessToken, etc.
  • change-tracking, replication and offline-sync
  • push notifications
  • storage component

Next level

Migrate LB 3.x .json and .js files to use @loopback/repository. The generated controllers should use typed Repositories instead of juggler 3.x based Model classes.

@raymondfeng

This comment has been minimized.

Copy link
Member

raymondfeng commented Oct 15, 2018

I have some initial ideas as follows:

  1. Create the swagger.json from an existing LB3 app
  2. Use lb4 openapi to generate controllers and models
  3. Capture datasources from LB3 as json
  4. Use lb4 datasource to recreate it from 3
  5. Use lb4 repository/service to create repos/services from model-config.json
  6. Handle other artifacts such as components, middleware, boot-scripts, hooks, ACLs, .... (TBD)
@marioestradarosa

This comment has been minimized.

Copy link
Contributor

marioestradarosa commented Oct 15, 2018

@bajtos , @raymondfeng for CRUD,

  • I believe that we should be able to read hook definitions and generate the controller automatically
  • The models can also be read and use either @model() decorator or generate the model class

In other words, we can run a command inside an lb3 app like so:

lb4 migrate --output ../mylb4app

The CLI could read the datasources, models, hooks for CRUD and generate the appropriate files.

thoughts?

@yagobski

This comment has been minimized.

Copy link

yagobski commented Dec 17, 2018

any progress about this task. How we can migrate large app from LB3?

@hacksparrow

This comment has been minimized.

Copy link
Member

hacksparrow commented Dec 18, 2018

@yagobski you can track the progress being made on feature parity with LB 3 at #1920.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment