Lazy Loading Engines Attack Plan!

[nathanhammond]

Come one, come all! It's time to begin the process of extending ember-engines to support lazy loading! This document should describe every single step to get us from beginning to end, provide points of contact, and help you identify places where you can help us deliver it sooner. This top-post will be continuously updated.

Background

As ember-engines exist right now you should consider them to be an isolation primitive. They allow you to provide guarantees as to the boundaries of your engine in its interaction with the host application and no more. This provides approximately zero benefit at runtime. However, this isolation primitive was built with the goal of being able to leverage that runtime isolation guarantee at build time to do clever things and allow us to asynchronously load engines. And if you're interested in that, you're in the right place!

Effort

Most of these changes will be made inside of the ember-engines addon. Our goal is to land most of the functionality there first to iron out how it is going to work. As we identify pieces that we can migrate into Ember and Ember CLI we will do so behind feature flags.

Participating

We want you to be able to dive in on any of this and are continuously updating this document with details on how to do particular pieces. Please feel free to claim portions of the effort by leaving a comment on this issue!

This effort is being organized by the Engine Lazy Loading Strike Team: Dan Gebhardt (@dgeb), Miguel Madero (@MiguelMadero), Nathan Hammond (@nathanhammond), Robert Jackson (@rwjblue), and Trent Willis (@trentmwillis). All of us will be able to review and provide guidance across most tasks. If you have detailed questions in specific areas try to track down the people that we've assigned to be responsible for particular sections first.

Issue Changelog

• 2016-07-13 10:11AM PDT - Upstream to Ember Core section added to track relevant issues to land prior to the 2.8 beta cutover.
• 2016-07-12 9:10PM PDT - Added clarity about the loader.js approach.
• No edits yet required! Substantive changes will appear here.

---

Build - @nathanhammond, @rwjblue

Ember CLI is responsible for doing all sorts of things at build time. This is area is concerned with what gets spit out after running ember build in an Ember application which includes lazy-loading engines. Note: engines may not have circular dependencies.

Goals

• Stick to familiar patterns from Ember.
• Enable caching of engine assets.
• Enable engines to be (mostly) separate deployable bundles.
  • The host application will still have to be the entry point for an engine rebuild.
  • The host application's index.html will contain the asset manifests for all engines using a meta config module.
  • The host application's vendor.js will be updated if the engine's routes.js file changes.

Approach

All occurrences of {engine} below are the name of the engine according to the name property in the module export of the index.js file inside of the engine.

Asset Manifest

• Constraint: The engine must enumerate its assets in order for them to be loadable after fingerprinting.
• Input: Broccoli tree after broccoli-asset-rev.
• Output: Asset manifest. See below section.
• Delivery: Inserted into a meta tag config inside of the host application's index.html

Router

• Constraint: {engine}/routes.js must be present at boot of the host application as well as anything it imports.
• Input: {engine}/routes.js
• Output: AMD modules in the engine namespace: {engine}/routes.
• Delivery: Bundled into the host application's vendor.js. Presence inside of the host application's vendor.js is considered undefined behavior and may change in the future.

Engine Code

• Constraint: Must be individually addressable by the asset loading service.
• Input: {engine}/addon/*
• Output: dist/assets/{engine}.(js|css)
• Delivery: Asset loading service when attempting to route to engine.

Engine Vendor

• Constraint: Must be individually addressable by the asset loading service.
• Input: {engine}/vendor/*
• Output: dist/assets/{engine}-vendor.(js|css)
• Delivery: Asset loading service when attempting to route to engine.

Engine Static Assets

• Constraint: Must follow current patterns of how addons place public assets.
• Input: {engine}/public/*
• Output: dist/{engine}/*
• Delivery: Assets in public have no default behavior and must be manually loaded by the engine code.

Tasks

• Make the ember-engines EngineAddon object handle the production of the correct assets. [PR] [Owner: @nathanhammond]
• Extract this section into documentation for the EnginesAddon build process.

---

Asset Manifest - @trentmwillis

It is possible that engines can be included into an application multiple times at different mount points. We should support the ability to do so. This is an easy area to participate in.

Goals

• Allow engine assets to be fingerprinted for caching reasons.
• Support multiple inclusion of same engine without duplicate backing engine load.

Approach

This is non-normative and provided as a sketch for the beginning of an implementation. It assuredly has gaps. We probably want to have individual named manifests for each engine. We will need a way to map route path microsyntax to the engine's moduleName which can come as a separate meta config item.

Sketch global engine config definition appearing at something like config/engines:

{
  engines: {
    'name-specified-by-mount': 'engine-name-from-package-json',
    'other-engine-mount': 'neat-engine',
    'duplicate-engine-mount': 'neat-engine',
    'other-engine-mount.nested-engine': 'foo-engine'
  }
}

Sketch engine manifest appearing at something like config/engines/neat-engine:

{
  'neat-engine': {
    app: 'neat-engine-2b00042f7481c7b056c4b410d28f33cf.js',
    appStyles: '...',
    vendor: '...',
    vendorStyles: '...'
  }
}

Tasks

• Specify the asset manifest in more detail. [Owner: @trentmwillis]
  • Land the associated RFC.
• Generate the manifest from build output. [PR] [Owner: @trentmwillis]
• Land multi meta module PR on Ember CLI.
• Insert the manifest into index.html as a meta tag from the ember-asset-loader addon using contentFor. [PR] [Owner: @trentmwillis]

---

Asset Loading Service - @trentmwillis

This is a service that should be implemented which receives an Asset Manifest and returns a promise which resolves once all of the assets from that manifest have finished loading. This is an easy area to participate in.

Goals

• Simple API.
• Prevent duplicate requests for the same engine.
• Supports whatever the Asset Manifest is from above.
• Identify engines by route path microsyntax from the host application root.

Approach

This is non-normative and provided as a sketch for the beginning of an implementation. It assuredly has gaps.

{
  lookup: {},
  load(bundle) {

    var loaded = new Promise(function(resolve, reject) {
      // Load the assets.
    });

    this.lookup[bundle.name] = loaded;

    loaded.then((response) => {
      this._populateRegistry(response);
      this.lookup[bundle.name] = true;
    })

    // Used to block the 
    return loaded;
  },
  _populateRegistry() {
    // TODO: Populate the loader.js aliases.
  }
}

Tasks

• Design the service's public API. [Owner: @trentmwillis]
  • Land the associated RFC.
• Implement the service. [Addon] [Owner: @trentmwillis]
• Register this service during the boot of Ember itself through an instance-initializer. [[PR]((https://github.com/trentmwillis/ember-asset-loader/pull/10)] [Owner: @trentmwillis]
• loader.js cache miss should probably be reset on each engine load.

The following may not work, but should be explored as an isolation guarantee:

• After script and style tags mount the engine modules which are registered will be duplicated inside of loader.js alias for the modules at the engine's mount point.
• The asset loading service rewrites the loading definition of all modules by getting a reference to the function and the module arguments and doing a new define for each of them.
• Duplicate engines will identify that modules are already loaded for neat-engine and will simply insert a duplicate module inside of loader.js at the appropriate mount point.

---

Runtime Routing - @trentmwillis, @nathanhammond

Support for async loading of route handlers. The route files themselves won't be present at time of transition, so routing will have different outcomes dependent upon host application or crossing engine boundaries.

Goals

• Minimize changes to existing routing behavior.
• Pause transition at engine boundaries while loading assets from the asset service.

Approach

This has been pretty-well-distilled into tasks in that there is little that still needs design work.

Tasks

• Optional - Deprecate default query params inside of Ember.
• Document route-serializer usage in ember-engines and how it differs from behavior outside of engines. [PR] [Owner: @trentmwillis]
• Document inaccessibleByURL behavior in ember-engines. [Owner: @trentmwillis]
• Document change in loading behavior for failed transitions when crossing an engine boundary. [PR] [Owner: @trentmwillis]
• Support simultaneous loading of engines if you cross multiple engine boundaries. Works by virtue of getHandler being called synchronously. No additional work needed. [Owner: @trentmwillis]
• Land async getHandler in Router.js and merge into Ember.js. [Owner: @trentmwillis]
• Address loading substates for this.mount. [PR] [Owner: @trentmwillis].

---

Community Breakage - @MiguelMadero

It's possible that some of these changes will break tooling that exists in the Ember community, such as the Ember Inspector. It is hard to anticipate what these may be. Plan to review behaviors in Ember Inspector and LOG_RESOLVER.

• Identify any issues with early resolution of routes in the ember-inspector.
• Identify any issues with LOG_RESOLVER eager resolution.

---

Routeless Lazy Engines - @dgeb, @rwjblue

Routeless lazy engines are engines which are mounted using the {{mount}} helper. They will be discovered at build time while walking the dependency tree and packaged identically to above. They can be thought of as "lazy components" which also bundle all of their own dependencies.

Goals

• Route transition will become dependent upon template content?.
• Support async loading at the moment the mount keyword is encountered in the template?

Approach

How this will function is poorly defined as template rendering is currently a synchronous behavior. The "easiest" solution likely requires us to know of {{mount}} usage in a template during the transition lifecycle.

Tasks

• ???

---

Upstream to Ember Core

• Partial rollback of route-serializers RFC.
• Upstream code into Ember prior to 2.8 beta.
  • Update ember-resolver addon to use resolveEngine. [PR]
  • Move mount keyword upstream to ember.js

---

Non-Goals

These are things we're not aiming to address at this time.

• CSS is order dependent and pollutes global scope. Write your CSS inside of your engines very carefully.
• Arbitrary segmentation of applications. We'll circle back to this just before landing to make sure we didn't paint ourselves into a corner.

[kiwiupover]

@nathanhammond thanks so much for putting this together.

[sivakumar-kailasam]

@nathanhammond Awesome doc! I'd like to help out on the ember-cli changes

[nathanhammond]

@sivakumar-kailasam which part? There are a bunch of pieces there; I'd encourage you to choose one of them and get a WIP PR open. 😊 (So that I can track progress and help keep the effort moving forward.)

[sivakumar-kailasam]

The first one to copy over engine-addons.js to ember-cli. Created ember-cli/ember-cli#6065 to start off. Will update and reach out when its good for review.

[mike183]

This is looking awesome, thanks @nathanhammond for putting so much time into this.

With regards to the build section, it states:

The host application will still have to be the entry point for an engine rebuild.

Does this mean that it will not be possible to build engines independently? For my use case, the engines would not be known or present at build-time of the application itself and would instead need to be built separately. The application would then need to "discover" the engines at run-time.

Is this something that will be at all possible?

[nathanhammond]

@mike183, since I know your use case let me expand a bit. For everybody else this is probably nonsense and you can ignore it.

---

No matter what you will need to modify at least one asset for the host application, either index.html or a separate assetmanifests.json, in order to be able to simply look up an engine. We elected for pushing the manifest(s) into index.html to prevent the extra round trip for the typical web scenario.

The other thing to address is the insertion of routes.js from the engine itself into the host application. This is required to be present before engines load and before the application boots.

For people who wish to extend your application you can write an addon which people must include if they're targeting your platform. It does two things:

1. Splits routes.js into a separate file and includes it in the manifest. That will be a custom extension of your own to the manifest format which specifies the files for routes.js.
2. Spits out this modified manifest file in a defined location inside of their build.

In your host application you:

1. Include an initializer which uses deferReadiness to load assetmanifests.json and immediately inserts all of the routes.js files for engines listed in it.
2. When somebody "installs" an engine you call a backend thing with the set of assets and append that engine's manifest file to assetmanifest.json. You'll need to manage the content of assetmanifests.json external to Ember.
3. You will need to dynamically insert a mount point for the engine into the host application's router.js based upon information in assetmanifest.json.

[mike183]

Thanks @nathanhammond, I've messaged you on slack to avoid creating too much noise for everyone else.

@peStoney - I think you had a similar use case to mine, you may also find the previous comment useful.

[nathanhammond]

@mike183 I updated the above comment for more clarity.

[mike183]

@nathanhammond Thanks, I'll attempt to take a deeper look into all of this in the next day or so.

[peStoney]

@nathanhammond I did go through the manifest recently and stumbed over the approach for the router:

Delivery: Bundled into the host application's vendor.js. Presence inside of the host application's vendor.js is considered undefined behavior and may change in the future.

Maybe my english capabilities are failing but this sounds contradictory (?) In our use case it would be important that the engines router.js is not bundled into the application's vendor.js...

[sglanzer-deprecated]

I believe we have a similar need to @peStoney and @mike183 - currently we're discovering micro-uis hosted from separate docker micro-services. We would like to convert this to discovery of engines, but we won't have prior knowledge of the engine in the main app.

At this stage we have a pre-engine solution to this that gives us a dynamically discovered navigation and dashboard, but the ultimate goal would be to use engines to provide a full SPA experience.

@cstolli @psbanka @sandersky @job13er this will be of interest

[nathanhammond]

@peStoney @sglanzer see the approach I described here. The key snippet is:

1. Include an initializer which uses deferReadiness to load assetmanifests.json and immediately inserts all of the routes.js files for engines listed in it.

You will need to change the default behavior in your build using an addon of your own writing but it will be entirely possible to accomplish. Further, we are aware of this use case and don't intend to make it impossibly difficult to implement. It will likely never be the default as a consequence of requiring additional network requests and out-of-band editing of a manifest file.

[sglanzer-deprecated]

@nathanhammond excellent, thanks for keeping this case in mind 👍

[nathanhammond]

Strike Team Meeting Notes: 2016-07-21

The strike team got together and had a conversation about the next steps and how we're progressing. These are the notes from that conversation. No timeline estimates yet, but the scope of work is better and better understood. Please review the new RFCs and give us feedback!

Asset Manifest RFC

• Link
• Serialize to assetmanifest.json every time. No reason not to have it.
  • Aside: primary config into separate JSON file using same technique.
• Last step to optionally insert into index.html
• Move default out of the environment config.
  • Overrides only go there.
  • May apply to non-engine loading things.

Asset Loading Service RFC

• Link
• Multiple manifests should definitely be supported.
• Open-ended, experiment in some separate addon.
  • Nest it into the ember-engines addon?
• May not land in Ember itself, possibly default Ember CLI blueprint.

Build Time

• Would return the addon bundle.
• Would need to fix the merge of all addons down together into an addon tree.
  • Need to change that behavior.

[nathanhammond]

Future investigation: notifying the Asset Loading Service of bundles that are reachable via link-to from the present route. Provide a hook that consumers can implement.

/cc Marc Lynch at one of his three GitHub handles: @marclynch @lynchbomb @Ginormous

[trentmwillis]

For the build assets, we need to determine where they go in dist. The constraints we've come up with are:

• Engine asset paths should be relative in relation to each other
• Assets should go into a non-root directory to minimize chances of collisions
  • Using assets results in redundancy in the paths
  • Name should not be specific to Engines

So ideally we have paths that look somewhat like:

dist/<asset-location>/<engine-name>/                 // root of directory
dist/<asset-location>/<engine-name>/assets/vendor.js // generated build assets
dist/<asset-location>/<engine-name>/img.jpg          // public assets

We need to determine what the name of asset-location is.

[lynchbomb]

@nathanhammond: "@marclynch" and "@Ginormous" are now but a faded ghost of the past and have been deleted ::tear::

[nathanhammond]

Strike Team Meeting Notes: 2016-08-10

Action Items

• Dan - Release 0.3 off of master.
• Rob - Start conversation about Asset Manifest and Asset Loading Service at Friday's Core Team meeting.
• Rob + Nathan - Conversation to walk through Lazy Load Build #173.

Discussion Topics

Router Changes

• Link
• Landed and upstreamed into Ember.

Lazy Load Build

• Link
• Two remaining things on the list, and then land.

RFCs

• Asset Manifest RFC
• Asset Loading Service RFC
• Discussing at core team meeting on Friday.

0.4 Alpha to Beta to Stable

• We'll release 0.4.0-alpha.1 upon landing Lazy Load Build #173 to make it easier for early adopters to test.
• We'll release 0.4.0-beta.1 once the effort for using lazy engines is approximately lazyLoading = true. This may simply be an exercise in documentation.
• We'll release stable once integrated into non-trivial applications and address build performance issues. (e.g. LinkedIn)

Road to 1.0

• Use only public APIs, upstream our modifications.
  • Resolver extension.
  • Route extension.
  • link-to-external
  • Figure out the solution for the route-serializer story.
• Specified Ember build system with extensible story for supporting engine needs. (lib/broccoli/ember-modules-app ember-cli/rfcs#64)
• Define how dependency declarations will work between hosts.
  • API for being declared.
  • Ways to support objects other than services.
• Have a solution for passing instance-specific variables forward from a host to a child engine.
  • Currently a shared service which is awkward.
• Specify how to handle multiple inclusion of routeless engines.

[villander]

@rwjblue this issue can be closed? Or Is there anything still needing to be done with these checklists?