Steal supports packages

[justinbmeyer]

Steal will be able to declare packages like:

steal.packages('contacts','login','filemanager')

This tells the builder to build contacts, login, and filemanager packages that can be progressively loaded later. In development, when code runs like:

steal('contacts', function(){ ... })

Steal will simply steal contacts/contacts.js and anything it steals that hasn't already been stolen.

But in production, instead of stealing contacts/contacts.js, it will steal app/packages/contacts.js. app/packages/contacts.js will be comprised of contacts/contacts.js and the steals unique to contacts/contacts.js.

In summary, packages tells steal that other resources will be loaded and to package and minify them in an efficient way for progressive loading. In the app's production.js, it tells it where to find these packages.

Example

Packages lets you define several parts of your application to progressively load. In this example, we want to load only what is needed to get the application started and defer the rest until we know if the user has logged in or not. The following is done with 3.3 code:

// tell steal that you want to build these as packages
steal.packages('contacts','login','filemanager')
      // load the minimum you need to start your app
      .then('jquery/controller/route',
              function(){

// checks if the user logged in or not
// we might get this from the page
$.ajax({
  url: '/curuser',
  success: function(){
    // if the user is, start routing
    new Router(window);
  },
  error : function(){
    // otherwise load the login code
    steal('login')
  }
});

// a router that listens to hashchange and loads modules on demand.
$.Controller('Router',{
  "contacts route" : function(){
    steal('contacts', function(){
      // setup contacts
    })
  },
  "filemanger route" : function(){
    steal('filemanager', function(){
      // setup filemanager
    })
  }
})

})

Impact

This makes progressive loading very simple. You define the parts of the app you want to load later and steal them naturally in development mode.

Steal's build system will make sure to package them as efficiently as possible. In fact, dependencies between packages are packaged themselves to improving caching and minimize load times.

When it comes time to build, everything just 'works'. No configuration files!

I believe this will be a game-changer. It provide the fastest possible load times for complex apps in a straightforward manner.

[justinbmeyer]

A short example of what could be done is:

1. The initial code loads with some basic understanding of routing, if the user is logged in, and the initial data requested for most routes.
2. The initial code checks the route and user, decides that they are wanting to view a filemanager, they can request the code and the data for the filemanager.
3. When the data, and the filemanager code are ready, the initial code populates the filemanger with the data, preventing it's first ajax request.

Compare this to what happens now:

1. All code is downloaded, all objects are built
2. The route and user are checked
3. The filemanager is created and makes the request for data.

The second case is, of course much more simple to understand. However, it is much slower. Potentially 2-3x the amount of code is downloaded as needed. JS does not start processing until the JS script is complete.

We can make all other packages start downloading in parallel, running in order, with the initial package. This would be the most ideal situation. The small script would load and run, firing the data request, while the other packages are loading. The other packages would then set themselves up while the ajax request is being sent.

Essentially, this moves-up the first data-ajax request much earlier while very likely keeping total application download times about the same.

[justinbmeyer]

Here's the test app:

https://github.com/jupiterjs/steal/blob/packages/build/packages/test/app.js

It progressively loads the accordion or table scroll as app.

[justinbmeyer]

Dealing with large numbers of packages

If the number of packages are large, say 7, there could be many sub-packages that need to get loaded for a single package. This is not ideal as the overhead would not be worth loading all the scripts.

What's needed is a DEPTH value that is used to limit the number of sub-packages any package needs to load.

Here's one way we can do it ... for each sub-package combination calculate the difference.

Select the min distance and 'combine' into a sub-package.

Repeat until every package only loads DEPTH number of scripts.

Issue

• it's possible to combine sub-packages so some packages will load less than DEPTH sub-packages
• This might not take into account a package that keeps getting 'screwed' ( repeatedly having packages part of the minimum difference )

A possible solution could be to select the minimum distance based on DEPTH and which packages are getting 'screwed'.

[jtsoi]

Not sure if this is the right place for this, but as JMVC gets traction and is used more and more in production, there is a clear need for cache busting in steal.

What do you guys think about this approach:

Current way to load an app in prod is:
<script type="text/javascript" src="steal.production.js?myApp"></script>

What if there would be an option to add:
<script type="text/javascript" src="steal.production.js?myApp&_=123456789"></script>

Inspired by jQuery getScript cache busting.

The _ variable would propagate to all dependency calls steal makes.
This way you could change the _ variable to the build number for your app and be sure that
users always get the latest code and we can turn up cache to forever.

(Maybe "v" is a better varible than "_" )

[justinbmeyer]

I agree this is needed, but I think Brian has a way to do this. Brian?

[justinbmeyer]

Oh, can you move this to another issue, also one for steal. This is about packages.