Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Mixed mode issues with Packages & Modules (JMH: we need application.json) #15

Closed
asilvas opened this Issue · 9 comments

2 participants

@asilvas

First of all, I want to clarify, "package" refers to a single javascript file which contains a compilation of two or more modules, for optimization purposes.

So lets just dig into an example, and it should be clear what the issue is in this situation:

<script src="curl.js"></script>
<script> // preloads!
curl("my-super-package"); // contains module1, 2, and 3
</script>

<!-- 
in a completely separate (and dynamic) part of my application, I'm going to dynamically inject content and load the individual modules required by the content
-->


<script>
curl("module2", function() { // use module2
});
</script>


<script>
curl("module4", function() { // use module4
});
</script>

OK. So the problem should already be clear, but lets talk this out. We've got a race condition, with module 2 (as it should not be downloading "module2" since it is already inside of the package). So why would we have code like this? You've got a huge application, where not all parts talk with one another, but you've got some form of "master" (potentially a few) where your common work takes place (such as including curl, and pre-loading common files such as this "package"). Ideally, we want to allow these types of disconnects between the master and the child pages, as it can be very difficult to keep everything in sync otherwise.

Of course, I could simply remove the inclusion of the "package", and all is well again. In a perfect world, developers can throw in their "curl" calls left and right, for each and every requirement they have, without concern over what has already been included, or what packages may have been included. And as "common" files change, we swap out with different packages in the header. The potential here is some substantial optimization's in the "master" with no impact on the remainder of the application -- if it worked, that is.

I have some ideas on how this could be addressed, but wanted to first throw the problem out there. I'm not sure if it is a limitation of AMD, or a misuse of it, or otherwise.

@asilvas

I want to point out that the above example would not fail (so long as they complied with AMD), the issue is effectively re-downloading the same resource twice (once from the package, once from the individual module request).

@asilvas

Also, I'm aware you technically could define each and every module in the "paths" configuration to point to the related package.. This would likely be the workaround if this is the path taken. Example:

curl = {
  paths: {
    "module1": "package1",
    "module2": "package1"
    "module3": "package1"
  }
}

I didn't test it, but believe it would work. Not a bad option for smaller applications. But when there are dozens of modules, that can quickly bloat your head (hah!), and be a pain to manage (since the app has to modify its config to correspond to the packages it's including). Ideally you can blindly include packages at will without further action required.

@unscriptable unscriptable was assigned
@unscriptable
Owner

I had a partial solution to this problem implemented back in curl.js 0.2. I yanked it for fear of not being AMD-compliant. (You can imagine that I was spending the vast majority of my coding time ensuring compatibility back then.)

Here's how it would work:

Requirements:

  1. you must namespace your module names, e.g. "my-super-package/module1"
  2. you must declare your packages in the packages curl config property
  3. you place a property in the package object, "curl-built": true
  4. any references to modules within a package must use the namespaced names:
define("someOtherModule", ["my-super-package/module1"], function (module1) {
    // do something with module1
});

curl watches for dependency requests for any modules that are within a package having the "curl-built" property and automatically adds the entire package as a dependency.

Note: "packages" in AMD-speak are a bit different than what you are calling "packages": Packages/1.1

Thoughts?

-- J

@unscriptable
Owner

Your definition of "packages" may not be very different, actually. CommonJS packages just don't have any notion of concatenation / compilation.

@unscriptable
Owner

Example:

packages: {
    myPackage: {
        path: "path/from/baseUrl/",
        lib: "my-super-package/", // other modules/resources are in my-super-package folder
        main: "../my-super-package.js", // main module is a peer of my-super-package folder
        "curl-built": true // true means use main module
    }
}

folder structure before files are built ("unbuilt"):

+ path/from/baseUrl/
  + my-super-package/
      myCss.css
      subModule.js
    my-super-package.js

folder structure when/if files are built/compiled:

+ path/from/baseUrl/
    my-super-package.js

The "curl-built": true property tells curl that the package's files were built into the main module, "my-super-package", at "path/from/baseUrl/my-super-package.js".

Note: if, in fact, a file in a package is not compiled into the main module, curl will attempt to fetch it at the unbuilt location. For example, if "my-super-package/subModule" wasn't built into "my-super-package.js", then curl will look for it at "path/from/baseUrl/my-super-package/subModule.js".

In item (3) above, there's no way to tell curl what the name of the built file is. It just assumes it's the main module. We could also support a different property to specify it explicitly: "curl-built": "name-of-built-file.js".

I hope this makes sense. :)

-- J

@unscriptable
Owner

I should tie my explanation back to your example. I modified yours to have it use a package, "my-super-package":

<script src="curl.js"></script>
<script> // preloads!
curl("my-super-package"); // contains my-super-package/module1, my-super-package/module2, and my-super-package/module3
</script>

<!-- 
in a completely separate (and dynamic) part of my application, I'm going to dynamically inject content and load the individual modules required by the content
-->


<script>
curl("my-super-package/module2", function() { // use module2
});
</script>

curl will see that "my-super-package/module2" is part of the "my-super-package" package via a package descriptor in the config. If the config parameter, "curl-built" is true, curl will place an implicit dependency on the main module of the package (or optionally the module/file of your choice) and wait for it to finish downloading/executing before proceeding.

AMD doesn't define how to handle this situation, but doesn't hinder it either, luckily. :)

Some clarification about packages and "main" modules (in case it's needed): In AMD/CommonJS, if a package is named as a dependency directly (e.g. define("myModule", ["my-super-package"], func); or curl("my-super-package");), its main module (if any) is fetched.

Fwiw, "curl-built" probably won't be the name we use. I'll find something better.

@asilvas

Actually that wouldn't really solve the case I gave, as the point is that the package didn't exist initially, and the curl calls for the individual modules have no concept of possible future package structures. Ideally none of the applications code ever changes (keeping in context that there are dozens of modules and a boat load of code to support it) for when I want to add in package support, especially since those packages are likely to come from a source outside of the application. These packages, per their purpose, are for optimization, nothing more -- so naturally I do not want to introduce a break to the application.

Another requirement is that I can keep changing the "core" package (or additional packages, if need be) that introduce a different set of modules (lets say, due to them being determined the "most popular"), again without worry or concern over breaking the application that has little to no knowledge of exactly what all is in the package.

So far, I see only two paths that can make this work (without introducing more round-trips for a manifest-like file):

  • Define modules locations in the initial config, before any requests are done -- example being the paths usage I gave above.
  • Provide a hook (possibly "pkg!" plugin) which introduces blocking. No other modules are permitted to load until the requested package loads.

The first path I gave might actually work reasonably well, though perhaps the same could be achieved via a new config such as "packages" that you gave above, but the definition of what you provided would not work for these cases. The downside of either of these approaches is that it can bloat your config. Dropping in a package would no longer be "just that easy", you'd also have to update the config accordingly. But still, not a terrible option.

The second path is to introducing some unique hook that tells curl "this request is a critical dependency with unknown contents, so we must block" -- such as a "pkg!" plugin. Normally I'd be adverse to ever use blocking, but being this is a "master" type of functionality, it might actually work quite well. I also want to clarify my usage of "blocking" -- I'm not referring to a synchronous operation in the normal sense of blocking -- instead I'm talking about more of a priority queue, so everything is still asynchronous, but no other requests will even start to be requested until the already pending "pkg!" finishes loading.

It's all about versatility for both application structures and external structures (shared entities outside of the apps control). Changing the mentality from "lets structure everything this way to comply with the new loader" to "use the loader this way and it'll do everything we need". You never know what the future holds, in and out of apps, which is why versatility for a loader is king.

Thoughts?

@asilvas

I understand the term "package", while has various definitions depending on the standard, from what I understand all differ from the usage patterns I gave. So another way to think of this concept I refer to, perhaps a bit more explicit, would be as a "bundle". I'm not trying to come up with something new, and certainly not to break any standards (which I know, you wouldn't consider anyway) -- just trying to solve a reasonably large problem (with hopefully a simple solution). Even if it means coming up with a plugin to keep it outside of the core.

@unscriptable
Owner

I really like your idea when you put it this way: "this request is a critical dependency with unknown contents". The concept of a priority queue is another idea -- one that RequireJS has -- and has given James no end of grief. :(

I'm going to dwell a bit more on the "unknown contents" concept and think about how to handle that without forcing the use of packages (in the CommonJS sense).

I'm moving this to milestone 0.6 so we have some time to contemplate it. :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.