Modularize code further to allow optimized one-projection builds #78
Comments
|
Sounds good @mourner. If you have ideas that you can turn into pull requests, that would be awesome! |
|
Honestly I'm afraid to get into Proj4js code because I'm a total noob in GIS and weird projections, so will probably break something without knowing. :) But I'll see what I can do to help. |
|
The projections are all loaded in one file (prrojections/index.js) a good
|
|
What was done to constants can be done to common, and then the projections
|
|
further to this, I'd float the idea of making the defs json objects, and then using one or more of those to determine which modules need to be included. |
|
So that was how it used to work, but this made it overly complicated to use
|
|
But we could make it easy for others to do async builds, like maybe can a
|
|
why would that make it complicated to use? ISTM it's much simpler. I just present the build process with a def.json that says for example: {
"proj": "utm",
"zone": "31",
"ellps": "GRS80",
"units": "m"
}and then say 'gimme a build that supports that - and assume I want to convert into/out of WGS84 coords'. I don't need to know what any of those properties means, nor do I need to know anything about the code. As @mourner says, only having one projection is an extremely common use case. Many of the possible projections are rarely used, and particularly in the browser, downloading a load of code you never use just wastes bandwidth. |
|
Oh I misunderstood you I thought you meant you wanted it to async load the
|
|
Sorry went back and read your original post. Here is the deal. Conditional loading only is helpful in the browser, not node. To load stuff That being said we can do a few things, we can add a way to catch a missing
|
|
I am not talking about async loading in the browser; that's an entirely separate issue. I am already (with version 1.1) loading custom proj builds in the browser asynchronously (i.e. as needed), but they are far larger than they need to be. I'm just suggesting that, if there is a build script to create a single-projection build, a proj.def be used as a config to that script. |
|
I'm getting you, I agree building with a proj string or wkt would be a Using the proj4 def json would be even better but is dependent on us
|
|
just noticed #27 (comment) |
|
Actually, I think conditional loading is a good way to address the custom build scenario, even without necessarily supporting the asynchronous / browser use case. For example, you might rewrite var projs = projections.registry = {}; // i.e. projStore
projections.get = function(name) {
if (!projs[name]) {
try {
// Conditional sync module load - will work in node,
// but will fail in the browser unless module is already in build
projs[name] = require('./' + name);
} catch (e) {
// Bail out with informative message
// (Caller could attempt an async load, then request the proj again)
}
}
return projs[name];
};When generating the custom build, you'd simply run You could even use this approach to build the main proj4 distribution: as long as you give it a complete set of representative projections, eventually all of the needed modules will be loaded. It's analogous to a coverage test - if any module is never included, that is an indicator that it might not actually be needed at all. This approach would also have the advantage of utilizing proj4's built in projection parsing code, rather than recreating parts of it in a separate build tool. I was going to conclude with another plug for async modules but I'll leave that for another time :) |
|
We are on same page in general, but we can end up doing it without the try.
|
|
In theory, once step 1 is solved you could just throw it a wkt and check the |
|
the relevant code is here but yes it's more or less exactly what your talking about the external call is |
|
a further point on this is that these aren't really 'custom' builds: they aren't site-specific. Once a build has been generated for a particular projection, it could be stored somewhere and then used by any site that needs that projection. |
|
@probins true, though when you have 2 and 3 projection builds is where we would run into problems, one thing we could have a core build with no projections and a file for each projection, then all you need to do is include the core and the varius projections you need (this would be in addition to the current all build) |
|
yes, but I think @mourner's point is that it's very common to only use 1 projection, and want to transform 4326 coords in and out of that projection. Having >1 projection is a different use case. |
|
nothing I'm planning is going to prevent us from doing one projection On Mon, Dec 9, 2013 at 9:20 AM, Peter Robins notifications@github.comwrote:
-Calvin W. Metcalf |
|
to summarize the ideas we all have had
1.i would be helped by cleaning up wkt and projString |
|
We probably don't need the try, though the dynamic require might still be useful. The example I gave didn't really demonstrate my reasoning very well - it comes from an earlier thought I had, which was that we could use the module system itself to determine which modules needed to be included in a build. For example, if someone requests a build for So, the build system I had in mind would actually run the proj4 source files on a projection, and then inspect the module system internals to see which files had actually been included. There at least two key problems with my approach:
At any rate, there are probably more elegant ways to do this with ASTs or something. My main point just is that we should be able to use the proj4 source itself when computing a build, rather than duplicating projection information in separate build tool. There are a number of ways this could be done. |
|
my thought was that if some one wanted just utm and tmerc we could either just use the browserify ability to alias files to replace https://github.com/proj4js/proj4js/blob/master/lib/projections/index.js with one that did have these lines https://github.com/proj4js/proj4js/blob/master/lib/projections/index.js#L4-L24 or just include 2 files in the build which had something like and in them, this should get browserify to include the relevant files |
|
Sure, that would work. I'm not so much talking about how to actually include the files (either of your examples seem fine), but more about how we would compute which files we actually needed. We need some way to take a projection definition (WKT etc.) and know just by looking at it what modules to include. I'm just proposing we use the actual module requires() in some way, rather than duplicating the dependency information somewhere else. |
|
Re: 3.iii (async loading), the most important takeaway from my example is that a hook won't really be able to do much except report which modules are missing - it won't be possible to switch to async mode while in the middle of loading a projection. proj4 would need to bail out and leave it to the user to request the modules async, and then call the sync API again to try loading the projection. They would need some indicator of which modules were actually missing so they knew what to load. The computation of which modules are missing might be very similar to the computation for a build, which is another motivation to include that computation code in proj4 itself rather than in a separate build script. (The other option is to tell the async loading user that they need to know which additional modules they need, and that they should load these modules before calling proj4.) Re: 3.ii (separate script tags) - I must admit this one makes me a bit sad, as an async (i.e. AMD) module loader would handle this use case just fine and use only one script tag (other than the ones it creates automatically to load deps). Also, I assume it is not possible to
At any rate, it will be hard to effectively support the async use case without adding an async API to the proj4 source in some way. It doesn't need to be a high priority now, but I don't think it can be put off indefinitely. At some point, I would like to explore the option of bringing back the AMD modules (probably just as define-wrapped source files, and leave it to AMD users to create a custom build). I think we could do this without much extra work by leveraging r.js' conversion tool in a supplemental build process. I understand that node integration is the priority now, but I would argue that the browser use case is at least equally valid and (as you know) am concerned about whether browserify can accomplish everything we need. |
|
so from the first one comment, I was envisioning calling lib/projString.js or lib/wkt.js to figure out which projection is necessary. so I was thinking that a hook could go here which could be handled by something async, even if it is something crude at first like loads the module and then tries again Re Re: 3.ii I should have also included, load via amd in that list,:) there is no reason you wouldn't be able to load proj4, and any modules you want through an async module loader there, we can write a script in grunt that allows us to do grunt build --include=tmerc,utm, you should also be able to do something like grunt build --nodeps and then latter do currently the projections are fairly self contained and only require functions out of the common module which I now split up making them easy to create individually. lastly we are not abandoning amd users, all of the files we are producing are wrapped in defines, thats the point of umd |
|
Sure, I didn't mean abandonment, just that certain workflows (in particular the cross-module, async ones) would be much easier with AMD source modules. If the projections are mostly self-contained anyway, then the UMD wrappers should work fine. Is the idea that you would include bits of the (former) common module in each standalone projection module? Anything not exposed via the public API will need to be included in each built projection module, which could lead to some redundancy as there would be a bunch of copies of e.g. Ok, I think I get what you're thinking with the initTransform hook. Presumably the user could just publish a folder with a build for each projName they might need (and each file would have all of the projName's deps inlined). Again, there is a potential for redundant common code, but maybe that's ok. |
|
I was thinking that for deployment purposes (where you'd care about the extra bits) you'd build it as one file and there would be minimal duplication, in any case as long as all the files where part of the same file, i.e. concated, gzip should handle it. |
|
if we clean up the proj4 constructor (which is on my list) it could be much easier to make it somewhat async |
|
You could imagine SaaS-type applications where you'd intentionally deploy a limited proj4 and load the projection modules on a per-user basis. So, the redundancy will be unavoidable in that case - but whether that matters depends on the amount of common overlap. Would it be useful to put some file size estimates together? (If it were me, I'd probably just deploy the complete proj4 anyway...) |
|
Yeah even "just" a callback version of the constructor would cover a lot of async use cases. (I say "just" since I'm not sure how much internal refactoring that would entail). |
|
it would probably be the kind of refactoring we'd want to do anyway On Mon, Dec 9, 2013 at 1:56 PM, S. Andrew Sheppard <notifications@github.com
-Calvin W. Metcalf |
|
Back to the single-projection build - might that just be concatenating proj4 core and one standalone projection module? |
|
lemme open a pull to show you On Mon, Dec 9, 2013 at 2:16 PM, S. Andrew Sheppard <notifications@github.com
-Calvin W. Metcalf |
|
@sheppard check out the But the good news is that basic one that can do just merc, 7101 byes gzipped and minified. yes you read that right less then 7 KB and that's including cruft like datum transformation, so you could use that to do nad83 lat longs to web mercator. |
|
@calvinmetcalf that is awesome! Great work! |
|
@calvinmetcalf nice |
Browsing through the code, it seems that in this state it can't satisfy an extremely common use case of making a custom build for one particular projection.
All projections share a huge "common" file, many other heavy things like "mgrs" with lots of functions for completely different cases are loaded in one monolith, many constants are defined in batch even if only a handful is used in a particular case, etc.
In a perfect Proj4.js, all the files would be split further and dependencies between different chunks defined at a small scale (which browserify makes very easy technically), which would in turn allow us to make a browserify-based build system that can make a perfect tiny build for each particular projection.
I know this has been briefly touched in #9 and #20, but since that issue was closed, I thought it would be nice to bring this up again as a clear goal.
cc @calvinmetcalf @ahocevar @sheppard
The text was updated successfully, but these errors were encountered: