Roadmap update

[filipenevola]

Roadmap update.

We are proposing a new format for the roadmap where community members are involved since the beginning.

This is a call for community leaders to step up and help in the future of Meteor.

[mitar]

I have looked into this few months ago and in fact it works pretty well, the only issue is that hot-reload currently uses DDP: #9960 I think that should be moved to its own socket and used that. See more discussion here:

• Provide a better way to use the Meteor Tool without Mongo meteor-feature-requests#31
• Move auto-reload functionality from app to Meteor Tool meteor-feature-requests#354

[filipenevola]

Thanks @mitar , I just added these links to the description.

Are you willing to be one of the leaders in this item?

[mitar]

I opened few related issues:

• Do not include empty packages on the client #10701
• Remove LaunchScreen from web clients #10702
• Not possible to disable dynamic-import #10704

[filipenevola]

Thanks @mitar , I also added these links to the description.

Are you willing to be one of the leaders in this item?

[mitar]

Sadly I am currently swamped with other projects, so my love for Meteor is on the side. :-(

[mitar]

I think it would be great to move some core Meteor packages to NPM first.

The question though is how to have features like legacy builds on NPM.

[hexsprite]

I think it would be great to move some core Meteor packages to NPM first.

The question though is how to have features like legacy builds on NPM.

Can't the packages be published as ES2019 or whatever and then Meteor can compile whatever bundle it needs?

[mitar]

Not really. The whole point of legacy targets in Meteor packages is that you can have literally different code for legacy. It is not just a question of compiling.

[sebakerckhof]

True, but not that many packages have separate legacy code. So all of the ejson/tracker/random/minimongo/... stuff could probably quite easily be published as npm packages since most has been modernized already. This would split Meteor more in framework functionality published on NPM and build system. Some stuff like the fetch package falls somewhat in between. So while far from complete, it could be a good first step.

[CaptainN]

Svelte uses a special key in package.json files. Something like that could work for meteor - it should be possible to map every meteor package API to a property on a meteor key. The specific bundles could still be created as they are now using tooling. I remember Ben had some issue with doing that. Maybe it's worth getting his 2 cents.

[mitar]

I agree with you both! To the list I would also add the server-side stuff. That also does not need legacy code. So it would be great it oplog tailing would be a NPM package. People could then use it in other contexts and I think that would bring a lot of optimizations into it.

[CaptainN]

I've recently been playing with integration into webpack for React Native and NativeScript. It'd be great if there was a top level way to pull assets into webpack for client bundles. If moving packages to npm would help do that, I'm all for it.

Enabling WebPack builds for the client bundle could also be a nice way to gain some features on this list which already exist over there - such as leveraging WebPack Offline for a Service Worker, and some PWA manifest tools. I've already tried it, and it's pretty nice. It just needs a few things - in particular, a meteor module loader (or reify-loader more specifically) would be nice, and maybe a dynamic-imports loader to keep perfect code splitting, and maybe port the modern/legacy bundle split to webpack somehow (webpacks method for modern/legacy bundles leaves a lot to be desired).

[mitar]

I think a low hanging fruit is: #10703

[mitar]

Also add Vue Tutorial?

[filipenevola]

yes @mitar , Vuejs tutorial is listed in the topic First-class citizen Technologies

[mitar]

Not really. One of big selling points of Meteor is drop-in accounts. I think those should also be made so that database connection can be swapped to different database backends.

[filipenevola]

I have a few items that I would like to add to the roadmap in the next cycle, one of them is the accounts system independence

• Improve accounts system
• HTTP/2 Websocket Support
• Improving MongoDB Oplog Tailing

If someone is willing to take the lead now we could include already.

[hexsprite]

what's the "routes dependency" referred to here? seems out of context...

[filipenevola]

you are right :) fixed.

[sebakerckhof]

Disabling the legacy build (at least in dev mode) could also help here for people that don't care about old browsers.

[filipenevola]

Added.

[filipenevola]

Do you want to help @zodern on this as a leader?

[sebakerckhof]

This PR of me is related to this: #10792

[filipenevola]

Great @sebakerckhof . do you want to be one of the leaders of this item?

[sebakerckhof]

What does it mean to be 'a leader of an item'? Because although I've done some related PR's recently (e.g. also the jquery stuff), my availability is highly dependent on workload of my actual job, so I might have some months without any time. If that's ok, you can add me.

[filipenevola]

The leader is someone interested to guide the work but not necessarily to work in the item. Usually would be someone who understands what is necessary to make an item reality.

[sebakerckhof]

Could also be done for isServer/isClient. But you can only eliminate these branches if everything within them is block-scoped. Function-scoped variable declarations are hoisted outside of them, so you may break stuff if you remove the whole branch.

[benjamn]

That's the kind of thing that a good minifier can figure out, once we tell it to consider expressions like Meteor.isServer as constants—which I agree is a great idea!

[CaptainN]

I'll hoist this question - what do you all think of adding first-class WebPack support to Meteor? The idea is not to replace meteor's build system with webpack, at least not right away. And this suggestion is not meant as any kind of condemnation of Meteor. The main thing I'm after is access to the wide swath of tools that are available for WebPack, in particular tools like WebPack Offline (which has a decent default Service Worker), and integration paths for 3rd party tools like React Native, NativeScript (and the various tools built on top of that like Svelte Native and Vue Native), and possibly even with Cordova.

I'm probably oversimplifying, but this is what it looks like it would take:

• Add a default WebPack configuration to take care of the last mile of compilation for Meteor - out of the box.
• Add/port a Meteor module resolver (or loader, or whatever), with support for reify (nested imports), and dynamic-imports as WebPack plugins.
• Possibly also add a modern/legacy splitter of some kind (I have no idea how this would be done - but Meteor's result is superior to what I've been able to achieve with WebPack - nice if we can keep it).

There is already a pretty nice nice tool for doing this in @ardatan's Meteor WebPack. It'd be great if that could be a first-class tool, integrated with Meteor. It'd also be great if the paradigm could be reversed (if needed) so that WebPack cold reach in to Meteor, instead of being part of Meteor the way it is in Meteor WebPack.

The reasoning for that reversal (or alternative) is to allow NativeScript and React Native (and maybe Cordova) to run unmodified on their own, outside of Meteor, and pull out the parts they need. There is another project which kind of does it that way, but it's more limited than Meteor WebPack. It's nice because once it's set up (and probably using something like react-native-meteor-polyfills, even for NativeScript), you can just use the core and atmosphere packages that Meteor manages directly, instead of forking things (this will probably still be necessary even if atmosphere packages are moved to npm, because of the multiple entry points, among other concerns).

I see this kind of 3rd party integration as an almost feature story for Meteor. It just needs a bit more work - it's seems close to me.

[benjamn]

@CaptainN I appreciate the depth of your question, but those bullet points are all deal breakers for me, for the following reasons:

• The last mile of compilation is a privileged part of any compiler pipeline. Every Babel plugin would love to run under the assumption that all previous concerns have already been taken care of, but of course only one of them can be last, and even that plugin cannot make any assumptions about the ordering of the plugins. When you're working with a framework like Meteor, the final compilation decisions must ultimately belong to the framework itself. That's just the nature of frameworks, and I'm not eager to increase the complexity and processing burden of the Meteor build tool by introducing another expensive phase. You could try to run Webpack on the JavaScript assets generated inside .meteor/local/programs/, but I think you'd have trouble deriving any benefit from that, since Webpack doesn't understand Meteor-specific things like module.link.
• This is something I thought about while developing the Reify compiler, because I hoped that Reify could become useful outside of Meteor and Node. Unfortunately, I couldn't find a way to use Reify as a Webpack plugin/loader, because Meteor's module compilation system relies on Meteor's module runtime API (the install package), which (like Node) has a Module constructor which is responsible for creating the module object exposed to each module, with a Module.prototype where we can put additional methods to manage ECMAScript-style imports and exports. Webpack's module system doesn't have such a constructor or prototype, so it's not a viable target for Meteor's module compilation. This is not the only missing feature (Meteor-style dynamic import() and exact code splitting also depend on the install package), but it's the biggest barrier in my mind.
• Other systems would have implemented differential modern/legacy bundling and delivery by now, if it was easy. Meteor has a lot of advantages because it's a full-stack framework that also controls the web server. I'm doubtful that it's possible to implement a stack-spanning system of that complexity within a standalone bundling tool. You may see people talking about differential bundling based on <script type=module> and <script nomodule>, but that's a different, much less flexible technique: https://twitter.com/benjamn/status/1005104014944538624

I know this isn't what you want to hear, but if you really want to use Webpack with Meteor, I recommend using it to publish npm packages that your Meteor application can consume, so that the Meteor development experience does not have to bear the cost of running Webpack (or any other additional build step) as part of its default build pipeline.

As for React Native, from my limited review of the options, I imagine Meteor could play a role similar to Haul, except with its own build system instead of Webpack. Have you heard/thought about that?

Storybook 5.2 demonstrated that it's possible for a tool that usually relies on Webpack to load its assets directly from a running web server (see storybookjs/storybook#5975 (comment)), which makes me optimistic about the future of using Storybook with Meteor. However, Storybook is loading compiled assets, whereas Webpack would presumably want to do some compilation itself, so I'm not sure this technique makes as much sense for Webpack.

In general, I would rather implement Webpack-like features within Meteor than tie the two projects together in any direct way, because I want to keep Meteor as nimble as possible, so we can embrace the big changes in JavaScript module delivery that are still over the horizon. It's almost 2020 and the JS community is still struggling to use ES2015 modules natively in the browser. Webpack is benefitting from this painful transition period, and it has good reason to preserve the status quo. The future of JavaScript modules does not belong to bundlers, and Webpack would have to transform itself dramatically in order to stop being a bundler.

tl;dr I still have big plans for the future of Meteor's module system, and I don't see a first-class role for Webpack in that vision, but I'm open to ideas about generic features that allow interop with third-party tools like Storybook or Webpack or Jest, to name just a few.

[CaptainN]

@benjamn I'm always interested in the truth - so I appreciate the in depth answer. I agree broadly about not wanting to add an expensive additional stage to the build system. What I'm ultimately after is a way to use Meteor, and it's internal client side libraries (and some atmosphere packages), for projects on external tools like RN and NS (and derivatives - and to a lesser extent, gaining access to WebPack features - obviously, a proper Meteor version of the various tools would be preferable). I suggested a way to get there because I almost have it working, not because its necessarily the most optimal way to get there. I guess don't actually care as much about how we do ultimately get it working, as long as it's not fragile or too slow (but I'm glad you do care!). I also understand what you are saying about being a good JS community citizen, and not abandoning some principles about how to help move everyone forward on modules - that's good stuff.

I haven't heard of haul - I actually thought React Native was using WebPack already - perhaps I'm wrong about that (it's been a while since I played with it - I've been playing more recently with Svelte-Native/NativeScript, which is built on WebPack).

I have thought about how Meteor might be the build tool for additional client targets - I'd love to see that actually, for a lot of targets - like service workers, web workers, etc. - making that open ended would be pretty awesome. To me from the 10,000 mile high perspective though, moving to WebPack, is kind of like decoupling Blaze. In a way, my suggestions have an unmentioned ulterior motive - which is to move some of the maintenance of core components out to third party efforts (like WebPack, or the way the view layer is now React or Vue - maintained by third party communities). I understand wanting to keep Meteor doing things the meteor way though, and the specific reasons you listed for doing that sound great to me!

Anyway - I still want to sue Meteor with RN and NS! As I mentioned, I almost got the two solutions I mentioned above working. Most of the problems I have had is with Meteor packages that assume a browser execution context - leading to errors like undefined globals like window and location or attempts to access a DOM which doesn't exist. I think react-native-meteor-polyfills can help with that. The one which works how I think it should (pulling libs from another meteor project's .meteor/web/programs) - meteor-imports-webpack-plugin (this fork) - has some limitations, in that it requires a full meteor project as a sub-folder in the parent WebPack project, and also bundles everything in the Meteor client bundle (including static assets and css files), instead of allowing direct imports of only the specific modules you import - I think this gets around the module.link problem you mentioned. But it means you have to set up the child Meteor app specifically for that project, instead of using a full web app (perhaps as a git-submodule or something) and remove anything you don't want bundled, which is onerous. But it does basically work - I can use that set up to pull core meteor packages in to an NS project. It doesn't change the Meteor build system at all - it doesn't even require Meteor be running, just that a build has happened once.

I wondered if it could be improved by adding a dependency tree as json to the output in .meteor/web/programs that the webpack (or any external) module importer could parse to include a subset of modules, instead of pulling in everything (we have the meteor list --tree option to build on). I guess we'd still have the module.link problem - maybe a bridge or polyfill could help there?

The other package, Meteor WebPack, does it the other way - it adds an additional build stage. It was WAY easier to set up than the other, in my limited testing, but has its own limitations (including losing perfect code splitting and modern bundles). But I'm not sure how I'd integrate that in to the NS or RN build stages, which run their own bundlers. If I wanted to gain access to additional webpack tools like WebPack Offline, this is how I'd probably do it.

[benjamn]

After a quick glance at Webpack Offline, that's definitely something I'd like to support natively in Meteor. We could either use a ServiceWorker and fall back to appcache like they do, or use only appcache if that works everywhere, or just say that offline support only works in browsers with ServiceWorker support. I know you've worked a bit with the Meteor appcache package in the past, so you might have feelings about how reliable it is. I'm down with any of these options.

I'm also very interested in making React Native work with Meteor's build tool, since the RN team has fixed most/all of the barriers that used to give their bundler an exclusive advantage, such as the @providesModule directive.

If we had a way to generate native.react.{ios,android} JS bundles and serve them from a local server, sort of like the web.cordova bundle, I think we'd be pretty close. This seems aligned with the goal of making it easy to build additional client targets, which you mentioned above.

I got in a little trouble with Cordova fans the last time I advocated for React Native support, so let me be clear: I am not hoping to replace Cordova with RN, though I do hope there is some overlap between them, so we can use the same tools to support them both simultaneously.

[CaptainN]

A couple of notes on appcache:

• appcache is deprecated.
• Service Workers are well supported at this point, so we probably don't really need to give appcache priority.
• It's API limited in that it only supports one caching list - everything in the list. No pre-fetch, or opportunistic caching, which makes it impossible to efficiently capture certain types of content, like a list of multiple webfonts, or an images's srcSet - it would have to cache all or none.
• I think it's blocking, but I can't remember or find any info on it - I think everything is loaded from in manifest before the app runs (the first time - additional times everything is run from the cache).
• Because it's deprecated, it kicks up an angry message in dev tools in all the browsers, and will presumably be removed at some point from browsers altogether.

WebPack Offline uses client side feature detection to determine whether to use appcache (by deciding whether to load an html file with a manifest in an iframe) - but Meteor should probably use server side user agent sniffing, and avoid all that complexity. This shouldn't actually be all that difficult, and maybe is the level of support we could give to allowing it to work with a newer service worker based offline package, when Meteor's appcache is installed next to offline (or whatever it's called).

WebPack Offline has 3 main levels of caching for defined assets:

• Must have assets - these load before the app starts.
• Assets which will be fetched after load
• Opportunistic assets - cached when or if the user grabs the data

In WebPack Offline, these are defined in a JSON structure in the webpack config, and we could keep that or do something similar. But maybe we could also add some active default folders for static assets in /public.

When Meteor's appcache package is installed with dynamic-imports, it actually has 2 levels for application code when installed with dynamic-imports - it loads the initial bundle as a must have, then preloads the remaining modules after the app is run in to indexeddb (that code is in the dynamic-imports package, the same logic should probably apply when an offline package is installed, unless the actual Service Worker is used instead of indexeddb for caching modules).

An additional nice to have would be the ability to cache an index page that is distinct from whatever the current route produces from SSR. That's another limitation of appcache it always caches the current page - even if it has serialized (and then stale) data in it from SSR.

[Fen747]

I'll all on into the thin Meteor thing, with pruning / DCE / tree shaking.

And I think that also needs some modification in some core packages like accounts- : those needs to be stripped from underscore and others now-useless libraries that weigh Meteor down and makes it bigger and less performant / thin that it should.

alanning:roles was recently updated in that way, what a pleasure it is.

As for DDP, Hot Code Reload shouldn't rely on it.

And, cherry on the cake, I'd be marvellous to have to possibility to load accounts- packages only on server, in order to make it easily integrable to Apollo without the cost of adding useless code to the client bundle.

[harryadel]

@Fen747
accounts-js got it done, I guess.

[CaptainN]

@benjamn I just took a closer look at what you were talking about above about adding a native.react.{ios,android} build target, and completely agree. Maybe the place to start would be to work through the meteor build core, and break out the Cordova stuff, and replace that code in core with a common interface? I assume most of the places where we'd need checks for alternative build targets (like RN, NS, or SW) would be in similar places to where Cordova already is. If we could get the generic hooks in those places, maybe that'll create a way forward. To be honest, I took a look at that stuff a year or two ago, and couldn't figure out how to get it done.

It'd be so great to use Meteor to build RN client, because let me tell you - Metro is flaky! (at least on Windows/Android). Haul demonstrates a scope of work to get Meteor to that point. For many of those listed packages, equivalents already exist in Meteor.

Pretty neat idea.

[arggh]

Just a heads-up regarding the discussion about AppCache, it will be removed in Chrome 82.

[yanickrochon]

Where can I find information on how to use this "ultra-thin" version of Meteor (without MongoDB)?