Move Tracker and all other dependencies into separate repositories and NPM

[mitar]

For #11 to really work, we have to move all dependencies into separate repositories/NPM.

[stubailo]

Should we create a Blaze GitHub org for this?

[mitar]

I am proposing to use meteor-package. Then git repository URLs would be pretty: https://github.com/meteor-package/blaze, https://github.com/meteor-package/tracker, https://github.com/meteor-package/dpp and so on. :-)

I would also do #12 if we are going into this direction.

[mitar]

But this ticket was mostly about the fact that we have to move tracker out of the Meteor core repository as well. Not necessary to do repository-per-package as #12 suggests.

So if do not want #12, it could be that we do meteor/blaze and meteor/tracker and this is it. And inside tracker there is more than just Tracker, but also its dependencies.

[martijnwalraven]

I agree that it makes sense to also split off tracker and related packages (reactive-var etc.).

[stubailo]

I don't see the benefit of having a new meteor-package org - then what would be left in meteor? It makes sense to me to have a new organization if it's for something that might become more of a standalone project, like Blaze.

[mitar]

Just because it would be easier to give permissions to the community to create new repositories inside meteor-package over meteor. Or would you be willing to allow community members to create repositories inside meteor?

[stubailo]

Ah, I see - creating new repositories at will would be useful. Something like the reactjs organization, which is for community React projects.

Then maybe meteor-community?

[mitar]

You mean https://github.com/MeteorCommunity ? ;-) If not, then this might be a bit confusing to people.

Maybe we could use some ideas from: https://forums.meteor.com/t/searching-for-a-name-for-a-meteor-fork/17897

https://github.com/asteroid-belt ? ;-)

[stubailo]

A better place to talk than meteor-core

Hmm, sounds like the forums kinda took that over! And I don't want some abstract name like asteroid where you can't even tell what it means.

Also, I just looked at the beginning of the thread - why do we need tracker moved out at this time? Just to publish it on NPM?

Anyway, I'm starting to think the best starting point would be a blaze github org, specifically for Blaze.

[mitar]

Just to publish it on NPM?

Isn't this the main thing we want to achieve? Stand-alone use?

Anyway, I'm starting to think the best starting point would be a blaze github org, specifically for Blaze.

I am not sure. Because there are many other packages which will be pulled out sooner or later, no?

[stubailo]

Yes, but if we really want them to get a life of their own, lumping them into one giant organization called meteor-* doesn't make sense. for example, maybe it should be tracker/tracker. Or meteor-accounts/accounts-*.

[mitar]

reactjs makes sense.

Anyway, I am mostly concerned with maintenance overhead of one organization, and you are proposing to have even more. I think we can start with one organization, no? But we could also start with just things under meteor and simply have an easy way to request a repository. The question is also how many new community projects should be under meteor.

Hm, not that I think, I think we really degressed here. In fact we are not talking here about community packages (for that community can create some organization, or more), but where to put core packages, which are moved out of meteor repository. So maybe meteor is OK for that.

[martijnwalraven]

Yeah, I don't think hosting new community projects under meteor makes sense. The way I see it, the main point of splitting off these core packages is to make them more maintainable.

Over time, we may want these split off projects to evolve to be more independent, but right now there are a lot of interdependencies that would make this hard. Would a blaze organization include tracker? But then what about other packages like minimongo, that also use tracker? So having everything under meteor makes more sense to me, as they are still all part of one family (meteor/blaze, meteor/tracker, meteor/ddp, meteor/minimongo, etc.)

[mitar]

Yes. But the question of this ticket is, is it OK to create 20+ repositories inside meteor for all 20+ NPM packages we will be creating just to port Blaze to NPM?

[martijnwalraven]

My preference would be to group packages into coherent repositories, as we've done with meteor/blaze. The only downside is that NPM does not allow you to install from Git when you put more than one package into a repository. But that might not be a problem as long as you have a good development workflow. This is the approach projects like React and Babel also take.

An interesting article comparing different approaches:
https://plot.ly/javascript/modularizing-monolithic-javascript-projects/

[mitar]

The only downside is that NPM does not allow you to install from Git when you put more than one package into a repository.

Which pretty limits how much you can use it for forks. If you cannot do a dependency against a git repository, then you have to publish a package. Pretty hard.

[murillo128]

The list of packages used by blaze is the one below:

• Core
• Base64
• EJSON
• Random
• MongoID
• IdMap
• OrderedDict
• Tracker
• ObserveSequence
• HTMLJS
• HMLTools
• ReactiveVar
• Blaze
• BlazeTools
• Templating
• Spacebars
• SpacebarsCompiler

Also, in ObserveSequence i had to add two files from minimongo:

• meteor/packages/minimongo/diff.js
• meteor/packages/minimongo/objectid.js

And tinytest for testing all the above

[murillo128]

I think the best solution for packaging is having 1 meteor package == 1 git repo == 1 npm package. This has the (big) downside that it is really difficult to work with branches/git repos directly.

I overcome it by not requiring each specific package, but exporting a function that expect a Meteor object in which I add the package into.

So instead of doing

var Tracker = require('meteor-tracker');
var Blaze = {};
/*......*/
module.export = Blaze;

I do the following:

module.export = function(Meteor)
{
var Tracker = Meteor.Tracker;
var Blaze = {};
/*......*/
Meteor.Blaze = Blaze;
return Blaze;
}

Then you need a master project my meteor-client [https://github.com/eface2face/meteor-client] that acts as entry point, instantiating all the references:

  var jQuery= require('jquery');
  var lodash = require('lodash');
  var Meteor = require("meteor-core")(lodash);
  require("meteor-base64")(Meteor);
  require("meteor-ejson-safe")(Meteor);
  require("meteor-random-window-crypto")(Meteor);
  require("meteor-id-map")(Meteor);
  require("meteor-ordered-dict")(Meteor);
  require("meteor-tracker")(Meteor);
  require("meteor-observe-sequence")(Meteor);
  require("meteor-htmljs")(Meteor);
  require("meteor-html-tools")(Meteor);
  require("meteor-reactive-var")(Meteor);
  require("meteor-reactive-object-map")(Meteor);
  require("meteor-blaze")(Meteor,jQuery);
  require("meteor-blaze-tools")(Meteor);
  require("meteor-templating")(Meteor);
  require("meteor-spacebars")(Meteor);
  require("meteor-spacebars-compiler")(Meteor);
module.export= Meteor;

It is kind of an ugly trick, and don't like it much, but it allows me to override the phisical location of each of the module, and work with branches or local copies. For example in the meteor-module-generator project [https://github.com/eface2face/meteor-module-generator/blob/meteor-update] I have each meteor package as a git submodule and them load each of the packages by requiring the filename, not the package reference:

var jquery = require("jquery");
var underscore = require("lodash");
var Meteor = require("./modules/meteor-core/dist/meteor.js")(underscore);
require("./modules/meteor-base64/dist/base64.js")(Meteor);
require("./modules/meteor-ejson/dist/ejson.js")(Meteor);
require("./modules/meteor-random/dist/random.js")(Meteor);
require("./modules/meteor-mongo-id/dist/mongo-id.js")(Meteor);
require("./modules/meteor-id-map/dist/id-map.js")(Meteor);
require("./modules/meteor-ordered-dict/dist/ordered_dict.js")(Meteor);
require("./modules/meteor-tracker/dist/tracker.js")(Meteor);
require("./modules/meteor-observe-sequence/dist/observe_sequence.js")(Meteor);
require("./modules/meteor-htmljs/dist/html.js")(Meteor);
require("./modules/meteor-html-tools/dist/html-tools.js")(Meteor);
require("./modules/meteor-reactive-var/dist/reactive-var.js")(Meteor);
require("./modules/meteor-blaze/dist/blaze.js")(Meteor,jquery);
require("./modules/meteor-blaze-tools/dist/blaze-tools.js")(Meteor);
require("./modules/meteor-templating/dist/templating.js")(Meteor);
require("./modules/meteor-spacebars/dist/spacebars.js")(Meteor);
require("./modules/meteor-spacebars-compiler/dist/spacebars-compiler.js")(Meteor);
require("./modules/meteor-tinytest/dist/tinytest.js")(Meteor);
require("./ext/meteor-reactive-object-map/reactive-object-map.js")(Meteor);

//Export to global
global._ = underscore;
global.jquery = jquery;
global.$ = jquery;
global.Meteor = Meteor;

As I said, it is a quite awful trick and but it does the job. If we could rewrite/override the location of a package at npm, then life would be much easier, but I don't know of other way of doing so.

By the way, any estimation about when 1.3 will be released?

[murillo128]

Well, after investigating further, this could help:

https://nodejs.org/en/blog/npm/managing-node-js-dependencies-with-shrinkwrap/

[martijnwalraven]

@murillo128: Can you explain why you think shrinkwrapping would help here?

[murillo128]

still don't know.. :)

...but i will investigate, I don't like to export the function and then pass the variable of Meteor, but somehow we need to replace the dependencies on the subprojects somehow, which seems that is what shinkwrap is doing (although just changing the version numbers not the location). Anyway, just a bit of brainstorming

[murillo128]

BTW, i had to add a couple more packages in order to make Blaze work and pass all the tests from the required packages:

• DiffSequence
• Minimongo (This could be removed if ObserveSequence didn't use some utility functions from LocalCollection or they are moved to an external package)

[mitar]

@stubailo, @zol, now that Meteor 1.3 is released, I think we should really focus on the effort of removing Blaze from Meteor repository and make packages stand-alone. Otherwise I think sooner or later it will fork.

[murillo128]

BTW, even if it is a nightmare to develop at the beginning, I think that what makes more sense is a 1 meteor package = 1 git repo = 1 npm package

[zol]

@mitar totally, that's what we'll be focusing on next.

[mitar]

My wish list:

• easy to run Meteor with your own fork of a core package (or in general any package, probably core packages should not be anything special anymore, just a default list of packages available)
• core packages being such forks
• each package should be in its own repository and its own NPM package, or find a way for NPM packages to work with subdirectories (which NPM does not yet support)

[murillo128]

I have been making some tests with npm packages and git branches, and it seems it would be easier than I though.

I have created three dummy projects a,b and c (https://github.com/murillo128/dummy-a https://github.com/murillo128/dummy-b https://github.com/murillo128/dummy-c).

c is just exporting the name of the github branch:

module.exports="master";

I have then published it to npm (https://www.npmjs.com/package/dummy-c) with v1.0.0

b requires the npm package for c and re-exports the the branch name

module.exports=require('dummy-c');

I have then published it to npm (https://www.npmjs.com/package/dummy-b) with v1.0.0

a requires the npm package b and prints branch name

console.log('Using : ' + require('dummy-b'));

Doing an npm ion a rpovides the expected output

> npm i
dummy-b@1.0.0 node_modules/dummy-b
└── dummy-c@1.0.0

> node index.js
Using : master

Then I created a branch on c changing the label of the branch (https://github.com/murillo128/dummy-c/tree/branch)

module.exports="master";

Then I used it c directly on a:

console.log('Using : ' + require('dummy-b'));
console.log('Using : ' + require('dummy-c'));

But installing the branch instead of the package:

> npm i --save murillo128/dummy-c#branch
dummy-c@1.0.0 node_modules/dummy-c

> node index.js
Using : master
Using : branch

As the version in the package have not changed, removing the node modules and installing them again, makes npm to use c from dependency in a (the branch one) and not the one from b (the npm package one):

> rm node_modules/ -rf && npm i
http GET https://registry.npmjs.org/dummy-b
http 304 https://registry.npmjs.org/dummy-b
dummy-b@1.0.0 node_modules/dummy-b

dummy-c@1.0.0 node_modules/dummy-c
> node index.js
Using : branch
Using : branch

Also, if version differs or it is possible to use npm shrinkwrap to get the tree of references to the packages and override both the version and the source:

{
  "name": "dummy-a",
  "version": "1.0.0",
  "dependencies": {
    "dummy-b": {
      "version": "1.0.0",
      "from": "dummy-b@^1.0.0",
      "resolved": "https://registry.npmjs.org/dummy-b/-/dummy-b-1.0.0.tgz"
    },
    "dummy-c": {
      "version": "1.0.0",
      "from": "dummy-c@git://github.com/murillo128/dummy-c#branch",
      "resolved": "git://github.com/murillo128/dummy-c#d07589983747f3492bf042f4eb0fe68039e340cf"
    }
  }
}

[murillo128]

So IMHO I think the best would be to have one master repo/package blazejs and then one meteor-xxxx repo/pacakge per current meteor package.

blazejs
└── meteor-blaze
    └── meteor-core
    └── meteor-htmljs
         └── meteor-core
         └── meteor-xxxxx
         └── meteor-tracker
              └── meteor-core
               .......

So if you want to work with a branch of one of the "core" packages, you just need to install the dependency on the master package and override with shrinkwrap if necessary.

[murillo128]

@stubailo, @zol, @mitar any feedback? What are your plans on how to proceed?

[mitar]

I am moving this issue to a later milestone. For now we are going with Meteor-only packages so dependencies on other Meteor packages are OK.

[mitar]

So there is interest in working on this, because it is a dependency for #11. I think this is great. It is a dependency and can be probably done with little coding effort, but more coordination effort. The issue is not clearly discussed and a lot of discussion is happening in #11 as well.

My proposal is to have a NPM scope for Meteor/MDG/core packages and then we could use the same pacakge names for both Atmosphere and NPM packages.

@zol, @stubailo, @tmeasday, could we obtain such a scope? And then start porting one package to NPM?

[tmeasday]

@mitar apologies on the silence on this, we do actually have a plan here and I was just waiting to workshop it with a couple of people before feeling confident that it was a good idea. You can read it here: https://paper.dropbox.com/doc/Transitioning-Tracker-to-npm-Hu4QtlCaHj9UWM00uYZ4f

We should probably get the meteorjs namespace, npm just want a CC to set up an organization, need to find one to use :)

In terms of following through on the plan, it seems like a good idea for one of us to do the first package (I nominate tracker), so we can test out the above and figure out if there are problems. I hope to look at it next week.

[mitar]

We should probably get the meteorjs namespace, npm just want a CC to set up an organization, need to find one to use :)

Yea, I think that having a namespace/scope would be great. :-)

In terms of following through on the plan, it seems like a good idea for one of us to do the first package (I nominate tracker), so we can test out the above and figure out if there are problems. I hope to look at it next week.

Sounds good. Also, this plan.

But I do have comment on the above document: I do not think it is a good one. :-) Because it solves the problem for Meteor, but not for NPM users. So NPM-only users using Blaze and Tracker will might get bitten by your solution.

I would propose a much simpler one: simply attach Tracker global state to one known global variable. Attach it to process? process._tracker? Or window in browser?

For server-side tracker we are attaching it to the current fiber. :-)

Or something like this?

[tmeasday]

If tracker or any other singleton-like package wants to solve the multiple loading problem in that way then that's on the maintainer, but it's not really a solution. If package X wants tracker 1 and package Y wants tracker 2, and the package manager lets you install both X and Y (cough cough npm), then using a global like that is going to be a big problem.

The idea of the solution outlined in that doc isn't to solve the singleton loading problem, it is to make it so that I don't have to both meteor add X and npm install @meteorjs/X (and all of X's dependencies) every time I add a core package.

[mitar]

If package X wants tracker 1 and package Y wants tracker 2, and the package manager lets you install both X and Y (cough cough npm), then using a global like that is going to be a big problem.

No, because tracker 2 would use a different variable.

[tmeasday]

No, because tracker 2 would use a different variable.

If it's not a singleton then everything breaks because your mongo queries no longer trigger re-renders in your templates, for instance. At the end of the day the only true solution is to enforce the singleton-ness in the package system.

(In this story X=mongo and Y=templates)

Someone needs to make npm work properly. But that's a story for another day. For now it's on the users.

[mitar]

If it's not a singleton then everything breaks because your mongo queries no longer trigger re-renders in your templates, for instance.

OK, but this is in a hypothetical situation that we release a major, backwards incompatible version. Based on current state of Tracker development I would guess nobody will be doing much development here, so if we go with a global variable for Tracker 1, it will just work for everyone, and if Tracker 2 comes around they can explain that user has to enforce singleton.

[tmeasday]

Like I said, if the maintainer of tracker wants to paint themself into that corner, that's on them (it would make it very hard to ever release a new even-slightly-breaking version of tracker again, I suspect). It's not something I would recommend we do for all of the "singleton-ish" meteor packages.

[nickredmark]

I don't know the process, but maybe if this was a ticket on the meteor project it would get a higher priority? As far as I understand it this is what would get the transition to npm started, which is not just relevant to blaze imho.

[mitar]

@nmaro: No need for this, MDG is active here.

@tmeasday: What is the status of this?

[tmeasday]

Hey @mitar we've been a little distracted with the upcoming GraphQL summit and the performance improvements for 1.4.2 but this is definitely in the near-term plan.

[devonbarrett]

@tmeasday has the status of this changed at all?

[tmeasday]

Nope, not yet, apologies

[mitar]

Still nothing. :-(

[MiroHibler]

Bump! ;)

[MiroHibler]

Hey guys, is there at least intention to solve this in the near future (v1.7-ish)?

[filipenevola]

I'm closing this issue because it's too old.

If you think this issue is still relevant please open a new one.

Why? We need to pay attention to all the open items so we should keep opened only items where people are involved.