Migrate community packages into the core and check Meteor 3.x compatibility

[nachocodoner]

See reasons, why and how these packages are included into the core: #13039 (comment)

Keeping history of those packages on getting them here into the core.

• Migrations - https://github.com/quavedev/meteor-migrations
• Cron - https://github.com/quavedev/meteor-synced-cron
• Collection hooks - https://github.com/Meteor-Community-Packages/meteor-collection-hooks

[CLAassistant]

All committers have signed the CLA.

[harryadel]

@nachocodoner Is meteor-synced-cron the best alternative out there? I think https://github.com/wildhart/meteor.jobs is the most performant meteor package out there but it's not async ready yet. What do you think?

EDIT: out of sheer curiosity, what's the reasoning behind adding these packages to the core? Can you lay out in bullet points style the criteria for packages to be qualified for adding to the core? So many packages are paramount to the Meteor ecosystem yet none was added to the core before 🤷

[nachocodoner]

EDIT: out of sheer curiosity, what's the reasoning behind adding these packages to the core? Can you lay out in bullet points style the criteria for packages to be qualified for adding to the core? So many packages are paramount to the Meteor ecosystem yet none was added to the core before 🤷

I discussed this task with the team, as it was planned before my joining.

The reasons behind the addition, how and why these are included into the core:

• Popularity: Packages serving as a foundation for many Meteor apps, packages that almost any app, from medium to big, use at least one of them. Supporting evidence in repository numbers and atmosphere package installs. Have work in maintenance, expanding features or migrating to Meteor 3.x.
• Loosely coupled: Adopted packages aren't included directly in Meteor apps by default. Meteor apps just include minimal packages, and then the developers choose and import the necessary based on their preferences.
• Maintenance: Moving packages to the core helps the core team gain familiarity and provide ongoing support.

Looking into these packages:

• meteor-migrations:
  • percolatestudio/meteor-migrations with 244 stars, 58 forks, and 63,963 installs
  • Recently migrated to Meteor 3.x: quavedev/meteor-migrations
• meteor-synced-cron:
  • percolatestudio/meteor-synced-cron with 501 stars, 105 forks, and 80,288 installs
  • Recently migrated to Meteor 3.x: quavedev/meteor-synced-cron
• meteor-collection-hooks:
  • Meteor-Community-Packages/meteor-collection-hooks with 658 stars, 90 forks, 381,822 installs and 40th most downloaded (top 1 versus alternatives like aldeed:collection2 which is in 45th).
  • Recently adopted by the Meteor community and in works to be migrated to Meteor 3.x.

Of course, the team can revisit this in the future and bring more packages to the core. For now, those are the ones chosen.

[nachocodoner]

@nachocodoner Is meteor-synced-cron the best alternative out there? I think wildhart/meteor.jobs is the most performant meteor package out there but it's not async ready yet. What do you think?

Regarding meteor-synced-cron, the team decided for a simple scalability solution to manage tasks by adopting this package, which implements a cron-based system. wildhart/meteor.jobs offers a more powerful yet complex queue-based system for handling jobs. As the way meteor-synced-cron is added to the core makes it not mandatory for any meteor app, any developer can still choose the solution that best fits your app's needs, such as the one you shared from the community.

[StorytellerCZ]

I'm a bit annoyed that there is talk to migrate the quave versions of migrations and synced cron for two reasons.
First, the original packages are under percolate namespace, which Meteor Software controls and there have been releases done in the past and there are PRs for Meteor 3 compatibility.
Second, it is news to me that Quave maintains versions of these packages that have Meteor 3 compatibility and other updates without at least providing PRs to upstream so that this benefit could be provided to the wider community. So this either means that their changes are specific to their systems and not compatible with the wider community. If that is not the case is it a license issue? Are the changes they made not ready to be merged upstream? Either case I don't think it is good to take on their changes until this is explained and rectified also in terms of CLA.

[harryadel]

@nachocodoner Thank for you thorough and thoughtful reply. I typically initiate these discussion in order for us to understand things better and reach the best solutions, thank you :)

I've a bit of epiphany with this, on one hand, I like the fact that the core team is removing a considerable amount of the load from the shoulders of community package maintainers, primarily @jankapunkt @StorytellerCZ
but on the other hand I fail to see a clear cut reason for considering on package over another 🤷

There has to be a very conclusive reason to merge a package into the core. For instance, a crucial package/lib to the community/Meteor that nobody is capable of maintaining regardless of the reasons (time, effort, technical aptitude). The best case in point for this is reify. A package so important to the Meteor ecosystem that when @benjamn left, it had to be maintained or Meteor would've suffered a great deal without it. Also may I add, how nobody dared to touch it but @zodern? I'm tired of Meteor being a one man show but that's a problem for another day.
To help illustrate this point further try to replace any of the packages you mentioned with another and yet it'd still fit the bill.

• collection2:
  aldeed/collection2 with 1k stars, 108 forks, and 355,917 installs
  Recently migrated to Meteor 3.x: Meteor 3 Migration

Matter of fact, I believe the case for collection2 is even stronger :\

So the question remains: why?

I discussed this task with the team, as it was planned before my joining.

If you had joined a little earlier, would this decision still be made? 🤔

What makes things worse is the fact that the core team is heavily understaffed - only three people are tasked with maintaining Meteor @denihs @Grubba27 @nachocodoner The odds gets thinner considering the fact that Gabriel now acts as a developer advocate and not as hands on deck as before. The ETA for 3.0 was February and I don't think we're there yet so why take the spotlight from delivering 3.0 to adding some third packages?? It doesn't make sense.

Meteor should be kept light and nimble and not burdened with additional baggage unless called for.

I rest my case.

EDIT:

I think if the core team is willing to undertake such burden meteor-desktop and redis-oplog should be the top picks and the reasons are:

• Absence of Original Author: Both packages original authors are no longer maintaining those packages. The left the entire Meteor community for good. @theodorDiaconu and @wojtkowiak are no longer around.
• Opportunity: Again, both packages either fix a critical problem with the platform or open up a completely new avenue never present before. redis-oplog solves the issue of scaling with pub/sub and meteor-desktop allows you to publish to the desktop completely uncharted territory before it came onto the scene.
• Lack of Alternatives: Can you mention proper alternatives to these packages? Change stream are nowhere as close and the best thing we have to meteor-desktop is Capacitor integration @nachocodoner mentioned on the forums.
• Lack of Attention: Ok, the original authors left but nobody signed up for the task. @StorytellerCZ is doing his best to help keep the in shape but he's also doing the same for so many more which stretches his efforts. @msavin & @a4xrbj1 are vocal about this issue on the forums. Fun fact, redis-oplog isn't async-compliant yet! I think the community would be more appreciative for migrating redis-oplog than adding already migrated packages to the core.

[StorytellerCZ]

I'm for migrations and hooks (which should be a default feature in Meteor IMHO. So I look at it also from that lense. This is where collection2 has a bit more discussion and other packages come up. So I'm very happy with the inclusion of these 3. @harryadel brings out a good point in timing for the inclusion. I think we could get them updated on their main repos and releases first, then maybe move this to v3.1 release, but that really depends on what is more time efficient to get 3.0 out there faster.

[harryadel]

I think we could get them updated on their main repos and releases first, then maybe move this to v3.1 release

I believe that's a good compromise for both side 👍

@StorytellerCZ Collection2 and it's future has a lot of talking that is way out of scope for this thread. I'll be discussing it soon on the forums.

[nachocodoner]

I agree on following the proper workflow to reach the releases on the respective upstream repositories, gathering the reviews from the community, and having this process more organically done.

I close this PR in favor on follow a better plan on migrate packages into the core.

[jamauro]

fwiw, I think bringing these packages under Meteor core support and having them as optional installs is great. Appreciate y'all doing the work to bring them in. Collection hooks and Migrations in particular make perfect sense. I think having some sort of officially supported Jobs package is a great idea too though I wish a more detailed evaluation was done on which one go with. Number of installs can be a good indication of popularity but doesn't necessarily mean it's the best option, particularly when many of the top installed packages were likely either touted in the Meteor Guide so naturally people are going to choose them or created when Meteor was at peak popularity. I've used wildhart's meteor.jobs package and was hoping it might receive Meteor core support. It's been nice to work with and feels a little more Meteor-like with it's API which feels similar to Meteor Methods in some regards. Maybe it or something similar could be a candidate to receive Meteor core support in the future since I imagine a queue-based Jobs package with error handling would be useful for a lot of people.

Regarding the points made above on timing, I would love to see some of the remaining bugs on 3.0 addressed particularly around Meteor.callAsync / Meteor.applyAsync. I trust that the Meteor team is highly prioritizing these. Thanks again for all your efforts on the 3.0 migration. Looking forward to a RC.

[wojtkowiak]

@harryadel for anyone curious I am still around :) I am not following Meteor's development closely these days but back in the days I did propose to build desktop target into the core, hell I did even apply for a job at MDG to which I never got any response ;)

Anyways, on a more serious note I will be giving Meteor 3 a shot once it's stable and if that goes well I would be interested in building a more versatile integration system around it so for example integrating with Electron or it's alternatives like https://github.com/tauri-apps/tauri etc. would become a breeze.

[jamauro]

@wojtkowiak nice! Using Tauri seems pretty compelling

[a4xrbj1]

Anyways, on a more serious note I will be giving Meteor 3 a shot once it's stable and if that goes well I would be interested in building a more versatile integration system around it so for example integrating with Electron

I’d love to see you do some improvements on the Meteor-desktop project, Bartosz. What we’re really doing mostly is updating packages but we’re very far behind with official Electron versions and falling further and further behind. Right now, we’re waiting for Meteor version 3.0 as we can’t update npm packages any further due to being stuck with nodeJS version 14 right now.

It still works though and it’s very simple to write and publish an Electron app, for all 3 OS (which is what I do).

I’d rather prefer to continue with ElectronJS vs trying something completely new like Tauri or Ionic Capacitor.

& @a4xrbj1 are vocal about this issue on the forums.

To @harryadel - I agree with your post and reasoning. There were actually two other devs (sorry, don’t have their names right now) who helped in fixing some code to ensure we could update to newer ElectronJS versions. IMO, they did the most work on the package in regards to updating it.

Unfortunately I’m not only a single programmer in my company (apart from a post-grad which helps a bit on frontend code) but I’m also realistic enough that as a self taught old fart I can do more harm than good to the source code of Meteor-Desktop (or other community projects).

[harryadel]

I trust that the Meteor team is highly prioritizing these. Thanks again for all your efforts on the 3.0 migration. Looking forward to a RC.

I'm too quite grateful for the core team on their efforts on 3.0 and don't want it to be undermined by other endeavors, at least for the time being.

for anyone curious I am still around :) I am not following Meteor's development closely these days but back in the days I did propose to build desktop target into the core, hell I did even apply for a job at MDG to which I never got any response ;)

@wojtkowiak The return of the Count!! Great to have you, Bartosz.

hell I did even apply for a job at MDG to which I never got any response ;)

Don't worry, it's a recurring theme. You're not the only one 🤣

Though on a more serious note, I believe in the same vein that @radekmie is now responsible for Blaze, maybe the same can be for you & @theodorDiaconu. Each one working on his area of responsibility. It'd be a great way to bring the community back together and get the core nucleus of core contributors back! Maybe @fredmaiaarantes has something to say about that!

All in all, I'd love to have you back on board with us to help us post 3.0.

To @harryadel - I agree with your post and reasoning. There were actually two other devs (sorry, don’t have their names right now) who helped in fixing some code to ensure we could update to newer ElectronJS versions. IMO, they did the most work on the package in regards to updating it.

Unfortunately I’m not only a single programmer in my company (apart from a post-grad which helps a bit on frontend code) but I’m also realistic enough that as a self taught old fart I can do more harm than good to the source code of Meteor-Desktop (or other community projects).

Don't be too harsh on yourself, though I definitely understand why meteor-desktop is crucial to your very small team. 👍

[fredmaiaarantes]

Though on a more serious note, I believe in the same vein that @radekmie is now responsible for Blaze, maybe the same can be for you & @theodorDiaconu. Each one working on his area of responsibility. It'd be a great way to bring the community back together and get the core nucleus of core contributors back! Maybe @fredmaiaarantes has something to say about that!

Definitely! I'm all for the idea and believe that having dedicated people for specific areas can be great!
It's nice to see you around, @wojtkowiak!