[EPIC] Moving the API off Meteor

[aldeed]

We have been slowly removing Meteor dependencies from Reaction API server code. We are now close enough to being done on the server side, that we can create this Epic to track the remaining work.

To be clear, the goal here is to get the API server running as a non-Meteor Node app. This means all GraphQL code, all events, all background jobs, all relevant startup code. It does not include necessarily all Meteor methods or publications, nor does it include operator UI client code. There will remain a Meteor app to run those things for some time yet, but it will be needed only for running the operator UI. The operator UI will eventually not need Meteor, but there is much work to be done there and no timeline yet.

Here is a preliminary list of the work that this Epic entails:

• Move all SimpleSchema indexes to use collectionIndex function, and remove aldeed:schema-index Meteor package
  • Issue Remove aldeed:schema-index Meteor package #5080
• Add a standard way to register collections and their indexes
  • Issue Create a system that that allows you create indexes on Collections without modifying core #4268
  • PR feat: define collections using registerPlugin #5196
• Add SimpleSchemas on context
  • Issue SimpleSchemas on context #5335
• De-Meteorize emailing
  • PR De-Meteorize emailing code in the API #4998
• De-Meteorize the job-collection job queue
  • Issue De-Meteorize job control system #5288
• Review core startup code
  • PR Demeteorize and reorg some core startup code #5212
• Non-meteor translations loading
  • PR feat: Translations without Meteor #5514
• Deal with template-type assets as in core/server/hooks.js
  • PR feat: remove startup template import #5430
• Replace setAppVersion on startup with getting the data from the new systemInfo GraphQL
  • PR chore: remove unused shop.appVersion #5429
• Create a registerPackage equivalent with no Meteor deps
• Update all plugins to use the non-Meteor registerPackage
• Move calling of registerPackageHandlers into NodeApp, or part of registerPackage replacement.
• Remove all use of context.callMeteorMethod and remove it
  • Issue Rewrite addAccountToGroup to not call through to Meteor method #5336
  • Issue Rewrite inviteShopMember to not call through to Meteor method #5337 - PR refactor: remove inviteShopMember meteor method and rewrite with GraphQL #5520
  • Issue Rewrite removeAccountAddressBookEntry to not call through to Meteor method #5338 - PR refactor: remove addressBookRemove meteor method and rewrite with GraphQL #5492
  • Issue Rewrite removeAccountFromGroup to not call through to Meteor method #5339 - PR refactor: remove removeAccountFromGroup meteor method and rewrite with GraphQL #5493
  • Issue Rewrite setAccountProfileCurrency to not call through to Meteor method #5340 - PR: refactor: add setAccountProfileCurrency GraphQL mutation #5488
  • Issue Rewrite updateAccountAddressBookEntry to not call through to Meteor method #5341 - PR: refactor: rewrite updateAccountAddressBookEntry to not call through to Meteor method #5484
  • Removal PR: refactor: remove context.callMeteorMethod #5524
• Remove all use of getGraphQLContextInMeteorMethod and remove it ( Rewrite any code which calls getGraphQLContextInMeteorMethod to not use Meteor methods #5529 )
  • inviteShopMember.js - refactor: remove inviteShopMember meteor method and rewrite with GraphQL #5520
  • inviteShopOwner.js - refactor: remove all code releated to inviting a shop owner #5553
  • sendResetPasswordEmail.js - refactor: rewrite accounts/sendResetPasswordEmail into a GraphQL mutation #5558
  • sendVerificationEmail.js - TODO
  • sendWelcomeEmail.js - TODO
  • catalog.js - feat: add product and productVariant related GraphQL mutations #5562
  • publishProducts.js - PR refactor: remove catalog/publish/products meteor method, use publishProductsToCatalog GQL Mutation instead #5541
  • cart.js - feat: add product and productVariant related GraphQL mutations #5562
  • getLocale.js - Removed by PR feat: Translations without Meteor #5514
  • index.js - refactor: remove Reaction.Email #5559
  • verifySettings.js - refactor: create verifySMTPEmailSettings GraphQL mutation, replace meteor version #5531
  • generate-sitemaps.js - refactor: remove getGraphQLContextInMeteorMethod from sitemaps-generator plugin #5530
• Move ReactionError to real npm package
  • Issue ReactionError move to npm package #4493
• Demeteorize file upload
  • PR feat: Image uploading without Meteor #5515
  • Second PR needed to port files methods and pubs to GraphQL queries and mutations
• Move to a non-Meteor migrations solution
  • Issue Move to a non-Meteor migrations solution #4416
• Create a high-level place for API plugins, separate from Meteor plugins that are used by operator UI.
• One by one, for each plugin:
  • Confirm full demeteorization of API files. Remove any remaining Meteor deps.
  • Remove all imports from outside the plugin. Consolidate util functions and use Babel to map those imports as if they are a package.
  • Except for methods and pubs that are still used by the operator UI, move everything into the separate API plugin/folder

This will be complete when the pure Node server, currently runnable for testing with npm run devserver, correctly handles all GraphQL requests, generates app events, creates and works background jobs, and sends emails.

Note: Long term, there will be more modular and scalable solutions for background jobs and emails, but there is no need to lump those larger projects in with this one. Porting the existing core plugins should be straightforward for now.

[zenweasel]

Epic in every sense of the word

[pmn4]

This is fantastic, @aldeed. Godspeed!

[dhonig]

This is very exciting work. Re: JobCollection library....Is there any interest in moving this to something like Agenda or Bull? https://github.com/agenda/agenda I've developed a love and hate relationship with job-collection over the course of a few projects.....

[aldeed]

@dhonig Long term it will definitely move to a different tool for queuing and scheduled jobs. It will most likely be a separate microservice, so essentially you could use any library, tool, or third-party provider you want as long as you tell Reaction how to add, monitor, and work jobs.

But that is long term and isn't on the near-term roadmap, so for this epic the goal is really just to keep the existing system working without any changes except removing Meteor dependencies.

[janus-reith]

Has Account management been left out here intentionally?

[aldeed]

@janus-reith Account management (identity provider features) will remain on Meteor, as will the Reaction Admin UI for now. Accounts are still heavily dependent on Meteor accounts packages. The plan is that eventually IDP will split to its own Meteor app, and then will be abstracted so that you can use something like Keycloak or third-party like auth0.

[janus-reith]

Ok, makes sense.
I was wondering initially how the API would deal with account data and permissions then, but realized that this is is how it works already using the the auth token.
Maybe this could allow for a simple transition: https://github.com/accounts-js/accounts

[aldeed]

Having now a release-3.0.0 branch, I'm closing this epic as done. There is some remaining work fixing bugs and getting everything working in that branch, which is now being tracked in a public GH project: https://github.com/orgs/reactioncommerce/projects/4

[vajda-media]

Sorry to ask here after closed this thread. Just wanted to know, what was the benefit of removing one of the best JS Frameworks and go on simple nodeJS? Was also the possiblity kept in focus to take ReactJS + Meteor combination?

[focusaurus]

@vajda-media There were many reasons but one key pain point was crushingly slow start/restart time for the meteor server making back end development extremely hindered. Having a GraphQL API was also a key component of our "headless" product direction and meteor being UI/back-end integrated was contrary to that.