New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide a better way to use the Meteor Tool without Mongo #31

Open
mitar opened this Issue Jun 7, 2017 · 35 comments

Comments

Projects
None yet
7 participants
@mitar
Collaborator

mitar commented Jun 7, 2017

Migrated from: meteor/meteor#4645

@dr-dimitru

This comment has been minimized.

dr-dimitru commented Jun 7, 2017

I have apps running without mongo, with next packages only:

meteor
underscore
ecmascript
blaze-html-templates
reactive-var
jquery
tracker
standard-minifier-js
standard-minifier-css

As you see there is no meteor-base, but meteor instead.
//cc @ccorcos

@zimme

This comment has been minimized.

zimme commented Jun 8, 2017

It would be nice to be able to disable the start of mongodb when running Meteor.
For example when you use Meteor as a build system/dev server.

@hwillson

This comment has been minimized.

Member

hwillson commented Jun 8, 2017

Or if you’re leveraging some totally random library to work with databases other than Mongo … 🙂

@stubailo

This comment has been minimized.

stubailo commented Jun 8, 2017

Yep, I think this would be a huge improvement for people looking to use Meteor as a build system. Perhaps we can factor out the database stuff as a hook somehow, so that it can be implemented as a package?

@zimme

This comment has been minimized.

zimme commented Jun 8, 2017

So can we split this work up in at least 2 phases.

  1. Allow Meteor to be run without starting MongoDB.
    MONGO_URL=false or something
  2. Make Meteor packages that don't actually need MongoDB work without MongoDB

The way I'm using Meteor in a few projects is as a build system with no packages that uses mongodb.

static-html
standard-minifier-css
standard-minifier-js
es5-shim
ecmascript
shell-server
webapp
hot-code-push

In these projects I'm not using anything Meteor specific and just using Meteor instead of webpack.

@stubailo

This comment has been minimized.

stubailo commented Jun 8, 2017

Right now passing MONGO_URL=false already works! It feels hacky though since it's an env var.

@zimme

This comment has been minimized.

zimme commented Jun 8, 2017

Re-reading your comment again @stubailo, I believe you weren't talking about Meteor packages that is relying on MongoDB without needing too. So my previous comment might have been unnecessary.

@zimme

This comment has been minimized.

zimme commented Jun 8, 2017

@stubailo, well... awesome! That's all I need 👍
So all my needs are solved by knowing that.

@stubailo

This comment has been minimized.

stubailo commented Jun 8, 2017

Yeah there's not much we can do about packages over-declaring dependencies IMO.

@zimme

This comment has been minimized.

zimme commented Jun 8, 2017

Ok, so with Meteor already being able to start without starting MongoDB alongside it.
This discussion, I guess, will be more about if we need to find a different way of doing it?
Do we believe --no-mongodb or some other flag is better than using an environment variable?

@stubailo

This comment has been minimized.

stubailo commented Jun 8, 2017

I think the ideal case would be that Mongo doesn't start by default if you don't have the "Mongo" package.

@hwillson

This comment has been minimized.

Member

hwillson commented Jun 8, 2017

Maybe we should step back a bit and clarify what this issue is about. The title says “Allow Meteor to run without a Mongo database”, but that is already achievable by removing the mongo package, and setting MONGO_URL=false (which I agree is quite hacky).

The intent of the original issue however (meteor/meteor#4645), was to find a way to use Meteor to build web apps, without requiring Mongo, while still retaining lots of Meteor’s other benefits (like livedata, accounts, etc.). So not just using Meteor as a build tool, but being able to leverage certain parts of the Meteor ecosystem with different databases. Re-reading the original issue, I wonder how much of this is still relevant now that we have Apollo (and alternate ways of handling livedata for example). I guess the accounts part is still valid, but that should probably be tracked in a separate issue (if it isn’t already - I’ll check shortly).

So, what’s remaining with this feature request? Is it just coming up with a better way to disable Mongo, like #31 (comment)?

@zimme

This comment has been minimized.

zimme commented Jun 8, 2017

@hwillson, I think that sounds about right.

@zimme

This comment has been minimized.

zimme commented Jun 8, 2017

So I'm thinking if we just check .meteor/packages for a mongo entry and if it's not there, set MONGO_URL to false. This way we don't have to rely on the code that parse the packages etc. which I believe is in the "app" which is started after mongodb.

@hwillson

This comment has been minimized.

Member

hwillson commented Jun 8, 2017

Yes, I think something like this could work. We'll have to keep in mind the upcoming meteor/meteor#8769 changes (which will allow packages to be used via the --extra-packages command line parameter), but wiring up a check for this should be pretty straightforward.

@hwillson hwillson changed the title from Allow Meteor to run without a Mongo database to Provide a better way to use the Meteor Tool without Mongo Jun 8, 2017

@zimme

This comment has been minimized.

zimme commented Jun 8, 2017

I'll try and get a PR together later this week.

What I've found so far is that we should be able to use projectContext.ProjectConstraintsFiles getConstraint function to check if the project have the mongo package and it seems that the run command registration function in commands.js would be a good place for this.

@hwillson

This comment has been minimized.

Member

hwillson commented Jun 8, 2017

Thanks @zimme - I think we’re really close to moving forward with this, but I haven’t marked it as pull-requests-encouraged yet, as I haven’t had a chance to think about any gotchas we might be missing. Maybe hold off a day or two before working on this; I’ll aim to take a closer look shortly (and we can give others a chance to comment about this idea well).

@zimme

This comment has been minimized.

zimme commented Jun 8, 2017

No problem, wrote down what I found so far as I don't know if I'll have time to do this or not 😜

@zimme

This comment has been minimized.

zimme commented Jun 9, 2017

One thing that I can think of that might trip this up is the following.
Can a package use MongoDB without using the mongo package?

@mitar

This comment has been minimized.

Collaborator

mitar commented Jun 9, 2017

What if we create a dummy package mongodb-dev-server which if it exists you start a server. And then mongo depends on it. Somebody can remove mongo, and add mongodb-dev-server if they just want to have it running.

@zimme

This comment has been minimized.

zimme commented Jun 9, 2017

I actually like that idea @mitar.
I guess the PR would involve us finding core packages that uses MongoDB without depending on mongo, if there are any, and adding mongo-dev-server as a dependency on those packages, if we choose this route.

@abernix

This comment has been minimized.

Member

abernix commented Jun 9, 2017

I like the idea of having mongo depend on mongodb-dev-server and having the server only start if that package is present. It at least makes the resource drain slightly more configurable (and sticky, per project).

Going a step further (probably too far), I think it'd be even better if mongodb-dev-server was not just a dummy-package but actually contained a lot of the (extensive) Mongo logic which currently lies in run-mongo.js, and maybe even the mongod executable. This would be a much larger undertaking so probably best to land the above recommendation first, then look further. It'd be nice to easily avoid the weight of mongod in the meteor-tool when it's not (ever!) going to be necessary!

As for implementation, I'm not sure that using projectContext's ProjectConstraintsFiles#getConstraint would work since that should reference packages which are directly listed in .meteor/packages. I believe the correct way to check would be to call projectContext.packageMapFile.getInfo('<packageName>'), which would properly check if it was included, as a dependency as well (for example, of a dependency of mongo, as suggested above).

@zimme

This comment has been minimized.

zimme commented Jun 9, 2017

Going a step further (probably too far), I think it'd be even better if mongodb-dev-server was not just a dummy-package but actually contained a lot of the (extensive) Mongo logic which currently lies in run-mongo.js, and maybe even the mongod executable. This would be a much larger undertaking so probably best to land the above recommendation first, then look further. It'd be nice to easily avoid the weight of mongod in the meteor-tool when it's not (ever!) going to be necessary!

Sounds good like a good direction forward.

Edit:
I'm not sure this is even possible, MongoDB is started by the Meteor tool before the processing of packages are even started. Maybe it could be done with an npm package?

As for implementation, I'm not sure that using projectContext's ProjectConstraintsFiles#getConstraint would work since that should reference packages which are directly listed in .meteor/packages. I believe the correct way to check would be to call projectContext.packageMapFile.getInfo(''), which would properly check if it was included, as a dependency as well (for example, of a dependency of mongo, as suggested above).

That sounds like a better approach 👍 I'm keeping an eye on meteor/meteor#8769 too, to see how that plays out. I'm hoping it will be implmented in a way that makes projectContext.packageMapFile.getInfo('<packageName>') include the extra packages. So we don't have to do multiple checks.

@zimme

This comment has been minimized.

zimme commented Jun 10, 2017

I believe the correct way to check would be to call projectContext.packageMapFile.getInfo(''), which would properly check if it was included, as a dependency as well (for example, of a dependency of mongo, as suggested above).

It was packageMap that had getInfo, projectContext.packageMap.getInfo('mongo-dev-server');.
This could work, but we would have to run projectContext.resolveConstraints() to have packageMap populated, I don't know if this is an expensive operation?

@zimme

This comment has been minimized.

zimme commented Jun 11, 2017

By using resolveConstraints() and packageMap.getInfo() we don't really have to wait for meteor/meteor#8769, as that will get the extra packages through eachConstraints once that PR is merged.

Also if we decide to check for the existence of a package and based on that start MongoDB or not, the developer needs to re-run meteor for it to take effect. This i have no problem with.

@zimme

This comment has been minimized.

zimme commented Jun 11, 2017

~~Looking into this a bit more, it would be better to use find/indexOf on projectContext.packageMapFile.getCachedVersions().keys() to see if mongo-dev-server is in the project. resolveConstraints() would add to the startup time which I don't think is good.~~~

There doesn't seem to be a better way than resolveConstraints() and packageMapFile.getInfo(), and the overhead of resolveConstraints() wasn't as bad as I thought because if used on it's own it just do the resolve part and doesn't try and download missing packages and the other stuff.

@zimme

This comment has been minimized.

zimme commented Jun 11, 2017

This is what I have so far while playing around with this.
https://github.com/zimme/meteor/tree/zimme/mongo-dev-server

Should start mongodb if mongo-dev-server is present in the project.
Shouldn't start mongodb if it's not present in the project.
If the package is added while the app is running, you should get a message
about restarting meteor to start the mongodb server.
If the package is removed while the app is running, the mongodb server will still run and you still have the correct MONGO_URL in the environment. You have to restart meteor for the change to take effect.

@zimme

This comment has been minimized.

zimme commented Jun 21, 2017

Setting MONGO_URL should take precedence over mongo-dev-server i.e. it should not start a local mongodb server just as before.

@zimme

This comment has been minimized.

zimme commented Jun 28, 2017

I've opened meteor/meteor#8853 with a proposal that seems to work, based on the discussions in here.

I would love to have ddp decoupled from mongo so one could use dynamic-imports, which depends on ddp, without using mongo.

@NitroBAY

This comment has been minimized.

NitroBAY commented Jul 3, 2017

Hello,
I want to disable mongo so I did as you said I've set MONGO_URL to false. The server didn't start MongoDB which is a good thing yet it says it needs it :

> 32-PHARMACIE-METEOR@ start C:\www\eds-www\32-PHARMACIE_METEOR
> meteor run --port 2000 --settings settings-development.json

[[[[[ C:\www\eds-www\32-PHARMACIE_METEOR ]]]]]

=> Started proxy.
W20170704-00:00:00.070(2)? (STDERR) C:\Users\Ilan Schemoul\AppData\Local\.meteor\packages\meteor-tool\1.5.0\mt-os.windows.x86_32\dev_bundle\server-lib\node_modules\fibers\future.js:280
W20170704-00:00:00.125(2)? (STDERR)                                             throw(ex);
W20170704-00:00:00.126(2)? (STDERR)                                             ^
W20170704-00:00:00.129(2)? (STDERR)
W20170704-00:00:00.130(2)? (STDERR) Error: invalid schema, expected mongodb
W20170704-00:00:00.132(2)? (STDERR)     at module.exports (C:\Users\Ilan Schemoul\AppData\Local\.meteor\packages\npm-mongo\2.2.24\npm\node_modules\mongodb\lib\url_parser.js:18:11)
W20170704-00:00:00.132(2)? (STDERR)     at connect (C:\Users\Ilan Schemoul\AppData\Local\.meteor\packages\npm-mongo\2.2.24\npm\node_modules\mongodb\lib\mongo_client.js:401:16)
W20170704-00:00:00.133(2)? (STDERR)     at Function.MongoClient.connect (C:\Users\Ilan Schemoul\AppData\Local\.meteor\packages\npm-mongo\2.2.24\npm\node_modules\mongodb\lib\mongo_client.js:225:3)
W20170704-00:00:00.134(2)? (STDERR)     at new MongoConnection (packages/mongo/mongo_driver.js:169:11)
W20170704-00:00:00.134(2)? (STDERR)     at new MongoInternals.RemoteCollectionDriver (packages/mongo/remote_collection_driver.js:4:16)
W20170704-00:00:00.135(2)? (STDERR)     at Object.<anonymous> (packages/mongo/remote_collection_driver.js:38:10) W20170704-00:00:00.136(2)? (STDERR)     at Object.defaultRemoteCollectionDriver (packages\underscore.js:784:19)
W20170704-00:00:00.136(2)? (STDERR)     at new Mongo.Collection (packages/mongo/collection.js:99:40)
W20170704-00:00:00.137(2)? (STDERR)     at AccountsServer.AccountsCommon (packages/accounts-base/accounts_common.js:23:18)
W20170704-00:00:00.137(2)? (STDERR)     at new AccountsServer (packages/accounts-base/accounts_server.js:18:5)
=> Exited with code: 1

I've removed meteor remove meteor-base (previous versions of the technique was to remove meteor-platform but it doesn't seem to exist anymore...). And adding back meteor add meteor webapp hot-code-push (I don't use mongo but apollo so I guess I do not need the DDP package anymore).
Here's my packages :

apollo                 0.9.1   � Add Apollo to your Meteor app
dynamic-import         0.1.1  Runtime support for Meteor 1.5 dynamic import(...) syntax
ecmascript             0.8.1  Compiler plugin that supports ES2015+ in all .js files
es5-shim               4.6.15  Shims and polyfills to improve ECMAScript 5 support
hot-code-push          1.0.4  Update the client in place when new code is available.
meteor                 1.6.1  Core Meteor environment
mobile-experience      1.0.4  Packages for a great mobile user experience
shell-server           0.2.3  Server-side component of the `meteor shell` command.
standard-minifier-css  1.3.4  Standard css minifier used with Meteor apps by default.
standard-minifier-js   2.1.0  Standard javascript minifiers used with Meteor apps by default.
static-html            1.2.2  Define static page content in .html files
tracker                1.1.3  Dependency tracker to allow reactive callbacks
webapp                 1.3.16  Serves a Meteor app over HTTP

Versions file :

accounts-base@1.3.0
allow-deny@1.0.5
apollo@0.9.1
autoupdate@1.3.12
babel-compiler@6.19.3
babel-runtime@1.0.1
base64@1.0.10
binary-heap@1.0.10
blaze@2.3.2
blaze-tools@1.0.10
boilerplate-generator@1.1.1
caching-compiler@1.1.9
caching-html-compiler@1.1.2
callback-hook@1.0.10
check@1.2.5
ddp@1.2.5
ddp-client@1.3.4
ddp-common@1.2.8
ddp-rate-limiter@1.0.7
ddp-server@1.3.14
deps@1.0.12
diff-sequence@1.0.7
dynamic-import@0.1.1
ecmascript@0.8.1
ecmascript-runtime@0.4.1
ecmascript-runtime-client@0.4.2
ecmascript-runtime-server@0.4.1
ejson@1.0.13
es5-shim@4.6.15
fastclick@1.0.13
geojson-utils@1.0.10
hot-code-push@1.0.4
html-tools@1.0.11
htmljs@1.0.11
http@1.2.12
id-map@1.0.9
jquery@1.11.10
launch-screen@1.1.1
localstorage@1.1.0
logging@1.1.17
meteor@1.6.1
minifier-css@1.2.16
minifier-js@2.1.0
minimongo@1.2.1
mobile-experience@1.0.4
mobile-status-bar@1.0.14
modules@0.9.2
modules-runtime@0.8.0
mongo@1.1.18
mongo-id@1.0.6
npm-mongo@2.2.24
observe-sequence@1.0.16
ordered-dict@1.0.9
promise@0.8.9
random@1.0.10
rate-limit@1.0.8
reactive-var@1.0.11
reload@1.1.11
retry@1.0.9
routepolicy@1.0.12
service-configuration@1.0.11
shell-server@0.2.3
spacebars@1.0.15
spacebars-compiler@1.1.2
standard-minifier-css@1.3.4
standard-minifier-js@2.1.0
static-html@1.2.2
templating-tools@1.1.2
tracker@1.1.3
ui@1.0.13
underscore@1.0.10
url@1.1.0
webapp@1.3.16
webapp-hashing@1.0.9

@abernix

This comment has been minimized.

Member

abernix commented Jul 4, 2017

@NitroBAY apollo depends on accounts-base and accounts-base depends on mongo. With those dependencies, you'll need a MongoDB running for the Meteor.users collection.

@NitroBAY

This comment has been minimized.

NitroBAY commented Sep 1, 2017

I've managed to remove apollo by duplicating only the necessary code from the package but meteor remove mongo gives 'mongo is not a direct dependency in this project.'

standard-minifier-css@1.3.4   # CSS minifier run for production mode
standard-minifier-js@2.1.1    # JS minifier run for production mode
es5-shim@4.6.15                # ECMAScript 5 compatibility for older browsers.
ecmascript@0.8.1              # Enable ECMAScript2015+ syntax in app code
shell-server@0.2.4            # Server-side component of the `meteor shell` command

static-html
meteor@1.7.0
webapp@1.3.17
hot-code-push@1.0.4
@NitroBAY

This comment has been minimized.

NitroBAY commented Sep 1, 2017

On atmosphere I haven't found any of these packages to have a dependency to mongo yet it mongo is still listed in the versions file

@zimme

This comment has been minimized.

zimme commented Sep 2, 2017

@NitroBAY hot-code-push depends on autoupdate which depend om ddp and mongo

@NitroBAY

This comment has been minimized.

NitroBAY commented Sep 2, 2017

Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment