Discussion: Iron as a build tool

[lirbank]

Just had a brief look at the new iron with new project structure.

I must admit I am somewhat split to the idea of having iron as a build tool and passthrough of commands to meteor. I am advocate of simplicity so I am asking myself, will there really be enough benefits to justify the added complexity? Not only code maintenance but also, we will not be able to use "standard" buildpacks for deploying on Heroku (and later on Galaxy), or no? If we keep the folder structure in a way that meteor can use natively I think a lot of future issues can be avoided.

[lirbank]

Ahh! Now I see, the complete meteor app is in /app/ instead of in the root. So the app can be built and run by meteor simply by cd in there. That makes sense. I thought iron was re-organizing the folders on the fly to create a meteor compatible dir structure, not so, phew.

I kind of like that, but still it will require custom heroku buildpacks but it shouldn't break to easily once done with this structure.

[cmather]

The meteor app is in the app directory as you pointed out. You can either cd into the app directory and run any meteor commands as usual, or you can just use the iron command line tool from the project folder. If you run one of the meteor commands it will be run in the context of the app folder (the iron cli just proxies any meteor commands to the meteor cli).

Does the heroku build pack rely on something about the old project structure? Like the config folder or something?

[cmather]

Btw the main reason for proxying commands to the meteor cli is because of the super project structure for config and other stuff we don't want to end up in the build process. For example, you might have some maintenance tools and scripts in the bin folder that aren't really part of the built app. But it's annoying to have to cd into the app directory just to run the app or add a package :).

[lirbank]

Testing deploying to Heroku with the new project structure. Heroku apps are deployed via git push of the actual source code so the buildpack need to have an understanding of what the project looks like and existing build packs assume the meteor project is in the root of the repo.

iron create test
cd test/
git init
git add .
git commit -am "init"
heroku create
heroku config:set ROOT_URL=https://damp-river-3902.herokuapp.com
heroku config:set BUILDPACK_URL=https://github.com/AdmitHub/meteor-buildpack-horse.git
git push heroku master

which will give you:

Counting objects: 45, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (31/31), done.
Writing objects: 100% (45/45), 4.55 KiB | 0 bytes/s, done.
Total 45 (delta 1), reused 0 (delta 0)
remote: Compressing source files... done.
remote: Building source:
remote: 
remote: -----> Fetching custom git buildpack... done
remote: -----> Node.js app detected
remote: -----> Installing node
remote: -----> Installing meteor
remote: Downloading Meteor distribution
remote: ######################################################################## 100.0%
remote: 
remote: Meteor 1.0.3.1 has been installed in your home directory (~/.meteor).
remote: 
remote: Now you need to do one of the following:
remote: 
remote:   (1) Add "$HOME/.meteor" to your path, or
remote:   (2) Run this command as root:
remote:         cp "/tmp/buildpack_4574857dbc9836db086457654ba60109/meteor-D9Wd/.meteor/packages/meteor-tool/1.0.40/meteor-tool-os.linux.x86_64/scripts/admin/launch-meteor" /usr/bin/meteor
remote: 
remote: Then to get started, take a look at 'meteor --help' or see the docs at
remote: docs.meteor.com.
remote: -----> Bundling bundle
remote: remove-platform: You're not in a Meteor project directory.
remote: 
remote: To create a new Meteor project:
remote:   meteor create <project name>
remote: For example:
remote:   meteor create myapp
remote: 
remote: For more help, see 'meteor --help'.
remote: 
remote:  !     Push rejected, failed to compile Node.js app
remote: 
remote: Verifying deploy...
remote: 
remote: !   Push rejected to damp-river-3902.
remote: 
To https://git.heroku.com/damp-river-3902.git
 ! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'https://git.heroku.com/damp-river-3902.git'

However if I move the content of /app/ to the root and push commit+push again it works.

Conclusion, we have to create our own buildpack or provide patches for the popular ones. We could first look for /.meteor (in the root) and if it's not there we look for /app/.meteor before we abort with failure.

[lirbank]

Heroku is weird in that it requires git push to deploy, so you will still push the whole project to Heroku (including tools and stuff you don't want in your bundle), but the final meteor bundle will be created from the /app folder (with our patch) so it will still make a clean app build and deployment.

Anyhow, I like the idea of the super project structure so I think we should just patch the Horse buildpack and we're good.

[lirbank]

Another potential risk with proxying commands to meteor is that if meteor add a new command name that we already use. But we could name space it so it's not likely that will happen, and worst case we'll just change our command name there is a future clash. Have some additional thoughts on commands/flags, I'll start a separate issue for it shortly.

[cmather]

Oh I see (re the buildpack). But don't we need the heroku build pack to recognize configuration settings? Or, do people using Heroku do something different? I would think the build pack should run the meteor app in production mode, bringing in the appropriate settings and env for the production environment.

Should we just write our own build pack that works? Or send a PR to the popular ones?

I agree there's a little risk of namespacing conflicts between iron and meteor. But I'm hoping the iron surface area is small enough that it won't be a real issue in practice. I do keep an array of all valid meteor methods, so if they add a new one, or remove one, we'll have to update that array. This is how I'm able to show the usage help for iron instead of just proxying any unknown thing to meteor and showing a random meteor failure.

[lirbank]

You configure the buildpack to do what ever you want, eg configuring a mongoDB, downloading meteor, creating a bundle and starting meteor, etc. The buildpack is run when you git push a new commit to Heroku. Environment vars are set from the command line like this (example with Meteor special vars):

heroku config:set ROOT_URL=https://damp-river-3902.herokuapp.com
heroku config:set BUILDPACK_URL=https://github.com/AdmitHub/meteor-buildpack-horse.git

So you can do heroku config:set environment=production or heroku config:set environment=staging to distinguish between different environments. But I asume this could be done from the buildpack as well (eg. export vars from a file). I've never done it that way, but I would actually be a nicer way of doing it...

I usually have two apps att heroku, production and staging, plus dev. locally. Also I've mostly done closed source project so then I just put a JS file in /server/lib/ with config (smtp keys etc) for all three environment and commit it. Then my app loads staging config for staging based on the environment the Heroku app has (eg heroku config:set environment=production). But that would not work for OSS. How do you usually separate configs from git for open source apps?

[lirbank]

Btw. I think we should choose an existing buildpack and contribute to it. Less work :) Better share the burden. I've lately used https://github.com/AdmitHub/meteor-buildpack-horse - it seems to be an active project.

It doesn't handle Iron's configs but at least you can deploy, so on par with what we had before the new project structure.

[lirbank]

A nice way would be to make another patch to the buildpack to make it handle environments and configs the iron way. Eg the build pack should export METEOR_SETTINGS="$(cat config/staging/settings.json)" before launch the app based on what env is set by the user (heroku config:set environment=production).

I'll look into it.

[cmather]

In production I symlink to a production config folder. For example, in production:

current/
  bin/
  app/
  config -> ../shared/config (symlink)
shared/
  config/
    production/

Then protect the shared folder so that it can only be ready by the deployer or administrator user.

Not sure what a good workflow should be for Heroku.

[lirbank]

As a first step I've modified the first Heroku buildpack to support the new project structure https://github.com/lirbank/meteor-buildpack-horse

I've created a PR too, let's see if the maintainer pull it in AdmitHub/meteor-buildpack-horse#18

[cmather]

Thanks Mikael!

On Feb 17, 2015, at 12:15 AM, Mikael Lirbank notifications@github.com wrote:

As a first step I've modified the first Heroku buildpack to support the new project structure https://github.com/lirbank/meteor-buildpack-horse

I've created a PR too, let's see if the maintainer pull it in AdmitHub/meteor-buildpack-horse#18

—
Reply to this email directly or view it on GitHub.

[cmather]

closing to keep track of other issues. can continue discussion though. deployment should work :)

[cmather]

or will keep open and mark as idea

[lirbank]

I've sent another message to the maintainer of the Horse buildpack to see if he can merge. @cmather or @scott-mcpherson, in the meantime can you merge this PR #90 so people using Heroku know there is a way to deploy projects created with Iron?

[cmather]

yep thanks