Support multiple jitsu apps in the same package.json #368

Open
jwietelmann opened this Issue Dec 7, 2012 · 44 comments
@jwietelmann

I imagine there's some closed-source work that would be required to make this happen, but this looked like the best place to document it.

What I want is to be able to jitsu deploy production where production maps to a different app. This better reflects the git workflow of many organizations and the next logical step would be for the webhooks to allow me to have one branch continuously deploy to one app and another to continuously deploy to another app. All without constantly having to change and resolve conflicts in package.json.

Consider the following setup: I have master branch (myapp), staging branch (myapp_staging), and feature branches (don't get deployed anywhere). When I merge feature -> staging, it could deploy to myapp_staging, and when I merge staging -> master it could deploy myapp.

Even if you want to manually use this kind of git workflow, right now you need to constantly change the package.json back and forth or remove it with .gitignore altogether, in which case you lose dependency versioning.

So anyway, I propose that jitsu's extensions to package.json be refactored to allow multiple, labeled apps so that we can jitsu deploy <app label> explicitly.

Here's just a rough example package.json I threw together:

{
  "version": "0.0.1-37",
  "apps": {
    "default": {
      "name": "myapp_staging",
      "subdomain": "myapp_staging",
      "private": true
    },
    "production": {
      "name": "myapp_production",
      "subdomain": "myapp"
    }
  },
  "scripts": {
    "start": "node app.js"
  },
  "engines": {
    "node": "0.8.x"
  },
  "dependencies": {
    "express": "3.0.3",
    "jade": "*",
    "less": "*",
    "mongoose": "*"
  }
}
@coen-hyde

This is attempting to tackle the same issue as #292. While I prefer issue's #292 approach I definitely would like some support from nodejitsu built into jitsu to help dealing with multiple environment deploys.

@jwietelmann

Yeah honestly it might be better to get jitsu's config out of package.json altogether. But at a minimum there needs to be some sort of support for deploying to different apps without constantly changing the package.json file. It just makes business sense, too. If Nodejitsu wants enterprise customers, jitsu has to support your typical production/staging/QA/etc. corporate setup in a way that doesn't interfere with that workflow. Making devs change and git add the package.json file in order to deploy to a different endpoint seriously screws with version control and workflow.

@coen-hyde

I've got a build process that builds the correct package.json, which is then git ignored. But it feels rather dirty. And it was more effort to setup than i'd like.

@constantx

+1 (same as #352)

@sbabigian

+1 for multiple app targets. I agree all around. This is really making me consider Heroku instead for enterprise code where we have dev, prod, testing environments each on its own git branch.

@blakmatrix

This is blocked pending business plan milestones, so we're looking at a month or two before we can make such a large closed sourced feature available.

@dscape
nodejitsu member

+1

@indexzero
nodejitsu member

@jwietelmann What does private: true do in app: { default: { ... ??

@Southern

@indexzero I believe he's talking about making it where you can also use this in NPM. It wouldn't be compatible with NPM anyway, because it would ignore the apps field... and that's a change that would have to be made with NPM, and probably not that well received. I also noticed this package.json doesn't have a name field in the root of the object. They're all under apps, so NPM would have to conform to this.. otherwise we get:

 ~ ❱ echo "{}" > package.json && cat package.json
{}
 ~ ❱ npm pack
npm ERR! addLocal Could not install .
npm ERR! Error: No 'name' field
npm ERR!     at validName (/usr/local/lib/node_modules/npm/node_modules/read-package-json/read-json.js:557:40)
npm ERR!     at final (/usr/local/lib/node_modules/npm/node_modules/read-package-json/read-json.js:340:27)
npm ERR!     at /usr/local/lib/node_modules/npm/node_modules/read-package-json/read-json.js:140:33
npm ERR!     at cb (/usr/local/lib/node_modules/npm/node_modules/slide/lib/async-map.js:48:11)
npm ERR!     at /usr/local/lib/node_modules/npm/node_modules/read-package-json/read-json.js:320:48
npm ERR!     at fs.js:117:20
npm ERR!     at /usr/local/lib/node_modules/npm/node_modules/graceful-fs/graceful-fs.js:53:5
npm ERR!     at /usr/local/lib/node_modules/npm/node_modules/graceful-fs/graceful-fs.js:62:5
npm ERR!     at Object.oncomplete (fs.js:297:15)
npm ERR! If you need help, you may report this log at:
npm ERR!     <http://github.com/isaacs/npm/issues>
npm ERR! or email it to:
npm ERR!     <npm-@googlegroups.com>

Ignoring the local error, name is the first thing NPM looks for.

@coen-hyde We could still use the fields in #292 in the apps field. @blakmatrix offered a jitsu.conf file rather than package.json.

I would suggest something like:

{
  "name": "test",
  "version": "0.0.1",
  "description": "",
  "main": "server.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1",
    "start": "node server.js"
  },
  "repository": "git://github.com/random/repo",
  "author": "Your Name",
  "license": "Your License",
  "engines": {
    "node": "0.8.16"
  },
  "dependencies": {
    "our-dep": "0.0.1"
  },
  "environments": {
    "default": "development",
    "development": {
      "name": "test-development",
      "subdomain": "test.dev",
      "version": "0.0.2",
      "dependencies": {
        "our-dep": "0.0.2",
        "our-dep2": "0.0.1"
      },
      "domains": [
        "dev.yourdomain.com"
      ]
    },
    "production": {
      "name": "test-production",
      "subdomain": "test.pro",
      "domains": [
        "yourdomain.com",
        "www.yourdomain.com"
      ]
    }
  }
}

Then make jitsu deploy <environment> and just overwrite any keys in the root that it's given. That uses the same type of layout as #292, and wouldn't cause problems if npm pack is called by itself. Calling jitsu deploy without an environment would use environments.default, so that we always have an environment to append changes to. That way we can leave the original NPM version alone when bumping during deploys.

Using the above package.json (minus the fake dependencies):

 ~/dev/node/projects/test ❱ tar -tf $(npm pack)
npm WARN package.json test@0.0.1 No README.md file found!
package/package.json
package/server.js
package/test-0.0.1.tgz
@coen-hyde

@Southern I like it.

@indexzero
nodejitsu member

There is value here, but everything I've seen it way over complicated. Lets talk about this at a higher level than what the package.json should look like and I believe the changes to the package.json will become obvious.

Users want to run the same application at multiple stages of the development process:
e.g. "I have this app running in production on Nodejitsu, but I want to test the new version before it goes live"

Given this fact the following properties seem totally unnecessary when considering a given application:

  {
    name: "myapp"
    subdomain: "myapp",
    domains: ["www.custom-domain.com"],
    version: "1.0.0",
    dependencies: {
      "our-dep": "0.0.2",
      "our-dep2": "0.0.1"
    },
  }
  1. name, version, subdomain, domains: I don't see any reason why the name, version, subdomains and domains of these non-production copies of an application cannot just be derived from the name of the copy itself e.g.
    • name: myapp-test is the "test" copy of myapp
    • version: 1.0.0-test is the "test" copy of myapp@1.0.0
    • subdomain: test.myapp.jitsu.com is the test copy of myapp.jitsu.com
    • domain: test.www.custom-domain.com is the test copy of the custom domain.
  2. Copies will only be allowed for the latest snapshot of an app. Once you deploy to the main copy, everything is blown away. Why would you want multiple things out of sync?

So the only actual chance to the package.json format I see is somewhere to specify dependencies for a given environment.

@Southern

@indexzero That's only one environment, though. I've had at least one person question how to run 3 environments at once. I want to say they were like.. staging, q&a and production.

@indexzero
nodejitsu member

@Southern simple:

  {
    name: "myapp"
    subdomain: "myapp",
    domains: ["www.custom-domain.com"],
    version: "1.0.0",
    environments: ['qa', 'staging', 'production']
    dependencies: {
      "our-dep": "0.0.2",
      "our-dep2": "0.0.1"
    },
  }
@coen-hyde

@indexzero Previously you mentioned having only copies for the latest snapshot then blowing them away once you deploy the main copy. How would you deal with hosting feature branches then? They'd be blown away each time the main trunk is deployed. How about we do this:

$ jitsu deploy [environment]

Where environment can be whatever you want. There would be no need for listing environments in the package.json. Then use all your predetermined attributes you mentioned above. eg
[environment].myapp.jitsu.com

It would then be up to the user to decide their workflow.

@coen-hyde

Something else to consider is environment variables. Each environment would need it's own environment variable space for things like database credentials.

@arush

+1 for this

@rcotter

+1

@Diggs

+1 desperately need this

@dscape
nodejitsu member

Bumping this up to medium priority now that multi datacenters are around the corner.

I know we are talking some time guys, thanks for being patient!

@thepipster

+1 really need this too

@kyledetella

+1 Important issue.

@julianduque

Not the best solution but this should work as a workaround

https://npmjs.org/package/penv

@neztec

+1

@rosskukulinski

+1.

We're building a new application that's going to have to scale much larger than our previous ones and we're shopping around the different hosting providers. One of the things we must have is separate staging/production systems so we can properly test deploys before they go live.

@simonsmith

Any news on this one? :)

@sambaker

+1
Not having the ability to manage stage and production deployments is a real problem. I'm using penv but I'd really prefer nodejitsu to support multiple environments.

Additionally having to pay more than 3 times as much ($33/month instead of $9/month) to add a staging environment is bizarre.

@lukemartin

Just adding my +1.

Shopping around for different Node hosting providers, and I'm surprised separate environment deployments are not currently supported with jitsu.

@pfrank8

+1

@kwhitley

+1! Any update on this? Last +1 is 2 months old...

@noducks

+1

@Fluidbyte

+1

As of right now I'm just scripting it out using this: Deploy.js and would love a 'native' method.

@saikojosh

+1

Great work @Fluidbyte, very useful!

@chrismatheson

yeah id have to add a +1 👍 for this. just not i have multiple branches of my repo with the separate package.json files and some git rules so as never to merge these files (i have to remember to do it manually) but this is definitely not optimum, i would very much like to have a single branch and use tags to designate staging & prod points

@wamoyo

A little late to the conversation. I made this last year: https://gist.github.com/wamoyo/dca0c3ea938d0ef0e2c8

It's a tiny command line thing, written in node, that pushes a version of the app for staging. In my case I used the name 'stajing' because it wasn't taken.

This script should also copy environment variables, it doesn't do that yet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment