Skip to content

Commit

Permalink
feat: Integrate cluster-info-client, automatically fetching and stori…
Browse files Browse the repository at this point in the history
…ng cluster info (#3)

* wip

* add logic to infer name/url

* wip: factor reconciling args/defaults into a module

* wip: merge defaults generate certain vals

* workign tests!

* lint

* refactor output data

* add store function

* 100% coverage

* fixes from running tests against a real api

* lint

* properly pass in credentials, rename subdomain -> domain

* update kubeform

* strip out fount, also scrub object credentials when saving

* fix: handle non-default common settings

* save progress between kubeform and hikaru

* make sure credentials make it to the right places in the right formats

* update hikaru

* docs

* add seed.js, with example object

* moar docs

* add output option
  • Loading branch information
aearly committed Apr 29, 2019
1 parent 4b70b7a commit 2df3712
Show file tree
Hide file tree
Showing 11 changed files with 1,242 additions and 81 deletions.
4 changes: 3 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,6 @@ node_modules/
package-lock.json
gs-auth.json
gs-test-meta.js
cluster-*.json
*cluster*.json
gke-common.js
.env
32 changes: 28 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ It works well in environments where you might want ephemeral clusters, clusters

### What It's **Not** For

`fabrik8` is not a CD solution (at least not presently). It is not meant to be run continuously against the same target (it cannot guarantee 100% idempotence). Running `fabrik8` multiple times _may_ yield unexpected results. For CD solutions, see [`hikaru`](https://github.com/npm-wharf/hikaru).
`fabrik8` is not a CD solution (at least not presently). It is not meant to be run continuously against the same target (it cannot guarantee 100% idempotence, but makes a best effort to be). Running `fabrik8` multiple times _may_ yield unexpected results. For CD solutions, see [`hikaru`](https://github.com/npm-wharf/hikaru).

## Approach

Expand All @@ -35,9 +35,9 @@ As noted in `kubeform`, many of the environment variables are cloud provider spe

## API

### `initiate(cluster, specification, data|onData)`
### `initialize(cluster, specification, data|onData)`

The `initiate` call requires three arguments and returns a promise.
The `initialize` call requires three arguments and returns a promise.

#### `cluster`

Expand Down Expand Up @@ -83,8 +83,32 @@ It is recommended that sensitive data (like the Kubernetes admin password) is st

A CLI is also provided for `fabrik8` that allows you to invoke the API from the command line:

### `fabrik8 create [--name name] [--url url] --spec ./path/to/spec`

Creates a full cluster, reading defaults and existing configuration securely from centralized cluster-info. The only options that are required are configuration for cluster-info, a name or cluster url, and the path to a McGonagall specification. If re-running, only a name is required -- options will be re-read from cluster-info.

* `--url`, `-u` the url of the cluster you wish to create, e.g. `mycluster.npme.io`
* `--name`, `-n` the name of the cluster. Can be inferred from the url
* `--domain` the domain of the cluster. Can be inferred from the url. Defaults to whatever is specified in the cluster-info defaults, if only a name is provided.
* `--projectId` the name of the gke project to use. Can be inferred from the cluster name
* `--environment` the environment of the cluster, e.g. development, production
* `--specification`, `-m`, `--spec` the path or URL to the mcgonagall specification
* `--verbose` output verbose logging (status check output for hikaru)
* `--redisUrl` the url of the redis containing cluster information. Can also be set through the `REDIS_URL` environment variable
* `--vaultHost` the host of the vault server containing sensitive cluster information, auth data, and defaults. Can also be set through the `VAULT_HOST` environment variable
* `--vaultToken` an auth token for the vault server. Can also be set through the `VAULT_TOKEN` environment variable
* `--provider` the cloud provider to use, defaults to `KUBE_SERVICE` environment variable or `GKE`
* `--output`, `-o` file to write cluster-info to, for debugging

Values from the defaults can also be overridden as command line args, by prefixing the key with `--arg-`, e.g. `--arg-cluster.worker.memory 26GB`, or `--arg-common.zones eu-central1-a`. Look at the cluster-info defaults for a list of values that can be overridden.

Command line arguments take precedence over saved cluster-info, which take precedence over default cluster-info. Cluster info-will be saved everytime you run `fabrik8`, so re-running `fabrik8 create` can be used to change values.


### `fabrik8 init ./path/to/config -a ./path/to/authFile -s ./path/to/spec -f ./path/to/data -p gke`

**DEPRECATED**

Similar to a blend of CLIs from its component libraries, `fabrik8` requires the following arguments:

* `./path/to/config`: configuration to base cluster provisioning on
Expand All @@ -107,4 +131,4 @@ Similar to a blend of CLIs from its component libraries, `fabrik8` requires the
[downloads-image]: https://img.shields.io/npm/dm/@npm-wharf/fabrik8.svg?style=flat
[downloads-url]: https://www.npmjs.com/package/@npm-wharf/fabrik8
[dependencies-image]: https://img.shields.io/david/npm-wharf/fabrik8.svg?style=flat
[dependencies-url]: https://david-dm.org/npm-wharf/fabrik8
[dependencies-url]: https://david-dm.org/npm-wharf/fabrik8
2 changes: 2 additions & 0 deletions bin/fabrik8.js
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

const fabricator = require('../src/index')
const chalk = require('chalk')
require('dotenv').config()

const levelColors = {
debug: 'gray',
Expand All @@ -21,6 +22,7 @@ const debugOut = {
require('yargs') // eslint-disable-line no-unused-expressions
.usage('$0 <command> [options]')
.command(require('../src/commands/init')(fabricator, debugOut))
.command(require('../src/commands/create')(fabricator, debugOut))
.demandCommand(1, 'A command is required.')
.help()
.version()
Expand Down
10 changes: 6 additions & 4 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -43,15 +43,17 @@
"standard-version": "^4.4.0"
},
"dependencies": {
"@npm-wharf/hikaru": "^1.14.0",
"@npm-wharf/kubeform": "^1.3.1",
"@npm-wharf/cluster-info-client": "^1.4.2",
"@npm-wharf/hikaru": "^1.15.0",
"@npm-wharf/kubeform": "^1.4.3",
"bluebird": "^3.5.2",
"chalk": "^2.4.1",
"dotenv": "^7.0.0",
"fauxdash": "^1.4.0",
"fount": "^1.1.4",
"inquirer": "^6.2.0",
"js-yaml": "^3.12.0",
"toml-j0.4": "^1.1.1"
"toml-j0.4": "^1.1.1",
"uuid": "^3.3.2"
},
"standard": {
"env": [
Expand Down
128 changes: 128 additions & 0 deletions seed.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,128 @@
const createClient = require('@npm-wharf/cluster-info-client')
const createVault = require('node-vault')
const fs = require('fs')
require('dotenv').config()

async function main () {
const redisUrl = process.env.REDIS_URL || 'redis://localhost:6379'
const vaultHost = process.env.VAULT_HOST || 'https://your.vault.server:8200'
const vaultToken = process.env.VAULT_TOKEN || 's.myVaultToken'

const client = createClient({
redisUrl,
vaultHost,
vaultToken,
vaultPrefix: 'kv/'
})

const vault = createVault({
endpoint: vaultHost,
token: vaultToken
})

const resourceManagerJson = JSON.parse(fs.readFileSync(`${process.env.HOME}/resource-manager.json`))
const wombotProdJson = JSON.parse(fs.readFileSync(`${process.env.HOME}/wombot-prod.json`))
await client.addServiceAccount(resourceManagerJson)
await client.addServiceAccount(wombotProdJson)

console.log('service accounts:\n')
console.log((await client.listServiceAccounts()).join('\n'))

const exampleGkeCommonData = {
allowedDomains: ['my-company.net'],
projectPrefix: 'my-project-',

// used by both kubeform and hikaru
common: {
billingAccount: '123456-123456-123456',
organizationId: '234523452345',
user: 'admin',
version: '1.10.12-gke.14',
basicAuth: true,
zones: ['us-central1-a']
},

applicationCredentials: 'resource-manager-svc@my-project.iam.gserviceaccount.com',

serviceAccounts: {
cluster_sa: 'some-service-account@my-project.iam.gserviceaccount.com',
applicationCredentials: 'resource-manager-svc@my-project.iam.gserviceaccount.com'
},

// used by hikaru
tokens: {
awsAccount: 'AASDGHJKASGDJKASGDJ',
awsSecret: 'asdfghjkasdfgjkasdfhjasdjkhfg',
awsZone: 'my-company.net',
bucketACL: 'private',
dashboardAdmin: 'admin',
nginx_upstream1: 'frontdoor.npm.svc.cluster.local:5000',
nginx_upstream2: 'rewrite.npm.svc.cluster.local:5001',
cluster_sa: 'some-service-account@my-project.iam.gserviceaccount.com'
},

// used by kubeform
cluster: {
worker: {
cores: 2,
memory: '13GB',
count: 3,
min: 3,
max: 6,
maxPerInstance: 4,
reserved: true,
storage: {
ephemeral: '0GB',
persistent: '160GB'
},
network: {},
maintenanceWindow: '08:00'
},

flags: {
alphaFeatures: false,
authedNetworksOnly: false,
autoRepair: true,
autoScale: false,
autoUpgrade: false,
basicAuth: true,
clientCert: true,
includeDashboard: false,
legacyAuthorization: false,
loadBalanceHTTP: true,
maintenanceWindow: '08:00:00Z',
networkPolicy: true,
privateCluster: false,
serviceLogging: false,
serviceMonitoring: false
},
manager: {
distributed: false,
network: {}
},
managers: 1
}
}

try {
var gkeCommonData = require('./gke-common')
} catch (e) {}

await vault.write('kv/data/clusters/common/gke', {
data: {
value: JSON.stringify(gkeCommonData || exampleGkeCommonData, null, 2)
}
})

console.log('\nGKE defaults:\n')
console.log(await client.getCommon())

client.close()
}

main()
.then(() => console.log('\nDone.'))
.catch(err => {
console.error(err.stack)
process.exit(1)
})

0 comments on commit 2df3712

Please sign in to comment.