deploy/int ignored ((xyz)) and didn't prompt for missing variable

[drnic]

I placed some ((xyz)) variables in a deployment manifest and expected bosh deploy manifest.yml and bosh int manifest.yml to fail and complain the variables hadn't been provided.

Instead, they appeared in the manifest unmodified.

I'm aware of --var-errs flag; but perhaps it should be the default to fail on missing variables? This would help new users "fail fast" when not providing important variables; rather than booting up a system only to eventually debug and find ((xyz)) is a meaningless/erroneous input value.

[pivotal-jamil-shamy]

@drnic having --var-errs as a default will present a problem when a user wants the director - not the cli - to interpolate the values using config server.
In this case the CLI should act as a pipe passing the variables to the director, and it should not fail.

But to your point, I can see that it can get confusing to the user ( interpolating in the CLI and the director, perhaps both ).

[drnic]

Couple of extra bits of info:* my directors in this case don't have a config-server so will not be interpolating any variables* these variables weren't defined in variable: section for interpolation by config-server In either of these situations is there a reason for cli to not fail fast with missing variables?

[cppforlife]

@drnic the reason why this is not currently happening in your director is because config server integration is not turned on (you can give it a try though through config-server or credhub ops files in bosh-deployment). since director is the end of the line for variable interpolation once we release config server integration officially, director will be always error before starting update procedure if it detects missing variables.

closing this since this is not a cli concern but director but feel free to reopen if plan is not clear.

[drnic]

Sorry I'm confused a bit; perhaps there's an assumption I'm missing: is the cli assuming that all directors have a config- server? Is that currently true for anyone? (credhub-release still isn't publishing non-draft releases; config-server-release doesn't publish releases at all) Second question: if a director did have a config-server - and I pass thru an unresolved variable that config- server isn't give a variables: to generate - then which part of the stack will eventually error? Another way to ask: if there is ((xyz)) and no /variables/name=xyz/ then why wait for the config- server to fail: the cli knows in advance that it's not a variable that can be auto generated? Sorry if I'm misunderstanding ((xyz)) with and without matching variables defn.

[cppforlife]

is the cli assuming that all directors have a config- server

it's for deploy and other director level commands because it will be default in the director at some point. dont want to add flags to cli that we will have to pull out in near future.

Is that currently true for anyone

nope, no one is using it.

Another way to ask: if there is ((xyz)) and no /variables/name=xyz/ then why wait for the config- server to fail: the cli knows in advance that it's not a variable that can be auto generated?

cli doesnt necessarily know all the director level intricacies. but before director starts with a deploy it will error if something is not found/generated. technically variables section doesnt have to define all variables that could be used by the director.

[drnic]

I feel like we're dancing around the user story:

"As a bosh2 deploy users, if any variables are not satisfied I want it the command to fail as fast as possible. I do not want it to render templates with ((missing_var)) inserted into my files - which can take me large amounts of time to diagnose that the issue was a misspelled or missing variable"

I'm sorry if I've approached this issue from the wrong angles. But this issue is painful in a way that I feel it shouldn't be.

[drnic]

Here's an example of something I found in one of my rendered templates from a failing errand:

export broker_uri=http://broker:eqlr4r1y1d05radscfum@((broker-route-uri)):80
export redacted_uri=http://broker:REDACTED@((broker-route-uri)):80

Debugging this is pretty painful - run the errand with --keep-alive and then look thru every file for rendering anomalies like above.

[drnic]

Ooh, bosh2 create-env is lovely like this - if you don't provide a variable it complains fast:

$ bosh2 create-env jumpbox.yml
Deployment manifest: '/Users/drnic/Projects/bosh_deployments/jumpbox-deployment/jumpbox.yml'
Deployment state: '/Users/drnic/Projects/bosh_deployments/jumpbox-deployment/jumpbox-state.json'

Started validating
Failed validating (00:00:00)

Parsing release set manifest '/Users/drnic/Projects/bosh_deployments/jumpbox-deployment/jumpbox.yml':
  Evaluating manifest:
    - Expected to find variables:
        - external_ip
        - internal_cidr
        - internal_gw
        - internal_ip
        - jumpbox_ssh
        - mbus_bootstrap_password

[drnic]

Another scenario where I would have delighted in bosh2 deploy failing fast because it didn't have a --vars-store provided and instead it just printed out the variables into the templates and failed slowly:

$ cat config/docker.key
((docker-tls.private_key))

[drnic]

This struck me again today:

$ bosh deploy mything.yml
...
21:10:22 | Updating instance docker: docker/ac47a498-ec61-4930-b608-55e4e6b49aa8 (0) (00:01:23)
            L Error: 'docker/0 (ac47a498-ec61-4930-b608-55e4e6b49aa8)' is not running after update. Review logs for failed jobs: docker, postgresql

And then realise it didn't ask/prompt/nor fail with missing variables.

It's pretty annoying. Can we please have the bosh2 int --var-errs error behavior please?