Generating variables with config server behavior

[MattSurabian]

Describe the bug
This is maybe less of a bug and more a question that might necessitate a documentation updates (which I'm happy to help contribute). It's my understanding based on this documentation in Credhub about variable interpolation that when deploying a manifest with variables and a variable block for generation in it the Bosh Director won't issue a POST request to the config server until after it issues a GET request to see if the variable already exists.

It looks like instead of that an undocumented mode parameter was added to config-server calls which ensures Credhub doesn't generate a new value if one already exists. Seems like this happened last August based on commits and pivotal tracker issue links.

My guess is that somewhere along the line the logic to issue the initial GET request was also removed from the director. I can't find the commit where it happened, but when logging traffic into my config server implementation I don't see anything coming through.

The behavior of Credhub makes sense, and the parameter is fine too but it doesn't seem to be documented anywhere.

To Reproduce
Steps to reproduce the behavior (example):

1. Deploy a bosh director anywhere setup to use config server
2. Upload any stemcell and cloud-config
3. Deploy any manifest using variables with a generation block. I used the default bosh cloud-config and deployed this basic redis manifest to test
4. Inspect the traffic between the Director and config server and do two deploys

Expected behavior
On the first deploy the Bosh director issues a GET request for variables in the manifest, receives 404's and then issues a POST request to generate the variables as per the documentation. Subsequent deploys issue a GET request, receive a 200 with data and do not issue POST requests as per the documentation.

Logs
Here is some traffic logging from my config server which logs all requests before routing them via a middleware and then logs the same request again when a handler has received it. I have tested this in Bosh Lite, VSphere, and AWS with the same results. Specifically that no initial GET request is ever sent. The logs below are from config server receiving traffic from a Bosh Lite director deployed using bosh-deployment manifests at hash eea8efcbce4da16d19584ebe31994218a1460531 (latest master).

First deploy:

{"level":"info","msg":"POST request to /v1/data","time":"2019-01-25T11:12:13-05:00"}
{"level":"info","msg":"POST request to PostHandler /v1/data {\"name\":\"/BLite/redis/redis_password\",\"type\":\"password\",\"parameters\":{},\"mode\":\"no-overwrite\"}","time":"2019-01-25T11:12:13-05:00"}
{"level":"info","msg":"PostHandler generating password /BLite/redis/redis_password","time":"2019-01-25T11:12:13-05:00"}
{"level":"info","msg":"GET request to /v1/data/eyJuYW1lIjoiL0JMaXRlL3JlZGlzL3JlZGlzX3Bhc3N3b3JkIiwidmVyc2lvbiI6MX0","time":"2019-01-25T11:12:13-05:00"}
{"level":"info","msg":"GET request to GetById handler /v1/data/eyJuYW1lIjoiL0JMaXRlL3JlZGlzL3JlZGlzX3Bhc3N3b3JkIiwidmVyc2lvbiI6MX0","time":"2019-01-25T11:12:13-05:00"}

Second deploy:

{"level":"info","msg":"POST request to /v1/data","time":"2019-01-25T11:45:20-05:00"}
{"level":"info","msg":"POST request to PostHandler /v1/data {\"name\":\"/BLite/redis/redis_password\",\"type\":\"password\",\"parameters\":{},\"mode\":\"no-overwrite\"}","time":"2019-01-25T11:45:20-05:00"}
{"level":"info","msg":"PostHandler generating password /BLite/redis/redis_password","time":"2019-01-25T11:45:20-05:00"}
{"level":"info","msg":"GET request to /v1/data/eyJuYW1lIjoiL0JMaXRlL3JlZGlzL3JlZGlzX3Bhc3N3b3JkIiwidmVyc2lvbiI6Mn0","time":"2019-01-25T11:45:20-05:00"}
{"level":"info","msg":"GET request to GetById handler /v1/data/eyJuYW1lIjoiL0JMaXRlL3JlZGlzL3JlZGlzX3Bhc3N3b3JkIiwidmVyc2lvbiI6Mn0","time":"2019-01-25T11:45:20-05:00"}

Versions (please complete the following information):

• Infrastructure: I've tested in AWS, VSphere, and Bosh Lite
• BOSH version I've tested 268.5 and 268.4
• BOSH CLI version 5.4.0

Deployment info:
redis manifest I used

Additional context
Just to note again I don't think it's a problem expecting config server to do it and letting the Bosh Director just always issue POST requests without a GET. I think the documentation should just reflect that for Credhub, Bosh, and the config server spec.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/163486586

The labels on this github issue will be updated when the story is started.

[jontours]

I've discovered the same behavior while migrating CF secrets to CredHub.

[jontours]

@MattSurabian did you find a way to ensure this mode parameter gets set?

[MattSurabian]

It seems like things will work as expected provided you have a latest credhub and director version. The latest bosh director always sends the parameter and the latest credhub respects it. If you run latest credhub with an older bosh director version that doesn't send the parameter things won't be so ideal.

I probably should make a PR to update the docs but just haven't found the time yet.

[MattSurabian]

Though the more I've thought about it the more I think it is the director's job to issue the GET first.

[kitsirota]

+1 it would be great to get an update on this

[siennathesane]

//cc @mfine30

[mrosecrance]

We colocate and usually co-deploy credhub (or a cred service) and the bosh director. I'm curious as to the use-case of when you would have a new director and old credhub.

As far as I can tell the director has always sent only a Post request to the config-server(from 2 years ago at least). The config-server, sends a get and then a post to whatever credential service you are using. https://github.com/cloudfoundry/config-server/blob/3dd2f57d7f595fc1cf90c444388e14ac63c2be76/server/request_handler.go#L138. We also make the assumption that the bosh director should be the only thing mucking around with the credential values and cache the responses, thus avoiding extra get calls.

Here's a little color behind the mode feature: https://www.pivotaltracker.com/story/show/158947047. In relation to the docs, we don't own the repo cloudfoundry-incubator/credhub but at the moment, I'm not convinced they're not still accurate and just mixing together the concept of director and config-server as one box called director.

[h4xnoodle]

As @mrosecrance stated above, this is expected behaviour since bosh started taking advantage of the three modes credhub variables can be created.

The director sends a POST for all variables, since the update_mode in the manifest may mean credhub could generate a new value. Since the director doesn't keep track of modes, we rely on credhub to respect the mode. We issue a POST for all variables since we don't keep track of which modes may have changed -- credhub remains the authority on value generation.

Closing this now, as it's intended behaviour. There is an issue opened for updating the credhub docs: cloudfoundry/credhub#57