Agent (facter?) doesn't find credentials in WinCred

[jodyhuntatx]

Initial run on new Winserver 2008 R2 VM - new host identity correctly created w/ host factory:

C:\Users\Administrator>puppet agent -t -v
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Enumerate credentials in WinCred failed. Error code: 1168
Info: Caching catalog for win-2re6mq9ddem.localdomain
Info: Applying configuration version '1565020354'
Notice: /Stage[main]/Conjur::Identity::Wincred/Credential[conjur-master:30443]/ensure: created (corrective)
Notice: Applied catalog in 0.14 seconds

Second run - identity not found, seems to be updating the existing credential:

C:\Users\Administrator>puppet agent -t -v
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Conjur identity not found on system
Info: Caching catalog for win-2re6mq9ddem.localdomain
Info: Applying configuration version '1565020487'
Notice: /Stage[main]/Conjur::Identity::Wincred/Credential[conjur-master:30443]/value: changed [redacted] to [redacted]
Notice: Applied catalog in 0.03 seconds

Test code in: https://github.com/jodyhuntatx/dap-demo-env/tree/master/PUPPET_demo

[jodyhuntatx]

Running above w/ PuppetMaster & Agent v6.7.2

[jodyhuntatx]

For comparison - a run against the same PuppetMaster instance from a Linux node:

~/Conjur/dap-demo-env/PUPPET_demo >> docker-compose exec prod-webapp puppet agent -t
Info: Downloaded certificate for ca from puppet
Info: Downloaded certificate revocation list for ca from puppet
Info: Creating a new RSA SSL key for prod-webapp.cluster.local
Info: csr_attributes file loading from /etc/puppetlabs/puppet/csr_attributes.yaml
Info: Creating a new SSL certificate request for prod-webapp.cluster.local
Info: Certificate Request fingerprint (SHA256): 57:AA:FD:FD:10:BC:D3:69:4B:87:FB:0C:F0:73:EB:10:08:D3:7D:12:CE:96:32:D8:70:FE:69:6F:46:8E:C3:1D
Info: Downloaded certificate for prod-webapp.cluster.local from puppet
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Notice: /File[/opt/puppetlabs/puppet/cache/lib/conjur]/ensure: created
Notice: /File[/opt/puppetlabs/puppet/cache/lib/conjur/puppet_module]/ensure: created
Notice: /File[/opt/puppetlabs/puppet/cache/lib/conjur/puppet_module/config.rb]/ensure: defined content as '{md5}b3780ec739325ca060b5c74e76fe7d55'
Notice: /File[/opt/puppetlabs/puppet/cache/lib/conjur/puppet_module/identity.rb]/ensure: defined content as '{md5}6e6a3f337b9fba99b0e422b2a0769c11'
Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter]/ensure: created
Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/conjur.rb]/ensure: defined content as '{md5}41be6fd2bc522f5be28eaf197eeaf86d'
Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet]/ensure: created
Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet/functions]/ensure: created
Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet/functions/conjur]/ensure: created
Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet/functions/conjur/client.rb]/ensure: defined content as '{md5}6305c573f56a6dd8810aabe636a194df'
Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet/functions/conjur/config_yml.rb]/ensure: defined content as '{md5}e58bbe1b6c25bed7f7a9bf96c7ffeda3'
Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet/functions/conjur/decrypt.rb]/ensure: defined content as '{md5}db122f2ca606cb8cf187ce535d533fb3'
Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet/functions/conjur/manufacture_host.rb]/ensure: defined content as '{md5}9e460f636128e0d7d45565d8180836da'
Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet/functions/conjur/netrc.rb]/ensure: defined content as '{md5}4b77091862ff01f2a3016c388c736b35'
Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet/functions/conjur/secret.rb]/ensure: defined content as '{md5}4d8a35b874b7214d7a3a2a2aedcbae01'
Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet/functions/conjur/token.rb]/ensure: defined content as '{md5}1af71b2ca11a01ec0ed25d18144e628f'
Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet/provider]/ensure: created
Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet/provider/credential]/ensure: created
Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet/provider/credential/wincred.rb]/ensure: defined content as '{md5}c7a85590f9f87226d74a1108e3ac3771'
Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet/provider/registry_key]/ensure: created
Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet/provider/registry_key/registry.rb]/ensure: defined content as '{md5}57d97e6cf5e20370af6caff63f575b6a'
Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet/provider/registry_value]/ensure: created
Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet/provider/registry_value/registry.rb]/ensure: defined content as '{md5}70f825ee0a196ea0e3030bafd979aec6'
Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet/type]/ensure: created
Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet/type/credential.rb]/ensure: defined content as '{md5}683259dee013d88421e096fca4585847'
Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet/type/registry_key.rb]/ensure: defined content as '{md5}833cac29fc9cbc761dfb1616d3c7cc8b'
Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet/type/registry_value.rb]/ensure: defined content as '{md5}9e7cafee6cc3a514e05b9a34c2e43292'
Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x]/ensure: created
Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs]/ensure: created
Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet_x/puppetlabs/registry.rb]/ensure: defined content as '{md5}1132a6f66c4e7d549ac90f9f58bd73e2'
Notice: /File[/opt/puppetlabs/puppet/cache/lib/wincred]/ensure: created
Notice: /File[/opt/puppetlabs/puppet/cache/lib/wincred/conversion.rb]/ensure: defined content as '{md5}6fe6013230d9459d2612927a36cd3d0b'
Notice: /File[/opt/puppetlabs/puppet/cache/lib/wincred/native.rb]/ensure: defined content as '{md5}374523d071d50d332c4283bfb52318d4'
Notice: /File[/opt/puppetlabs/puppet/cache/lib/wincred/wincred.rb]/ensure: defined content as '{md5}91cc3df0325a86d1915d934833764418'
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for prod-webapp.cluster.local
Info: Applying configuration version '1565021519'
Notice: /Stage[main]/Conjur::Config::Files/File[/etc/conjur.pem]/ensure: defined content as '{md5}13f76ab141e9bd445651273f41e6a63c'
Notice: /Stage[main]/Conjur::Config::Files/File[/etc/conjur.conf]/ensure: defined content as '{md5}28a69a4b6eff0e6d04dd14d43aa1594b'
Notice: /Stage[main]/Conjur::Identity::Files/File[/etc/conjur.identity]/ensure: changed [redacted] to [redacted]
Notice: *** Secret key - encrypted:  Sensitive [value redacted]
Notice: /Stage[main]/Main/Node[__node_regexp__prod-webapp.]/Notify[*** Secret key - encrypted:  Sensitive [value redacted] ]/message: defined 'message' as '*** Secret key - encrypted:  Sensitive [value redacted] '
Notice: *** Secret key - unencrypted:  Se(re1Fr0mConjur
Notice: /Stage[main]/Main/Node[__node_regexp__prod-webapp.]/Notify[*** Secret key - unencrypted:  Se(re1Fr0mConjur ]/message: defined 'message' as '*** Secret key - unencrypted:  Se(re1Fr0mConjur '
Notice: *** DB Password - encrypted:  Sensitive [value redacted]
Notice: /Stage[main]/Main/Node[__node_regexp__prod-webapp.]/Notify[*** DB Password - encrypted:  Sensitive [value redacted] ]/message: defined 'message' as '*** DB Password - encrypted:  Sensitive [value redacted] '
Notice: *** DB Password - unencrypted:  white rabbit
Notice: /Stage[main]/Main/Node[__node_regexp__prod-webapp.]/Notify[*** DB Password - unencrypted:  white rabbit ]/message: defined 'message' as '*** DB Password - unencrypted:  white rabbit '
Notice: ******* Writing secret key to file /etc/mysecretkey ******
Notice: /Stage[main]/Main/Node[__node_regexp__prod-webapp.]/Notify[******* Writing secret key to file /etc/mysecretkey ******]/message: defined 'message' as '******* Writing secret key to file /etc/mysecretkey ******'
Notice: /Stage[main]/Main/Node[__node_regexp__prod-webapp.]/File[/etc/mysecretkey]/ensure: changed [redacted] to [redacted]
Info: Creating state file /opt/puppetlabs/puppet/cache/state/state.yaml
Notice: Applied catalog in 0.04 seconds

value stored in /etc/mysecretkey:
Se(re1Fr0mConjur

[jodyhuntatx]

Because of this:

Notice: /Stage[main]/Conjur::Identity::Wincred/Credential[conjur-master:30443]/value: changed [redacted] to [redacted]

I suspected the API key is getting rotated on every run. The following workflow seems to confirm that.

1. On windows host win-2re6mq9ddem - create identity :

<puppet run creates creds in WinCred>

2. In Conjur CLI container - rotate host API key & login as host:

# conjur host rotate_api_key -h win-2re6mq9ddem
3y5zx0f36t1sf3dj8vpc2qmsdbz27mthwexkrdar2p0sb051pr68nw

# conjur authn login -u host/win-2re6mq9ddem -p 3y5zx0f36t1sf3dj8vpc2qmsdbz27mthwexkrdar2p0sb051pr68nw
Logged in

# conjur authn whoami
{"account":"dev","username":"host/win-2re6mq9ddem"}

# conjur authn logout
Logged out

3. On Windows host - update API key and do another Puppet run:

<API key pasted into WinCred>
<puppet run pulls secrets>

4. In Conjur CLI - try to login as host using same API key as before:

# conjur authn login -u host/win-2re6mq9ddem -p 3y5zx0f36t1sf3dj8vpc2qmsdbz27mthwexkrdar2p0sb051pr68nw
Unable to authenticate with Conjur. Please check your credentials.

[jodyhuntatx]

Net-net is while the host identity is created and secrets are pulled correctly, the identity is effectively being recreated on every run, which w/ HF token simply rotates the API key on the second-nth run. At scale this will cause concurrency conflict at the Master, and Puppet nodes will be unable to retrieve secrets at all once the HF token expires.

This needs to be addressed before customers can use the Puppet Windows module in production.

[doodlesbykumbi]

This bit of discussion from Slack gives some insight into this bug

---

re:

suspected current:

• HFT expires
• host that was created with HFT can’t have puppet reapply manifest

desired:

• HFT expires
• host that was created with HFT can have the manifests reapplied (since identity should not be tied to the HFT expiration)

I tried the flow and on first glance it seems as though suspected current IS current, but it’s not. It only seems that way because there’s a bug in how machine identity is retrieved/persisted into wincreds.

Before digging into the wincred issue it’s worth mentioning that the logic responsible for machine identity taking precedence over the host factory token can be found here https://github.com/cyberark/conjur-puppet/blob/conjur-puppet/manifests/init.pp#L20. It shows that host factory tokens are the last to be considered.

The reason why suspected current SEEMS like current right now when you run through the flow with an expired HFT after the first successful run is because there’s a mismatch between persistence and retrieval. Suppose you have Appliance URL http://conjur.host:8080

1. Persistence of machine identity. The target name is taken from the appliance URL. It is transformed and stored using this logic found at https://github.com/cyberark/conjur-puppet/blob/conjur-puppet/manifests/identity/wincred.pp#L7. The target name generated for our example in windows credentials will be conjur.host:8080 . The regex strips the appliance URL of the scheme and the anything beyond the port. The regex is actually quite brittle. It’d be better to parse the appliance URL as a URI perhaps followed by concatenating the host and port.
2. Retrieval of machine identity. The code for selecting the right credentials in wincreds based on target name https://github.com/cyberark/conjur-puppet/blob/conjur-puppet/lib/conjur/puppet_module/identity.rb#L45. Note that it looks through all the credentials in wincreds then selects the first one that matches either of the following conditions. (1) Target name prefix-matches the appliance URL (e.g. conjur.host:8080 (target name) will match with conjur (appliance URL)). This likely won’t ever be true because appliance URLs are parsed as URLs and so come with a scheme e.g. http://… , while target names are stripped of their schemes as shown in the persistence section above because of the regex. (2) Target name is equal to the host component of the parsed appliance URL. For our example we’d be comparing the host part of the appliance URL conjur.host with the target name conjur.host:8080 . This can only ever be true when the appliance URL specifies no port.

You’ll see from above that for appliance URLs (with ports) the machine identity persisted (BY the conjur module from Puppet Server) can never be retrieved. This actually means that any subsequent runs beyond the initial one where machine identity is established will fail. Pre-provisioned machine identity however might work for multiple runs if you happen to have used a manually set target name that can be retrieved from the appliance URL (e.g. respectively http://conjur.host:8080 and http://conjur.host:8080 … this is actually what our README suggests).

NOTE that persistence (potentially overwriting) is only carried out by the conjur module when the server has the authn_api_key as a fact. This happens when

1. Manifest on server is given the literal values of the authn_api_key credentials
2. Manifest on server is given the host factory token. For this it will create a host and get the authn_api_key credential

Once persistence occurs authn_token takes precedence and does not invoke persistence (e.g. https://github.com/cyberark/conjur-puppet/blob/conjur-puppet/manifests/identity/wincred.pp#L4 because $conjur::api_key can only be set on the server when authn_token is not https://github.com/cyberark/conjur-puppet/blob/conjur-puppet/manifests/init.pp#L18). authn_token
is generated on the agent and sent to the server.

To reiterate, the bug here is that whenever persistence of machine identity is carried out by the conjur module it is highly likely that there will be a mismatch of the wincred target name used to store machine identity and the rules used to retrieve the machine identity, there are certain values of appliance URLs where this isn’t an issue. Where the mismatch IS an issue it results in machine identity not being retrievable on subsequent puppet runs. This results in an auth error where the agent says it can’t find machine identity. In the case where the manifest on the server has a host factory token then the precedence (https://github.com/cyberark/conjur-puppet/blob/conjur-puppet/manifests/init.pp#L20) of machine identity is lost and the logic falls back to using the host factory token. THIS is what makes it seems as though the suspected current IS current.

To fix this we simply need to reconcile how machine identity is stored and retrieved. For persistence this must be reflected both in the conjur module and our recommendations for the target name to use when preprovisioning machine identity.

[izgeri]

@jodyhuntatx we believe we fixed this in #124. Do you still have an environment that you could try our local version of the plugin with to validate that you are no longer impacted by this issue? Please let us know if you'd like to try to see if your issue is resolved, either now or in a week or so once we have a new tag with the fix. If not, we'll close this issue - please report if you see any kind of problem like this again, we'll be glad to get it fixed up!