fixes #25065 - use puppetca http api

[timogoebel]

This is required for Puppet 6 support as the legacy cli commands aren't available anymore.

[timogoebel]

cc: @quba42

[timogoebel]

Yesterday I added a lot of tests and refactored the implementation to providers. I also added a settings migration.
One open question is: Do we want to use the setting migration to copy the puppetmaster url and certificate settings from the puppet module to the smart proxy http module? @ekohl: Any thoughs?

[timogoebel]

For reference: This is the (current) documentation of the API endpoint. https://puppet.com/docs/puppet/6.0/http_api/http_certificate_status.html

[trenta]

Would it be possible to get an indication of when this fix will be released?

[timogoebel]

@trenta: Would you be able to assist with testing this?

[trenta]

I'd like to but probably a bit beyond my current skill set. I could do some testing if that helps?

[timogoebel]

@trenta: You should be able to apply this patch to an rpm deployment by something like this:

yum install patch -y
wget -O /tmp/615.patch https://github.com/theforeman/smart-proxy/pull/615.patch
cd /usr/share/foreman-proxy
patch -p1 < /tmp/615.patch
systemctl restart foreman-proxy.service

Let us know if this works for you.

[trenta]

@timogoebel Thanks for that. I've tried applying the patch and receive the following:

patching file 2018101600000_migrate_puppetca_puppet_cert_settings.rb
can't find file to patch at input line 123
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|diff --git a/config/settings.d/puppetca.yml.example b/config/settings.d/puppetca.yml.example
|index 0d2ebf34..3c0fb348 100644
|--- a/config/settings.d/puppetca.yml.example
|+++ b/config/settings.d/puppetca.yml.example
--------------------------
File to patch:

[timogoebel]

@trenta: Just ignore the files it cannot find. The example files do not exist on a production deployment. Tests will be missing as well. You can skip them, just press enter a couple of times. :-D

[trenta]

@timogoebel After patching and restarting, I refreshed the smart-proxy features and got the following error.

Disabling all modules in the group ['puppetca'] due to a failure in one of them: Parameter ':puppet_version' is expected to have a non-empty value

The relevant line in /etc/foreman-proxy/settings.d/puppet.yml is:
:puppet_version: 6.0.3
So it is there.

I’m using the puppet module https://forge.puppet.com/theforeman/foreman_proxy to manage this.

And for more info I’m running Ubuntu 18.04

[timogoebel]

@trenta: You need to also set this in /etc/foreman-proxy/settings.d/settings.d/puppetca.yml. Please take a look at the code of this PR for how the setting should look.

[trenta]

@timogoebel I got it working thanks. One thing I noticed and I'm not sure how much of a problem it would be but when I went to rebuild a machine that didn't have a certificate in the ca it failed and wouldn't go into boot mode. Deleting the machine had the same error. The message was

ERROR -- : Attempt to remove nonexistent client autosign for <fqdn>
INFO -- : 10.85.0.10 - - [01/Nov/2018:09:59:29 +1100] "DELETE /puppet/ca/autosign/<fqdn> HTTP/1.1" 404 69 0.0008
Failed to remove certificate(s) for <fqdn>: Failed to set Puppet CA certificate_status v1 API: 404 Invalid certificate subject.
DELETE /puppet/ca/<fqdn> HTTP/1.1" 406 140 0.0455

[trenta]

@timogoebel That certificate deletion issue if there isn't one is a problem. If you initiate a build, make a change, cancel build and try to build again it tries to remove the certificate that isn't there and because it fails it doesn't go into build mode.

[timogoebel]

@trenta: Thanks for testing this. This is a huge help.

If I see it correctly, the issue is that the API returns "404" when we clean the certificate. Our code should probably simply accept this instead of throwing an error.

[timogoebel]

@trenta: Pushed a version that contains a fix for the issue you have found. Do you mind to retest?

You can revert the first version using patch and then apply the new version.

[trenta]

@timogoebel Yep that patch fixes that problem. Nice work!
After my testing I'd my opinion on your question "migration to copy the puppetmaster url and certificate settings from the puppet module to the smart proxy http module" would be to migrate the settings. Because for me to get it working I just ran cp /etc/foreman-proxy/settings.d/puppet_proxy_legacy.yml /etc/foreman-proxy/settings.d/puppetca_http_api.yml

Thanks for your work.

[ekohl]

@timogoebel I have most of the Puppet 6 support in the installer (critically theforeman-puppet) so I intend to test this soon and work on the installer configuration aspect.

[juliantodt]

@timogoebel When trying to test this I ran into an issue from puppet Failed to list certificates: Failed to query Puppet CA search v1 API: 500 Internal Server Error: java.lang.IllegalArgumentException: The PEM stream contains more than one object when having an intermediate CA (which puppet recommends in their doc).
This is not an issue for non /puppet-ca/ API endpoints for puppet (e.g. getting environments via a smartproxy using an intermediate CA works) and not an issue when only using a Root-CA without an intermediate CA.
I tested this under the latest puppet 5 - Their documentation said they changed some things to puppet 6, so it might work there but I wasn't able to test.
This is definitely something to keep in mind when setting this as the default for puppet 5.

[timogoebel]

@juliantodt: That's very interesting. Thanks for finding this. Do you have a recommendation on how to fix this?

[juliantodt]

@timogoebel I would consider this a bug in the puppet-ca-API of the puppetserver - so no fixing this on the smartproxy's side. I would test this with puppet 6. If it works there then set the new provider as default only for puppet_version >= 6 and keep the old provider for the older puppet versions. If it also doesn't work on puppet 6, open an issue with puppet. I don't think there is a lot we can do about that.
Fortunately, most people won't have an intermediated CA, because the default for puppet to create on startup is without it. Only if you manually do that - or use puppetserver ca setup to create your CA then you have that intermediate CA.

[ekohl]

Could you change this 6.0 per @juliantodt's suggestion? I'd comfortable merging this after that.

[timogoebel]

@ekohl: Done.

[trenta]

When the patch is at this stage of testing is it still the same method to apply it?

wget -O /tmp/615.patch https://github.com/theforeman/smart-proxy/pull/615.patch
cd /usr/share/foreman-proxy/
patch -p1 < /tmp/615.patch
answer yes to all
restart service

I don't seem to be able to get it working again after the latest upgrade

[trenta]

Sorry missed the step
cp /etc/foreman-proxy/settings.d/puppet_proxy_legacy.yml /etc/foreman-proxy/settings.d/puppetca_http_api.yml

[tbrisker]

1.21 branching is behind us so i'm fine with merging this now (though i did not test it). @ekohl i believe your concerns have been addressed?

[ekohl]

It's on my agenda to test this

[ekohl]

Perhaps this needs quotes? Otherwise it'll be a float and 4.10 will be 4.1.

[ekohl]

Comparison using Gem::Version is probably better so we can deal with 4.10.1 as well

[ekohl]

Looks like they will be. Should we already start reading them anyway so we'll support it when it is? Should return nil if it's not found either way.

[timogoebel]

Let's wait until a version is released that contains this data.

[ekohl]

It was just merged so I think we can start assuming the field names.

[timogoebel]

I won't add any new features to this PR. Let's merge this first.
We have to make sure our code can handle the old and new versions, so we need appropriate tests etc. I'd prefer to do this later. Keep in mind this PR was opened in October, we have user testing, unit tests, code reviews. What is keeping us from merging?

[ekohl]

Could you copy this test and add 4.10.1 as well? Our installer uses the puppetversion fact which is a full version.

[timogoebel]

@ekohl: Thanks for reviewing. Addressed your comments. The lines you commented about are actually copy-and-paste-code, we also have it in other places.

[ekohl]

I've tried this using forklift:

centos7-foreman-puppet-6:
  box: centos7-foreman-nightly
  ansible:
    variables:
      # foreman-proxy with PR 615
      koji_task_ids:
        - 172818
      foreman_installer_module_prs:
        - theforeman/foreman_proxy/488
      # Needed for puppet-systemd: https://github.com/camptocamp/puppet-systemd/pull/97
      foreman_installer_options:
        - "--skip-puppet-version-check"
      puppet_repositories_version: 6

This applies two few PRs:

• Add Puppet 6 repositories forklift#922
• Add puppet http api support puppet-foreman_proxy#488

Initial observation are UI bugs on the smart proxy detail page:

theforeman/foreman#6494 fixes the iso8601 issue. Then I see the following:

If you select all instead of valid or pending then I do see the certificate. It looks like valid changed to signed. This is also the cause of the CA expiry issue in the UI. I think it's cleaner to fix this in the proxy so we provide a unified API to Foreman.

I haven't had time to provision nodes yet. On a side note: Kafo needs some Puppet 6 love in the progress bar.

[ekohl]

After a bit of research it looks like the possible states are requested, signed and revoked (https://github.com/puppetlabs/puppetserver/blob/948190ddf39a3b8d229decc7d29dec11b9eb0782/src/clj/puppetlabs/puppetserver/certificate_authority.clj#L94-L96). Looks like you need to do the valid calculation yourself. Without the not_before and not_after fields this is impossible.

I'd propose we map signed to valid. If no dates are available you're using a too old version of puppetserver and we mention that in the release notes. If the dates are available then we do the calculation in the smart-proxy.

Whether this should block the merge, I'm not sure.

[alexjfisher]

As well as mapping signed to valid, if you don't also map requested to pending, the option to sign certs in foreman is missing.

[ekohl]

@timogoebel I was thinking about something like ekohl@a7c682b but I don't know if it's really correct since I can't find how the old way handled expired certs.

[timogoebel]

@ekohl: Thanks for testing this! The code you sent me is an excellent starting point. If you can send me some data I can use for fixtures (basically a api response for 6.3.0 is fine), I should be able to write tests for this tomorrow. I won't have time or connectivity to set up a proper development environment myself 🚆 . Hope we can get this in after that.

[alexjfisher]

If the cert is already in the requested state clean fails here.
https://github.com/theforeman/smart-proxy/pull/615/files#diff-2b3c9fe953a79a2f98f3586b2b26589fR15

Puppetserver logs the following.

Failed to remove certificate(s) for foreman-test.example.com: Failed to set Puppet CA certificate_status v1 API: 409 Cannot revoke certificate for host foreman-test.example.com without a signed certificate

[alexjfisher]

Fortunately, I can confirm that calling DELETE on a certificate in a requested state does work.

[timogoebel]

@alexjfisher: We should be able to mitigate that by allowing Net::HTTPConflict to be a valid response.

[alexjfisher]

Net::HTTPConflict

@timogoebel Yes, I've just tested that change, and it works great now.

[timogoebel]

Amended these changes:

• certificate state mapping, thanks @ekohl
• the new fields not_before, not_after and serial + tests
• a fix for what @alexjfisher reported + a test for that

f0218175 ~/foremandev/smart-proxy (25065-puppetca-http-api)  $ git diff
diff --git a/modules/puppetca_http_api/ca_v1_api_request.rb b/modules/puppetca_http_api/ca_v1_api_request.rb
index 3aa4e69..84976ce 100644
--- a/modules/puppetca_http_api/ca_v1_api_request.rb
+++ b/modules/puppetca_http_api/ca_v1_api_request.rb
@@ -12,7 +12,7 @@ module ::Proxy::PuppetCa::PuppetcaHttpApi
     end

     def clean(certname)
-      handle_response(put_data("/puppet-ca/v1/certificate_status/#{certname}", 'desired_state' => 'revoked'), [Net::HTTPNoContent, Net::HTTPNotFound], 'Failed to set Puppet CA certificate_status v1 API')
+      handle_response(put_data("/puppet-ca/v1/certificate_status/#{certname}", 'desired_state' => 'revoked'), [Net::HTTPNoContent, Net::HTTPNotFound, Net::HTTPConflict], 'Failed to set Puppet CA certificate_status v1 API')
       handle_response(delete("/puppet-ca/v1/certificate_status/#{certname}"), [Net::HTTPNoContent, Net::HTTPNotFound], 'Failed to delete Puppet CA certificate_status v1 API')
     end

diff --git a/modules/puppetca_http_api/puppetca_impl.rb b/modules/puppetca_http_api/puppetca_impl.rb
index 9ec3476..3fcaf0f 100644
--- a/modules/puppetca_http_api/puppetca_impl.rb
+++ b/modules/puppetca_http_api/puppetca_impl.rb
@@ -17,15 +17,39 @@ module ::Proxy::PuppetCa::PuppetcaHttpApi
       response.each_with_object({}) do |entry, hsh|
         name = entry['name']
         # serial, not_before and not_after are not available via http api
+        # for puppetserver < 6.3
         # see https://tickets.puppetlabs.com/browse/SERVER-2370
         hsh[name] = {
-          'state' => entry['state'],
+          'state' => normalized_state(entry),
           'fingerprint' => entry['fingerprint'],
-          'serial' => nil,
-          'not_before' => nil,
-          'not_after' => nil
+          'serial' => entry['serial_number'],
+          'not_before' => entry['not_before'],
+          'not_after' => entry['not_after']
         }
       end
     end
+
+    private
+
+    # Normalize the state to match the Puppet < 6 command line values since
+    # that's what Foreman expects.
+    #
+    # Puppet CA API sends: requested, signed, revoked
+    # Foreman expects: pending, valid, revoked
+    def normalized_state(entry)
+      case entry['state']
+      when 'signed'
+        # Versions before puppetserver 6.3 do not send not_after
+        if entry['not_after'] && Time.parse(entry['not_after']) > Time.now
+          'revoked'
+        else
+          'valid'
+        end
+      when 'requested'
+        'pending'
+      else
+        entry['state']
+      end
+    end
   end
 end
diff --git a/test/puppetca_http_api/ca_v1_api_request_test.rb b/test/puppetca_http_api/ca_v1_api_request_test.rb
index 398fae4..a302753 100644
--- a/test/puppetca_http_api/ca_v1_api_request_test.rb
+++ b/test/puppetca_http_api/ca_v1_api_request_test.rb
@@ -37,6 +37,17 @@ class CaApiv1RequestTest < Test::Unit::TestCase
     assert_nil @client.clean('puppet.example.com')
   end

+  def test_clean_with_certs_in_requested_state
+    stub_request(:put, 'https://puppet:8140/puppet-ca/v1/certificate_status/puppet.example.com').
+      with(:body => "{\"desired_state\":\"revoked\"}").
+      to_return(:status => 409, :body => 'Cannot revoke certificate for host puppet.example.com without a signed certificate')
+
+    stub_request(:delete, 'https://puppet:8140/puppet-ca/v1/certificate_status/puppet.example.com').
+      to_return(:status => 204)
+
+    assert_nil @client.clean('puppet.example.com')
+  end
+
   def test_search
     stub_request(:get, 'https://puppet:8140/puppet-ca/v1/certificate_statuses/foreman').
       to_return(:status => 200, :body => fixture('ca_search.json'))
diff --git a/test/puppetca_http_api/puppetca_http_impl_test.rb b/test/puppetca_http_api/puppetca_http_impl_test.rb
index 87ea1be..5058829 100644
--- a/test/puppetca_http_api/puppetca_http_impl_test.rb
+++ b/test/puppetca_http_api/puppetca_http_impl_test.rb
@@ -26,8 +26,32 @@ class PuppetCaHttpImplTest < Test::Unit::TestCase
         }
       ]
     end
-  end
   # rubocop:enable Metrics/LineLength
+  end
+
+  class FakeCaApiV1Request63 < FakeCaApiV1Request
+    # rubocop:disable Metrics/LineLength
+    def search(key = 'foreman')
+      [
+        {
+          'name' => 'puppet.example.com',
+          'state' => 'signed',
+          'dns_alt_names' => ['DNS:puppet', 'DNS:puppet.example.com'],
+          'subject_alt_names' => ['DNS:puppet', 'DNS:puppet.example.com'],
+          'fingerprint' => 'F8:DA:15:EA:BD:2F:2D:D3:05:71:73:55:96:74:A4:97:2B:04:06:47:A8:8E:D2:C4:AB:8F:EC:3B:7C:0F:0A:EE',
+          'fingerprints' => {
+            'SHA1' => '4F:C2:4B:C5:B3:AD:36:64:8D:70:65:85:0B:F9:29:9E:96:67:4B:6F',
+            'SHA256' => 'F8:DA:15:EA:BD:2F:2D:D3:05:71:73:55:96:74:A4:97:2B:04:06:47:A8:8E:D2:C4:AB:8F:EC:3B:7C:0F:0A:EE',
+            'SHA512' => '83:BD:D2:32:30:F3:3E:69:7D:61:ED:A8:3F:3D:29:81:1C:96:AC:39:9B:A3:09:9E:61:9F:17:78:91:69:73:12:84:51:59:EE:93:42:AB:A8:34:72:41:43:B5:48:32:E7:3C:DE:85:13:5E:78:A5:C9:FD:A3:FF:54:53:7C:E6:03',
+            'default' => 'F8:DA:15:EA:BD:2F:2D:D3:05:71:73:55:96:74:A4:97:2B:04:06:47:A8:8E:D2:C4:AB:8F:EC:3B:7C:0F:0A:EE'
+          },
+          'not_after' => '2019-08-25T19:25:29UTC',
+          'not_before' => '2014-08-25T19:25:29UTC',
+          'serial_number' => 4
+        }
+      ]
+    end
+  end

   def setup
     @api = Proxy::PuppetCa::PuppetcaHttpApi::PuppetcaImpl.new
@@ -51,7 +75,25 @@ class PuppetCaHttpImplTest < Test::Unit::TestCase
           'not_after' => nil,
           'not_before' => nil,
           'serial' => nil,
-          'state' => 'signed'
+          'state' => 'valid'
+      }
+    }
+
+    assert_equal expected, @api.list
+  end
+
+  def test_list_puppetserver_6_3
+    client = FakeCaApiV1Request63.new
+    @api.stubs(:client).returns(client)
+
+    expected = {
+      'puppet.example.com' => {
+        'fingerprint' =>
+        'F8:DA:15:EA:BD:2F:2D:D3:05:71:73:55:96:74:A4:97:2B:04:06:47:A8:8E:D2:C4:AB:8F:EC:3B:7C:0F:0A:EE',
+          'not_after' => '2019-08-25T19:25:29UTC',
+          'not_before' => '2014-08-25T19:25:29UTC',
+          'serial' => 4,
+          'state' => 'revoked'
       }
     }

[alexjfisher]

Looking good. valid and revoked are now showing up properly in Foreman (not tried pending yet).

[alexjfisher]

Should it be possible to delete/clean a revoked certificate? I'm not getting an option to.

[alexjfisher]

I've successfully provisioned a new host with both host whitelisting and token based provisioning

[ekohl]

Test failures are unrelated. I'm going to spend time today to finish the puppet-foreman_proxy change today and then we should merge it.

[ekohl]

I think theforeman/puppet-foreman_proxy#488 is now ready

[ekohl]

Thanks!

[tbrisker]

Thanks everyone! I'm assuming this also needs updates in the manual?