(SIMP-4158) k8s: integrate with iptables (#2)

* Adds parameters and code to manage ports related to
  kubernetes with simp/iptables
* Tests will fail on checking 'componentstatus' using 'kubectl' due to a
  known bug in Kubernetes < 1.7 per
  kubernetes/kubernetes#29330

SIMP-4158 #close
SIMP-4187 #close

.fixtures.yml

@@ -11,7 +11,7 @@ fixtures:
       ref: 3.3.1
     stdlib:
       repo: https://github.com/simp/puppetlabs-stdlib
-      ref: 4.19.1
+      ref: 4.24.0
     # integration deps
     iptables:
       repo: https://github.com/simp/pupmod-simp-iptables

.travis.yml

@@ -46,19 +46,19 @@ jobs:
 
     - stage: spec
       rvm: 2.1.9
-      env: STRICT_VARIABLES=yes TRUSTED_NODE_DATA=yes PUPPET_VERSION="~> 4.10.0"
+      env: STRICT_VARIABLES=yes TRUSTED_NODE_DATA=yes PUPPET_VERSION="~> 4.10.9"
       script:
         - bundle exec rake spec
 
     - stage: spec
       rvm: 2.1.9
-      env: STRICT_VARIABLES=yes TRUSTED_NODE_DATA=yes PUPPET_VERSION="~> 4.9.2"
+      env: STRICT_VARIABLES=yes TRUSTED_NODE_DATA=yes PUPPET_VERSION="~> 4.9.4"
       script:
         - bundle exec rake spec
 
     - stage: spec
       rvm: 2.1.9
-      env: STRICT_VARIABLES=yes TRUSTED_NODE_DATA=yes PUPPET_VERSION="~> 4.7.0"
+      env: STRICT_VARIABLES=yes TRUSTED_NODE_DATA=yes PUPPET_VERSION="~> 4.7.1"
       script:
         - bundle exec rake spec
 

Gemfile

@@ -38,5 +38,5 @@ end
 group :system_tests do
   gem 'beaker'
   gem 'beaker-rspec'
-  gem 'simp-beaker-helpers', ENV.fetch('SIMP_BEAKER_HELPERS_VERSION', '~> 1.7')
+  gem 'simp-beaker-helpers', ENV.fetch('SIMP_BEAKER_HELPERS_VERSION', '~> 1.9')
 end

README.md

@@ -26,7 +26,7 @@ services required to get a basic Kubernetes cluster up and running.
 
 This module can set up:
 * Every host
-  - `etcd`
+  - `etcd` (only one peer has been tested)
   - `flanneld`
 * Master hosts
   - `kube-apiserver`
@@ -35,6 +35,10 @@ This module can set up:
 * Node hosts
   - `kubelet`
   - `kube-proxy`
+* Kubeconfigs
+
+Using the following packages from the CentOS Extras repo:
+* kubernetes
 
 This module will not interact with Kubernetes in any way, besides what can be
 configured on a system level.
@@ -50,13 +54,12 @@ compliance-management framework built on Puppet.
 If you find any issues, they may be submitted to our [bug
 tracker](https://simp-project.atlassian.net/).
 
-**FIXME:** Ensure the *This is a SIMP module* section is correct and complete, then remove this message!
-
 This module is optimally designed for use within a larger SIMP ecosystem, but
 it can be used independently:
 
  * When included within the SIMP ecosystem, security compliance settings will
-   be managed from the Puppet server.
+   be managed from the Puppet server, but only on the systems running kubernetes,
+   not the content in kubernetes.
  * If used independently, all SIMP-managed security subsystems are disabled by
    default and must be explicitly opted into by administrators.  Please review
    the parameters in
@@ -67,57 +70,80 @@ it can be used independently:
 
 ### What simp_kubernetes affects
 
-**FIXME:** Ensure the *What simp_kubernetes affects* section is correct and complete, then remove this message!
-
-If it's obvious what your module touches, you can skip this section. For
-example, folks can probably figure out that your mysql_instance module affects
-their MySQL instances.
-
-If there's more that they should know about, though, this is the place to
-mention:
-
- * A list of files, packages, services, or operations that the module will
-   alter, impact, or execute.
- * Dependencies that your module automatically installs.
- * Warnings or other important notices.
-
-### Setup Requirements **OPTIONAL**
 
-**FIXME:** Ensure the *Setup Requirements* section is correct and complete, then remove this message!
+simp_kubernetes only sets up the services listed above, with a few catches:
 
-If your module requires anything extra before setting up (pluginsync enabled,
-etc.), mention it here.
+* The integration with SIMP's PKI may give you trouble, because kubernetes
+  expects certificates with IP subjectAltNames
+* This module does not deploy any services on kubernetes, even if they are
+  required for a complete set up
+* There are no Load Balancers available in the CentOS repos, so there is not one
+  available out of the box using this module
 
-If your most recent release breaks compatibility or requires particular steps
-for upgrading, you might want to include an additional "Upgrading" section
-here.
+ **NOTE:** This module only supports EL7.  **It does not support EL6.**
 
-### Beginning with simp_kubernetes
+### Usage
 
-**FIXME:** Ensure the *Beginning with simp_kubernetes* section is correct and complete, then remove this message!
+The minimal set up would look as follows:
 
-The very basic steps needed for a user to get the module up and running. This
-can include setup steps, if necessary, or it can be an example of the most
-basic use of the module.
-
-## Usage
+```puppet
+class { 'simp_kubernetes':
+  etcd_peers   => ['kube-master01.domain.net'],
+  kube_masters => ['kube-master01.domain.net','kube-master02.domain.net','kube-master03.domain.net']
+}
+```
 
-**FIXME:** Ensure the *Usage* section is correct and complete, then remove this message!
+Where `etcd_peers` is a list of hostnames that will be etcd servers, and
+`kube_masters` is the list of servers where the kubernetes API will be available.
+These parameters will result in a kubernetes cluster using flannel as a network
+backend with all communication over http.
+
+If you need a HA etcd, you will probably be better off setting that up elsewhere
+for now, and setting `manage_etcd` to `false`.
+
+
+To enable TLS, call your class like this:
+
+```puppet
+class { 'simp_kubernetes':
+  etcd_peers           => ['kube-master01.domain.net'],
+  etcd_peer_protocol   => 'https',
+  etcd_client_protocol => 'https',
+  etcd_app_pki_key     => '/path/to/cert',
+  etcd_app_pki_cert    => '/path/to/cert',
+  etcd_app_pki_ca      => '/path/to/cert',
+  kube_masters         => ['kube-master01.domain.net','kube-master02.domain.net','kube-master03.domain.net'],
+  kube_api_protocol    => 'https',
+  kubelet_protocol     => 'https',
+  kube_api_port        => 6443,
+  app_pki_key          => '/path/to/cert',
+  app_pki_cert         => '/path/to/cert',
+  app_pki_ca           => '/path/to/cert',
+}
+```
 
-This section is where you describe how to customize, configure, and do the
-fancy stuff with your module here. It's especially helpful if you include usage
-examples and code samples for doing things with your module.
+There are many parameters in this module that are useful for tweaking every
+service in this module:
+* `flannel_network_config`
+* `flannel_args`
+* `etcd_options`
+* `api_args`
+* `master_api_args`
+* `scheduler_args`
+* `controller_args`
+* `proxy_args`
+* `kubelet_args`
 
 ## Reference
 
-**FIXME:** Ensure the *Reference* section is correct and complete, then remove this message!  If there is pre-generated YARD documentation for this module, ensure the text links to it and remove references to inline documentation.
-
 Please refer to the inline documentation within each source file, or to the
 module's generated YARD documentation for reference material.
 
 ## Limitations
 
-**FIXME:** Ensure the *Limitations* section is correct and complete, then remove this message!
+This module is only intended to work with RedHat's distribution of kubernetes.
+If a newer version is required or a version from a different source, this module
+may not work as intended.
 
 SIMP Puppet modules are generally intended for use on Red Hat Enterprise Linux
 and compatible distributions, such as CentOS. Please see the
@@ -126,8 +152,6 @@ supported operating systems, Puppet versions, and module dependencies.
 
 ## Development
 
-**FIXME:** Ensure the *Development* section is correct and complete, then remove this message!
-
 Please read our [Contribution Guide](http://simp-doc.readthedocs.io/en/stable/contributors_guide/index.html).
 
 ### Acceptance tests
@@ -144,7 +168,8 @@ bundle install
 bundle exec rake beaker:suites
 ```
 
-**FIXME:** Ensure the *Acceptance tests* section is correct and complete, including any module-specific instructions, and remove this message!
+There is also a suite in simp-core (kubernetes) that tests this component in the
+context of a SIMP system.
 
 Please refer to the [SIMP Beaker Helpers documentation](https://github.com/simp/rubygem-simp-beaker-helpers/blob/master/README.md)
 for more information.

manifests/flannel.pp

@@ -33,7 +33,16 @@
   service { 'flanneld':
     ensure    => running,
     enable    => true,
-    subscribe => File['/etc/sysconfig/flanneld']
+    subscribe => File['/etc/sysconfig/flanneld'],
+    require   => Package['flannel'],
+  }
+
+  if $::simp_kubernetes::flannel_manage_firewall {
+    $port = $::simp_kubernetes::flannel_network_config['Port']
+    iptables::listen::udp { 'simp_kubernetes flannel':
+      trusted_nets => $::simp_kubernetes::trusted_nets,
+      dports       => [$port],
+    }
   }
 
 }

manifests/init.pp

@@ -60,6 +60,9 @@
 # @param etcd_peer_protocol
 #   `http` or `https`. Be sure to specify certificates if this is set to `https`
 #
+# @param etcd_client_protocol
+#   `http` or `https`. Be sure to specify certificates if this is set to `https`
+#
 # @param etcd_app_pki_key
 #   Path and name of the private SSL key file for etcd
 #
@@ -69,9 +72,6 @@
 # @param etcd_app_pki_ca
 #   Path to the CA for etcd
 #
-# @param etcd_client_protocol
-#   `http` or `https`. Be sure to specify certificates if this is set to `https`
-#
 # @param etcd_options
 #   Hash of extra options to be passed along to cristifalcas/etcd
 #
@@ -97,7 +97,7 @@
 #   Set the level of log output to debug-level (0~4) or trace-level (5~10)
 #
 # @param api_args
-#   Hash of extra arguments to be sent to any Kubernetes service
+#   Hash of extra arguments to be sent to all Kubernetes services
 #
 # @param service_addresses
 #   Virtual IP range that will be used by Kubernetes services
@@ -157,9 +157,18 @@
 # @param flannel_package_ensure
 #   Forwarded to the package resource for flannel
 #
+# @param etcd_manage_firewall
+#   Open up the firewall for ports used for etcd in this module using simp/iptables
+#
+# @param kube_manage_firewall
+#   Open up the firewall for ports used for kubernetes in this module using simp/iptables
+#
+# @param flannel_manage_firewall
+#   Open up the firewall for ports used for flannel in this module using simp/iptables
+#
 # @param trusted_nets
 #   The address range(s) to allow connections from for host to host
-#   communication.
+#   communication
 #
 # @author https://github.com/simp/pupmod-simp-simp_kubernetes/graphs/contributors
 #
@@ -221,10 +230,12 @@
 # SIMP Catalysts
   Variant[Boolean,Enum['simp']] $use_simp_certs = simplib::lookup('simp_options::pki', { 'default_value' => false }),
   Stdlib::Absolutepath $app_pki_external_source = simplib::lookup('simp_options::pki::source', { 'default_value' => '/etc/pki/simp/x509' }),
-  String $package_ensure         = simplib::lookup('simp_options::package_ensure', {'default_value' => 'installed' }),
+  String $package_ensure = simplib::lookup('simp_options::package_ensure', {'default_value' => 'installed' }),
   String $flannel_package_ensure = simplib::lookup('simp_options::package_ensure', {'default_value' => 'installed' }),
+  Boolean $etcd_manage_firewall = simplib::lookup('simp_options::firewall', {'default_value' => false }),
+  Boolean $kube_manage_firewall = simplib::lookup('simp_options::firewall', {'default_value' => false }),
+  Boolean $flannel_manage_firewall = simplib::lookup('simp_options::firewall', {'default_value' => false }),
   Simplib::Netlist $trusted_nets = simplib::lookup('simp_options::trusted_nets', {'default_value' => ['127.0.0.1/32'] }),
-
 ) {
 
   $etcd_advertise_client_urls = $etcd_peers.map |$peer| {
@@ -251,12 +262,11 @@
   -> Class['simp_kubernetes::common_config']
 
   if $is_master {
-    include '::simp_kubernetes::master'
-    Class['simp_kubernetes::etcd'] -> Class['simp_kubernetes::flannel']
+    contain '::simp_kubernetes::master'
     Class['simp_kubernetes::common_config'] -> Class['simp_kubernetes::master']
   }
   else {
-    include '::simp_kubernetes::node'
+    contain '::simp_kubernetes::node'
     Class['simp_kubernetes::common_config'] -> Class['simp_kubernetes::node']
   }
 

manifests/master.pp

@@ -4,15 +4,22 @@
 
 
   if $::simp_kubernetes::manage_etcd {
-    include '::simp_kubernetes::etcd'
+    include '::simp_kubernetes::master::etcd'
 
     if $::simp_kubernetes::network_tech == 'flannel' {
-      Class['simp_kubernetes::etcd'] -> Class['simp_kubernetes::flannel']
+      Class['simp_kubernetes::master::etcd'] -> Class['simp_kubernetes::flannel']
     }
   }
 
-  include '::simp_kubernetes::master::apiserver'
-  include '::simp_kubernetes::master::controller_manager'
-  include '::simp_kubernetes::master::scheduler'
+  if $::simp_kubernetes::kube_manage_firewall {
+    iptables::listen::tcp_stateful { 'simp_kubernetes kube_api_port':
+      trusted_nets => $::simp_kubernetes::trusted_nets,
+      dports       => [$::simp_kubernetes::kube_api_port],
+    }
+  }
+
+  contain '::simp_kubernetes::master::apiserver'
+  contain '::simp_kubernetes::master::controller_manager'
+  contain '::simp_kubernetes::master::scheduler'
 
 }

manifests/etcd.pp → manifests/master/etcd.pp

@@ -1,6 +1,6 @@
 # Manage etcd
 #
-class simp_kubernetes::etcd {
+class simp_kubernetes::master::etcd {
 
   $client_protocol   = $::simp_kubernetes::etcd_client_protocol
   $client_listen_url = "${client_protocol}://${::simp_kubernetes::etcd_client_listen_address}:${::simp_kubernetes::etcd_client_port}"
@@ -13,6 +13,8 @@
   $etcd_listen_peer_urls = $peers.map |$peer| {
     "${peer_protocol}://${peer}:${peer_port}"
   }
+  # this is probably wrong, but there is an issue with the way we start etcd with puppet
+  # and how etcd does leader election and cluster bootstrapping
   $etcd_cluster = zip($peers,$etcd_listen_peer_urls).map |$url| {
     if $url[0] == $facts['fqdn'] {
       "${url[0]}=${peer_protocol}://0.0.0.0:${peer_port}"
@@ -73,6 +75,16 @@
   }
 
 
+  if $::simp_kubernetes::etcd_manage_firewall {
+    iptables::listen::tcp_stateful {
+      default: trusted_nets => $::simp_kubernetes::trusted_nets;
+
+      'simp_kubernetes etcd_peer_port':   dports => [$::simp_kubernetes::etcd_peer_port];
+      'simp_kubernetes etcd_client_port': dports => [$::simp_kubernetes::etcd_client_port];
+    }
+  }
+
+
   if $::simp_kubernetes::inject_network_config {
     if $::simp_kubernetes::etcd_client_protocol == 'https' {
       $etcdctl_pki = {