(maint) Add puppetserver alias puppet.internal

 - Remove the domain introspection / setting of AZURE_DOMAIN env var
   as this does not work as originally thought.

   Instead, hardcode the DNS suffix `.internal` to each service in the
   compose stack, and make sure that `dns_search` for `internal` will
   use the Docker DNS resolver when dealing with these hosts. Note that
   these compose file settings only affect the configuration of the
   DNS resolver, *not* resolv.conf. This is different from the
   docker run behavior, which *does* modify resolv.conf. Also note,
   config file locations vary depending on whether or not systemd is
   running in the container.

   It's not "safe" to refer to services in the cluster by only their
   short service names like `puppet`, `puppetdb` or `postgres` as they
   can conflict with hosts on the external network with these names
   when `resolv.conf` appends DNS search suffixes.

   When docker compose creates the user defined network, it copies the
   DNS settings from the host to the `resolv.conf` in each of the
   containers. This often takes search domains from the outside network
   and applies them to containers.

   When network resolutions happen, any default search suffix will be
   applied to short names when the dns option for ndots is not set to 0.
   So for instance, given a `resolv.conf` that contains:

   search delivery.puppetlabs.net

   A DNS request for `puppet` becomes `puppet.delivery.puppetlabs.net`
   which will fail to resolve in the Docker DNS resolver, then be sent
   to the next DNS server in the `nameserver` list, which may resolve it
   to a different host in the external network. This behaves this way
   because `resolv.conf` also sets secondary DNS servers from the host.

   While it is possible to try and service requests for an external
   domain like `delivery.puppetlabs.net` with the embedded Docker DNS
   resolver, it's better to instead choose a domain suffix to use inside
   the cluster.

   There are some good details on how various network types configure:
   docker/for-linux#488 (comment)

 - Note that the .internal domain is typically not recommended for
   production given the only IANA reserved domains are .example, .test,
   .invalid or .localhost. However, given the DNS resolver is set to
   own the resolution of .internal, this is a compromise.

   In production its recommended to use a subdomain of a domain that
   you own, but that's not yet configurable in this compose file. A
   future commit will make this configurable.

 - Another workaround for this problem would be to set the ndots option
   in resolv.conf to 0 per the documentation at
   http://man7.org/linux/man-pages/man5/resolv.conf.5.html

   However that can't be done for two reasons:

   - docker-compose schema doesn't actually support setting DNS options
     docker/cli#1557

   - k8s sets ndots to 5 by default, so we don't want to be at odds

 - A further, but implausible workaround would be to modify the host DNS
   settings to remove any search suffixes.

 - The original FQDN change being reverted in this commit was introduced
   in 8b38620

   "
   Lastly, the Windows specific docker-compose.windows.yml sets up a
   custom alias in the "default" network so that an extra DNS name for
   puppetserver can be set based on the FQDN that Facter determines.
   Without this additional DNS reservation, the `puppetserver ca`
   command will be unable to connect to the REST endpoint.

   A better long-term solution is making sure puppetserver is setup to
   point to `puppet` as the host instead of an FQDN.
   "

   With the PUPPETSERVER_HOSTNAME value set on the puppetserver
   container, both certname and server are set to puppet.internal,
   inside of puppet.conf, preventing a need to inject a domain name as
   was done previously.

   This is necessary because of a discrepancy in how Facter 3 behaves vs
   Facter 2, which creates a mismatch between how the host cert is
   initially generated (using Facter 3) and how `puppetserver ca`
   finds the files on disk (using Facter 2), that setting
   PUPPETSERVER_HOSTNAME will explicitly work around.

   Specifically, Facter 2 may return a different Facter.value('domain')
   than calling `facter domain` using Facter 3 at the command line.
   Such is the case inside the puppet network, where Facter 2 returns
   `ops.puppetlabs.net` while Facter 3 returns `delivery.puppetlabs.net`

	 Without explicitly setting PUPPETSERVER_HOSTNAME, this makes cert
   files on disk get written as *.delivery.puppetlabs.net, yet the
   `puppetserver ca` application looks for the client certs on disk as
   *.ops.puppetlabs.net, which causes `puppetserver ca` to fail.

 - Facter 2 should not be included in the puppetserver packages, and
   changes have been made to packaging for future releases, which may
   remove the need for the above.

 - This PR is also made possible by switching over to using the Ubuntu
   based container from the Alpine container (performed in a prior
   commit), due to DNS resolution problems with Alpine inside LCOW:

   moby/libnetwork#2371
   microsoft/opengcs#303

 - Another avenue that was investigated to resolve the DNS problem in
   Alpine was to feed host:ip mappings in through --add-host, but it
   turns out that Windows doesn't yet support that feature per

   docker/for-win#1455

 - Finally, these changes are also made in preparation of switching the
   pupperware-commercial repo over to a private builder

 - Additionally update k8s / Bolt specs to be consistent with updated
   naming

commercial/Boltdir/modules/stack/tasks/manage.sh

@@ -7,7 +7,7 @@ die() {
 
 pdb_running() {
     docker-compose exec -T puppet \
-        curl -s 'http://puppetdb:8080/status/v1/services/puppetdb-status' | \
+        curl -s 'http://puppetdb.internal:8080/status/v1/services/puppetdb-status' | \
     python -c 'import json, sys; print json.load(sys.stdin)["state"]'
 }
 
@@ -27,7 +27,7 @@ wait_for_it() {
 cd pupperware || die no-repo "run the clone task first to set up pupperware"
 
 host=$(getent hosts "$(hostname -s)")
-export DNS_ALT_NAMES="puppet,${host##* }"
+export DNS_ALT_NAMES="puppet,puppet.internal,${host##* }"
 
 case $PT_action in
     up)

commercial/Boltdir/modules/stack/tasks/pdb_node.sh

@@ -7,7 +7,7 @@ body=$(printf '{ "query": "nodes { certname = \\"%s\\" }" }' "$PT_agent")
 out=""
 while [ -z "$out" ]; do
     out=$(docker-compose exec -T puppet \
-                         curl -s -X POST http://puppetdb:8080/pdb/query/v4 \
+                         curl -s -X POST http://puppetdb.internal:8080/pdb/query/v4 \
                          -H 'Content-Type:application/json' \
                          -d "$body")
     if [ -z "$out" ]; then

commercial/README.md

@@ -31,12 +31,12 @@ Once you have Docker Compose installed, you can start the stack on Linux with:
 
 The value of `DNS_ALT_NAMES` must list all the names, as a comma-separated
 list, under which the Puppet server in the stack can be reached from
-agents. It will have `puppet` prepended to it as that name is used by PuppetDB
-to communicate with the Puppet server. The value of `DNS_ALT_NAMES` only has an
-effect the first time you start the stack, as it is placed into the server's SSL
-certificate. If you need to change it after that, you will need to properly
-revoke the server's certificate and restart the stack with the changed
-`DNS_ALT_NAMES` value.
+agents. It will have `puppet` and `puppet.internal` prepended to it as that
+name is used by PuppetDB to communicate with the Puppet server. The value of
+`DNS_ALT_NAMES` only has an effect the first time you start the stack, as it
+is placed into the server's SSL certificate. If you need to change it after
+that, you will need to properly revoke the server's certificate and restart
+the stack with the changed `DNS_ALT_NAMES` value.
 
 When you first start the Puppet Infrastructure, the stack will create a
 `volumes/` directory with a number of sub-directories to store the

commercial/azure-pipelines.yml

@@ -65,12 +65,8 @@ steps:
   name: test_prepare
 
 - powershell: |
-    $domain = Get-WmiObject -Class Win32_NetworkAdapterConfiguration |
-      Select -ExpandProperty DNSDomain |
-      Select -First 1
     Write-Host 'Writing compose config to disk'
     $content = @"
-    AZURE_DOMAIN=$domain
     VOLUME_ROOT=$ENV:TempVolumeRoot
     "@
     $Utf8NoBomEncoding = New-Object System.Text.UTF8Encoding $False

commercial/docker-compose.yml

@@ -2,26 +2,30 @@ version: '3'
 
 services:
   puppet:
-    hostname: puppet
+    hostname: puppet.internal
     image: puppet/puppetserver
     ports:
       - 8140:8140
     environment:
+      # necessary to set certname and server in puppet.conf, required by
+      # puppetserver ca cli application
+      - PUPPETSERVER_HOSTNAME=puppet.internal
       # DNS_ALT_NAMES must be set before starting the stack the first time,
       # and must list all the names under which the puppetserver can be
-      # reached. 'puppet' must be one of them, otherwise puppetdb won't be
+      # reached. 'puppet.internal' must be one of them, otherwise puppetdb won't be
       # able to get a cert. Add other names as a comma-separated list
-      - DNS_ALT_NAMES=puppet,${DNS_ALT_NAMES:-}
+      - DNS_ALT_NAMES=puppet,puppet.internal,${DNS_ALT_NAMES:-}
       - PUPPERWARE_ANALYTICS_ENABLED=${PUPPERWARE_ANALYTICS_ENABLED:-true}
-      - PUPPETDB_SERVER_URLS=https://puppetdb:8081
+      - PUPPETDB_SERVER_URLS=https://puppetdb.internal:8081
     volumes:
       - ${VOLUME_ROOT:-.}/volumes/code:/etc/puppetlabs/code/
       - ${VOLUME_ROOT:-.}/volumes/puppet:/etc/puppetlabs/puppet/
       - ${VOLUME_ROOT:-.}/volumes/serverdata:/opt/puppetlabs/server/data/puppetserver/
+    dns_search: internal
     networks:
       default:
         aliases:
-         - puppet.${AZURE_DOMAIN:-}
+         - puppet.internal
 
   postgres:
     image: postgres:9.6
@@ -34,14 +38,20 @@ services:
     volumes:
       - ${VOLUME_ROOT:-.}/volumes/puppetdb-postgres/data:/var/lib/postgresql/data
       - ./postgres-custom:/docker-entrypoint-initdb.d
+    dns_search: internal
+    networks:
+      default:
+        aliases:
+         - postgres.internal
 
   puppetdb:
-    hostname: puppetdb
+    hostname: puppetdb.internal
     image: puppet/puppetdb
     environment:
       - PUPPERWARE_ANALYTICS_ENABLED=${PUPPERWARE_ANALYTICS_ENABLED:-true}
       # This name is an FQDN so the short name puppet doesn't collide outside compose network
-      - PUPPETSERVER_HOSTNAME=puppet.${AZURE_DOMAIN:-}
+      - PUPPETSERVER_HOSTNAME=puppet.internal
+      - PUPPETDB_POSTGRES_HOSTNAME=postgres.internal
       - PUPPETDB_PASSWORD=puppetdb
       - PUPPETDB_USER=puppetdb
     ports:
@@ -52,3 +62,8 @@ services:
       - puppet
     volumes:
       - ${VOLUME_ROOT:-.}/volumes/puppetdb/ssl:/etc/puppetlabs/puppet/ssl/
+    dns_search: internal
+    networks:
+      default:
+        aliases:
+         - puppetdb.internal

commercial/gem/lib/pupperware/spec_helper.rb

@@ -225,11 +225,10 @@ def wait_on_puppetserver_status(seconds = 180, service_name = 'puppet')
     end
   end
 
+  # agent_name is the fully qualified name of the node
   def clean_certificate(agent_name)
-    result = run_command('docker-compose --no-ansi exec -T puppet facter domain')
-    domain = result[:stdout].chomp
-    STDOUT.puts "cleaning cert for #{agent_name}.#{domain}"
-    result = run_command("docker-compose --no-ansi exec -T puppet puppetserver ca clean --certname #{agent_name}.#{domain}")
+    STDOUT.puts "cleaning cert for #{agent_name}"
+    result = run_command("docker-compose --no-ansi exec -T puppet puppetserver ca clean --certname #{agent_name}")
     return result[:status].exitstatus
   end
 
@@ -253,11 +252,10 @@ def run_agent(agent_name, network, server = get_container_hostname(get_service_c
     return result[:status].exitstatus
   end
 
+  # agent_name is the fully qualified name of the node
   def check_report(agent_name)
     pdb_uri = URI::join(get_service_base_uri('puppetdb', 8080), '/pdb/query/v4')
-    result = run_command("docker-compose --no-ansi exec -T puppet facter domain")
-    domain = result[:stdout].chomp
-    body = "{ \"query\": \"nodes { certname = \\\"#{agent_name}.#{domain}\\\" } \" }"
+    body = "{ \"query\": \"nodes { certname = \\\"#{agent_name}\\\" } \" }"
 
     return retry_block_up_to_timeout(120) do
       Net::HTTP.start(pdb_uri.hostname, pdb_uri.port) do |http|

commercial/k8s/README.md

@@ -14,7 +14,7 @@ running Kubernetes via Docker for Mac, this will be the FQDN of your Mac. Note t
 
 ```yaml
   - name: DNS_ALT_NAMES
-    value: puppet,myworkstation.domain.net
+    value: puppet,puppet.internal,myworkstation.domain.net
 ```
 
 Then create the Pupperware resources:

commercial/k8s/bin/puppet-query

@@ -1,3 +1,3 @@
 #! /bin/sh
 
-kubectl get pods --selector=svc=puppetdb -o name | cut -d '/' -f 2 | xargs -I '%' kubectl exec '%' -- curl -s -X GET http://puppetdb:8080/pdb/query/v4 --data-urlencode "query=$@"
+kubectl get pods --selector=svc=puppetdb -o name | cut -d '/' -f 2 | xargs -I '%' kubectl exec '%' -- curl -s -X GET http://puppetdb.internal:8080/pdb/query/v4 --data-urlencode "query=$@"

commercial/k8s/postgres.yaml

@@ -44,6 +44,7 @@ spec:
         app: pupperware
         svc: postgres
     spec:
+      hostname: postgres.internal
       containers:
         - image: puppet/puppetdb-postgres
           name: postgres

commercial/k8s/puppetdb.yaml

@@ -47,11 +47,15 @@ spec:
         app: pupperware
         svc: puppetdb
     spec:
-      hostname: puppetdb
+      hostname: puppetdb.internal
       containers:
         - image: puppet/puppetdb
           name: puppetdb
           env:
+            - name: PUPPETSERVER_HOSTNAME
+              value: puppet.internal
+            - name: PUPPETDB_POSTGRES_HOSTNAME
+              value: postgres.internal
             - name: PUPPETDB_PASSWORD
               valueFrom:
                 secretKeyRef:

commercial/k8s/puppetserver.yaml

@@ -71,15 +71,19 @@ spec:
         app: pupperware
         svc: puppet
     spec:
-      hostname: puppet
+      hostname: puppet.internal
       containers:
         - image: puppet/puppetserver
           name: puppet
           env:
+            # necessary to set certname and server in puppet.conf, required by
+            # puppetserver ca cli application
+            - name: PUPPETSERVER_HOSTNAME
+              value: puppet.internal
             - name: DNS_ALT_NAMES
-              value: puppet
+              value: puppet,puppet.internal
             - name: PUPPETDB_SERVER_URLS
-              value: https://puppetdb:8081
+              value: https://puppetdb.internal:8081
           ports:
             - containerPort: 8140
           volumeMounts:

commercial/spec/dockerfile_spec.rb

@@ -14,7 +14,9 @@
   ]
 
   before(:all) do
-    @test_agent = "puppet_test#{Random.rand(1000)}"
+    # append .internal to ensure domain suffix for Docker DNS resolver is used
+    # since search domains are not appended to /etc/resolv.conf
+    @test_agent = "puppet_test#{Random.rand(1000)}.internal"
     @timestamps = []
     status = run_command('docker-compose --no-ansi version')[:status]
     if status.exitstatus != 0