Ensure amd64 containers are pulled when needed

[genebean]

Prior to this, asking for a centos8-64 host while running Beaker on a machine with an ARM processor such the M1 baesd Macs would pull the ARM version of CentOS instead of the x86-64 based one.

[genebean]

I am testing this now to make sure I works as anticipated

[genebean]

This seems to work as expected when running on an Apple Silicon (M1 aka ARM) MacBook Pro:

$ uname -a
Darwin beanm1 20.4.0 Darwin Kernel Version 20.4.0: Thu Apr 22 21:46:41 PDT 2021; root:xnu-7195.101.2~1/RELEASE_ARM64_T8101 arm64

$ BEAKER_setfile=centos7-64 pdk bundle exec rake beaker
pdk (INFO): Using Ruby 2.7.2
pdk (INFO): Using Puppet 7.5.0
TEST_TIERS env variable not defined. Defaulting to run all tests.
/opt/puppetlabs/pdk/private/ruby/2.7.2/bin/ruby -I/opt/puppetlabs/pdk/share/cache/ruby/2.7.0/gems/rspec-core-3.10.1/lib:/opt/puppetlabs/pdk/share/cache/ruby/2.7.0/gems/rspec-support-3.10.2/lib /opt/puppetlabs/pdk/share/cache/ruby/2.7.0/gems/rspec-core-3.10.1/exe/rspec spec/acceptance
/Users/geneliverman/.pdk/cache/ruby/2.7.0/gems/beaker-rspec-6.3.0/lib/beaker-rspec/helpers/serverspec.rb:43: warning: already initialized constant Module::VALID_OPTIONS_KEYS
/opt/puppetlabs/pdk/share/cache/ruby/2.7.0/gems/specinfra-2.82.2/lib/specinfra/configuration.rb:4: warning: previous definition of VALID_OPTIONS_KEYS was here

Hosts file 'centos7-64' does not exist.
Trying as beaker-hostgenerator input.

Hypervisor for centos7-64-1 is docker
Beaker::Hypervisor, found some docker boxes to create
Provisioning docker
provisioning centos7-64-1
Creating image
Dockerfile is         FROM amd64/centos:7
        ENV container docker
RUN yum clean all

[ekohl]

I dislike this implementation. I think what's better is to do it here:

beaker-hostgenerator/lib/beaker-hostgenerator/hypervisor/docker.rb

Lines 12 to 14 in 4144af6

	base_config['image'] = node_info['ostype'].sub(/(\d)/, ':\1')
	base_config['image'].sub!(/(\w+)/, '\1/leap') if node_info['ostype'] =~ /^opensuse/
	base_config['image'].sub!(/(\d{2})/, '\1.') if node_info['ostype'] =~ /^ubuntu/

There we can automatically prefix the image with amd64/. For example:

$ git diff
diff --git a/lib/beaker-hostgenerator/hypervisor/docker.rb b/lib/beaker-hostgenerator/hypervisor/docker.rb
index 18d362e..e9187a4 100644
--- a/lib/beaker-hostgenerator/hypervisor/docker.rb
+++ b/lib/beaker-hostgenerator/hypervisor/docker.rb
@@ -12,6 +12,9 @@ module BeakerHostGenerator
         base_config['image'] = node_info['ostype'].sub(/(\d)/, ':\1')
         base_config['image'].sub!(/(\w+)/, '\1/leap') if node_info['ostype'] =~ /^opensuse/
         base_config['image'].sub!(/(\d{2})/, '\1.') if node_info['ostype'] =~ /^ubuntu/
+        if node_info['bits'] == '64'
+          base_config['image'] = "amd64/#{base_config['image']}"
+        end
 
         return base_generate_node(node_info, base_config, bhg_version, :docker)
       end

This uses that any arch suffix which is 64 is prefixed with amd64. However, with other arches (-32 or -POWER) it's not.

It does rely on https://github.com/docker-library/official-images#architectures-other-than-amd64.

You can also take the same approach to map 32 to i386 and POWER to ppc64le.

The benefit of my method is that it also works for generated nodes (currently only Fedora) and requires less work when adding a new release (like Debian 11, etc).

[genebean]

I like the sound of that @ekohl. Do you know how I would account for when the name here and of the image are different? For example, oracle7-64 == amd64/oraclelinux:7. I assume it's like the other regex entries but I am just not groking what is happening there totally

[ekohl]

First of all, I'd recommend using the console tool. For example:

$ bundle exec beaker-hostgenerator -t docker debian9-64
---
HOSTS:
  debian9-64-1:
    pe_dir:
    pe_ver:
    pe_upgrade_dir:
    pe_upgrade_ver:
    docker_cmd:
    - "/sbin/init"
    image: debian:9
    platform: debian-9-amd64
    packaging_platform: debian-9-amd64
    docker_image_commands:
    - cp /bin/true /sbin/agetty
    - rm -f /usr/sbin/policy-rc.d
    - apt-get update && apt-get install -y cron locales-all net-tools wget systemd-sysv
      gnupg apt-transport-https
    hypervisor: docker
    roles:
    - agent
CONFIG:
  nfs_server: none
  consoleport: 443

Then you can do puts-based debugging and add puts statements in generate_node I linked. Of course you can also use a tool like pry if you're familiar with that. That should give you the context. Then I chose to use sub! when I did some replacements (as you can see for opensuse and ubuntu). You can replace oracle with oraclelinux there.

[genebean]

Thanks! I’ll do that tomorrow. I really appreciate all the tips - Beaker has been something I’ve kinda been overwhelmed by historically but now I’m slowly trying to wrap my head around it.

[genebean]

@ekohl Updated as suggested

[ekohl]

Turns out that if you use podman, it doesn't understand the amd64/ prefix. I'm wondering how you'd detect this because at this stage you don't see anything and it's just a drop in replacement.

Any idea @genebean?

[genebean]

The prefix is an account on Docker Hub, effectively. I don't know beyond that as I have not used Podman yet