HBASE-25375 Provide a VM-based release environment (#2754)

This adds a Vagrantfile and supporting automation that creates a virtual machine environment
suitable for running the create-release scripting.

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Michael Stack <stack@apache.org>

dev-support/release-vm/.gitignore

@@ -0,0 +1,3 @@
+.vagrant/
+*.log
+*.patch

dev-support/release-vm/README.md

@@ -0,0 +1,141 @@
+<!---
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# HBase Release Env
+
+This is a vagrant project that provides a virtual machine environment suitable
+for running an Apache HBase release.
+
+Requires:
+* [VirtualBox](http://virtualbox.org)
+* [Vagrant](http://virtualbox.org)
+* The private portion of your signing key avilable in the local GPG agent
+* The private portion of your Github authentication key available in either the local GPG agent or
+  local SSH agent
+
+## Usage
+
+Unlock the local keyring before proceeding (this should prompt you for your GPG passphrase). For
+example, assuming you have an authentication key configured in your keyring, this will do the
+trick.
+
+All terminal commands used below are assumed to be run with the current working directory as the
+location containing the `Vagrantfile`.
+
+The term "Host" is used to mean the environment that runs the Vagrant process. The term "Guest" is
+used to mean the virtual machine managed by the Host.
+
+### Ensure credentials work from the Host OS
+
+The ssh- and gpg-agent forwarding configuration used here assumes that your credentials work
+on the Host. Verify both are working before you proceed with the Guest. Additionally, using the
+credentials requires you to unlock the respective keyring, the state of which is persisted by the
+agent process or processes running on the Host.
+
+See instructions in [`create-release`](../create-release/README.txt) regarding proper
+configuration of ssh- and gpg-agents.
+
+Assuming the git repo origin is on GitHub, the following command will ensure that your ssh
+credentials are working. On the Host, run:
+
+```sh
+host:~$ ssh -T git@github.com
+Hi <you>! You've successfully authenticated, but GitHub does not provide shell access.
+```
+
+Likewise, ensure you have an encryption key that can be used to sign a file. Again, on the Host,
+run:
+
+```sh
+host:~$ gpg --detach --armor --sign Vagrantfile
+host:~$ gpg --verify Vagrantfile.asc
+gpg: assuming signed data in 'Vagrantfile'
+...
+host:~$ rm Vagrantfile.asc
+```
+
+### Make public keyring available to the VM
+
+Export the public portion of your signing credentials where the Guest can access it. Vagrant
+(+VirtualBox) shares the directory of the `Vagrantfile` with the Linux Guest via the `/vagrant`
+mount point. Any files present in this working directory on the Host are available to the Guest.
+
+From the Host, run:
+
+```sh
+host:~$ gpg --export <you>@apache.org > gpg.<you>.apache.pub
+```
+
+### Launch the Guest VM
+
+Launch the Guest VM by running
+
+```sh
+host:~$ vagrant up
+```
+
+If anything about the Vagrant or VirtualBox environment have changed since you last used this VM,
+it's best to `vagrant destroy -f` all local state and `vagrant up` a fresh instance.
+
+### Verify the Guest VM
+
+Connect to the Guest. This should forward your ssh- and gpg-agent session, as configured in the
+`Vagrantfile`.
+
+```sh
+host:~$ vagrant ssh
+```
+
+Now that you're in the Guest VM, be sure that all `gpg` command you issue include the
+`--no-autostart` flag. This ensures that the `gpg` process in the Guest communicates with the
+agent running on the Host OS rather than launching its own process on the Guest OS.
+
+From the Guest, verify that ssh-agent forwarding is working, using the same test performed on the
+Host,
+
+```sh
+guest:~$ ssh -T git@github.com
+Hi <you>! You've successfully authenticated, but GitHub does not provide shell access.
+```
+
+From the Guest, import your exported public identity and verify the gpg-agent passthrough is
+working correctly.
+
+```sh
+guest:~$ gpg --no-autostart --import /vagrant/gpg.<you>.apache.pub
+...
+gpg: Total number processed: 1
+gpg:               imported: 1
+guest:~$ gpg --no-autostart --detach --armor --sign repos/hbase/pom.xml
+guest:~$ gpg --no-autostart --verify repos/hbase/pom.xml.asc
+gpg: assuming signed data in 'repos/hbase/pom.xml'
+...
+guest:~$ rm repos/hbase/pom.xml.asc
+```
+
+### Build a Release Candidate
+
+Finally, you can initiate the release build. Follow the instructions in
+[`create-release`](../create-release/README.txt), i.e.,
+
+```sh
+guest:~$ mkdir ~/build-2.3.1-rc0
+guest:~$ cd repos/hbase
+guest:~/repos/hbase$ ./dev-support/create-release/do-release-docker.sh -d ~/build-2.3.1-rc0/ ...
+```

dev-support/release-vm/Vagrantfile

@@ -0,0 +1,50 @@
+# -*- mode: ruby -*-
+# vi: set ft=ruby :
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+Vagrant.configure("2") do |config|
+
+  config.vm.define "rmvm" do |rmvm|
+    rmvm.vm.box = "ubuntu/focal64"
+    rmvm.vm.hostname = "rmvm"
+
+    rmvm.vm.provision "shell", path: "provision/focal.sh", run: "once"
+
+    rmvm.vm.provision "puppet", run: "always" do |puppet|
+      puppet.environment = "production"
+      puppet.environment_path = "puppet"
+      puppet.working_directory = "/tmp/vagrant-puppet"
+      puppet.options = "--test"
+    end
+
+    rmvm.vm.provider "virtualbox" do |vb|
+      vb.name = "rmvm"
+      vb.cpus = 2
+      vb.memory = (4 * 1024).to_s
+    end
+  end
+
+  # pass through ssh-agent for github interaction
+  config.ssh.forward_agent = true
+  # pass through gpg-agent for artifact signing
+  config.ssh.extra_args = [
+      "-R", "/run/user/1000/gnupg/S.gpg-agent:#{%x(gpgconf --list-dirs agent-extra-socket).strip}",
+      "-R", "/run/user/1000/gnupg/S.gpg-agent.extra:#{%x(gpgconf --list-dir agent-extra-socket).strip}",
+  ]
+end

dev-support/release-vm/provision/focal.sh

@@ -0,0 +1,25 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Bootstrap provisioner for a Ubuntu Bionic host.
+#
+
+sudo apt-get update -qq
+# puppet lets us manage the host, librarian-puppet lets us download puppet libraries
+sudo apt-get install -y --no-install-recommends puppet librarian-puppet
+cd /tmp/vagrant-puppet/environments/production && sudo librarian-puppet install --verbose

dev-support/release-vm/puppet/production/.gitignore

@@ -0,0 +1,3 @@
+.tmp/
+modules/
+**/*.lock

dev-support/release-vm/puppet/production/.librarian/puppet/config

@@ -0,0 +1,21 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+---
+LIBRARIAN_PUPPET_DESTRUCTIVE: 'false'
+LIBRARIAN_PUPPET_USE_V1_API: '1'
+LIBRARIAN_PUPPET_TMP: "/tmp/librarian_puppet/tmp"

dev-support/release-vm/puppet/production/Puppetfile

@@ -0,0 +1,27 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+forge "https://forgeapi.puppetlabs.com"
+
+mod 'puppet-packages',
+  :git => "https://github.com/greenaar/puppet-packages.git",
+  :ref => '8d6b8a85eea931e4cd045884d5786c1c1ff0df4c'
+mod 'puppetlabs-docker', '3.10.1'
+mod 'puppetlabs-stdlib', '5.2.0'
+mod 'puppetlabs-vcsrepo', '3.1.0'
+mod 'saz-ssh', '6.2.0'

dev-support/release-vm/puppet/production/data/nodes/rmvm.yaml

@@ -0,0 +1,31 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+---
+classes:
+  - docker
+  - packages::manage
+  - ssh
+
+packages::latest:
+  - curl
+  - git
+  - gnupg
+  - gnupg-agent
+
+ssh::server_options:
+  StreamLocalBindUnlink: 'yes'

dev-support/release-vm/puppet/production/environment.conf

@@ -0,0 +1,20 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+modulepath = modules:site:$basemodulepath
+manifest = manifests

dev-support/release-vm/puppet/production/hiera.yaml

@@ -0,0 +1,23 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+---
+version: 5
+
+hierarchy:
+  - name: "Per-node data"
+    path: "nodes/%{facts.hostname}.yaml"

dev-support/release-vm/puppet/production/manifests/default.pp

@@ -0,0 +1,44 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+lookup('classes', Array[String], 'unique').include
+
+node rmvm {
+  $user = 'vagrant'
+
+  # include the default `vagrant` user in the `docker` group
+  user { $user:
+    groups => ['docker'],
+    require => Package['docker'],
+  }
+
+  # allow managing git repos in puppet
+  vcsrepo { "/home/${user}/repos/hbase":
+    ensure             => latest,
+    branch             => 'master',
+    group              => $user,
+    owner              => $user,
+    keep_local_changes => true,
+    provider           => git,
+    remote             => 'origin',
+    source             => {
+      'origin' => 'https://github.com/apache/hbase.git',
+    },
+    depth              => 1,
+  }
+}