HBASE-25375 Provide a VM-based release environment #2754

ndimiduk · 2020-12-08T19:01:57Z

This adds a Vagrantfile and supporting automation that creates a virtual machine environment
suitable for running the create-release scriping.

Apache9

I'm not familiar with this usage of Virtualbox although I've used it under linux...

Anyway, if you think this is useful then let's try it.

Just one question, after launching the VM, we will call a release script diectly, or we still need to use docker based release script instead the VM?

ndimiduk · 2020-12-09T18:20:53Z

Anyway, if you think this is useful then let's try it.

It's been useful for me. I don't know if it's useful to other release managers. I don't want to add unused code to our repository, so if there isn't another release manager who thinks this will be helpful, let's not merge it.

Just one question, after launching the VM, we will call a release script diectly, or we still need to use docker based release script instead the VM?

I run the docker-based release script inside of this VM. I mention this at the very end of the README.md file, but maybe this information should be stated earlier, in a kind of summary usage statement.

I'll wait for other RM's to chime in.

saintstack

Trying it. I have virtual box locally left over messing w/ vagrant/virtualbox to do a cluster on a node. I have just finished a few runs using docker desktop for Mac OS and it has been really slow. Usually I've gone to GCE when I wanted to go fast (network especially). If this gets us over slow Mac docker and is easier to set up, means less futzing with the environment to make sure all is good for signing and ssh'ing, we should consider it (would be good to integrate it more w/ the create-release stuff... rather than have it over here on the side). Will be back.

saintstack · 2020-12-09T19:29:22Z

FYI @ndimiduk

$ vagrant up
Bringing machine 'rmvm' up with 'virtualbox' provider...
==> rmvm: Checking if box 'ubuntu/bionic64' version '20201207.0.0' is up to date...
==> rmvm: Clearing any previously set forwarded ports...
==> rmvm: Clearing any previously set network interfaces...
==> rmvm: Preparing network interfaces based on configuration...
    rmvm: Adapter 1: nat
==> rmvm: Forwarding ports...
    rmvm: 22 (guest) => 2222 (host) (adapter 1)
==> rmvm: Running 'pre-boot' VM customizations...
==> rmvm: Booting VM...
There was an error while executing `VBoxManage`, a CLI used by Vagrant
for controlling VirtualBox. The command and stderr is shown below.

Command: ["startvm", "e3d1d940-168e-4eb4-9fc8-ff4e46d02203", "--type", "headless"]

Stderr: VBoxManage: error: The virtual machine 'rmvm' has terminated unexpectedly during startup with exit code 1 (0x1)
VBoxManage: error: Details: code NS_ERROR_FAILURE (0x80004005), component MachineWrap, interface IMachine

I'd started virtualbox ...

I updated it, and got this:

$ vagrant up
The provider 'virtualbox' that was requested to back the machine
'rmvm' is reporting that it isn't usable on this system. The
reason is shown below:

Vagrant has detected that you have a version of VirtualBox installed
that is not supported by this version of Vagrant. Please install one of
the supported versions listed below to use Vagrant:

4.0, 4.1, 4.2, 4.3, 5.0, 5.1, 5.2, 6.0

A Vagrant update may also be available that adds support for the version
you specified. Please check www.vagrantup.com/downloads.html to download
the latest version.

(I'd started it and see the rmvm as stopped in listing of machines).

Updated vagrant... brew upgrade vagrant. 2.2.4 to 2.2.14.

Then it did this again...

There was an error while executing `VBoxManage`, a CLI used by Vagrant
for controlling VirtualBox. The command and stderr is shown below.

Command: ["startvm", "e3d1d940-168e-4eb4-9fc8-ff4e46d02203", "--type", "headless"]

Stderr: VBoxManage: error: The virtual machine 'rmvm' has terminated unexpectedly during startup with exit code 1 (0x1)
VBoxManage: error: Details: code NS_ERROR_FAILURE (0x80004005), component MachineWrap, interface IMachine

Reinstalled (because net says might work...)

brew reinstall virtualbox

... hmm... still same complaint.

This is enough comment for one text box.

ndimiduk · 2020-12-09T20:16:40Z

VirtualBox is really painful to use. It doesn't help that Vagrant and VirtualBox don't always update together. If it helps at all, the vm is working for my with this version pair,

$ VBoxManage --version                                                                                                                                                   
6.1.16r140961                                                                                                                                                                                            
$ vagrant --version                                                 
Vagrant 2.2.14

ndimiduk · 2020-12-09T20:19:19Z

Oh, one other idea: sometimes VirtualBox gets stuck in such a way that the only way to resolve it is by launching it's GUI and clicking through whatever error messages it displays. Sometimes I have to delete a VM from the GUI and then resend vagrant in order to get state cleared out of both systems.

saintstack · 2020-12-09T20:25:45Z

On another host (my build machine as it happens), it seems to just work. Trying test build...

Looked like this when I logged in...

$ vagrant ssh
Welcome to Ubuntu 18.04.3 LTS (GNU/Linux 4.15.0-72-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  System information as of Wed Dec  9 19:47:22 UTC 2020

  System load:  0.22              Processes:              110
  Usage of /:   29.8% of 9.63GB   Users logged in:        0
  Memory usage: 7%                IP address for enp0s3:  10.0.2.15
  Swap usage:   0%                IP address for docker0: XXX.17.0.1


168 packages can be updated.
121 updates are security updates.

New release '20.04.1 LTS' available.
Run 'do-release-upgrade' to upgrade to it.


*** System restart required ***

Seemed good bug gpg was not present in env.... so closed down... cleaned that up and then on restart got this.

stack:release-env stack$ vagrant up
Bringing machine 'rmvm' up with 'virtualbox' provider...
==> rmvm: Checking if box 'ubuntu/bionic64' version '20191218.0.0' is up to date...
==> rmvm: Machine already provisioned. Run `vagrant provision` or use the `--provision`
==> rmvm: flag to force provisioning. Provisioners marked to run always will still run.
==> rmvm: Running provisioner: puppet...
==> rmvm: Running Puppet with environment production...
==> rmvm: Info: Loading facts
==> rmvm: Info: Loading facts
==> rmvm: Info: Loading facts
==> rmvm: Info: Loading facts
==> rmvm: Warning: /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5
==> rmvm:    (file: /etc/puppet/hiera.yaml)
==> rmvm: Notice: VersionRanges will always be strict when using non-vendored SemanticPuppet gem, version 1.0.2
==> rmvm: Warning: The function 'hiera_array' is deprecated in favor of using 'lookup'. See https://docs.puppet.com/puppet/5.4/reference/deprecated_language.html\n   (file & line not available)
==> rmvm: Warning: The function 'hiera_hash' is deprecated in favor of using 'lookup'. See https://docs.puppet.com/puppet/5.4/reference/deprecated_language.html\n   (file & line not available)
==> rmvm: Notice: Compiled catalog for rmvm.apple.com in environment production in 2.01 seconds
==> rmvm: Info: Applying configuration version '1607545060'
==> rmvm: Notice: /Stage[main]/Main/Node[rmvm]/Vcsrepo[/home/vagrant/repos/hbase]/ensure: Updating to latest 'master' revision
==> rmvm: Error: Execution of '/usr/bin/git stash pop' returned 1: No stash entries found.
==> rmvm: Error: /Stage[main]/Main/Node[rmvm]/Vcsrepo[/home/vagrant/repos/hbase]/ensure: change from 'present' to 'latest' failed: Execution of '/usr/bin/git stash pop' returned 1: No stash entries found.
==> rmvm: Notice: Applied catalog in 10.30 seconds
The SSH command responded with a non-zero exit status. Vagrant
assumes that this means the command failed. The output for this command
should be in the log above. Please read the output to determine what
went wrong.

saintstack · 2020-12-09T22:33:22Z

Build fails for me because no gpg key in vagrant env...

$ gpg --sign repos/hbase/pom.xml
gpg: no default secret key: No secret key
gpg: signing failed: No secret key

When I run vagrant, there is a gpg-agent up that works.. has cached key...

Vagrant supposed to hoist it up into the vm (the key may not have registered in agent when I did 'vagrant up'... maybe it has to be then?)

saintstack · 2020-12-09T23:48:23Z

Ok. Sorted out the virtualbox and vagrant stuff... Thats all good. Just stuck now on the gpg secret passing bit (Nick has been helping me out offline).

ndimiduk · 2020-12-10T00:45:16Z

I think this project has bit-rot since I used it last (which has been several months). My current best guess is that gpg version miss-match is preventing agent forwarding to work. My host machine has gpg 2.2.20 while the guest bionic vm install gpg 2.2.4. According to packages.u.o, I can get gpg 2.2.19 if I upgrade the vm to Ubuntu focal (20.04LTS).

The alternative is to downgrade the gpg version running on the host OS. I don't think that's a good long-term solution though.

ndimiduk · 2020-12-10T00:48:53Z

Head's up though -- if the issue here is indeed gpg agent version mismatch, create-release in docker mode will have the same problems because it's also using an ubuntu bionic base for it's agent forwarding tests...

saintstack · 2020-12-10T01:47:38Z

On Mac its $ gpg --version
gpg (GnuPG) 2.2.25

Trying later ubuntu...

saintstack · 2020-12-10T05:15:33Z

Tried upping the vagrantfile ubuntu... removed the old image first.

vagrant@rmvm:~$ more /etc/issue
Ubuntu 20.04.1 LTS \n \l

vagrant@rmvm:$ gpg --sign repos/hbase/pom.xml
gpg: no default secret key: No secret key
gpg: signing failed: No secret key
vagrant@rmvm:$ ps auxwww|grep gpg-agent
vagrant 21315 0.0 0.0 82948 3496 ? SLs 05:08 0:00 /usr/bin/gpg-agent --supervised

I didn't check to see if the gpg-agent process was present before I went about my business.

ndimiduk · 2021-01-05T22:29:21Z

Upgrading this VM to 20.04 (focal) gives me gpg 2.2.19 on the VM side. with this, and 2.2.26 on the mac side, i’m able to forward my agent and ssh -T git@github.com run from the vm passes through my credentials.

dev-support/release-vm/README.md

ndimiduk · 2021-01-06T17:42:22Z

I've updated the readme based on @saintstack 's comments.

ndimiduk · 2021-01-06T17:56:37Z

Maybe some other release managers could take another look here? Maybe someone is feeling ambitious and would like to add box definitions for public clouds (i.e., via vagrant-google or vagrant-aws). In theory this is trivial, assuming an equivalent base box to the one used with the VirtualBox provider.

dev-support/release-vm/puppet/production/data/nodes/rmvm.yaml

dev-support/release-vm/README.md

dev-support/release-vm/puppet/production/data/nodes/rmvm.yaml

This adds a Vagrantfile and supporting automation that creates a virtual machine environment suitable for running the create-release scriping.

Apache-HBase · 2021-01-07T17:45:32Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 5s	Docker mode activated.
-0 ⚠️	yetus	0m 4s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+0 🆗	mvndep	0m 23s	Maven dependency ordering for branch
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 8s	Maven dependency ordering for patch
		_ Other Tests _
		2m 32s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2754/9/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#2754
Optional Tests
uname	Linux ae33baaed4bc 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `2444d26`
Max. process+thread count	49 (vs. ulimit of 30000)
modules	C: U:
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2754/9/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2021-01-07T17:46:14Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 46s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+0 🆗	mvndep	0m 22s	Maven dependency ordering for branch
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 8s	Maven dependency ordering for patch
		_ Other Tests _
		3m 10s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2754/9/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#2754
Optional Tests
uname	Linux ba6ec0b1b0a8 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `2444d26`
Max. process+thread count	58 (vs. ulimit of 30000)
modules	C: U:
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2754/9/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2021-01-07T17:47:16Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	2m 49s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 1s	No case conflicting files found.
+0 🆗	shelldocs	0m 1s	Shelldocs was not available.
+0 🆗	yamllint	0m 1s	yamllint was not available.
+0 🆗	markdownlint	0m 1s	markdownlint was not available.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+0 🆗	mvndep	0m 12s	Maven dependency ordering for branch
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 8s	Maven dependency ordering for patch
+1 💚	shellcheck	0m 0s	There were no new shellcheck issues.
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
		_ Other Tests _
+0 🆗	asflicense	0m 0s	ASF License check generated no output?
		4m 14s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2754/9/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#2754
Optional Tests	dupname asflicense shellcheck shelldocs yamllint markdownlint
uname	Linux c938f713b205 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `2444d26`
Max. process+thread count	43 (vs. ulimit of 30000)
modules	C: U:
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2754/9/console
versions	git=2.17.1 maven=3.6.3 shellcheck=0.4.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

saintstack · 2021-01-09T04:27:08Z

Left some advocacy for this addition up on the JIRA

ndimiduk · 2021-01-11T21:54:22Z

Thanks for giving it a proper spin, @saintstack . Given that it has meaningful utility beyond myself, I'm good with merging this patch. It also provides a reference point for diagnosing what's slow on docker vs. the vm with the same hardware.

ndimiduk · 2021-01-11T22:02:46Z

@Apache9 do you have any further comments before merge? Hopefully the readme is more clear.

ndimiduk · 2021-01-14T19:22:29Z

This is merged. Thanks for taking time with it!

@Apache9 I included a signed-off-by line with your name on it because you did approve an early version of this. If you get back to this ticket and have some reservations, let me know and I'll revert the commit and continue the discussion.

Apache9 · 2021-01-15T01:14:28Z

No problem. Just go ahead.

Thanks.

ndimiduk requested review from saintstack, apurtell, infraio, Apache9 and virajjasani December 8, 2020 19:01