Test VM: updated base box (centos/7), cookbooks and packages #513

MMatten · 2016-10-20T11:13:46Z

Update Vagrant VB base box, cookbook versions and packages versions where appropriate:

box based on CentOS 7.2. changed VG/LV names.
VBox disk expansion now handled via vagrant-newdisk plugin
mysql 5.7
postgresql 9.6
git 2.10.1
eliminate resource cloning in mysql and postgresql cookbooks (CHEF-3694)
swap, postgresql - switch back to upstream cookbooks
update .gitignore list (for the extra VBox disk, vagrant temp files, informix installer)
disable selinux and firewalls
fix vbox shared mountpoint options (nfs v4 requires udp_off)

TODO:

Check if we can switch swap cookbook to upstream (instead of dbfit fork)
Try to deal with the CHEF-3694 in our mysql cookbook
Try to deal with the CHEF-3694 in our postgres cookbook
Tidy up the commits in this PR
Tidy up the PR title and description

resolves #510

javornikolov

Since we merged the informix mode changes with the other PR I guess we should rebase to remove them from this one?

javornikolov · 2016-10-20T18:40:44Z

test_vm/vendor/cookbooks/dbfit_test/recipes/default.rb

@@ -1,6 +1,8 @@
 include_recipe "dbfit_test::diskspace"
-include_recipe "dbfit_test::firewalld"
+include_recipe 'firewalld::disable'


How is disabling firewall useful?

Why do we need the firewall active on the guest?

Actually on this box it's disabled by default. On the old one it was turned on.

javornikolov · 2016-10-20T18:41:09Z

test_vm/vendor/cookbooks/dbfit_test/recipes/default.rb

@@ -1,6 +1,8 @@
 include_recipe "dbfit_test::diskspace"
-include_recipe "dbfit_test::firewalld"
+include_recipe 'firewalld::disable'


It's good to be consistent on quotations: ' vs "

MMatten · 2016-10-20T20:18:12Z

Actually regarding the firewalld service - it looks like port forwarding of 8085 to the host still isn't quite working. After turning off the iptables service I was able to connect to FitNesse on the guest from the host.

I could turn off this service as well (with https://supermarket.chef.io/cookbooks/iptables). What do you think?

javornikolov · 2016-10-20T21:43:40Z

OK, let's disable the firewalls. In general it depends if VM network interfaces are exposed to external world (depends how the VM box is configured).

javornikolov · 2016-10-20T21:55:50Z

So... I managed to arrange a cloud box on https://zrh.cloudsigma.com to try the VM setup for a few days :-) I'm getting the following error on vagrant up from this branch:

==> default: Running 'pre-boot' VM customizations...
A customization command failed:

["storageattach", :id, "--storagectl", "IDE Controller", "--port", 1, "--device", 0, "--type", "hdd", "--medium", "./disk1.vdi"]

The following error was experienced:

#<Vagrant::Errors::VBoxManageError: There was an error while executing `VBoxManage`, a CLI used by Vagrant
for controlling VirtualBox. The command and stderr is shown below.

Command: ["storageattach", "acdec18e-b21f-47e2-9349-89ccfa2500ff", "--storagectl", "IDE Controller", "--port", "1", "--device", "0", "--type", "hdd", "--medium", "./disk1.vdi"]

Stderr: VBoxManage: error: Could not find a controller named 'IDE Controller'
>

Please fix this customization and try again.

javornikolov · 2016-10-20T22:34:11Z

test_vm/Vagrantfile

@@ -13,15 +13,15 @@ Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
    # Add another disk.
    if ARGV[0] == "up" && !File.exist?(file_to_disk)
        vb.customize ['createhd', '--filename', file_to_disk, '--size', 10 * 1024]
-        vb.customize ['storageattach', :id, '--storagectl', 'SATA Controller', '--port', 1, '--device', 0, '--type', 'hdd', '--medium', file_to_disk]
+        vb.customize ['storageattach', :id, '--storagectl', 'IDE Controller', '--port', 1, '--device', 0, '--type', 'hdd', '--medium', file_to_disk]


For me it happens that the controller name is IDE (not IDE Controller). I'm using VirtualBox 5.1.8.

MMatten · 2016-10-21T09:28:28Z

That's strange. Let me double check I'm up to date with VB.

MMatten · 2016-10-21T09:31:14Z

In general it depends if VM network interfaces are exposed to external world (depends how the VM box is configured).

Isn't that ultimately governed by the host's firewall?

javornikolov · 2016-10-21T13:06:55Z

That's strange. Let me double check I'm up to date with VB.

That indeed depends on VBox version - controller name is IDE for 5.1.8 and IDE Controller For VBox 5.0. Similar topic is discussed here: https://www.virtualbox.org/ticket/5994.

MMatten · 2016-10-21T13:16:08Z

That indeed depends on VBox version - controller name is IDE for 5.1.8 and IDE Controller For VBox 5.0. Similar topic is discussed here: https://www.virtualbox.org/ticket/5994.

That's nasty. But this ticket is ancient.

Have you tried with both recent VB versions?

javornikolov · 2016-10-21T15:03:29Z

Have you tried with both recent VB versions?

Versions were 5.1.8 and 5.0.28.

MMatten · 2016-10-21T20:28:28Z

I'm actually on actually 5.0.25 for VB. I might have installed from a public repo earlier instead of installing the download that I'd got.

So we could perhaps add our own disk controller to avoid name variations in VB versions. Can you try the latest commit? I think it might need a better name though.

javornikolov · 2016-10-21T21:09:51Z

I switched to VB 5.0.10 - there are some bugs when running newer versions of VB inside KVM virtual machine...

The last commit is a great idea 👍 Looks like it would be compatible with the different VB versions. I just tested it - looks working fine with VBox 5.0.10.

MMatten · 2016-10-22T09:53:50Z

The last commit is a great idea 👍 Looks like it would be compatible with the different VB versions. I just tested it - looks working fine with VBox 5.0.10.

This is weird. Today, the additional disk now appears as /dev/sda and the diskpace recipe fails.

May be the device names are unpredictable with different controller types.

javornikolov · 2016-10-22T10:04:44Z

Hmm, I also had some issue with the device name but I thought it was due to running vagrant provision while the vagrant box is already up (versus creating the box from scratch via vagrant up).

javornikolov · 2016-10-22T10:08:34Z

Port and device numbers maybe have some influence: '--port', 1, '--device', 0

Another interesting question is - which is the controller of the disk which already exists in the vm box?

MMatten · 2016-10-22T10:13:54Z

Port and device numbers maybe have some influence: '--port', 1, '--device', 0

I'll try playing with these.

MMatten · 2016-10-22T10:15:20Z

Another interesting question is - which is the controller of the disk which already exists in the vm box?

The base box comes up with "IDE Controller" on my VB version. Do you mean port/device numbers? There's nothing visible in the VB UI.

MMatten · 2016-10-22T10:17:36Z

But I can try these options: http://www.virtualbox.org/manual/ch08.html#vboxmanage-storageattach

javornikolov · 2016-10-22T10:40:30Z

Once upon a time IDE controllers used to lead to hdX* device names instead of sdX. However that changed at some point...

I guess the way to deal with that is to exploit uuid options in some of the storageattach options.

javornikolov · 2016-10-22T11:37:28Z

Here is an idea: if the new controller is of the same type as the existing one (IDE) - maybe it's more likely to come on 2nd place and get named as /sdb*.

javornikolov · 2016-10-22T11:39:59Z

Here is an idea: if the new controller is of the same type as the existing one (IDE) - maybe it's more likely to come on 2nd place and get named as /sdb*.

That unfortunately fails (there can be only one IDE controller):

Stderr: VBoxManage: error: Too many storage controllers of this type

MMatten · 2016-10-22T12:52:48Z

Here is an idea: if the new controller is of the same type as the existing one (IDE) - maybe it's more likely to come on 2nd place and get named as /sdb

I tried that too.

MMatten · 2016-10-22T14:17:38Z

I'll try adding a SCSI controller.

javornikolov · 2016-10-22T14:24:03Z

SCSI controllers naming is also /dev/sd*.
I'm already in a way of writing a Vagrant plugin :-) I'm just exploring https://github.com/sprotheroe/vagrant-disksize as baseline. The idea is to create something similar with following syntax:

config.newdisk.size 4000 # size in MB
config.newdisk.path file_to_disk

Within plugin there is an access to the VM internals.

MMatten · 2016-10-22T14:33:02Z

Well, regardless of the controller type, the additional disk seems to now consistently appear at /dev/sda. May be this is actually reliable after all.

What was the exact device name issue you observed?

The old one has been moved, is a bit older and and has some issues. The new box is effectively 7.2.1511 now. * change vm.box from chef/centos-7.1 * update diskspace recipe: since the volume group name is different

javornikolov · 2016-10-23T14:51:00Z

@MMatten, I'm going to tidy up the commit history in the PR and then I'll force-push.

javornikolov

@MMatten, hopefully we're ready to merge this. Do you think there is anything more to add?

* Limit mysql recipe dependency to version < 8.0 (it's some mess with it).

* disable firewalls to allow host access to guest FitNesse server * disable selinux to deal with the needs of some packages like mysql

lvm cookbook already handles setting up the gem automatically. Such use of chef_gem is reported as deprecated since Chef 12

* Bump up version to 9.6 (settings in Vagrantfile since latest versions are still not in upstream) * Go back to upstream postgresql cookbook

javornikolov · 2016-10-23T16:38:18Z

One thing I noticed is that the vagrant-librarian-chef plugin is not maintained any more. I gave a quick experiment with the berkshelf alternative but for some reason it results in pulling different (lower) versions for some cookbooks. So far vagrant-librarian-chef works fine for us so let's stick to it until we have a good enough replacement.

MMatten · 2016-10-23T17:57:04Z

Nothing else to add. I've tested with everything apart from Teradata and Netezza and as those are nit gusted in our VM I'm guessing no issues there.

javornikolov · 2016-10-23T18:50:52Z

Great :-) Thank you @MMatten!
I'm merging...

I have some progress with the librarian-chef -> bershelf migration but it will be a subject of separate PR.

Use explicit database_name parameter when creating postgres changelog table. The issue has been introduced with 904ced7 (#513).

Use explicit database_name parameter when creating postgres changelog table. The issue has been introduced with 904ced7 (dbfit#513).

MMatten mentioned this pull request Oct 20, 2016

Test guest VM fails to mount host filesystem to /var/dbfit #510

Closed

MMatten force-pushed the use-new-testvm-base-box branch from 0b78bc4 to e0f4604 Compare October 20, 2016 11:31

javornikolov reviewed Oct 20, 2016

View reviewed changes

MMatten force-pushed the use-new-testvm-base-box branch from e0f4604 to 7e9876f Compare October 20, 2016 19:11

javornikolov reviewed Oct 20, 2016

View reviewed changes

test_vm: change VB base box to centos/7

e289df8

The old one has been moved, is a bit older and and has some issues. The new box is effectively 7.2.1511 now. * change vm.box from chef/centos-7.1 * update diskspace recipe: since the volume group name is different

javornikolov force-pushed the use-new-testvm-base-box branch from f5b5f17 to 22f2674 Compare October 23, 2016 16:13

javornikolov changed the title ~~Updates for new Vagrant VB base box~~ Test VM updates: base box centos/7, updated cookbooks and packages Oct 23, 2016

javornikolov changed the title ~~Test VM updates: base box centos/7, updated cookbooks and packages~~ Test VM: updated base box (centos/7), cookbooks and packages Oct 23, 2016

javornikolov approved these changes Oct 23, 2016

View reviewed changes

javornikolov and others added 11 commits October 23, 2016 19:30

test_vm: limit mysql recipe < 8.0

ebf21bc

* Limit mysql recipe dependency to version < 8.0 (it's some mess with it).

test_vm: update Cheffile.lock

0de3415

test_vm: disable selinux, iptables, firewalld

f7959da

* disable firewalls to allow host access to guest FitNesse server * disable selinux to deal with the needs of some packages like mysql

test_vm: bump up mysql to 5.7

1d13bd7

test_vm: bump up git to 2.10.1

a709896

test_vm: bump up mysql2 gem version to 0.4.4

f994771

test_vm: rm chef_gem 'di-ruby-lvm' from diskspace.rb

2b2b1d3

lvm cookbook already handles setting up the gem automatically. Such use of chef_gem is reported as deprecated since Chef 12

test_vm: bump up postgresql to 9.6

0b4a174

* Bump up version to 9.6 (settings in Vagrantfile since latest versions are still not in upstream) * Go back to upstream postgresql cookbook

test_vm: go back to upstream swap cookbook

107334d

test_vm: rm resource cloning in mysql.rb CHEF-3694

b595ca9

test_vm: rm resource cloning in postgres.rb CHEF-3694

904ced7

javornikolov force-pushed the use-new-testvm-base-box branch from 22f2674 to 904ced7 Compare October 23, 2016 16:31

javornikolov approved these changes Oct 23, 2016

View reviewed changes

javornikolov added the techdebt label Oct 23, 2016

javornikolov added this to the Next milestone Oct 23, 2016

javornikolov merged commit eb7aeb0 into master Oct 23, 2016

javornikolov deleted the use-new-testvm-base-box branch October 23, 2016 18:51

javornikolov added a commit that referenced this pull request Nov 8, 2016

test_vm: fix postgres changelog table creation

747b825

Use explicit database_name parameter when creating postgres changelog table. The issue has been introduced with 904ced7 (#513).

javornikolov mentioned this pull request Nov 8, 2016

test_vm: fix postgres changelog table creation #519

Merged

javornikolov added a commit that referenced this pull request Nov 8, 2016

test_vm: fix postgres changelog table creation

3355fe0

Use explicit database_name parameter when creating postgres changelog table. The issue has been introduced with 904ced7 (#513).

bhuesemann pushed a commit to bhuesemann/dbfit that referenced this pull request Aug 23, 2020

test_vm: fix postgres changelog table creation

b0d6664

Use explicit database_name parameter when creating postgres changelog table. The issue has been introduced with 904ced7 (dbfit#513).

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Test VM: updated base box (centos/7), cookbooks and packages #513

Test VM: updated base box (centos/7), cookbooks and packages #513

MMatten commented Oct 20, 2016 •

edited by javornikolov

javornikolov left a comment

javornikolov Oct 20, 2016 •

edited

MMatten Oct 20, 2016

javornikolov Oct 20, 2016

MMatten Oct 20, 2016

MMatten commented Oct 20, 2016

javornikolov commented Oct 20, 2016

javornikolov commented Oct 20, 2016

javornikolov Oct 20, 2016

MMatten commented Oct 21, 2016

MMatten commented Oct 21, 2016

javornikolov commented Oct 21, 2016

MMatten commented Oct 21, 2016

javornikolov commented Oct 21, 2016

MMatten commented Oct 21, 2016

javornikolov commented Oct 21, 2016

MMatten commented Oct 22, 2016

javornikolov commented Oct 22, 2016

javornikolov commented Oct 22, 2016

MMatten commented Oct 22, 2016

MMatten commented Oct 22, 2016

MMatten commented Oct 22, 2016

javornikolov commented Oct 22, 2016 •

edited

javornikolov commented Oct 22, 2016

javornikolov commented Oct 22, 2016

MMatten commented Oct 22, 2016

MMatten commented Oct 22, 2016

javornikolov commented Oct 22, 2016

MMatten commented Oct 22, 2016 •

edited

javornikolov commented Oct 23, 2016

javornikolov left a comment

javornikolov commented Oct 23, 2016 •

edited

MMatten commented Oct 23, 2016

javornikolov commented Oct 23, 2016

Test VM: updated base box (centos/7), cookbooks and packages #513

Test VM: updated base box (centos/7), cookbooks and packages #513

Conversation

MMatten commented Oct 20, 2016 • edited by javornikolov

javornikolov left a comment

Choose a reason for hiding this comment

javornikolov Oct 20, 2016 • edited

Choose a reason for hiding this comment

MMatten Oct 20, 2016

Choose a reason for hiding this comment

javornikolov Oct 20, 2016

Choose a reason for hiding this comment

MMatten Oct 20, 2016

Choose a reason for hiding this comment

MMatten commented Oct 20, 2016

javornikolov commented Oct 20, 2016

javornikolov commented Oct 20, 2016

javornikolov Oct 20, 2016

Choose a reason for hiding this comment

MMatten commented Oct 21, 2016

MMatten commented Oct 21, 2016

javornikolov commented Oct 21, 2016

MMatten commented Oct 21, 2016

javornikolov commented Oct 21, 2016

MMatten commented Oct 21, 2016

javornikolov commented Oct 21, 2016

MMatten commented Oct 22, 2016

javornikolov commented Oct 22, 2016

javornikolov commented Oct 22, 2016

MMatten commented Oct 22, 2016

MMatten commented Oct 22, 2016

MMatten commented Oct 22, 2016

javornikolov commented Oct 22, 2016 • edited

javornikolov commented Oct 22, 2016

javornikolov commented Oct 22, 2016

MMatten commented Oct 22, 2016

MMatten commented Oct 22, 2016

javornikolov commented Oct 22, 2016

MMatten commented Oct 22, 2016 • edited

javornikolov commented Oct 23, 2016

javornikolov left a comment

Choose a reason for hiding this comment

javornikolov commented Oct 23, 2016 • edited

MMatten commented Oct 23, 2016

javornikolov commented Oct 23, 2016

MMatten commented Oct 20, 2016 •

edited by javornikolov

javornikolov Oct 20, 2016 •

edited

javornikolov commented Oct 22, 2016 •

edited

MMatten commented Oct 22, 2016 •

edited

javornikolov commented Oct 23, 2016 •

edited