Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add images build capability to vsphere #1183

Closed

Conversation

mynameisrufus
Copy link
Contributor

Successfully cloning machines based on images in the hosts ui.

Cluster option currently not working because of the way the fog provider is attempting to retrieve the resourcePool.

http://projects.theforeman.org/issues/2438

@ohadlevy
Copy link
Member

[test]

@domcleal
Copy link
Contributor

Thanks for the contribution @mynameisrufus, this will be much appreciated by our users.

I'm worried about the approach though, since it's a new workflow for something that already exists in Foreman. There are currently two provisioning methods for new hosts on compute resources (the code refers to these as capabilities): build and images, so the VMware CR currently only implements the "build" method (PXE).

I was imagining that implementing hosts from templates would use the existing images mechanism, rather than adding a new VMware-specific workflow.

Have you had a look at this method - what do you think?

@domcleal
Copy link
Contributor

#1184 is a really good example of adding image provisioning to oVirt - many of the general UI and JS fixes that @jimmidyson's added in the process would be useful here.

@mynameisrufus
Copy link
Contributor Author

Thanks @domcleal, I was looking for feedback on this as I have only just started looking at the codebase. I will update this PR over the coming week.

@domcleal
Copy link
Contributor

Thanks @mynameisrufus. Please just ping me when you'd like another look, or if you have any questions.

@mynameisrufus
Copy link
Contributor Author

@domcleal I think I have this working at some level, I am successfully cloning vms through hosts but my request times out and a Host does not get created but the vm does. What is the approach for long running requests?

I also needed to make this patch to fog https://github.com/fog/fog/pull/2642/files

@dLobatog
Copy link
Member

dLobatog commented Feb 4, 2014

@mynameisrufus You can possibly try https://github.com/iNecas/foreman-tasks to get your cloning (or host creation) to run asynchronously.

@mynameisrufus
Copy link
Contributor Author

@elobato that approach seems to be solid however this PR will be used by everybody so adding that dependency does not seem like an option?

@domcleal
Copy link
Contributor

domcleal commented Feb 6, 2014

Yeah, relying on foreman-tasks isn't an option at the moment. Maybe in future releases.

I'm surprised it's so slow - how long are you talking, minutes just for VM creation? Is there any sort of COW functionality we can use instead?

@jimmidyson
Copy link
Contributor

@domcleal There are multiple places where timeouts can affect the creation of systems. We hit timeouts in DHCP reservations for example, this was limited to 60 seconds (hardcoded in lib/proxy_api/resource.rb), but due to the configuration of our DHCP server managing multiple subnets this was taking a long time.

Also, it would be have the option to request a host creation asynchronously, especially through the API. Kick off host creation, keep checking for progress, or use the tasks API to keep track.

@domcleal
Copy link
Contributor

domcleal commented Feb 6, 2014

@jimmidyson indeed, we're working on foreman-tasks etc to fix these very problems. We're not going to add it as a dependency today though.

Successfully cloning machines based on images in the hosts ui.
@mynameisrufus
Copy link
Contributor Author

@domcleal we've won! Please review. The Fog adaptor is a little beta, I stayed away from altering it so this PR can actually get merged.

@domcleal
Copy link
Contributor

Nice, let's [test].

# +searchIndex.FindChild("Resources")+ in RbVmomi that then returns nil
# because it has no children.
def clone_vm args
vm = client.servers.get(args["image_id"], datacenter)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm having issues with this line, and the one below for looking up the template by the image_id, as neither of them return the template "VM". Which identifier are you using when creating your images?

Related is that I tried implementing #available_images (so it shows a list of templates in the "new image" form) for Foreman::Model::Vmware by simply returning client.templates (with fog/fog#2702) and it returns this:

=> [  <Fog::Compute::Vsphere::Template
    id="5030c304-48b1-b2ce-867e-5ab4a8889637",
    name="dcleal-tmpl-rhel6",
    uuid="4230b500-017b-e739-4cf4-cec65f389c78"
  >,

Though none of the three IDs work for either client.servers.get or client.templates.get. Any ideas how this should work?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right now, it looks like the 'template' has actually to be a real VM in the list of servers. So the ID must be the UUID of an actual VM. Though, if we could get the list of actual VmWare template through client.templates, it would be great

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Our use case is currently to build a VM with packer and clone off that, in this case the VM is not a template but we are treating as such.

@domcleal due to some lack of implementation around the clone code in fog (network device defaults to VIM::VirtualE1000 but we need VIM::VirtualVmxnet3) we have rolled our own microservice using rbvmomi directly. A pull request exists on fog that has been open for a year that addresses many of these things but it has diverged too much and I don't think it will ever make it in. The fog chaps highest priority is to create a core gem with adapter gems to make the codebase not so monolithic, I would like to work on the fog-vshpere gem when it drops but until then you might want to close this PR.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, that makes sense, thanks. The more I look at the fog code, the less likely it looks to work with templates anyway - I just can't see any capability for cloning or creating VMs from templates, so I think this'll have to do for now.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or not, it looks like calling client.vm_clone() with "template_path" => "/foo" should work, but I can't get it to work for my template that's in a few folders deep, having tried lots of permutations.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ugh, it does actually work, when you spell the template_path correctly. But I don't think we have the relative path to templates available from vSphere for this to work end to end, as Fog's template object only stores the name.

@domcleal
Copy link
Contributor

I also changed app/views/images/_form.html.erb:9 to read:

<% if ['Ovirt', 'Vmware'].include?(@compute_resource.provider) %>

(so you could set a password for SSH finish provisioning)

@RonnyMaas
Copy link

Any News about this, because I was hoping this would become available in foreman 1.4.2 :)

@domcleal
Copy link
Contributor

We won't include new features like this in a point release (.2), but if we can get it into a mergeable state by the end of March, then it could be released in 1.5.0.

@RonnyMaas
Copy link

Ok, thanks for the info. Thought It was panned for 1.4.2 because it is an item in sprint 21 as Feature #1966

@domcleal domcleal self-assigned this Mar 19, 2014
"network_adapter_device_key" => network_adapter_device_key
}

vm.relative_path = vm.folder.path.gsub(path_replace, '')
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not working for me. Folders are present only in relative_path attribute (which starts with dc name), and this line is overridding it. As such this can work only if the VM to be cloned is in /.
This is working for me :
dc_replace = /#{datacenter}//
vm.relative_path = vm.folder.path.gsub(path_replace, '')+vm.relative_path.gsub(dc_replace,'')

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@couniojc I'm not quite sure how your fix works, it produces a string like this for me, doubling up the name:

2.0.0-p247 :018 > vm.folder.path.gsub(path_replace, '')+vm.relative_path.gsub(dc_replace,'')
 => "Scratch/Training - SA3Scratch/Training - SA3" 

The original seems to do something sensible to me:

2.0.0-p247 :019 > vm.folder.path.gsub(path_replace, '')
 => "Stratch/Training - SA3" 

Or even just vm.relative_path.gsub(dc_replace,'') has the same result.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@domcleal path is '/' and vm.folder.path is '/Datacenters/{dcname}/vm/' for all the VMs on my side. So relative_path is the only variable that has the folders.

I suppose yours are correct then

Strangely, the folders (path attribute) are correct when listed in 'virtual machines' list but in this case data is coming from MiniServers.

If it works for you, then that's fine, you can drop that one. If it still happens in 1.5.0, at worse we can live with VMs at root for a while (or I will patch our install)

@domcleal
Copy link
Contributor

domcleal commented Apr 9, 2014

I've created this branch which changes the behaviour to enable cloning of templates by entering the path to the template on the new image form (I think the same will work for a VM too, actually). What do you think?

https://github.com/domcleal/foreman/compare/vmware-clone-template

I agree with @couniojc's comments on the issue, this is useful enough to get into 1.5 in time.

@couniojc
Copy link

couniojc commented Apr 9, 2014

@domcleal Thanks ! I tested with the new branch and it's working on my side ! Excellent !

@domcleal
Copy link
Contributor

domcleal commented Apr 9, 2014

Thanks for testing @couniojc, much appreciated!

Merged as 3304049 for Foreman 1.5.0. Thanks @mynameisrufus - you've solved our highest-voted issue!

@domcleal domcleal closed this Apr 9, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
7 participants