Unable to find uploaded source_image

[phantium]

Hi,

I am experiencing an issue with packer-plugins-ddcloud.
I have uploaded a customer image to use as a source_image, however the plugin does not seem to find it.

Expected result is that it should find the customer image and use this as a source_image to modify (provisioners e.g. Ansible) and build and upload a new customer image on the Dimension Data cloud.

Packer output:

# packer build debian9.json
ddcloud-customerimage output will be in this color.

    ddcloud-customerimage: Resolving datacenter 'EU7'...
    ddcloud-customerimage: Resolved datacenter 'EU7'.
==> ddcloud-customerimage: Image 'Debian9-base' not found in datacenter 'EU7'.
Build 'ddcloud-customerimage' errored: One or more steps failed to complete

Used builder configuration:

  "builders": [{
        "type": "ddcloud-customerimage",
        "mcp_region": "EU",
        "mcp_user": "MYLOGIN",
        "mcp_password": "MYPASSWORD",
        "datacenter": "EU7",
        "networkdomain": "develop",
        "vlan": "prod-vlan-120",
        "source_image": "Debian9-base",
        "target_image": "Debian 9 test",
        "use_private_ipv4": "true",
        "communicator" : "ssh"
    }

Thanks in advance!

[tintoy]

Hi looks like it should work - I'll check this out first thing tomorrow (am in AU and it's almost bedtime)

[tintoy]

Ok, had a glance at the code and it looks like you've hit a bug - will sort it out first thing tomorrow; sorry for the inconvenience :)

[phantium]

Thank you!

[tintoy]

No problem :)

In the meanwhile, would you mind running packer again with the following environment variables set and then attaching the resulting log file to this issue? I'm off to bed but will have a look tomorrow.

export PACKER_LOG=1
export PACKER_LOG_PATH=$PWD/packer.log
export MCP_EXTENDED_LOGGING=1

[phantium]

packer.log

I have installed your latest release beta4 which shows that it is unable to find an OS Image or Customer Image named Debian9-base. Which I did upload as a customer image.

[tintoy]

From the log, it looks like the CloudControl API says there is no image, but I suspect that this relates to guest OS customisation (older API doesn't return images with GOC disables), had to deal with this in our Terraform provider a while back. Will check it out tomorrow

[tintoy]

Ok, I'm going to try updating to the current version of the CloudControl client library (should be able to see images that have GOC disabled) and rebuild. Will post here when a new release is out.

[tintoy]

Ok, so I'm about to create the release (thanks for bearing with me). This won't actually fix your problem but will at least confirm for us that the problem relates to the newer style of images (where you can disable GOC). If that's the case, it should only take a couple of hours to implement.

Sorry about that, BTW - it looks like a case of our Packer plugins not keeping up with new features in CloudControl (I didn't realise anyone was using the plugins, but if you are then I'm happy to keep them up to date).

[tintoy]

Ok @phantium, could you try running v1.0.3-beta5 with logging and posting the log, please?

[phantium]

Just woke up and about to go to work, will run it immediately once I get there.

[phantium]

I have run beta5, it seems to work alright. But cleanup on failure goes wrong.
I still have a network issue as ddcloud does not seem to support DHCP for some reason, hoping to figure out how to get around that...

packer.log

[tintoy]

Yeah, they don't support DHCP unfortunately - I've been caught out by that one too 😁

Fortunately, I wrote a service that'll do what you want:

https://github.com/DimensionDataResearch/mcp2-dhcp-server

[tintoy]

(sorry, I know it's slightly more awkward than native support within CloudControl but once it's set up, it's pretty much set-and-forget)

[tintoy]

(just turn off PXE / iPXE features as needed)

[phantium]

Thanks 👍

[tintoy]

No worries - give us a yell if you have any trouble with it.

[phantium]

I am a bit disappointed as to needing to run a VM with a DHCP server in order to deploy VMs naturally.
How do the Dimension Data images work to circumvent this?

[tintoy]

Yeah, it's not the best :-/

As I understand it (I'm not part of the team that does the MCP and CloudControl), CloudControl uses VMWare's Guest OS Customisation facility to achieve configuration of stuff like IP addresses (more here and distro support matrix here).

When you initially imported your image, were you presented with a choice to enable / disable guest OS customisation? I could be wrong, but if not then you might be using a distro that VMWare doesn't know how to customise.

[tintoy]

I'll reach out to the relevant team to confirm this, BTW.

[phantium]

Thanks I'll give that a try, yes I did disable guest OS customisation as I did not have vmware tools installed.

[tintoy]

Ah, sorry, just looked through the support matrix myself - looks like Debian is supported, but not for GOC :(

[tintoy]

So if you're using Debian, you'll probably need either DHCP or static IPs baked into the image (yuck).

[phantium]

I tried to fake it being Ubuntu 16.04 instead, giving the following error:

ddcloud-customerimage output will be in this color.

    ddcloud-customerimage: Resolving datacenter 'EU7'...
    ddcloud-customerimage: Resolved datacenter 'EU7'.
    ddcloud-customerimage: Deploying server 'packer-build-bddaa7002f' in network domain 'develop' ('e7e74af0-6dcd-41a9-8c12-9e60ad24c6a5')...
==> ddcloud-customerimage: Request to deploy server 'packer-build-bddaa7002f' failed with status code 400 (INVALID_INPUT_DATA): administratorPassword must not be provided if the imageId corresponds to a Linux Customer Image or Windows 2003 Customer Image.
Build 'ddcloud-customerimage' errored: unexpected EOF

==> Some builds didn't complete successfully and had errors:
--> ddcloud-customerimage: unexpected EOF

Not quite sure what this is supposed to mean.

[phantium]

I did not provide "initial_admin_password" as part of the builder in Packer.

[tintoy]

Ah - ok, that might be a bug (or undefined behaviour at least); let me have a look.

[phantium]

I prefer httpie or postman as well.

[tintoy]

Try this:

http get 'https://api-EU.dimensiondata.com/caas/2.5/{orgId}/server/server/{serverId}' Accept:application/json --auth-type basic --auth 'user:password'

Where {orgId} is, from the logs, 799dda5b-93ba-411c-a2a3-61c4c6e20c54, and {serverId} is the Id of the server (you can see that in the CloudControl UI).

[phantium]

Thanks, that resulted in giving me the following information.

    "progress": {
        "action": "DEPLOY_SERVER",
        "numberOfSteps": 14,
        "requestTime": "2017-11-09T07:54:52.000Z",
        "step": {
            "name": "WAIT_FOR_GUEST_IP_ADDRESS",
            "number": 11
        },
        "updateTime": "2017-11-09T08:29:00.000Z",
        "userName": "x"
    },

Doesn't really say why, but I guess I should make a ticket with this information.

[tintoy]

Yeah - that part of the process is a little opaque, but essentially it means that the server has booted but has not picked up the configured IP address. I believe the customisation process modifies stuff in /etc and if your distro has unexpected stuff in there it may not be successful in doing so. I'd say at this stage yes, the best option is to raise a ticket; someone with knowledge of the system internals will need to take a look at it.

[tintoy]

BTW, you can see a list of the steps and what they do here (search for "Guest OS Customization Server Deployment Process").

(sorry I couldn't more directly helpful but I have no deeper access to the system than you do)

[tintoy]

BTW, it looks you do need to have VMWare tools or open-vm-tools installed for GOC to work correctly.

[phantium]

I did have them installed, I am seeing the following issue still when using Ubuntu 16.04 as a base image.

    ddcloud-customerimage: Destroying server 'packer-build-c2da04e43b' ('6a2c6ef9-ddd0-4e41-88a5-0717e9ae7053')...
==> ddcloud-customerimage: Request to delete server failed with unexpected status code 400 (SERVER_STARTED): Server with id 6a2c6ef9-ddd0-4e41-88a5-0717e9ae7053 is started but must be stopped to perform this operation. Please Power Off or Shutdown the Server (as appropriate) and try again.
Build 'ddcloud-customerimage' errored: unexpected EOF

Could you make it so that it forces deletion or stops and then deletes?

It causes Packer to fail.

[tintoy]

Oops, I would have though it would already do that - I'll look into it first thing tomorrow. Thanks for sticking with it!

[tintoy]

I've made the change (turns out it was done correctly in another step, but not this one), but have to get to bed; will create a release as soon as I wake up (sorry about that).

[tintoy]

Never mind - just created beta9 release. Go for it. Will be back online tomorrow morning :)

PS. If you still have any problems, attach a log and I'll see what's going on.

[phantium]

Thanks!! Sleep well :)

[phantium]

packer.log
Getting a segmentation fault now.

[tintoy]

😭

[tintoy]

Believe it or not, we're making progress! The server deploy / destroy is working now and it's the destroying the firewall rule that is causing problems. Looking into it now.

[tintoy]

Turns out the code path for use_private_ipv4 hasn't been used before and it was trying to delete a non-existent firewall rule. Fixing now.

[tintoy]

Right, beta10 is ready to go - fingers crossed this is the last release you'll have to try.

[phantium]

Thanks, sorry for the lack of response. You are a day ahead of us and by the time you are at work I am home and unable to run Packer. 😄 I am running it now and waiting for the results!

[tintoy]

No worries - I'm used to the time-zone thing and besides, it seems like you're the one who's having to wait for me rather than the other way around :)

[phantium]

The build has finished succesfully now!

Build 'ddcloud-customerimage' finished.

==> Builds finished. The artifacts of successful builds are:
--> ddcloud-customerimage: Customer image 'Ubuntu 16.04 test' ('ea3e0f47-d598-4135-8943-0f5b60c11d67') in datacenter 'EU7'.

[tintoy]

Sorry again for all the trouble; I built this more than a year ago, and nobody used it so it didn't receive much testing. I'll give it a little love in the coming months if I can (will probably get it merged into Packer as a built-in module).

[tintoy]

(let me know if the image deploys successfuly, BTW)

[phantium]

No worries! Glad to be able to help test it! In fact used to do a lot of testing for companies, and worked in software development myself.

I'll test if the image deploys now, I didn't do much to it besides install python and run Ansible debug returning which network it's in.

[tintoy]

If it helps, BTW, there's a libcloud driver for CloudControl, and some (rather basic because we've had a couple of PRs stuck in limbo for a while) Ansible modules for it too.

[phantium]

Haven't heard of it before, what is it's use case?

[tintoy]

Libcloud's a Python library that provides an abstraction over most of the cloud providers out there. If you can write Python it's not a bad way to automate things (it's how we wrote those Ansible modules, for example).

https://docs.mcp-services.net/display/LPC/LibCloud+Python+Client

[phantium]

Ah cool, thanks I'll give it a look! Yes I do write Python.. for quite a few years already, but you probably checked my profile? 👍

[tintoy]

Yep, that's why I suggested it 😉