Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

provisioning of new zones on live image 20121130T194746Z #148

Closed
MerlinDMC opened this Issue Dec 2, 2012 · 4 comments

Comments

Projects
None yet
2 participants
Contributor

MerlinDMC commented Dec 2, 2012

After updating to the latest release new zones get provisioned, change to running state and are accessible via zlogin. After about 4 minutes they automatically shut down, get marked as "failed" and switch to state uninitialized.

That's reproducible using the vmware image - here is the console output plus the output of /usr/vm/sbin/zoneevent:

console:

[root@00-0c-29-18-ec-10 ~]# imgadm update
updating local images database...
Get https://datasets.joyent.com/datasets...
done
[root@00-0c-29-18-ec-10 ~]# imgadm import 60a3b1fa-0674-11e2-abf5-cb82934a8e24
60a3b1fa-0674-11e2-abf5-cb82934a8e24 doesnt exist. continuing with install
60a3b1fa-0674-11e2-abf5-cb82934a8e24 successfully installed
image 60a3b1fa-0674-11e2-abf5-cb82934a8e24 successfully imported
[root@00-0c-29-18-ec-10 ~]# vmadm create << EOF
> {
>   "brand": "joyent",
>   "image_uuid": "60a3b1fa-0674-11e2-abf5-cb82934a8e24",
>   "autoboot": true,
>   "alias": "test",
>   "max_physical_memory": 256,
>   "max_swap": 512,
>   "nics": [
>     {
>       "nic_tag": "admin",
>       "ip": "192.168.1.112",
>       "netmask": "255.255.255.0",
>       "gateway": "192.168.1.1",
>       "primary": true
>     }
>   ]
> }
> EOF
Successfully created 31fd6a56-91e5-494a-813c-05b49935680f
[root@00-0c-29-18-ec-10 ~]# vmadm list
UUID                                  TYPE  RAM      STATE             ALIAS
31fd6a56-91e5-494a-813c-05b49935680f  OS    256      running           test

zoneevent:

{"zonename": "31fd6a56-91e5-494a-813c-05b49935680f", "newstate": "initialized", "oldstate": "uninitialized", "zoneid": "1", "when": "1354473021921918505", "channel": "com.sun:zones:status", "class": "status", "subclass": "change"}
{"zonename": "31fd6a56-91e5-494a-813c-05b49935680f", "newstate": "ready", "oldstate": "initialized", "zoneid": "1", "when": "1354473021923283364", "channel": "com.sun:zones:status", "class": "status", "subclass": "change"}
{"zonename": "31fd6a56-91e5-494a-813c-05b49935680f", "newstate": "ready", "oldstate": "ready", "zoneid": "1", "when": "1354473022274936395", "channel": "com.sun:zones:status", "class": "status", "subclass": "change"}
{"zonename": "31fd6a56-91e5-494a-813c-05b49935680f", "newstate": "running", "oldstate": "ready", "zoneid": "1", "when": "1354473022276100146", "channel": "com.sun:zones:status", "class": "status", "subclass": "change"}
{"zonename": "31fd6a56-91e5-494a-813c-05b49935680f", "newstate": "shutting_down", "oldstate": "running", "zoneid": "1", "when": "1354473064430294385", "channel": "com.sun:zones:status", "class": "status", "subclass": "change"}
{"zonename": "31fd6a56-91e5-494a-813c-05b49935680f", "newstate": "shutting_down", "oldstate": "shutting_down", "zoneid": "1", "when": "1354473065437023466", "channel": "com.sun:zones:status", "class": "status", "subclass": "change"}
{"zonename": "31fd6a56-91e5-494a-813c-05b49935680f", "newstate": "shutting_down", "oldstate": "shutting_down", "zoneid": "1", "when": "1354473065630964013", "channel": "com.sun:zones:status", "class": "status", "subclass": "change"}
{"zonename": "31fd6a56-91e5-494a-813c-05b49935680f", "newstate": "shutting_down", "oldstate": "shutting_down", "zoneid": "1", "when": "1354473065746953643", "channel": "com.sun:zones:status", "class": "status", "subclass": "change"}
{"zonename": "31fd6a56-91e5-494a-813c-05b49935680f", "newstate": "uninitialized", "oldstate": "shutting_down", "zoneid": "1", "when": "1354473065747310520", "channel": "com.sun:zones:status", "class": "status", "subclass": "change"}
{"zonename": "31fd6a56-91e5-494a-813c-05b49935680f", "newstate": "initialized", "oldstate": "uninitialized", "zoneid": "2", "when": "1354473066257171977", "channel": "com.sun:zones:status", "class": "status", "subclass": "change"}
{"zonename": "31fd6a56-91e5-494a-813c-05b49935680f", "newstate": "ready", "oldstate": "initialized", "zoneid": "2", "when": "1354473066259245960", "channel": "com.sun:zones:status", "class": "status", "subclass": "change"}
{"zonename": "31fd6a56-91e5-494a-813c-05b49935680f", "newstate": "ready", "oldstate": "ready", "zoneid": "2", "when": "1354473066758332489", "channel": "com.sun:zones:status", "class": "status", "subclass": "change"}
{"zonename": "31fd6a56-91e5-494a-813c-05b49935680f", "newstate": "running", "oldstate": "ready", "zoneid": "2", "when": "1354473066782835287", "channel": "com.sun:zones:status", "class": "status", "subclass": "change"}

after about 4 minutes delay (without doing anything in the GZ or the provisioned zone state switches:

console:

[root@00-0c-29-18-ec-10 ~]# vmadm list
UUID                                  TYPE  RAM      STATE             ALIAS
31fd6a56-91e5-494a-813c-05b49935680f  OS    256      failed            test

zoneevent:

{"zonename": "31fd6a56-91e5-494a-813c-05b49935680f", "newstate": "shutting_down", "oldstate": "running", "zoneid": "2", "when": "1354473320596502176", "channel": "com.sun:zones:status", "class": "status", "subclass": "change"}
{"zonename": "31fd6a56-91e5-494a-813c-05b49935680f", "newstate": "shutting_down", "oldstate": "shutting_down", "zoneid": "2", "when": "1354473321617987860", "channel": "com.sun:zones:status", "class": "status", "subclass": "change"}
{"zonename": "31fd6a56-91e5-494a-813c-05b49935680f", "newstate": "shutting_down", "oldstate": "shutting_down", "zoneid": "2", "when": "1354473321619438578", "channel": "com.sun:zones:status", "class": "status", "subclass": "change"}
{"zonename": "31fd6a56-91e5-494a-813c-05b49935680f", "newstate": "shutting_down", "oldstate": "shutting_down", "zoneid": "2", "when": "1354473321640351710", "channel": "com.sun:zones:status", "class": "status", "subclass": "change"}
{"zonename": "31fd6a56-91e5-494a-813c-05b49935680f", "newstate": "uninitialized", "oldstate": "shutting_down", "zoneid": "2", "when": "1354473321640630140", "channel": "com.sun:zones:status", "class": "status", "subclass": "change"}
Contributor

MerlinDMC commented Dec 2, 2012

it looks like this commit fixes the issue MerlinDMC/smartos-live@537b8f1

@joshwilsdon should the joyent brand not have this file created while provisioning? If this was missing intentionally then something else seems to be wrong with 1663802

Owner

joshwilsdon commented Dec 3, 2012

It was intentional that the joyent brand we should not have
var_svc_provisioning. The reason for this is that when you use the joyent
brand you rely on the zoneinit to setup the zone and older datasets will
have a zoneinit that would never enable the services which will lead to the
zone to come out of provisioning. So you'll see this logic:

if (BRAND_OPTIONS[vmobj.brand].features.var_svc_provisioning) {
// these brands always handle /var/svc/provisioning on this
// platform.
var_svc_provisioning = true;
} else {
if (zoneinit.hasOwnProperty('features')) {
if (zoneinit.features.var_svc_provisioning) {
VM.log.debug('features.var_svc_provisioning == true');
var_svc_provisioning = true;
}
} else {
// didn't load zoneinit features, so check for datasets that
// have 04-mdata.sh
filename = path.join(vmobj.zonepath,
'/root/root/zoneinit.d/04-mdata.sh');
if (fs.existsSync(filename)) {
VM.log.debug('/root/zoneinit.d/04-mdata.sh exists');
var_svc_provisioning = true;
} else {
VM.log.debug('/root/zoneinit.d/04-mdata.sh does not '
+ 'exist');
}
}
}

which detects whether the zoneinit in the dataset will be ever moving the
provisioning file to indicate the end of provisioning. When those files
don't exist we rely on the old method which is to wait for the zone to
reboot after zoneinit runs.

One big question I have is how this platform was built which I will have to
check on tomorrow. This build appears to have several changes which were
not intended to be in the 2012-11-29 release and indeed are intentionally not
in the release branch:

https://github.com/joyent/smartos-live/commits/release-20121129

Anyway I've just reproduced this on my system and figured out what's going
on. Apparently in the smartos-1.8.1 dataset, zoneinit has been moved to
/var/zoneinit which I did not know. The information I had was that the
/root/zoneinit.d/ the place for these files in all the datasets we had
released. I'll have to work with the people building datasets to figure
out if they're going to leave it in /var/zoneinit or whether this was just
a mistake in this dataset. Either way I'll try to get it fixed asap.

If you want to work around this for now, you can use MerlinDMC's patch if
you don't want to provision any old datasets such as smartos-1.4.x or
earlier. Those would break with this patch. Another solution would be to
just run:

mkdir -p /zones/60a3b1fa-0674-11e2-abf5-cb82934a8e24/root/root/zoneinit.d;
touch /zones/60a3b1fa-0674-11e2-abf5-cb82934a8e24/root/root/zoneinit.d/04-mdata.sh

which I've confirmed will allow the zone to be created properly but will
leave that directory and file around in your zone after provisioning so
you'd have to delete it.

Thanks,
Josh

@ghost ghost assigned joshwilsdon Dec 3, 2012

Contributor

MerlinDMC commented Dec 3, 2012

zoneinit moved to /var/zoneinit while switching from smartos(64) to base(64) datasets

all is packaged up in the smtools package that gets installed to /opt/local/lib/zoneinit ... sm-prepare-image puts a copy of those files to /var/zoneinit

Owner

joshwilsdon commented Dec 3, 2012

This should be fixed with baa6e13.

@joshwilsdon joshwilsdon closed this Dec 3, 2012

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment