[sled agent] Consider setting uniform coreadm values to extract info from terminating processes?

[smklein]

In the 8/16 control plane sync, we discussed the possibility of using https://illumos.org/man/8/coreadm to set a filter to extract core files from crashing non-global zones into the global zone.

Currently, when non-global zone services terminate, Sled Agent stops and deletes the underlying zone. This helps avoid leakage of that resource - we have no further execution-time usage for it - but limits visibility.

By dumping core files into the global zone, we'd be able to inspect errors, even after the zone is destroyed.

[rmustacc]

In particular, we want to enable global cores and use the %z token to include the zone name for disambiguation. In the past we've done things like /var/xxx/%z/core.%f.%p.

[jclulow]

A request: probably please don't hard code /var paths (or use of rpool specifically) for any more large files. It seems fine as a default but in the ramdisk environment we're going to want to direct those files to specific tmpfs or other mounted pools, etc. The current use of /var/oxide for writing a bunch of larger files is something we'll probably have to unwind so it'll be good to avoid adding more things like that.

[leftwo]

In the past we've done things like /var/xxx/%z/core.%f.%p.

coreadm wants the directory to exist before it will create a core. If we use /%z/ as part of the path, then I believe something outside coreadm will need to create that directory. In previous use cases, was there another subsystem that created the %z directory, or did I just miss the coreadm option that would create on demand?

[davepacheco]

I believe that in the case being referenced (Joyent's SmartOS), the path was really /zones/%z/cores/core.%f.%p, and yes, that directory was created by the machinery that created the zone (vmadm(1M)).

[leftwo]

I believe that in the case being referenced (Joyent's SmartOS), the path was really /zones/%z/cores/core.%f.%p, and yes, that directory was created by the machinery that created the zone (vmadm(1M)).

https://www.illumos.org/issues/2123
That vmadm?

[jclulow]

If we need to create a cores directory we can do that in the brand code. It has hooks for installing and for booting and so on.

[leftwo]

If we do want a .../%z/... in the path, then something will need to both create that directory on zone creation, and remove it (if empty) when the zone goes away. Otherwise we are left with a record of every zone ever created.

[davepacheco]

I believe that in the case being referenced (Joyent's SmartOS), the path was really /zones/%z/cores/core.%f.%p, and yes, that directory was created by the machinery that created the zone (vmadm(1M)).

https://www.illumos.org/issues/2123 That vmadm?

I expect so. I'm not sure where the path was actually managed, though. Maybe as Josh suggested it was done in the brand code.