-
Notifications
You must be signed in to change notification settings - Fork 0
/
crashdump-restructuring-in-solaris.html
7 lines (7 loc) · 4.94 KB
/
crashdump-restructuring-in-solaris.html
1
2
3
4
5
6
7
<html><body>
<head>
<title>Crashdump restructuring in Solaris | Oracle Czech techie's adventures Blog</title>
<meta name="publish_date" content="2014-07-24 14:19:58">
</head>
<p>In Solaris 11.2 the crashdump restructuring project changed the way how dump data are stored. Data which are not pure kernel pages now go into separate files. Together with my colleague Sriman we made it to happen.</p><p>The first noticeable change was a change in the layout of the crash directory. The files are stored under <tt>/var/crash/data/uuid/</tt> directory.<br/>The long hexadecimal string (uuid) was added to better align with FMA - it's actually uuid (universally unique ID) of the crash event which can be found in <tt>fmadm faulty</tt> output. Actually, if you look at FMA panic events from earlier versions you can see that the resource string for the event was already designed this way, it's just materialized with this project.</p><p>For example, after 2 panic events the /var/crash directory will look like this:</p><pre>0 -> data/404778fb-88da-4188-f222-8da174d44fa4<br/>1 -> data/6e50417e-95fc-4ab4-e9a8-abbe80bc6b48<br/>bounds<br/>data/<br/> 404778fb-88da-4188-f222-8da174d44fa4/<br/> vmcore-zfs.0<br/> vmcore.0<br/> vmdump-zfs.0<br/> vmdump.0<br/> 6e50417e-95fc-4ab4-e9a8-abbe80bc6b48/<br/> vmdump-zfs.1<br/> vmdump.1</pre><p>The <tt>0</tt>, <tt>1</tt> symlinks maintain the sequential ordering of the old layout.</p><p>The example reflects a configuration when savecore is not automatically run after boot (i.e. <tt>dumpadm -n</tt> is in effect) and the administrator has extracted the first crash dump by hand (running <tt>savecore 0</tt> in <tt>/var/crash/0/</tt> directory). If you take a look at the console after the system rebooted after panic there are commands which you can copy-n-paste to the terminal to perform the extraction.</p><p>The other change in the above example is that there is new <tt>vmcore-zfs.N</tt> file. This is not the only new file which can appear. Depending on dumpadm(1M) configuration there can be files like:</p><ul><li><tt>vmcore.N</tt> - core kernel pages<br/></li><li><tt>vmcore-zfs.N</tt> - ZFS metadata (ZIO buffers)<br/></li><li><tt>vmcore-proc.N</tt> - process pages<br/></li><li><tt>vmcore-other.N</tt> - other pages (ZFS data, free pages)<br/></li></ul><p>By splitting the dump into multiple files it is possible to transfer just <tt>vmcore.N</tt> file for analysis to quickly assess what caused the panic and transfer the rest of the files later on.</p><p>If any of the "auxiliary" files is missing, <tt>mdb</tt> will report it:</p><pre>root@va64-v20zl-prg06:/var/crash/0# mdb 0<br/>mdb: failed to locate file ./vmcore-zfs.0. Contents of ZFS metadata (ZIO buffers) <br/>will not be available<br/>Loading modules: [ unix genunix specfs dtrace mac cpu.generic <br/>cpu_ms.AuthenticAMD.15 uppc pcplusmp zvpsm scsi_vhci zfs mpt sd ip hook neti arp<br/> usba kssl sockfs lofs idm cpc nfs fcip fctl ufs logindmux ptm sppp ipc ]<br/>> ::status<br/>debugging crash dump vmcore.0 (64-bit) from va64-v20zl-prg06<br/>operating system: 5.12 dump.on12 (i86pc)<br/>usr/src version: 23877:f2e76e2d0329:on12_51+48<br/>usr/closed version: 1961:cad017e4c7e4:on12_51+4<br/>image uuid: 404778fb-88da-4188-f222-8da174d44fa4<br/>panic message: forced crash dump initiated at user request<br/>complete: yes, all pages present as configured<br/>dump content: kernel [LOADED,UNVERIFIED] (core kernel pages)<br/> zfs [<b>MISSING</b>] (ZFS metadata (ZIO buffers))<br/>panicking PID: 101575 (not dumped)</pre><p>Another choice is not to dump some of the sections at all. E.g. to dump only pages kernel and currently running process at the time of panic but not ZFS metadata the system can be configured as:</p><pre>dumpadm -c curproc-zfs</pre><p>Also, the <tt>unix.N</tt> file is no longer extracted automatically (it can be done with <tt>-u</tt> option for savecore(1M) if you need the file for some reason) since it is embedded in <tt>vmcore.N</tt> file; mdb will find it automatically.</p><p>How to load the files into mdb with all these files around ? The easiest way how to access an extracted dump is to use just the suffix, i.e.<p><pre> mdb N</pre><p>which will pick up all the files based on metadata in the main <tt>vmcore.N</tt> file. This worked even before this change, except there was just one file (2 if counting <tt>unix.N</tt>).</p><p>It is still possible to specify the files by hand, just remember to put the main file (<tt>vmcore.N</tt>) as the first argument:</p><pre> mdb vmcore.N vmcore-zfs.N ...</pre><p>The other change (which is hard to notice unless you're dealing with lots of crash dump files) was that we laid out the infrastructure in kernel/mdb/libkvm to be properly backwards compatible w.r.t. on-disk crash dump format. As a result mdb will automatically load crash dump files produced in earlier Solaris versions. Currently it supports 3 latest versions.</p>
</body></html>