Create plugin to monitor snapshots

[atc0005]

Overview

In the old codebase this was implemented as two plugins:

• size
• age

Both plugins allowed excluding individual VMs or resource pools as did other plugins in the set. I'm not sure yet whether this project will have two plugins or a shared plugin to handle both items. The check-path project uses a shared plugin approach where monitoring criteria can be specified as needed. If not specified, those thresholds are not checked.

Goals

• accept CRITICAL/WARNING threshold values (with useful default values)
• (IncludeRP) allow restricting VMs to select Resource Pools
• optional User Domain (with automatic selection applied if not given)
• (ExcludeRP) allow excluding a list of Resource Pools
  • reverse mode where VMs from all pools are checked, except for any VMs in this optional list of Resource Pools
• (IgnoreVM) allow excluding a list of individual VMs
• skip cert validation
• emit ManagedObjectReference ID value in the Long Service Output
  • won't be needed for the vast majority of use cases, but could be useful with troubleshooting work

References

• PowerCLI
  • https://code.vmware.com/web/tool/12.1.0/vmware-powercli
  • https://code.vmware.com/doc/preview?id=6330#/doc/Get-Snapshot.html
  • https://vdc-repo.vmware.com/vmwb-repository/dcr-public/d402b7ed-b345-4fda-880d-a48e8885e910/b6ff10a7-3769-4346-8a83-d92d99d6caf3/doc/Snapshot.html
• govmomi
  • vmware/govmomi 1197
  • vmware/govmomi 2243

[atc0005]

I'm not sure yet whether this project will have two plugins or a shared plugin to handle both items. The check-path project uses a shared plugin approach where monitoring criteria can be specified as needed. If not specified, those thresholds are not checked.

I'm going to give the shared binary approach a try, but will likely require that only one set (age or size) be used at a time.

[atc0005]

I'm going to give the shared binary approach a try, but will likely require that only one set (age or size) be used at a time.

Working on this plugin now.

I'm likely going to stick with one focus per plugin. Thus far the idea of one "thing" per plugin has been followed; I think it's worth keeping the expected pattern as it will likely make the most sense for new users of the project plugins.

[atc0005]

I've been spinning my wheels on the "size" aspect of this issue for a number of days now and have hit a wall. I'm able to get the size value for snapshotData files, but not for the files which make up the snapshot content. I've reached out to VMware {code} Slack and official project for some guidance. Hopefully one of those paths will yield some insight.

After digging into the simulator, I found this:

https://github.com/vmware/govmomi/blob/50c576d6470e7ab9803ceb466ac34f936971dfc1/simulator/virtual_machine.go#L548

func (vm *VirtualMachine) addSnapshotLayout(snapshot types.ManagedObjectReference, dataKey int32) {
	for _, snapshotLayout := range vm.Layout.Snapshot {
		if snapshotLayout.Key == snapshot {
			return
		}
	}

	var snapshotFiles []string
	for _, file := range vm.LayoutEx.File {
		if file.Key == dataKey || file.Type == "diskDescriptor" {
			snapshotFiles = append(snapshotFiles, file.Name)
		}
	}

	vm.Layout.Snapshot = append(vm.Layout.Snapshot, types.VirtualMachineFileLayoutSnapshotLayout{
		Key:          snapshot,
		SnapshotFile: snapshotFiles,
	})

	vm.updateStorage()
}

func (vm *VirtualMachine) addSnapshotLayoutEx(snapshot types.ManagedObjectReference, dataKey int32, memoryKey int32) {
	for _, snapshotLayoutEx := range vm.LayoutEx.Snapshot {
		if snapshotLayoutEx.Key == snapshot {
			return
		}
	}

	vm.LayoutEx.Snapshot = append(vm.LayoutEx.Snapshot, types.VirtualMachineFileLayoutExSnapshotLayout{
		DataKey:   dataKey,
		Disk:      vm.LayoutEx.Disk,
		Key:       snapshot,
		MemoryKey: memoryKey,
	})

	vm.LayoutEx.Timestamp = time.Now()

	vm.updateStorage()
}

I think the first is for a deprecated type, the latter is applicable here.

The arguments for the addSnapshotLayoutEx method seem to suggest that having the ManagedObjectReference and dataKey values are sufficient to build a comprehensive listing of files associated with a snapshot. I don't know yet whether those files also include the original data, or just the difference (what I'm after).

[atc0005]

Replicating some notes I posted to the official repo here, for context if nothing else.

---

Hierarchy of types

Levels above and some below skipped for simplicity and due to my ignorance.

• VirtualMachine
  • VirtualMachineSnapshotInfo
    • VirtualMachineSnapshotTree
  • VirtualMachineFileLayoutEx
    • VirtualMachineFileLayoutExFileInfo
    • VirtualMachineFileLayoutExSnapshotLayout

Fields expanded

• vm
  • Name (Used in output)
  • Snapshot
    • RootSnapshotList (VirtualMachineSnapshotTree)
      • Name (Used in output)
      • Id (int32)
      • CreateTime (Used in output)
      • Snapshot (ManagedObjectReference) (e.g., snapshot-229099)
        • Value
          • The specific instance of Managed Object this
            ManagedObjectReference refers to.
          • Links to LayoutEx.Snapshot.Key.Value
      • ChildSnapshotList
        • can be multiple levels deep, or null
  • LayoutEx
    • File
      • type (filtered to snapshotData)
      • Key (e.g., 40)
        • links to LayoutEx.Snapshot.DataKey
      • Size (Used in output; by itself & aggregate)
    • Snapshot
      • DataKey (e.g., 40)
        • links to LayoutEx.File.Key
      • Key (ManagedObjectReference) (e.g., snapshot-229099)
        • Value
          • links to vm.Snapshot.Value

---

Unfortunately this didn't work. I ended up with the size of the vmsn files associated with the snapshots instead of the snapshots themselves.

---

Downloaded, compiled and ran the VMSnapshot C# project from the vSphere Management SDK and it did not report the size of snapshots, so no go on using it as a guide for an equivalent govmomi approach.

I pulled this from page 155 of vSphere Web Services SDK Programming Guide - VMware vSphere 7.0 (U1):

File Extension	Usage	File Description
.vmsd	vmname.vmsd	Virtual machine snapshot file.
.vmsn	vmname.vmsn	Virtual machine snapshot data file.
**.delta.vmdk		Snapshot difference file. A number preceding the extension increases with more snapshots.
**.vmdk		Metadata about a snapshot.
-Snapshot#.vmsn		Snapshot of virtual machine memory. Snapshot size is equal to the size of your virtual machine's maximum memory.

This is starting to look like I'll need to match all of those file types in order to calculate snapshot size. The challenge will be tying the specific files back to a specific snapshot ID value. Digging further.

I still can't escape the feeling that I'm overlooking something obvious.

[atc0005]

I've hit a wall on checking the size of a snapshot, but I've got enough to work with to complete an age-based monitoring plugin. I'll do that, then hit pause on further size-based monitoring until I get further feedback on the GH issue I opened in the vmware/govmomi project repo.