invalid memory address or nil pointer dereference azure.mapFromVMScaleSetVM

[McHammar]

What did you do?

The prometheus pod can't start anymore and gets this error message before dying

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x50 pc=0x1211d1b]
goroutine 762 [running]:
github.com/prometheus/prometheus/discovery/azure.mapFromVMScaleSetVM({0xc0447b9710, 0x0, 0x0, 0x0, 0x0, 0xc0447b9740, 0xc0447b9720, 0xc0447b9730, {0x0, 0x0, ...}, ...}, ...)
	/app/discovery/azure/azure.go:609 +0x5b
github.com/prometheus/prometheus/discovery/azure.(*azureClient).getScaleSetVMs(0xc04b804540, {0x3f6e318, 0xc04abdf7c0}, {0xc07b482290, 0x0, 0x0, 0x0, 0xc0796e28c0, 0xc07b1e5410, 0x0, ...})
	/app/discovery/azure/azure.go:562 +0x49b
github.com/prometheus/prometheus/discovery/azure.(*Discovery).refresh(0xc04ab9f890, {0x3f6e318?, 0xc04abdf7c0})
	/app/discovery/azure/azure.go:354 +0x8b0
github.com/prometheus/prometheus/discovery/refresh.(*Discovery).refresh(0xc04ab93a40, {0x3f6e318, 0xc04abdf7c0})
	/app/discovery/refresh/refresh.go:121 +0xcf
github.com/prometheus/prometheus/discovery/refresh.(*Discovery).Run(0xc04ab93a40, {0x3f6e318, 0xc04abdf7c0}, 0xc04a2bf140)
	/app/discovery/refresh/refresh.go:77 +0x66
created by github.com/prometheus/prometheus/discovery/legacymanager.(*Manager).startProvider in goroutine 107
	/app/discovery/legacymanager/manager.go:200 +0x3a5

If I remove all Azure sd configs the pod is able to start again

What did you expect to see?

Pod starting

What did you see instead? Under which circumstances?

Crashloop

System information

AKS 1.27.3

Prometheus version

v2.49.1

Prometheus configuration file

No response

Alertmanager version

No response

Alertmanager configuration file

No response

Logs

No response

[McHammar]

If the azure service discovery finds a VM scale set then we get this error. If the service discovery is for a subscription without one it still works

[darshanime]

from the stacktrace, the issue seems to be here. let me raise a PR

 func mapFromVMScaleSetVM(vm armcompute.VirtualMachineScaleSetVM, scaleSetName string) virtualMachine {
-	osType := string(*vm.Properties.StorageProfile.OSDisk.OSType)
+	osType := getOSTypeFromVirtualMachineScaleSetVM(vm)
 	tags := map[string]*string{}
 	networkInterfaces := []string{}
 	var computerName string
@@ -702,3 +702,23 @@ func (d *Discovery) getFromCache(nicID string) (*armnetwork.Interface, bool) {
 	net, found := d.cache.Get(nicID)
 	return net, found
 }
+
+func getOSTypeFromVirtualMachineScaleSetVM(vm armcompute.VirtualMachineScaleSetVM) string {
+	if vm.Properties == nil {
+		return ""
+	}
+
+	if vm.Properties.StorageProfile == nil {
+		return ""
+	}
+
+	if vm.Properties.StorageProfile.OSDisk == nil {
+		return ""
+	}
+
+	if vm.Properties.StorageProfile.OSDisk.OSType == nil {
+		return ""
+	}
+
+	return string(*vm.Properties.StorageProfile.OSDisk.OSType)
+}

[bboreham]

Closed by #13578

[dpericaxon]

Hello I just updated to v2.50.1 from v2.49.1 and started seeing an error that looked similar:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x123e50a]

goroutine 1605 [running]:
github.com/prometheus/prometheus/discovery/azure.(*Discovery).refresh.func1({{0xc180dc21e0, 0x92}, {0xc180db2de0, 0xe}, {0xc180db3362, 0xe}, {0xc179d65620, 0x21}, {0xc180db2df0, 0xd}, ...})
	github.com/prometheus/prometheus/discovery/azure/azure.go:438 +0xb4a
created by github.com/prometheus/prometheus/discovery/azure.(*Discovery).refresh in goroutine 1174
	github.com/prometheus/prometheus/discovery/azure/azure.go:375 +0xa55

Is it possibly related? Has anyone else thats updated seen this?

[bboreham]

I think it's more like #13643, which was a bad merge, fixed by #13702.

There is a release candidate for 2.51.0 out now which has this fix.

[dpericaxon]

I think it's more like #13643, which was a bad merge, fixed by #13702.

There is a release candidate for 2.51.0 out now which has this fix.

Great! Will there be an official release for 2.51.0 today/tomorrow? Thank you for the quick response by the way!

[bboreham]

We need to wait longer than 1 day for feedback to come in on the release candidate.
Please try it out! If nobody has any feedback this then will become 2.51.0, you just get it faster.