Unable to restart mackerel-agent with slow cloud meta-data service

[hanazuki]

Since recent update Custom Identifier (eg. AWS instance id) check is introduced( #405 ). This check ensures that the Custom Identifier of the current instance matches the one previously stored on the mackerel.io server for the HostID. In our setup, this check prevents mackerel-agent to start up.

The Custom Identifier of the current instance is fetched from the cloud meta-data service (i.e. http://169.25.169.254 in case of AWS). The meta-data service is expected to return response within 100ms:

mackerel-agent/spec/cloud.go

Line 47 in 6798d53

var timeout = 100 * time.Millisecond

When the meta-data service is unable to respond within 100ms, an empty string is used as the Custom Identifier of the current instance, which does not match the one successfully stored on the server in the previous agent run.

mackerel-agent/command/command.go

Lines 108 to 113 in 369b932

	if result.CustomIdentifier != "" && result.CustomIdentifier != customIdentifier {
	if fsStorage, ok := conf.HostIDStorage.(*config.FileSystemHostIDStorage); ok {
	return nil, fmt.Errorf("custom identifiers mismatch: this host = \"%s\", the host whose id is \"%s\" on mackerel.io = \"%s\" (File \"%s\" may be copied from another host. Try deleting it and restarting agent)", customIdentifier, hostID, result.CustomIdentifier, fsStorage.HostIDFile())
	}
	return nil, fmt.Errorf("custom identifiers mismatch: this host = \"%s\", the host whose id is \"%s\" on mackerel.io = \"%s\" (Host ID file may be copied from another host. Try deleting it and restarting agent)", customIdentifier, hostID, result.CustomIdentifier)
	}

(customIdentifier = "" and result.CustomIdentifier = (previously stored instance-id))

We use OpenStack, which provides AWS-compatible meta-data service at 169.254.169.254, but in our setup it sometimes take a bit longer than 100ms to respond. Due to the reason described above, we are having trouble that mackerel-agent often fails to restart.

I have some idea to deal with this problem,

• Add an option to extend the HTTP timeout for the meta-data service,
• Add an option to disable cloud meta-data retrieval, or
• Skip the check when the Custom Identifier cannot be fetched for some reason.

Which one is preferred? I can provide a patch for the solutions if necessary.

[itchyny]

Thank you. I think we can simply extend the default timeout (adding an option for this would not be a good idea). IMO 5 seconds for timeout is acceptable. What do you think? @mechairoi @Songmu

[hanazuki]

I have missed #401#398 . My description above seems wrong.

Before #401#398, VM instances on our OpenStack pass isEC2 check since the AWS-compatible meta-data service returns ami-id. After the change, they don't because the kernel do not have /sys/hypervisor/uuid (we use qemu-kvm instead of Xen) and thus Custom Identifier seems not fetched from the meta-data service.

Update: refer to the correct PR

[Songmu]

The timeout duration is 3 seconds now. There is also a way to explicitly specify cloud_platform =" ec2 " in the configuration file.

Close this issue as it has already been resolved.