customData in metadata service doesn't work

[Shell32-Natsu]

I tried the newly added customData in the metadata service. But in my VM it seems that it doesn't work. The response is empty but the status code is 200.

$ curl -v -H "Metadata:true" "http://169.254.169.254/metadata/instance/compute/customData?api-version=2019-02-01&format=text"
* About to connect() to 169.254.169.254 port 80 (#0)
*   Trying 169.254.169.254...
* Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)
> GET /metadata/instance/compute/customData?api-version=2019-02-01&format=text HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 169.254.169.254
> Accept: */*
> Metadata:true
>
< HTTP/1.1 200 OK
< Content-Type: text/plain; charset=utf-8
< Server: Microsoft-IIS/10.0
< Date: Tue, 30 Apr 2019 18:40:35 GMT
< Content-Length: 0
<
* Connection #0 to host 169.254.169.254 left intact

I am sure that custom data is set for the vm and it's downloaded to /var/lib/waagent/CustomData.

---

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

• ID: 0fdce995-5347-32c4-119f-6e8484250859
• Version Independent ID: 3dcfe7f4-cbb8-6323-5d93-b7e6b7f74db7
• Content: Azure Instance Metadata Service
• Content Source: articles/virtual-machines/windows/instance-metadata-service.md
• Service: virtual-machines-windows
• GitHub Login: @KumariSupriya
• Microsoft Alias: sukumari

[mimckitt]

Thanks for the feedback! We are currently investigating and will update you shortly.

[Karishma-Tiwari-MSFT]

@kenchinlai @KumariSupriya Can either of you share your insights on this? Thanks :)

[Karishma-Tiwari-MSFT]

@Shell32-Natsu Is your VM running or in stop state? We would be able to get the information from your VM which will help us troubleshoot, if it is in running state.

On your running VM, please query metadata from within the VM with the following and provide the information back to us.

curl -H Metadata:true "http://169.254.169.254/metadata/instance/compute?api-version=2019-02-01"

As an example, see: https://docs.microsoft.com/en-us/azure/virtual-machines/linux/instance-metadata-service#getting-more-information-about-the-vm-during-support-case

[Shell32-Natsu]

It is in the running state. The VM ID is 29a5dc1e-8ce1-4064-8371-f24776b11a10.

The output of the command

curl -H Metadata:true "http://169.254.169.254/metadata/instance/compute?api-version=2019-02-01"

is:

{
  "azEnvironment": "AzurePublicCloud",
  "customData": "",
  "location": "westus2",
  "name": "***-64pky7iqnjuts-vm-***",
  "offer": "",
  "osType": "Linux",
  "placementGroupId": "",
  "plan": {
    "name": "",
    "product": "",
    "publisher": ""
  },
  "platformFaultDomain": "0",
  "platformUpdateDomain": "0",
  "provider": "Microsoft.Compute",
  "publicKeys": [
    {
      "keyData": "ssh-rsa ***",
      "path": "/home/ml-user/.ssh/authorized_keys"
    }
  ],
  "publisher": "",
  "resourceGroupName": "em-rg",
  "sku": "",
  "subscriptionId": "b143bcf7-***",
  "tags": "",
  "version": "",
  "vmId": "29a5dc1e-8ce1-4064-8371-f24776b11a10",
  "vmScaleSetName": "",
  "vmSize": "Standard_D8s_v3",
  "zone": "3"
}

[Karishma-Tiwari-MSFT]

@Shell32-Natsu I am working with the product team on your issue. They would need the 'name' and 'SubscriptionId' fields value as well from the output you shared.
Can you please share your Subscription Id and name values over an email to Azure CXP Community Connect AzCommunity@microsoft.com. Please mention my name and this GitHub issue link in the email as well.

[Shell32-Natsu]

@Karishma-Tiwari-MSFT Hi Karishma, the email has been sent.

[Karishma-Tiwari-MSFT]

The product team engineer is helping out the customer offline.

[Karishma-Tiwari-MSFT]

@Shell32-Natsu As per discussion offline with the product team, we have identified a product issue and the need to update the document. I have assigned the issue to the content author to update the document as appropriate.

[Shell32-Natsu]

@Karishma-Tiwari-MSFT Thanks!

[NingKuang]

@Shell32-Natsu
Hi, to help us accelerate the resolution, could you share more information on how this issue is impacting your product? Are you blocked?
Thanks.

[Shell32-Natsu]

@NingKuang
We want to access the custom data after the VM booting. But now the custom data is downloaded by WAagent asynchronously so we have to block and wait for it. We want to use meta data service to make this progress faster which means we don't need to block.

[Shell32-Natsu]

@NingKuang @Karishma-Tiwari-MSFT Hi, any update for this problem?

[mimckitt]

I am in contact with the right people and they are taking a look.

An update will be shared here when available.

[NingKuang]

Hi, The custom data via IMDS is temporarily offline, due to a product issue. We are working on getting it back.
As a workaround, you can still read the custom data from local files, see this article for the file locations: https://azure.microsoft.com/en-us/blog/custom-data-and-cloud-init-on-windows-azure/ Thanks.

[chepepa]

@NingKuang , if this is a temporary shutdown, do you guys have any ETA? Or do you even plan to share it with us? Thanks!

[NingKuang]

We don't have an ETA yet. We will give you an update when we have more information. Thank you for your patience.

[stevefan1999-personal]

So this has been almost half a year and this bug is still not fixed

[stevefan1999-personal]

okay, fine, I would like to go the pragmatic way: can someone reverse-engineer WALinuxAgent so as to see how does it handle custom data?

[mimckitt]

We are currently cleaning up old issues and closing out items that are greater than 90 days old. If an issue is still present, please open a new feedback item on the document so we can prioritize correctly.

#please-close

[schnerring]

okay, fine, I would like to go the pragmatic way: can someone reverse-engineer WALinuxAgent so as to see how does it handle custom data?

I've done that by reading through WALinuxAgent's code. See linuxkit/linuxkit#3585 for a very simplistic Go implementation. The custom data is provided via CDROM attached by the Azure hypervisor. See also cloud-init Azure docs for a high level overview how cloud-init does it.

#64154 was also closed without any explanation other than "please close", I don't know where we stand on this issue.

[VikasPullagura-MSFT]

@mimckitt Can you please check and add your comments.

[mimckitt]

@VikasPullagura-MSFT please reach out to @KumariSupriya offline.

[lifeofguenter]

Is this issue still existent? We are experiencing a similar problem but unsure due to this ticket being closed.

[schnerring]

Is this issue still existent? We are experiencing a similar problem but unsure due to this ticket being closed.

There is a new issue: #64154

However, it was also closed without any explanation. I have not found a more up-to-date issue...

[schnerring]

So I haven't seen any updates on GitHub, however, the IMDS documentation has been updated:

customData: This feature is deprecated and disabled in IMDS. It has been superseded by userData

userData: The set of data specified when the VM was created for use during or after provisioning (Base64 encoded)

userData was introduced with version 2021-01-01 of the IMDS API. I'm excited to be testing this soon.

[KumariSupriya]

@schnerring : thanks for follow up. Yes user data is now available https://azure.microsoft.com/en-us/updates/user-data-through-azure-instance-metadata-service-now-generally-available/ and as you pointed out we would not support custom data via IMDS but recommend users to use UserData for the same.

[NingKuang]

For details on how to access user data through instance metadata service, see:
User data on Azure
Thanks!