Can't guest.mkdir - ServerFaultCode: A specified parameter was not correct

[atodorov]

I am trying to upload an ssh key to a newly provisioned VM (I saw an example script on the web):

# govc guest.mkdir -vm=myVmName -p=true  /root/.ssh
govc: ServerFaultCode: A specified parameter was not correct:

my next step was going to upload the ssh key file but duh.

# govc about
Name:         VMware vCenter Server
Vendor:       VMware, Inc.
Version:      6.7.0
Build:        16046713
OS type:      linux-x64
API type:     VirtualCenter
API version:  6.7.3
Product ID:   vpx
UUID:         13dc2663-db18-4499-a65d-ee5c79f2f09a

# govc vm.info myVmName
Name:           myVmName
  Path:         /Engineering/vm/Team/myVmName
  UUID:         420d3986-9b44-62ef-f77d-149f48878f60
  Guest name:   Red Hat Enterprise Linux 8 (64-bit)
  Memory:       2048MB
  CPU:          1 vCPU(s)
  Power state:  poweredOn
  Boot time:    2020-07-09 10:55:15.300016 +0000 UTC
  IP address:   10.8.29.87
  Host:         esxi1.lab.example.com

[dougm]

Not sure if the soap response includes more detail, but you can check using -debug: https://github.com/vmware/govmomi/blob/master/govc/README.md#troubleshooting

And/or check vmware.log for the VM.

Can you create directories outside of /root, such as /tmp/.ssh ?

Can you create the directory using: govc guest.run -vm=myVmName mkdir -p /root/.ssh ?

[chris1984]

I tried doing the following and they give the same error that @atodorov was getting:

[chrobert@Area51 ~]$ govc guest.run -vm=osbuild-image-tests-vmware-image-42c3ccaf-a2be-4890-97b2-c30dc634ce14 mkdir -p /root/.ssh
govc: ServerFaultCode: A specified parameter was not correct:
[chrobert@Area51 ~]$ govc guest.run -vm=osbuild-image-tests-vmware-image-42c3ccaf-a2be-4890-97b2-c30dc634ce14 mkdir -p /tmp/.ssh
govc: ServerFaultCode: A specified parameter was not correct:

I am seeing this in the hostd logs from the ESXi host:

c30dc634ce14.vmx opID=748741e1-c3-135f user=vpxuser:REDHAT.COM\composer] Starting [N8Guestsvc30CreateTemporaryFileRequestImplE:0x000000251ca632d0] opCode=7 auth=<hidden> prefix=govmomi- suffix= directoryPath=
2020-07-10T01:15:18.304Z trivia hostd[2232646] [Originator@6876 sub=Vigor.Vmsvc.vm:/vmfs/volumes/6a084a39-13bd5e02-0000-000000000000/osbuild-image-tests-vmware-image-42c3ccaf-a2be-4890-97b2-c30dc634ce14_2/osbuild-image-tests-vmware-image-42c3ccaf-a2be-4890-97b2-c30dc634ce14.vmx opID=748741e1-c3-135f user=vpxuser:REDHAT.COM\composer] op "Create temporary file" calling VigorOnlineRPC_GuestOpsCreateTemporaryFile
2020-07-10T01:15:18.405Z trivia hostd[2232752] [Originator@6876 sub=Vigor.Vmsvc.vm:/vmfs/volumes/6a084a39-13bd5e02-0000-000000000000/osbuild-image-tests-vmware-image-42c3ccaf-a2be-4890-97b2-c30dc634ce14_2/osbuild-image-tests-vmware-image-42c3ccaf-a2be-4890-97b2-c30dc634ce14.vmx] Create temporary file vigor error: vix error codes = (3, 2).
-->
2020-07-10T01:15:18.405Z verbose hostd[2232752] [Originator@6876 sub=Vigor.Vmsvc.vm:/vmfs/volumes/6a084a39-13bd5e02-0000-000000000000/osbuild-image-tests-vmware-image-42c3ccaf-a2be-4890-97b2-c30dc634ce14_2/osbuild-image-tests-vmware-image-42c3ccaf-a2be-4890-97b2-c30dc634ce14.vmx] Create temporary file translated error to vmodl.fault.InvalidArgument
2020-07-10T01:15:18.405Z trivia hostd[2232752] [Originator@6876 sub=Vigor.Vmsvc.vm:/vmfs/volumes/6a084a39-13bd5e02-0000-000000000000/osbuild-image-tests-vmware-image-42c3ccaf-a2be-4890-97b2-c30dc634ce14_2/osbuild-image-tests-vmware-image-42c3ccaf-a2be-4890-97b2-c30dc634ce14.vmx] Create temporary file failed: vmodl.fault.InvalidArgument
2020-07-10T01:15:18.405Z verbose hostd[2232752] [Originator@6876 sub=Vigor.Vmsvc.vm:/vmfs/volumes/6a084a39-13bd5e02-0000-000000000000/osbuild-image-tests-vmware-image-42c3ccaf-a2be-4890-97b2-c30dc634ce14_2/osbuild-image-tests-vmware-image-42c3ccaf-a2be-4890-97b2-c30dc634ce14.vmx] Create temporary file message:
2020-07-10T01:15:18.405Z trivia hostd[2232752] [Originator@6876 sub=Vigor.Vmsvc.vm:/vmfs/volumes/6a084a39-13bd5e02-0000-000000000000/osbuild-image-tests-vmware-image-42c3ccaf-a2be-4890-97b2-c30dc634ce14_2/osbuild-image-tests-vmware-image-42c3ccaf-a2be-4890-97b2-c30dc634ce14.vmx] Signalling Create temporary file

Then I noticed this:

2020-07-10T01:15:18.413Z info hostd[2232736] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/6a084a39-13bd5e02-0000-000000000000/osbuild-image-tests-vmware-image-42c3ccaf-a2be-4890-97b2-c30dc634ce14_2/osbuild-image-tests-vmware-image-42c3ccaf-a2be-4890-97b2-c30dc634ce14.vmx] [N8Guestsvc30CreateTemporaryFileRequestImplE:0x000000251ca632d0] opCode=7 auth=<hidden> prefix=govmomi- suffix= directoryPath= failed
2020-07-10T01:15:18.413Z trivia hostd[2232736] [Originator@6876 sub=Default] Completing task haTask--vim.vm.guest.FileManager.createTemporaryFile-98684243 as 3
2020-07-10T01:15:18.413Z trivia hostd[2232736] [Originator@6876 sub=Dispatcher] Request removed on completion : 21358 - (failed)
2020-07-10T01:15:18.413Z verbose hostd[2232736] [Originator@6876 sub=PropertyProvider] RecordOp ASSIGN: info, haTask--vim.vm.guest.FileManager.createTemporaryFile-98684243. Applied change to temp map.
2020-07-10T01:15:18.413Z trivia hostd[2232736] [Originator@6876 sub=PropertyProvider] [RecordInvalidateNotLocked] Removing stored value for property at index 2
2020-07-10T01:15:18.413Z trivia hostd[2232736] [Originator@6876 sub=PropertyProvider] [RecordAndNotifyChangeInt] No listeners on haTask--vim.vm.guest.FileManager.createTemporaryFile-98684243 - bailing out
2020-07-10T01:15:18.413Z trivia hostd[2232736] [Originator@6876 sub=Vimsvc.DynMethodInvalidationBroker] Filtering method : vim.vm.guest.FileManager-createTemporaryFile for target : ha-guest-operations-file-manager
2020-07-10T01:15:18.413Z info hostd[2232736] [Originator@6876 sub=Solo.Vmomi] Activation [N5Vmomi10ActivationE:0x000000251cc6b6a0] : Invoke done [createTemporaryFile] on [vim.vm.guest.FileManager:ha-guest-operations-file-manager]
2020-07-10T01:15:18.413Z verbose hostd[2232736] [Originator@6876 sub=Solo.Vmomi] Arg vm:
--> 'vim.VirtualMachine:1732'
2020-07-10T01:15:18.413Z verbose hostd[2232736] [Originator@6876 sub=Solo.Vmomi] Arg auth:
--> (vim.vm.guest.NamePasswordAuthentication) {
-->    interactiveSession = false,
-->    username = "",
-->    password = (not shown)
--> }
2020-07-10T01:15:18.413Z verbose hostd[2232736] [Originator@6876 sub=Solo.Vmomi] Arg prefix:
--> "govmomi-"
2020-07-10T01:15:18.413Z verbose hostd[2232736] [Originator@6876 sub=Solo.Vmomi] Arg suffix:
--> ""
2020-07-10T01:15:18.413Z verbose hostd[2232736] [Originator@6876 sub=Solo.Vmomi] Arg directoryPath:
--> (null)
2020-07-10T01:15:18.413Z info hostd[2232736] [Originator@6876 sub=Solo.Vmomi] Throw vmodl.fault.InvalidArgument
2020-07-10T01:15:18.413Z info hostd[2232736] [Originator@6876 sub=Solo.Vmomi] Result:
--> (vmodl.fault.InvalidArgument) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>,
-->    invalidProperty = <unset>
-->    msg = ""
--> }

[chris1984]

Here is from the vpxa log showing the error and the params being sent in:

020-07-10T01:32:35.535Z info vpxa[2232779] [Originator@6876 sub=Default opID=ca8eec7-ec] [VpxLRO] -- ERROR lro-1927 -- guestOperationsFileManager -- vim.vm.guest.FileManager.createTemporaryFile: vmodl.fault.InvalidArgument:
--> Result:
--> (vmodl.fault.InvalidArgument) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>,
-->    invalidProperty = <unset>
-->    msg = "Received SOAP response fault from [<cs p:0000008ab108eb20, TCP:localhost:8307>]: createTemporaryFile
--> A specified parameter was not correct: "
--> }
--> Args:
-->
--> Arg vm:
--> 'vim.VirtualMachine:1732'
--> Arg auth:
--> (vim.vm.guest.NamePasswordAuthentication) {
-->    interactiveSession = false,
-->    username = "",
-->    password = (not shown)
--> }
--> Arg prefix:
--> "govmomi-"
--> Arg suffix:
--> ""
--> Arg directoryPath:
-->

I can bump the log level up if that helps

[dougm]

@chris1984 thanks for the log details. Not sure what the issue is, but:

Create temporary file vigor error: vix error codes = (3, 2).

I think these are:
3 == VIX_E_INVALID_ARG
2 == VIX_E_OUT_OF_MEMORY

As defined here:
https://github.com/vmware/open-vm-tools/blob/f72e314e8b0df4e80c6b5f9b0c66ad2e9ce02e19/open-vm-tools/lib/include/vix.h#L92-L93

Maybe the vmware-tools log has some more detail in the VM? And/or vmware.log you can find in the VM's datastore directory.

[atodorov]

Not sure if the soap response includes more detail, but you can check using -debug: https://github.com/vmware/govmomi/blob/master/govc/README.md#troubleshooting

Adding -debug to the cli doesn't have any effect. The output is still the same.

And/or check vmware.log for the VM.

This is the only thing I see:

2020-07-10T09:27:26.180Z| vmx| I125: VigorTransportProcessClientPayload: opID=19fa4cc5-44-d820 seq=1013: Receiving GuestOps.MakeDirectory request.
2020-07-10T09:27:26.195Z| vcpu-0| I125: VigorTransport_ServerSendResponse opID=19fa4cc5-44-d820 seq=1013: Completed GuestOps request with messages.

Can you create directories outside of /root, such as /tmp/.ssh ?

No, same error.

Can you create the directory using: govc guest.run -vm=myVmName mkdir -p /root/.ssh ?

No, same error.

It's up to @chris1984 to dig out more details from the logs, that's all I have access to from my side.

[dougm]

Any luck with this? Sounds like the VM just needs more memory, but not sure. Feel free to re-open if this is still an issue.

[arnegroskurth]

Likely solution: #2689 (comment)