Error " The given command 'readyForShutdown' either does not exist" on 4.19.1.0 ACS UI every Second

[biswajitban]

ISSUE TYPE

• Bug Report

COMPONENT NAME

Cloudstack UI

CLOUDSTACK VERSION

ACS - 4.19.1.0
Upgrade from ACS 4.15.1
KVM Hosts with Centos 7.9

CONFIGURATION

Advance Networking

OS / ENVIRONMENT

Centos 7.9

SUMMARY

We have Upgraded ACS 4.15.1 to 4.19.1.0 . We are getting repeated error on ACS webUI with admin users saying " The given command 'readyForShutdown' either does not exist, is not available for user. Unable to proceed. Please contact your administrator" every Second . This is Happening only with the account whose privileges are Root Admin users .

STEPS TO REPRODUCE

None

EXPECTED RESULTS

Should Not have the mentioned error on ACS UI way root admin user account 

ACTUAL RESULTS

Error Free

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[rohityadavcloud]

@biswajitban is this use a root admin, what is the role of the logged in user where you see this? And/or, if the account uses a custom Role?

[biswajitban]

@rohityadavcloud
is this use a root admin? Yes , all user with root admin role are facing this issue . A user , not a root admin role , does not face this issue .
what is the role of the logged in user where you see this ? only in case of user having root Admin as role
And/or, if the account uses a custom Role? There is no defined custom Roles in our cloudstack

[filesfm]

+1. Same issues, multiple/different messages depending on user type:

Root admin: The given command 'readyForShutdown' either does not exist, is not available for user. Unable to proceed. Please contact your administrator.

User: The given command 'listAnnotations' either does not exist, is not available for user. Unable to proceed. Please contact your administrator.

https://cloudstack_host/client/api/?command=readyForShutdown&response=json

{
"readyforshutdownresponse": {
"uuidList": [],
"errorcode": 401,
"cserrorcode": 9999,
"errortext": "The given command 'readyForShutdown' either does not exist, is not available for user."
}
}

2024-08-06 16:24:30,351 DEBUG [c.c.a.ApiServlet] (qtp1386883398-1735:ctx-929d21c8) (logid:c95b3248) ===START=== 10.10.10.31 -- GET command=readyForShutdown&response=json
2024-08-06 16:24:30,351 DEBUG [c.c.a.ApiServlet] (qtp1386883398-1735:ctx-929d21c8) (logid:c95b3248) Two factor authentication is already verified for the user 2, so skipping
2024-08-06 16:24:30,375 DEBUG [c.c.a.ApiServer] (qtp1386883398-1735:ctx-929d21c8 ctx-0017b621) (logid:c95b3248) CIDRs from which account 'Account [{"accountName":"admin","id":2,"uuid":"f0e7fdf6-48f1-11e6-a9f1-00163e44393e"}]' is allowed to perform API calls: 0.0.0.0/0
2024-08-06 16:24:30,391 DEBUG [c.c.a.ApiServlet] (qtp1386883398-1735:ctx-929d21c8 ctx-0017b621) (logid:c95b3248) ===END=== 10.10.10.31 -- GET command=readyForShutdown&response=json

[rajujith]

@biswajitban The global configuration 'dynamic.apichecker.enabled' enabled?

[rohityadavcloud]

@biswajitban @filesfm can you check & enable:

(a) dynamic.apichecker.enabled (what @rajujith suggested) -> true

(b) the Root Admin role is allowed all APIs:

(c) if your account is using any customer Root Admin roles, check and add rules like ^^.

Please keep us posted, thanks.

[biswajitban]

@rohityadavcloud @rajujith
We are getting error while enabling the dynamic.apichecker.enable

Can you please Suggest way ahead

[rohityadavcloud]

Can you share the management server logs for when you attempt to set this global setting @biswajitban ? You may also try API (cmk) client to update configuration using:

(cmk) > update configuration name=dynamic.apichecker.enable value=true

[biswajitban]

@rohityadavcloud
Please find the Result as Below

(cmk) > update configuration name=dynamic.apichecker.enabled value=true
Error: (HTTP 431, error code 9999) Restricted configuration update not allowed

Log Details As Below r

2024-08-07 14:59:21,247 DEBUG [c.c.a.ApiServlet] (qtp1742920067-411:ctx-8211b3aa ctx-9784aaa9) (logid:3582be66) ===END=== 10.1.50.7 -- GET command=listIdps&response=json
2024-08-07 14:59:21,251 DEBUG [c.c.a.ApiServlet] (qtp1742920067-18:ctx-ff632b39) (logid:6bdfa5a9) ===END=== 10.1.50.7 -- GET command=listOauthProvider&response=json
2024-08-07 14:59:21,275 DEBUG [c.c.a.t.Request] (StatsCollector-5:ctx-216ea4b9) (logid:24608deb) Seq 94-8162774324609024029: Received: { Ans: , MgmtId: 154880089438368, via: 94(Synergy-bla
de02.csccloud.in), Ver: v1, Flags: 10, { GetVmStatsAnswer } }
2024-08-07 14:59:21,464 DEBUG [c.c.a.t.Request] (StatsCollector-5:ctx-216ea4b9) (logid:24608deb) Seq 96-26458647810801693: Received: { Ans: , MgmtId: 154880089438368, via: 96(synergy-blade
03.csccloud.in), Ver: v1, Flags: 10, { GetVmStatsAnswer } }
2024-08-07 14:59:21,700 DEBUG [c.c.a.ApiServlet] (qtp1742920067-409:ctx-21f6fabc) (logid:2a0d6206) ===START=== 127.0.0.1 -- GET apiKey=K3JQlPMZIN97wOTUveewM_5dYlhPe26fvbNc4InqZefW0yWY-42F
fGYRKGPtfKQUTL-I4TCdjCuTNGKNwa_T1Q&command=updateConfiguration&name=dynamic.apichecker.enabled&response=json&value=true&signature=PvZU9zA3CccPskb1uoHEu3zaj6A%3D
2024-08-07 14:59:21,721 DEBUG [c.c.a.ApiServer] (qtp1742920067-409:ctx-21f6fabc ctx-152f235f) (logid:2a0d6206) CIDRs from which account 'Account [{"accountName":"leela.aipl","id":51,"uuid":
"73bb58f6-4c49-4fca-85b6-fc1c924da889"}]' is allowed to perform API calls: 0.0.0.0/0,::/0
2024-08-07 14:59:21,738 DEBUG [c.c.a.t.Request] (StatsCollector-5:ctx-216ea4b9) (logid:24608deb) Seq 98-8668021907804651547: Received: { Ans: , MgmtId: 154880089438368, via: 98(Synergy-Bla
de05.csccloud.in), Ver: v1, Flags: 10, { GetVmStatsAnswer } }
2024-08-07 14:59:21,744 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1742920067-409:ctx-21f6fabc ctx-152f235f) (logid:2a0d6206) API rate limiting is disabled. We will not use ApiRateLimitSer
vice.
2024-08-07 14:59:21,747 INFO [c.c.a.ApiServer] (qtp1742920067-409:ctx-21f6fabc ctx-152f235f ctx-032d89ff) (logid:2a0d6206) Restricted configuration update not allowed
2024-08-07 14:59:21,747 DEBUG [c.c.a.ApiServlet] (qtp1742920067-409:ctx-21f6fabc ctx-152f235f ctx-032d89ff) (logid:2a0d6206) ===END=== 127.0.0.1 -- GET apiKey=K3JQlPMZIN97wOTUveewM_5dYlhP
e26fvbNc4InqZefW0yWY-42FfGYRKGPtfKQUTL-I4TCdjCuTNGKNwa_T1Q&command=updateConfiguration&name=dynamic.apichecker.enabled&response=json&value=true&signature=PvZU9zA3CccPskb1uoHEu3zaj6A%3D
2024-08-07 14:59:21,848 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:ctx-366c344f) (logid:3559677d) ClusterDRS.poll is being called at 2024-08-07 09:29:00 GMT
2024-08-07 14:59:21,881 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:ctx-366c344f) (logid:3559677d) Removed 0 old drs migration plans
2024-08-07 14:59:22,036 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:ctx-256fa524) (logid:26a29b01) VM scheduler.poll is being called at 2024-08-07 09:29:00 GMT
2024-08-07 14:59:22,040 DEBUG [c.c.v.VirtualMachineManagerImpl] (Vm-Operations-Cleanup-1:ctx-a31b287e) (logid:944b6ebb) VM Operation Thread Running
2024-08-07 14:59:22,044 DEBUG [c.c.a.t.Request] (StatsCollector-5:ctx-216ea4b9) (logid:24608deb) Seq 100-4150630006575333405: Received: { Ans: , MgmtId: 154880089438368, via: 100(Synergy-B
lade06.csccloud.in), Ver: v1, Flags: 10, { GetVmStatsAnswer } }
2024-08-07 14:59:22,058 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:ctx-256fa524) (logid:26a29b01) Got 0 scheduled jobs to be executed at 2024-08-07 09:29:00 GMT
2024-08-07 14:59:22,065 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:ctx-256fa524) (logid:26a29b01) Cleaned up 0 VM scheduled job entries
2024-08-07 14:59:22,298 DEBUG [c.c.a.t.Request] (StatsCollector-5:ctx-216ea4b9) (logid:24608deb) Seq 102-4216495151125626913: Received: { Ans: , MgmtId: 154880089438368, via: 102(Synergy-B
lade07.csccloud.in), Ver: v1, Flags: 10, { GetVmStatsAnswer } }
2024-08-07 14:59:22,462 DEBUG [c.c.a.t.Request] (StatsCollector-5:ctx-216ea4b9) (logid:24608deb) Seq 104-4332744316507127841: Received: { Ans: , MgmtId: 154880089438368, via: 104(Synergy-b
lade08.csccloud.in), Ver: v1, Flags: 10, { GetVmStatsAnswer } }
2024-08-07 14:59:22,643 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-3:ctx-376c5f63) (logid:5d0999d4) HA health check task is running...
2024-08-07 14:59:22,800 DEBUG [c.c.a.t.Request] (StatsCollector-5:ctx-216ea4b9) (logid:24608deb) Seq 106-2447143447522443296: Received: { Ans: , MgmtId: 154880089438368, via: 106(Synergy-B
lade09.csccloud.in), Ver: v1, Flags: 10, { GetVmStatsAnswer } }
2024-08-07 14:59:23,037 DEBUG [c.c.a.t.Request] (StatsCollector-5:ctx-216ea4b9) (logid:24608deb) Seq 108-3228236507894513701: Received: { Ans: , MgmtId: 154880089438368, via: 108(Synergy-b
lade10.csccloud.in), Ver: v1, Flags: 10, { GetVmStatsAnswer } }
2024-08-07 14:59:23,192 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (Timer-15:ctx-b02eca32) (logid:7c97180e) getCommandHostDelegation: class org.apache.cloudstack.storage.command.DownloadProgressC
ommand
2024-08-07 14:59:23,196 DEBUG [c.c.h.XenServerGuru] (Timer-15:ctx-b02eca32) (logid:7c97180e) We are returning the default host to execute commands because the command is not of Copy type.
2024-08-07 14:59:23,196 DEBUG [o.a.c.s.RemoteHostEndPoint] (Timer-15:ctx-b02eca32) (logid:7c97180e) Sending command org.apache.cloudstack.storage.command.DownloadProgressCommand to host: 18
3
2024-08-07 14:59:23,201 DEBUG [c.c.a.t.Request] (Timer-15:ctx-b02eca32) (logid:7c97180e) Seq 183-4320640892508569802: Sending { Cmd , MgmtId: 154880089438368, via: 183(s-856-VM), Ver: v1,
Flags: 100011, [{"org.apache.cloudstack.storage.command.DownloadProgressCommand":{"jobId":"a00a5420-0ea9-4334-941c-1fe70e769b10","request":"GET_STATUS","hvm":"true","description":"systemvm-
kvm-4.15.1","checksum":"0e9f9a7d0957c3e0a2088e41b2da2cec","maxDownloadSizeInBytes":"(50.00 GB) 53687091200","id":"292","resourceType":"TEMPLATE","installPath":"template/tmpl/8/292","_store"
:{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.1.51.29/export/secondary","_role":"Image"}},"followRedirects":"true","url":"https://download.cloudstack.org/systemvm/4.15/systemvmtemplate
-4.15.1-kvm.qcow2.bz2","format":"QCOW2","accountId":"8","name":"292-8-811f19e6-339a-3c7a-8e10-5324108af4b4","secUrl":"nfs://10.1.51.29/export/secondary","wait":"0","bypassHostMaintenance":"
false"}}] }
2024-08-07 14:59:23,244 DEBUG [c.c.a.t.Request] (AgentManager-Handler-11:null) (logid:) Seq 183-4320640892508569802: Processing: { Ans: , MgmtId: 154880089438368, via: 183, Ver: v1, Flags:

[biswajitban]

As suggested by @rohityadavcloud to update dynamic.apichecker.enabled via mysql client
update cloud.configuration set value='true' where name='dynamic.apichecker.enabled';
And then restart your management servers.
Solved the reported issue .
Thanks @rohityadavcloud @rajujith

[rohityadavcloud]

update cloud.configuration set value='true' where name='dynamic.apichecker.enabled';

Restart your management servers.

Further reading

• https://www.shapeblue.com/dynamic-roles-in-cloudstack/
• https://docs.cloudstack.apache.org/en/4.19.1.1/adminguide/accounts.html#using-dynamic-roles

Closing the issue on these remarks.