Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

VM scheduled actions via Sunstone 3/3: Unexpected behavior on VMs older than a month when clicked #5209

Closed
7 tasks
atodorov-storpool opened this issue Dec 16, 2020 · 10 comments
Closed
7 tasks

VM scheduled actions via Sunstone 3/3: Unexpected behavior on VMs older than a month when clicked #5209

atodorov-storpool opened this issue Dec 16, 2020 · 10 comments
Assignees
@jloboescalona2
Labels
Category: Core & System Category: Sunstone Priority: Normal Status: Accepted Type: Bug
Milestone
Release 6.0

Comments

@atodorov-storpool
Copy link
Contributor

Description
(bug 3/3)
Not sure how to name the blue "thing" next to the green "Add Action" button but it has a lot of troubles...
When you click on it two actions are defined without a warning or confirmation - just committed to the core...
image

To Reproduce
Perform the following on a VM that was instantiated more than a month ago:

  1. Select VM -> Actions tab
  2. Click the blue clock-like thing
  3. two actions are created one with wrong name "suspense", but the second one is entirely valid "terminate" action.
  4. Check the VM state after max 30 seconds... And the VM is Terminated

Expected behavior
Not sure what I expect here:

  • There should be a warning/confirmation dialog before committing the scheduled actions?
  • The time should be 2w/4w from the moment when the action is applied, not now?
  • What is the purpose of this button anyway? Should it exists at all?

Details

  • Affected Component: [Sunstone/Scheduler]
  • Hypervisor: [KVM]
  • Version: [5.12.6]

Additional context
Missing a warning even every operator could click there and terminate a production VM in a glipse...
Please note that I totally agree that this case is hard to catch with common CI/automated testing because usually the testing VMs are not older enough to catch this. I had the chance to play with a staging environment where i have older VMs.

Progress Status

  • Branch created
  • Code committed to development branch
  • Testing - QA
  • Documentation
  • Release notes - resolved issues, compatibility, known issues
  • Code committed to upstream release/hotfix branches
  • Documentation committed to upstream release/hotfix branches
@atodorov-storpool
Copy link
Contributor Author

@rsmontero, @tinova, I think this is very important if not critical...

@christian7007
Copy link
Contributor

Hello @atodorov-storpool,

Just FYI, we call charters to "the blue thing". You can find the docs here: http://docs.opennebula.io/5.12/operation/vm_management/vm_instances.html?highlight=charters#vm-charter

Regarding the bugs you mentioned:

  • The suspense instead of suspend typo will be fixed in master.
  • We agree with you, that a warning message requiring a confirmation for the user before creating the actions will be more than useful.
  • The VM shouldn't be terminated after 30s. Could you send us the XML of the VM to take a look at the schedule actions defined. Also it would be useful to know the aprox time when you clicked the button.

@christian7007 christian7007 mentioned this issue Dec 16, 2020
Closed
7 tasks
@christian7007 christian7007 added this to the Release 6.0 milestone Dec 16, 2020
@atodorov-storpool
Copy link
Contributor Author

atodorov-storpool commented Dec 16, 2020 •
edited
Loading

Sure @christian7007. here it is

Clicked on the charters in ~17:38:25
refreshed few seconds later (it is seen from the screenshot that the VM has been triggered 17:38:27 (just 2-3 seconds later ...)
image

[root@one512 ~]# cat vm-7-1608133136.xml
<VM>
  <ID>7</ID>
  <UID>0</UID>
  <GID>0</GID>
  <UNAME>oneadmin</UNAME>
  <GNAME>oneadmin</GNAME>
  <NAME>Ubuntu 20.04-7</NAME>
  <PERMISSIONS>
    <OWNER_U>1</OWNER_U>
    <OWNER_M>1</OWNER_M>
    <OWNER_A>0</OWNER_A>
    <GROUP_U>0</GROUP_U>
    <GROUP_M>0</GROUP_M>
    <GROUP_A>0</GROUP_A>
    <OTHER_U>0</OTHER_U>
    <OTHER_M>0</OTHER_M>
    <OTHER_A>0</OTHER_A>
  </PERMISSIONS>
  <LAST_POLL>1608133104</LAST_POLL>
  <STATE>6</STATE>
  <LCM_STATE>0</LCM_STATE>
  <PREV_STATE>6</PREV_STATE>
  <PREV_LCM_STATE>0</PREV_LCM_STATE>
  <RESCHED>0</RESCHED>
  <STIME>1603121479</STIME>
  <ETIME>1608133118</ETIME>
  <DEPLOY_ID>d574b352-5e26-4fa9-874e-461827423330</DEPLOY_ID>
  <MONITORING>
    <CPU><![CDATA[1.0]]></CPU>
    <DISKRDBYTES><![CDATA[141615508]]></DISKRDBYTES>
    <DISKRDIOPS><![CDATA[6415]]></DISKRDIOPS>
    <DISKWRBYTES><![CDATA[4637696]]></DISKWRBYTES>
    <DISKWRIOPS><![CDATA[242]]></DISKWRIOPS>
    <DISK_SIZE>
      <ID><![CDATA[0]]></ID>
      <SIZE><![CDATA[1085]]></SIZE>
    </DISK_SIZE>
    <ID><![CDATA[7]]></ID>
    <MEMORY><![CDATA[417392]]></MEMORY>
    <NETRX><![CDATA[0]]></NETRX>
    <NETTX><![CDATA[0]]></NETTX>
    <TIMESTAMP><![CDATA[1608133104]]></TIMESTAMP>
  </MONITORING>
  <TEMPLATE>
    <AUTOMATIC_DS_REQUIREMENTS><![CDATA[("CLUSTERS/ID" @> 0) & (TM_MAD = "storpool")]]></AUTOMATIC_DS_REQUIREMENTS>
    <AUTOMATIC_NIC_REQUIREMENTS><![CDATA[("CLUSTERS/ID" @> 0)]]></AUTOMATIC_NIC_REQUIREMENTS>
    <AUTOMATIC_REQUIREMENTS><![CDATA[(CLUSTER_ID = 0) & !(PUBLIC_CLOUD = YES) & !(PIN_POLICY = PINNED)]]></AUTOMATIC_REQUIREMENTS>
    <CONTEXT>
      <DISK_ID><![CDATA[1]]></DISK_ID>
      <NETWORK><![CDATA[YES]]></NETWORK>
      <SSH_PUBLIC_KEY><![CDATA[]]></SSH_PUBLIC_KEY>
      <TARGET><![CDATA[hda]]></TARGET>
    </CONTEXT>
    <CPU><![CDATA[1]]></CPU>
    <DISK>
      <ALLOW_ORPHANS><![CDATA[yes]]></ALLOW_ORPHANS>
      <CLONE><![CDATA[YES]]></CLONE>
      <CLONE_TARGET><![CDATA[SELF]]></CLONE_TARGET>
      <CLUSTER_ID><![CDATA[0]]></CLUSTER_ID>
      <DATASTORE><![CDATA[default]]></DATASTORE>
      <DATASTORE_ID><![CDATA[1]]></DATASTORE_ID>
      <DEV_PREFIX><![CDATA[vd]]></DEV_PREFIX>
      <DISK_ID><![CDATA[0]]></DISK_ID>
      <DISK_SNAPSHOT_TOTAL_SIZE><![CDATA[0]]></DISK_SNAPSHOT_TOTAL_SIZE>
      <DISK_TYPE><![CDATA[BLOCK]]></DISK_TYPE>
      <DRIVER><![CDATA[raw]]></DRIVER>
      <IMAGE><![CDATA[Ubuntu 20.04]]></IMAGE>
      <IMAGE_ID><![CDATA[0]]></IMAGE_ID>
      <IMAGE_STATE><![CDATA[2]]></IMAGE_STATE>
      <LN_TARGET><![CDATA[NONE]]></LN_TARGET>
      <ORIGINAL_SIZE><![CDATA[2252]]></ORIGINAL_SIZE>
      <READONLY><![CDATA[NO]]></READONLY>
      <SAVE><![CDATA[NO]]></SAVE>
      <SIZE><![CDATA[2252]]></SIZE>
      <SOURCE><![CDATA[ans-ds-1/ans-img-0]]></SOURCE>
      <TARGET><![CDATA[vda]]></TARGET>
      <TM_MAD><![CDATA[storpool]]></TM_MAD>
      <TYPE><![CDATA[BLOCK]]></TYPE>
    </DISK>
    <GRAPHICS>
      <LISTEN><![CDATA[0.0.0.0]]></LISTEN>
      <TYPE><![CDATA[vnc]]></TYPE>
    </GRAPHICS>
    <MEMORY><![CDATA[768]]></MEMORY>
    <OS>
      <ARCH><![CDATA[x86_64]]></ARCH>
    </OS>
    <TEMPLATE_ID><![CDATA[0]]></TEMPLATE_ID>
    <TM_MAD_SYSTEM><![CDATA[storpool]]></TM_MAD_SYSTEM>
    <VMID><![CDATA[7]]></VMID>
  </TEMPLATE>
  <USER_TEMPLATE>
    <INFO><![CDATA[Please do not use this VM Template for vCenter VMs. Refer to the documentation https://bit.ly/37NcJ0Y]]></INFO>
    <LOGO><![CDATA[images/logos/ubuntu.png]]></LOGO>
    <LXD_SECURITY_PRIVILEGED><![CDATA[true]]></LXD_SECURITY_PRIVILEGED>
    <SCHED_ACTION>
      <ACTION><![CDATA[suspense]]></ACTION>
      <ID><![CDATA[0]]></ID>
      <MESSAGE><![CDATA[Wed Dec 16 17:38:27 2020 : This action is not supported.]]></MESSAGE>
      <TIME><![CDATA[+1209600]]></TIME>
    </SCHED_ACTION>
    <SCHED_ACTION>
      <ACTION><![CDATA[terminate]]></ACTION>
      <DONE><![CDATA[1608133107]]></DONE>
      <ID><![CDATA[1]]></ID>
      <TIME><![CDATA[+2419200]]></TIME>
    </SCHED_ACTION>
    <SCHED_REQUIREMENTS><![CDATA[HYPERVISOR!="vcenter"]]></SCHED_REQUIREMENTS>
  </USER_TEMPLATE>
  <HISTORY_RECORDS>
    <HISTORY>
      <OID>7</OID>
      <SEQ>0</SEQ>
      <HOSTNAME>one512kvm1</HOSTNAME>
      <HID>0</HID>
      <CID>0</CID>
      <STIME>1608121489</STIME>
      <ETIME>1608132782</ETIME>
      <VM_MAD><![CDATA[kvm]]></VM_MAD>
      <TM_MAD><![CDATA[storpool]]></TM_MAD>
      <DS_ID>0</DS_ID>
      <PSTIME>1608121489</PSTIME>
      <PETIME>1608121492</PETIME>
      <RSTIME>1608121492</RSTIME>
      <RETIME>1608132779</RETIME>
      <ESTIME>1608132779</ESTIME>
      <EETIME>1608132782</EETIME>
      <ACTION>6</ACTION>
      <UID>0</UID>
      <GID>0</GID>
      <REQUEST_ID>8544</REQUEST_ID>
    </HISTORY>
    <HISTORY>
      <OID>7</OID>
      <SEQ>1</SEQ>
      <HOSTNAME>one512kvm2</HOSTNAME>
      <HID>1</HID>
      <CID>0</CID>
      <STIME>1608132987</STIME>
      <ETIME>1608133118</ETIME>
      <VM_MAD><![CDATA[kvm]]></VM_MAD>
      <TM_MAD><![CDATA[storpool]]></TM_MAD>
      <DS_ID>0</DS_ID>
      <PSTIME>1608132987</PSTIME>
      <PETIME>1608132991</PETIME>
      <RSTIME>1608132991</RSTIME>
      <RETIME>1608133113</RETIME>
      <ESTIME>1608133113</ESTIME>
      <EETIME>1608133118</EETIME>
      <ACTION>27</ACTION>
      <UID>0</UID>
      <GID>0</GID>
      <REQUEST_ID>6816</REQUEST_ID>
    </HISTORY>
  </HISTORY_RECORDS>
</VM>

[root@one512 ~]# onevm show 7
VIRTUAL MACHINE 7 INFORMATION                                                   
ID                  : 7                   
NAME                : Ubuntu 20.04-7      
USER                : oneadmin            
GROUP               : oneadmin            
STATE               : DONE                
LCM_STATE           : LCM_INIT            
LOCK                : None                
RESCHED             : No                  
START TIME          : 10/19 18:31:19      
END TIME            : 12/16 17:38:38      
DEPLOY ID           : d574b352-5e26-4fa9-874e-461827423330

VIRTUAL MACHINE MONITORING                                                      
CPU                 : 1.0                 
MEMORY              : 407.6M              
NETTX               : 0K                  
NETRX               : 0K                  
DISKRDBYTES         : 141615508           
DISKRDIOPS          : 6415                
DISKWRBYTES         : 4637696             
DISKWRIOPS          : 242                 
ID                  : 7                   
TIMESTAMP           : 1608133104          

PERMISSIONS                                                                     
OWNER               : um-                 
GROUP               : ---                 
OTHER               : ---                 

VM DISKS                                                                        
 ID DATASTORE  TARGET IMAGE                               SIZE      TYPE SAVE
  0 default    vda    Ubuntu 20.04                        1.1G/2.2G bloc   NO
  1 -          hda    CONTEXT                             -/-       -       -

VIRTUAL MACHINE HISTORY                                                         
SEQ UID  REQ   HOST         ACTION       DS           START        TIME     PROLOG
  0 0    8544  one512kvm1   undeploy-h    0  12/16 14:24:49   0d 03h08m   0h00m03s
  1 0    6816  one512kvm2   terminate     0  12/16 17:36:27   0d 00h02m   0h00m04s

SCHEDULED ACTIONS                                                               
   ID    ACTION  ARGS   SCHEDULED REPEAT   END        DONE                             MESSAGE CHARTER        
    0  suspense     - 01/15 02:00                       - Wed Dec 16 17:38:27 2020 : This act Already done *
    1 terminate     - 01/29 02:00             12/16 17:38                                   - Already done *

Ammm it looks like the scheduler become crazy after first try with VM 6 (when reproduced the problem for reporting)...
sched.log

root@one512 one]# cat sched.log 
Wed Dec 16 12:24:15 2020 [Z0][SCHED][I]: Init Scheduler Log system
Wed Dec 16 12:24:15 2020 [Z0][SCHED][I]: Starting Scheduler Daemon
----------------------------------------
     Scheduler Configuration File       
----------------------------------------
DEFAULT_DS_SCHED=POLICY=1
DEFAULT_NIC_SCHED=POLICY=1
DEFAULT_SCHED=POLICY=1
DIFFERENT_VNETS=YES
LIVE_RESCHEDS=1
LOG=DEBUG_LEVEL=3,SYSTEM=file
MAX_DISPATCH=30
MAX_HOST=1
MAX_VM=5000
MEMORY_SYSTEM_DS_SCALE=0
MESSAGE_SIZE=1073741824
ONE_XMLRPC=http://localhost:2633/RPC2
SCHED_INTERVAL=30
TIMEOUT=60
----------------------------------------
Wed Dec 16 12:24:15 2020 [Z0][SCHED][I]: XML-RPC client using 1073741824 bytes for response buffer.

Wed Dec 16 12:24:15 2020 [Z0][SCHED][I]: oned successfully contacted.
Wed Dec 16 12:24:15 2020 [Z0][SCHED][I]: Configuring scheduler for Zone ID: 0
Wed Dec 16 12:24:15 2020 [Z0][SCHED][I]: Starting scheduler loop...
Wed Dec 16 12:24:15 2020 [Z0][SCHED][I]: Scheduler loop started.
Wed Dec 16 12:49:16 2020 [Z0][VM][D]: Found 1 pending/rescheduling VMs.
Wed Dec 16 12:49:16 2020 [Z0][HOST][D]: Discovered 2 enabled hosts.
Wed Dec 16 12:49:16 2020 [Z0][VM][D]: VMs in VMGroups:

Wed Dec 16 12:49:16 2020 [Z0][VNET][D]: Discovered 2 vnets.
Wed Dec 16 12:49:16 2020 [Z0][SCHED][D]: Dispatching VMs to hosts:
	VMID	Priority	Host	System DS
	--------------------------------------------------------------
	5	0		1	0

Wed Dec 16 14:24:19 2020 [Z0][VM][D]: Found 1 pending/rescheduling VMs.
Wed Dec 16 14:24:19 2020 [Z0][HOST][D]: Discovered 2 enabled hosts.
Wed Dec 16 14:24:19 2020 [Z0][VM][D]: VMs in VMGroups:

Wed Dec 16 14:24:19 2020 [Z0][VNET][D]: Discovered 2 vnets.
Wed Dec 16 14:24:19 2020 [Z0][SCHED][D]: Dispatching VMs to hosts:
	VMID	Priority	Host	System DS
	--------------------------------------------------------------
	6	0		1	0

Wed Dec 16 14:24:49 2020 [Z0][VM][D]: Found 3 pending/rescheduling VMs.
Wed Dec 16 14:24:49 2020 [Z0][HOST][D]: Discovered 2 enabled hosts.
Wed Dec 16 14:24:49 2020 [Z0][VM][D]: VMs in VMGroups:

Wed Dec 16 14:24:49 2020 [Z0][VNET][D]: Discovered 2 vnets.
Wed Dec 16 14:24:49 2020 [Z0][SCHED][D]: Dispatching VMs to hosts:
	VMID	Priority	Host	System DS
	--------------------------------------------------------------
	7	0		0	0
	8	0		1	0

Wed Dec 16 14:25:19 2020 [Z0][VM][D]: Found 1 pending/rescheduling VMs.
Wed Dec 16 14:25:19 2020 [Z0][HOST][D]: Discovered 2 enabled hosts.
Wed Dec 16 14:25:19 2020 [Z0][VM][D]: VMs in VMGroups:

Wed Dec 16 14:25:19 2020 [Z0][VNET][D]: Discovered 2 vnets.
Wed Dec 16 14:25:19 2020 [Z0][SCHED][D]: Dispatching VMs to hosts:
	VMID	Priority	Host	System DS
	--------------------------------------------------------------
	9	0		0	0

Wed Dec 16 14:25:49 2020 [Z0][VM][D]: VMs with scheduled actions:
 6
Wed Dec 16 14:26:19 2020 [Z0][VM][D]: VMs with scheduled actions:
 6
Wed Dec 16 14:26:49 2020 [Z0][VM][D]: VMs with scheduled actions:
 6
Wed Dec 16 14:27:19 2020 [Z0][VM][D]: VMs with scheduled actions:
 6
Wed Dec 16 14:27:49 2020 [Z0][VM][D]: VMs with scheduled actions:
 6
Wed Dec 16 14:28:19 2020 [Z0][VM][D]: VMs with scheduled actions:
 6
Wed Dec 16 14:28:49 2020 [Z0][VM][D]: VMs with scheduled actions:
 6
Wed Dec 16 14:29:19 2020 [Z0][VM][D]: VMs with scheduled actions:
 6
Wed Dec 16 14:29:49 2020 [Z0][VM][D]: VMs with scheduled actions:
 6
Wed Dec 16 14:30:19 2020 [Z0][VM][D]: VMs with scheduled actions:
 6
Wed Dec 16 14:30:49 2020 [Z0][VM][D]: VMs with scheduled actions:
 6
Wed Dec 16 14:31:19 2020 [Z0][VM][D]: VMs with scheduled actions:
 6
Wed Dec 16 14:31:49 2020 [Z0][VM][D]: VMs with scheduled actions:
 6
Wed Dec 16 14:32:19 2020 [Z0][VM][D]: VMs with scheduled actions:
 6
Wed Dec 16 14:32:49 2020 [Z0][VM][D]: VMs with scheduled actions:
 6
Wed Dec 16 14:33:19 2020 [Z0][VM][D]: VMs with scheduled actions:
 6
Wed Dec 16 14:33:49 2020 [Z0][VM][D]: VMs with scheduled actions:
 6
Wed Dec 16 14:34:19 2020 [Z0][VM][D]: VMs with scheduled actions:
 6
Wed Dec 16 14:34:49 2020 [Z0][VM][D]: VMs with scheduled actions:
 6
Wed Dec 16 14:35:19 2020 [Z0][VM][D]: VMs with scheduled actions:
 6
Wed Dec 16 14:35:49 2020 [Z0][VM][D]: VMs with scheduled actions:
 6
Wed Dec 16 14:36:19 2020 [Z0][VM][D]: VMs with scheduled actions:
 6
Wed Dec 16 14:36:49 2020 [Z0][VM][D]: VMs with scheduled actions:
 6
Wed Dec 16 14:37:19 2020 [Z0][VM][D]: VMs with scheduled actions:
 6
Wed Dec 16 14:37:19 2020 [Z0][VM][I]: Executing action 'terminate' for VM 6 : Success.
Wed Dec 16 14:51:20 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 14:51:50 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 14:52:20 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 14:52:50 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 14:53:20 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 14:53:50 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 14:54:20 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 14:54:50 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 14:55:20 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 14:55:50 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 14:56:20 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 14:56:50 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 14:57:20 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 14:57:50 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 14:58:20 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 14:58:50 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 14:59:20 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 14:59:50 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:00:20 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:00:50 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:01:20 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:01:50 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:02:20 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:02:50 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:03:20 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:03:50 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:04:20 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:04:50 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:05:20 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:05:50 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:06:20 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:06:50 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:07:20 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:07:50 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:08:20 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:08:50 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:09:20 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:09:50 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:10:20 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:10:50 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:11:20 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:11:50 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:12:20 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:12:50 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:13:20 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:13:50 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:14:21 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:14:51 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:15:21 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:15:51 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:16:21 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:16:51 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:17:21 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:17:51 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:18:21 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:18:51 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:19:21 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:19:51 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:20:21 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:20:51 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:21:21 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:21:51 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:22:21 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:22:51 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:23:21 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:23:51 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:24:21 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:24:51 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:25:21 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:25:51 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:26:21 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:26:51 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:27:21 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:27:51 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:28:21 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:28:51 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:29:21 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:29:51 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:30:21 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:30:51 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:31:21 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:31:51 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:32:21 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:32:51 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:33:21 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:33:51 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:34:21 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:34:51 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:35:21 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:35:51 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:36:21 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:36:51 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:37:21 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:37:51 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:38:21 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:38:51 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:39:21 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:39:51 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:40:21 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:40:51 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:41:21 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:41:52 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:42:22 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:42:52 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:43:22 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:43:52 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:44:22 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:44:52 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:45:22 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:45:52 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:46:22 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:46:52 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:47:22 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:47:52 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:48:22 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:48:52 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:49:22 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:49:52 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:50:22 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:50:52 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:51:22 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:51:52 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:52:22 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:52:52 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:53:22 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:53:52 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:54:22 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:54:52 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:55:22 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:55:52 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:56:22 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:56:52 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:57:22 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:57:52 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:58:22 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:58:52 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:59:22 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 15:59:52 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:00:22 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:00:52 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:01:22 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:01:52 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:02:22 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:02:52 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:03:22 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:03:52 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:04:22 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:04:52 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:05:22 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:05:52 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:06:22 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:06:52 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:07:23 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:07:53 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:08:23 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:08:53 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:09:23 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:09:53 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:10:23 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:10:53 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:11:23 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:11:53 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:12:23 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:12:53 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:13:23 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:13:53 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:14:23 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:14:53 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:15:23 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:15:53 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:16:23 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:16:53 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:17:23 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:17:53 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:18:23 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:18:53 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:19:23 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:19:53 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:20:23 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:20:53 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:21:23 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:21:53 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:22:23 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:22:53 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:23:23 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:23:53 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:24:23 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:24:53 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:25:23 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:25:53 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:26:23 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:26:53 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:27:23 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:27:53 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:28:23 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:28:53 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:29:23 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:29:53 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:30:23 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:30:54 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:31:24 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:31:54 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:32:24 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:32:54 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:33:24 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:33:54 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:34:24 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:34:54 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:35:24 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:35:54 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:36:24 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:36:54 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:37:24 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:37:54 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:38:24 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:38:54 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:39:24 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:39:54 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:40:24 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:40:54 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:41:24 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:41:54 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:42:24 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:42:54 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:43:24 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:43:54 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:44:24 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:44:54 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:45:24 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:45:54 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:46:24 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:46:54 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:47:24 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:47:54 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:48:24 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:48:54 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:49:24 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:49:54 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:50:24 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:50:54 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:51:24 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:51:55 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:52:25 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:52:55 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:53:25 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:53:55 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:54:25 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:54:55 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:55:25 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:55:55 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:56:25 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:56:55 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:57:25 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:57:55 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:58:25 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:58:55 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:59:25 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 16:59:55 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:00:25 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:00:55 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:01:25 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:01:55 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:02:25 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:02:55 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:03:25 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:03:55 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:04:25 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:04:55 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:05:25 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:05:55 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:06:25 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:06:55 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:07:25 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:07:55 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:08:25 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:08:55 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:09:25 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:09:55 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:10:25 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:10:55 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:11:25 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:11:55 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:12:25 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:12:56 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:13:26 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:13:56 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:14:26 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:14:56 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:15:26 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:15:56 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:16:26 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:16:56 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:17:26 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:17:56 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:18:26 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:18:56 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:19:26 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:19:56 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:20:26 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:20:56 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:21:26 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:21:56 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:22:26 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:22:56 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:23:26 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:23:56 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:24:26 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:24:56 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:25:26 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:25:56 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:26:26 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:26:56 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:27:26 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:27:56 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:28:26 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:28:56 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:29:26 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:29:56 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:30:26 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:30:56 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:31:26 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:31:56 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:32:26 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:36:26 2020 [Z0][VM][D]: Found 1 pending/rescheduling VMs.
Wed Dec 16 17:36:26 2020 [Z0][HOST][D]: Discovered 2 enabled hosts.
Wed Dec 16 17:36:26 2020 [Z0][VM][D]: VMs in VMGroups:

Wed Dec 16 17:36:26 2020 [Z0][VNET][D]: Discovered 2 vnets.
Wed Dec 16 17:36:27 2020 [Z0][SCHED][D]: Dispatching VMs to hosts:
	VMID	Priority	Host	System DS
	--------------------------------------------------------------
	7	0		1	0

Wed Dec 16 17:38:27 2020 [Z0][VM][D]: VMs with scheduled actions:
 7
Wed Dec 16 17:38:27 2020 [Z0][VM][I]: Executing action 'suspense' for VM 7 : Failure. This action is not supported.
Wed Dec 16 17:38:27 2020 [Z0][VM][I]: Executing action 'terminate' for VM 7 : Success.

@atodorov-storpool
Copy link
Contributor Author

for faster reproduction I'am instantiating a VM, undeploy, then editing /VM/STIME to make the VM older, then resume the VM, and click on the charter

Same reproduced and in a customer installation too - I had a testing VM left from the initial tests. just resumed the VM to become in running state, clicked on the charter and the VM was almost immediately in state6 (DONE)

@atodorov-storpool
Copy link
Contributor Author

And The VM XML in question before clicking on the charter:

[root@one512 ~]# cat vm-7-1608133057.xml 
<VM>
  <ID>7</ID>
  <UID>0</UID>
  <GID>0</GID>
  <UNAME>oneadmin</UNAME>
  <GNAME>oneadmin</GNAME>
  <NAME>Ubuntu 20.04-7</NAME>
  <PERMISSIONS>
    <OWNER_U>1</OWNER_U>
    <OWNER_M>1</OWNER_M>
    <OWNER_A>0</OWNER_A>
    <GROUP_U>0</GROUP_U>
    <GROUP_M>0</GROUP_M>
    <GROUP_A>0</GROUP_A>
    <OTHER_U>0</OTHER_U>
    <OTHER_M>0</OTHER_M>
    <OTHER_A>0</OTHER_A>
  </PERMISSIONS>
  <LAST_POLL>1608133043</LAST_POLL>
  <STATE>3</STATE>
  <LCM_STATE>3</LCM_STATE>
  <PREV_STATE>3</PREV_STATE>
  <PREV_LCM_STATE>3</PREV_LCM_STATE>
  <RESCHED>0</RESCHED>
  <STIME>1603121479</STIME>
  <ETIME>0</ETIME>
  <DEPLOY_ID>d574b352-5e26-4fa9-874e-461827423330</DEPLOY_ID>
  <MONITORING>
    <CPU><![CDATA[1.0]]></CPU>
    <DISKRDBYTES><![CDATA[141351256]]></DISKRDBYTES>
    <DISKRDIOPS><![CDATA[6345]]></DISKRDIOPS>
    <DISKWRBYTES><![CDATA[4555776]]></DISKWRBYTES>
    <DISKWRIOPS><![CDATA[233]]></DISKWRIOPS>
    <DISK_SIZE>
      <ID><![CDATA[0]]></ID>
      <SIZE><![CDATA[1085]]></SIZE>
    </DISK_SIZE>
    <ID><![CDATA[7]]></ID>
    <MEMORY><![CDATA[417392]]></MEMORY>
    <NETRX><![CDATA[0]]></NETRX>
    <NETTX><![CDATA[0]]></NETTX>
    <TIMESTAMP><![CDATA[1608133043]]></TIMESTAMP>
  </MONITORING>
  <TEMPLATE>
    <AUTOMATIC_DS_REQUIREMENTS><![CDATA[("CLUSTERS/ID" @> 0) & (TM_MAD = "storpool")]]></AUTOMATIC_DS_REQUIREMENTS>
    <AUTOMATIC_NIC_REQUIREMENTS><![CDATA[("CLUSTERS/ID" @> 0)]]></AUTOMATIC_NIC_REQUIREMENTS>
    <AUTOMATIC_REQUIREMENTS><![CDATA[(CLUSTER_ID = 0) & !(PUBLIC_CLOUD = YES) & !(PIN_POLICY = PINNED)]]></AUTOMATIC_REQUIREMENTS>
    <CONTEXT>
      <DISK_ID><![CDATA[1]]></DISK_ID>
      <NETWORK><![CDATA[YES]]></NETWORK>
      <SSH_PUBLIC_KEY><![CDATA[]]></SSH_PUBLIC_KEY>
      <TARGET><![CDATA[hda]]></TARGET>
    </CONTEXT>
    <CPU><![CDATA[1]]></CPU>
    <DISK>
      <ALLOW_ORPHANS><![CDATA[yes]]></ALLOW_ORPHANS>
      <CLONE><![CDATA[YES]]></CLONE>
      <CLONE_TARGET><![CDATA[SELF]]></CLONE_TARGET>
      <CLUSTER_ID><![CDATA[0]]></CLUSTER_ID>
      <DATASTORE><![CDATA[default]]></DATASTORE>
      <DATASTORE_ID><![CDATA[1]]></DATASTORE_ID>
      <DEV_PREFIX><![CDATA[vd]]></DEV_PREFIX>
      <DISK_ID><![CDATA[0]]></DISK_ID>
      <DISK_SNAPSHOT_TOTAL_SIZE><![CDATA[0]]></DISK_SNAPSHOT_TOTAL_SIZE>
      <DISK_TYPE><![CDATA[BLOCK]]></DISK_TYPE>
      <DRIVER><![CDATA[raw]]></DRIVER>
      <IMAGE><![CDATA[Ubuntu 20.04]]></IMAGE>
      <IMAGE_ID><![CDATA[0]]></IMAGE_ID>
      <IMAGE_STATE><![CDATA[2]]></IMAGE_STATE>
      <LN_TARGET><![CDATA[NONE]]></LN_TARGET>
      <ORIGINAL_SIZE><![CDATA[2252]]></ORIGINAL_SIZE>
      <READONLY><![CDATA[NO]]></READONLY>
      <SAVE><![CDATA[NO]]></SAVE>
      <SIZE><![CDATA[2252]]></SIZE>
      <SOURCE><![CDATA[ans-ds-1/ans-img-0]]></SOURCE>
      <TARGET><![CDATA[vda]]></TARGET>
      <TM_MAD><![CDATA[storpool]]></TM_MAD>
      <TYPE><![CDATA[BLOCK]]></TYPE>
    </DISK>
    <GRAPHICS>
      <LISTEN><![CDATA[0.0.0.0]]></LISTEN>
      <PORT><![CDATA[5907]]></PORT>
      <TYPE><![CDATA[vnc]]></TYPE>
    </GRAPHICS>
    <MEMORY><![CDATA[768]]></MEMORY>
    <OS>
      <ARCH><![CDATA[x86_64]]></ARCH>
    </OS>
    <TEMPLATE_ID><![CDATA[0]]></TEMPLATE_ID>
    <TM_MAD_SYSTEM><![CDATA[storpool]]></TM_MAD_SYSTEM>
    <VMID><![CDATA[7]]></VMID>
  </TEMPLATE>
  <USER_TEMPLATE>
    <INFO><![CDATA[Please do not use this VM Template for vCenter VMs. Refer to the documentation https://bit.ly/37NcJ0Y]]></INFO>
    <LOGO><![CDATA[images/logos/ubuntu.png]]></LOGO>
    <LXD_SECURITY_PRIVILEGED><![CDATA[true]]></LXD_SECURITY_PRIVILEGED>
    <SCHED_REQUIREMENTS><![CDATA[HYPERVISOR!="vcenter"]]></SCHED_REQUIREMENTS>
  </USER_TEMPLATE>
  <HISTORY_RECORDS>
    <HISTORY>
      <OID>7</OID>
      <SEQ>0</SEQ>
      <HOSTNAME>one512kvm1</HOSTNAME>
      <HID>0</HID>
      <CID>0</CID>
      <STIME>1608121489</STIME>
      <ETIME>1608132782</ETIME>
      <VM_MAD><![CDATA[kvm]]></VM_MAD>
      <TM_MAD><![CDATA[storpool]]></TM_MAD>
      <DS_ID>0</DS_ID>
      <PSTIME>1608121489</PSTIME>
      <PETIME>1608121492</PETIME>
      <RSTIME>1608121492</RSTIME>
      <RETIME>1608132779</RETIME>
      <ESTIME>1608132779</ESTIME>
      <EETIME>1608132782</EETIME>
      <ACTION>6</ACTION>
      <UID>0</UID>
      <GID>0</GID>
      <REQUEST_ID>8544</REQUEST_ID>
    </HISTORY>
    <HISTORY>
      <OID>7</OID>
      <SEQ>1</SEQ>
      <HOSTNAME>one512kvm2</HOSTNAME>
      <HID>1</HID>
      <CID>0</CID>
      <STIME>1608132987</STIME>
      <ETIME>0</ETIME>
      <VM_MAD><![CDATA[kvm]]></VM_MAD>
      <TM_MAD><![CDATA[storpool]]></TM_MAD>
      <DS_ID>0</DS_ID>
      <PSTIME>1608132987</PSTIME>
      <PETIME>1608132991</PETIME>
      <RSTIME>1608132991</RSTIME>
      <RETIME>0</RETIME>
      <ESTIME>0</ESTIME>
      <EETIME>0</EETIME>
      <ACTION>0</ACTION>
      <UID>-1</UID>
      <GID>-1</GID>
      <REQUEST_ID>-1</REQUEST_ID>
    </HISTORY>
  </HISTORY_RECORDS>
</VM>

@atodorov-storpool
Copy link
Contributor Author

Just FYI, we call charters to "the blue thing". You can find the docs here: http://docs.opennebula.io/5.12/operation/vm_management/vm_instances.html?highlight=charters#vm-charter

Thanks! Will disable them in the customer installations until this is resolved.

@christian7007
Copy link
Contributor

I've been looking into it and it seems that the root cause is that the scheduled actions used are relative actions (which are relatives to the VM creation time).

I've discussed this with the team and we thing that the following improvements should be added:

  • Add the warning message previously mentioned, explaining that charters can have impact on the VM.
  • Fail when trying to create an schedule action scheduled for a past time.

Also, thank you very much for the feedback! Really nice catch!

@atodorov-storpool
Copy link
Contributor Author

Hi @christian7007,
I want to clarify the other issues discovered in one place here:

  1. The API accepts any string for action name, and this is discoverable only when the action is triggerred - IMO there should be validation on the API level, and an error returned for unknown actions. The reason for this is because the action could be registered via the API without sunstone, and the mistake will be detected only when the action is expected to be triggered.
  2. The API registers same actions multiple times(with same parameters) - IMO the name+time+etc should be unique to be accepted. The API should return an error for the duplicated actions.
  3. The scheduler's log is too noisy. IMO the log message Wed Dec 16 17:27:26 2020 [Z0][VM][D]: VMs with scheduled actions: should be with higher debug level (4?) instead of the default. There is no point having this logged every 30 seconds in a production system.

@christian7007
Copy link
Contributor

Hello @atodorov-storpool, thanks again for the feedback. We'll take this into account when fixing this issue.

rsmontero pushed a commit that referenced this issue Jan 5, 2021
@rsmontero
B #5209 fix sched action suspend
d97cea3 
Signed-off-by: Jorge Lobo <jlobo@opennebula.io>
atodorov-storpool pushed a commit to storpool/one that referenced this issue Jan 6, 2021
@atodorov-storpool
B OpenNebula#5209 fix sched action suspend
76bd7ee 
Signed-off-by: Jorge Lobo <jlobo@opennebula.io>
atodorov-storpool pushed a commit to storpool/one that referenced this issue Jan 7, 2021
@atodorov-storpool
B OpenNebula#5209 fix sched action suspend
cdea141 
Signed-off-by: Jorge Lobo <jlobo@opennebula.io>
rsmontero pushed a commit that referenced this issue Jan 20, 2021
@tinova
B #5209 fix sched action suspend
b04acde 
Signed-off-by: Jorge Lobo <jlobo@opennebula.io>
jloboescalona2 pushed a commit to jloboescalona2/docs that referenced this issue Feb 9, 2021
F OpenNebula/one#5209: fix error sched in vm
9a960c2 
Signed-off-by: Jorge Lobo <jlobo@opennebula.io>
@jloboescalona2
Copy link
Contributor

jloboescalona2 commented Feb 9, 2021 •
edited
Loading

PRs to merge in master:

tinova pushed a commit to OpenNebula/docs that referenced this issue Feb 9, 2021
@jloboescalona2
F OpenNebula/one#5209: fix error sched in vm (#1455)
3dd54bb 
Signed-off-by: Jorge Lobo <jlobo@opennebula.io>
rsmontero pushed a commit that referenced this issue Feb 10, 2021
@jloboescalona2
F #5209: add modal and fix time in vm (#730)
9b9141b 
Signed-off-by: Jorge Lobo <jlobo@opennebula.io>
rsmontero pushed a commit that referenced this issue Feb 10, 2021
@jloboescalona2
F #5209: Show schedule actions in service (#740)
b319722 
Co-authored-by: Jorge Lobo <jlobo@opennebula.systems>
rsmontero pushed a commit that referenced this issue Feb 10, 2021
@jloboescalona2
F #5209: fix schedule actions in service (#769)
bd814db 
Signed-off-by: Jorge Lobo <jlobo@opennebula.io>
@rsmontero rsmontero reopened this Mar 10, 2021
@atodorov-storpool atodorov-storpool mentioned this issue Mar 11, 2021
Closed
7 tasks
@tinova tinova mentioned this issue Mar 11, 2021
Closed
7 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jloboescalona2 jloboescalona2

Labels
Category: Core & System Category: Sunstone Priority: Normal Status: Accepted Type: Bug
Projects
None yet
Milestone
Release 6.0
Development

No branches or pull requests
4 participants
@rsmontero @atodorov-storpool @christian7007 @jloboescalona2