SCE:2022-03-24T21:28:00 2022/03/24 21:28:49 region.go:417: Enabling group ASG-ONE for processing because its tags, the currently configured  filtering mode (opt-in) and tag filters are aligned
SCE:2022-03-24T21:28:00 2022/03/24 21:28:49 region.go:417: Enabling group ASG-TWO for processing because its tags, the currently configured  filtering mode (opt-in) and tag filters are aligned
SCE:2022-03-24T21:28:00 2022/03/24 21:28:51 autoscaling.go:277: Adding instances to ASG-TWO
SCE:2022-03-24T21:28:00 2022/03/24 21:28:51 autoscaling_configuration.go:486: Loaded default value (XX+1) from conf percentage.
SCE:2022-03-24T21:28:00 2022/03/24 21:28:51 autoscaling_configuration.go:195: Loaded MinOnDemand value to 0 from tag autospotting_min_on_demand_number
SCE:2022-03-24T21:28:00 2022/03/24 21:28:51 autoscaling_configuration.go:161: Loaded MinOnDemand value to 0.000000 from tag autospotting_min_on_demand_percentage
SCE:2022-03-24T21:28:00 2022/03/24 21:28:51 autoscaling.go:200: Finding spot instances created for ASG-TWO
SCE:2022-03-24T21:28:00 2022/03/24 21:28:51 autoscaling.go:277: Adding instances to ASG-ONE
SCE:2022-03-24T21:28:00 2022/03/24 21:28:51 autoscaling_configuration.go:486: Loaded default value (XX+1) from conf percentage.
SCE:2022-03-24T21:28:00 2022/03/24 21:28:51 autoscaling_configuration.go:195: Loaded MinOnDemand value to 2 from tag autospotting_min_on_demand_number
SCE:2022-03-24T21:28:00 2022/03/24 21:28:51 autoscaling_configuration.go:161: Loaded MinOnDemand value to 0.000000 from tag autospotting_min_on_demand_percentage
SCE:2022-03-24T21:28:00 2022/03/24 21:28:51 autoscaling.go:200: Finding spot instances created for ASG-ONE
SCE:2022-03-24T21:28:00 2022/03/24 21:28:51 autoscaling.go:246: Found unattached spot instance i-03ec3b0486b2fffc1
SCE:2022-03-24T21:28:00 2022/03/24 21:28:51 autoscaling.go:693: ASG-TWO Counting already running on-demand instances
SCE:2022-03-24T21:28:00 2022/03/24 21:28:51 autoscaling.go:706: ASG-TWO Found (XX+1) on-demand instances running on a total of (YY)
SCE:2022-03-24T21:28:00 2022/03/24 21:28:51 autoscaling.go:132: Currently more than enough OnDemand instances running
SCE:2022-03-24T21:28:00 2022/03/24 21:28:51 instance_queries.go:420: Considering  i-03ec3b0486b2fffc1 for attaching to ASG-TWO
SCE:2022-03-24T21:28:00 2022/03/24 21:28:51 instance_queries.go:426: Instance uptime: 27m57s
SCE:2022-03-24T21:28:00 2022/03/24 21:28:51 instance_queries.go:432: The spot instance i-03ec3b0486b2fffc1  has passed grace period and is ready to attach to the group.
SCE:2022-03-24T21:28:00 2022/03/24 21:28:51 autoscaling.go:267: us-east-1 Found spot instance: i-03ec3b0486b2fffc1 Attaching it to ASG-TWO
SCE:2022-03-24T21:28:00 2022/03/24 21:28:51 autoscaling.go:319: ASG-TWO Retrieving instance details for  i-03ec3b0486b2fffc1
SCE:2022-03-24T21:28:00 2022/03/24 21:28:51 autoscaling.go:246: Found unattached spot instance i-07ea92597e523c2d1
SCE:2022-03-24T21:28:00 2022/03/24 21:28:51 autoscaling.go:693: ASG-ONE Counting already running on-demand instances
SCE:2022-03-24T21:28:00 2022/03/24 21:28:51 autoscaling.go:706: ASG-ONE Found (XX) on-demand instances running on a total of (YY+1)
SCE:2022-03-24T21:28:00 2022/03/24 21:28:51 autoscaling.go:132: Currently more than enough OnDemand instances running
SCE:2022-03-24T21:28:00 2022/03/24 21:28:51 instance_queries.go:420: Considering  i-07ea92597e523c2d1 for attaching to ASG-ONE
SCE:2022-03-24T21:28:00 2022/03/24 21:28:51 instance_queries.go:426: Instance uptime: 13m57s
SCE:2022-03-24T21:28:00 2022/03/24 21:28:51 instance_queries.go:437: The spot instance i-07ea92597e523c2d1 is still in the grace period, waiting for it to be ready before we can attach it to the group...
SCE:2022-03-24T21:28:00 2022/03/24 21:28:51 autoscaling.go:261: Spot instance i-07ea92597e523c2d1 not yet ready, waiting for next run while processing ASG-ONE
SCE:2022-03-24T21:28:00 2022/03/24 21:28:51 autoscaling.go:277: Adding instances to ASG-TWO
SCE:2022-03-24T21:28:00 2022/03/24 21:28:52 autoscaling_configuration.go:486: Loaded default value (XX+1) from conf percentage.
SCE:2022-03-24T21:28:00 2022/03/24 21:28:52 autoscaling_configuration.go:195: Loaded MinOnDemand value to 0 from tag autospotting_min_on_demand_number
SCE:2022-03-24T21:28:00 2022/03/24 21:28:52 autoscaling_configuration.go:161: Loaded MinOnDemand value to 0.000000 from tag autospotting_min_on_demand_percentage
SCE:2022-03-24T21:28:00 2022/03/24 21:28:52 instance_queries.go:124: us-east-1 instace i-06d19bedb0d4c1900 belongs to enabled ASG ASG-TWO
SCE:2022-03-24T21:28:00 2022/03/24 21:28:52 autoscaling.go:693: ASG-TWO Counting already running on-demand instances
SCE:2022-03-24T21:28:00 2022/03/24 21:28:52 autoscaling.go:706: ASG-TWO Found (XX+1) on-demand instances running on a total of (YY)
SCE:2022-03-24T21:28:00 2022/03/24 21:28:52 autoscaling.go:132: Currently more than enough OnDemand instances running
SCE:2022-03-24T21:28:00 2022/03/24 21:28:52 autoscaling.go:712: Suspending processes on ASG ASG-TWO
SCE:2022-03-24T21:28:00 2022/03/24 21:28:52 instance_actions.go:105: Attaching spot instance i-03ec3b0486b2fffc1 to the group ASG-TWO
SCE:2022-03-24T21:28:00 2022/03/24 21:28:54 autoscaling.go:431: Waiting for instance i-03ec3b0486b2fffc1 to be in status InService [Pending]
SCE:2022-03-24T21:28:00 2022/03/24 21:29:04 instance_actions.go:116: Terminating on-demand instance i-06d19bedb0d4c1900 from the group ASG-TWO
SCE:2022-03-24T21:28:00 2022/03/24 21:29:04 autoscaling.go:638: us-east-1 ASG-TWO Terminating instance: i-06d19bedb0d4c1900
SCE:2022-03-24T21:28:00 2022/03/24 21:29:04 autoscaling.go:677: Terminating EC2 instance: i-06d19bedb0d4c1900
SCE:2022-03-24T21:28:00 2022/03/24 21:29:04 autoscaling.go:726: Resuming processes on ASG ASG-TWO
SCE:2022-03-24T21:28:00 2022/03/24 21:29:04 main.go:84: us-east-1 ASG-TWO OnDemand instance i-06d19bedb0d4c1900 replaced with spot instance i-03ec3b0486b2fffc1
END RequestId: 9d91c759-fcb7-47c6-a8e2-268571f7a153


SCE:2022-03-24T21:34:00 2022/03/24 21:34:51 region.go:417: Enabling group ASG-ONE for processing because its tags, the currently configured  filtering mode (opt-in) and tag filters are aligned
SCE:2022-03-24T21:34:00 2022/03/24 21:34:51 region.go:417: Enabling group ASG-TWO for processing because its tags, the currently configured  filtering mode (opt-in) and tag filters are aligned
SCE:2022-03-24T21:34:00 2022/03/24 21:34:52 autoscaling.go:277: Adding instances to ASG-TWO
SCE:2022-03-24T21:34:00 2022/03/24 21:34:52 autoscaling_configuration.go:486: Loaded default value (XX+1) from conf percentage.
SCE:2022-03-24T21:34:00 2022/03/24 21:34:52 autoscaling_configuration.go:195: Loaded MinOnDemand value to 0 from tag autospotting_min_on_demand_number
SCE:2022-03-24T21:34:00 2022/03/24 21:34:52 autoscaling_configuration.go:161: Loaded MinOnDemand value to 0.000000 from tag autospotting_min_on_demand_percentage
SCE:2022-03-24T21:34:00 2022/03/24 21:34:52 autoscaling.go:200: Finding spot instances created for ASG-TWO
SCE:2022-03-24T21:34:00 2022/03/24 21:34:52 autoscaling.go:277: Adding instances to ASG-ONE
SCE:2022-03-24T21:34:00 2022/03/24 21:34:52 autoscaling_configuration.go:486: Loaded default value (XX+1) from conf percentage.
SCE:2022-03-24T21:34:00 2022/03/24 21:34:52 autoscaling_configuration.go:195: Loaded MinOnDemand value to 2 from tag autospotting_min_on_demand_number
SCE:2022-03-24T21:34:00 2022/03/24 21:34:52 autoscaling.go:246: Found unattached spot instance i-02afc905985407021
SCE:2022-03-24T21:34:00 2022/03/24 21:34:52 autoscaling.go:693: ASG-TWO Counting already running on-demand instances
SCE:2022-03-24T21:34:00 2022/03/24 21:34:52 autoscaling_configuration.go:161: Loaded MinOnDemand value to 0.000000 from tag autospotting_min_on_demand_percentage
SCE:2022-03-24T21:34:00 2022/03/24 21:34:52 autoscaling.go:706: ASG-TWO Found (XX) on-demand instances running on a total of (YY)
SCE:2022-03-24T21:34:00 2022/03/24 21:34:52 autoscaling.go:132: Currently more than enough OnDemand instances running
SCE:2022-03-24T21:34:00 2022/03/24 21:34:52 instance_queries.go:420: Considering  i-02afc905985407021 for attaching to ASG-TWO
SCE:2022-03-24T21:34:00 2022/03/24 21:34:52 instance_queries.go:426: Instance uptime: 21m58s
SCE:2022-03-24T21:34:00 2022/03/24 21:34:52 instance_queries.go:432: The spot instance i-02afc905985407021  has passed grace period and is ready to attach to the group.
SCE:2022-03-24T21:34:00 2022/03/24 21:34:52 autoscaling.go:267: us-east-1 Found spot instance: i-02afc905985407021 Attaching it to ASG-TWO
SCE:2022-03-24T21:34:00 2022/03/24 21:34:52 autoscaling.go:319: ASG-TWO Retrieving instance details for  i-02afc905985407021
SCE:2022-03-24T21:34:00 2022/03/24 21:34:52 autoscaling.go:200: Finding spot instances created for ASG-ONE
SCE:2022-03-24T21:34:00 2022/03/24 21:34:52 autoscaling.go:214: No spot instances were found for  ASG-ONE
SCE:2022-03-24T21:34:00 2022/03/24 21:34:52 autoscaling.go:693: ASG-ONE Counting already running on-demand instances
SCE:2022-03-24T21:34:00 2022/03/24 21:34:52 autoscaling.go:706: ASG-ONE Found 7 on-demand instances running on a total of (YY+1)
SCE:2022-03-24T21:34:00 2022/03/24 21:34:52 autoscaling.go:132: Currently more than enough OnDemand instances running
SCE:2022-03-24T21:34:00 2022/03/24 21:34:52 autoscaling.go:277: Adding instances to ASG-TWO
SCE:2022-03-24T21:34:00 2022/03/24 21:34:53 instance_queries.go:338:    MATCH FOUND, added m5.4xlarge to launch candidates list for instance i-00e3237b888ca98b3
SCE:2022-03-24T21:34:00 2022/03/24 21:34:53 instance_queries.go:338:    MATCH FOUND, added m5a.4xlarge to launch candidates list for instance i-00e3237b888ca98b3
SCE:2022-03-24T21:34:00 2022/03/24 21:34:53 instance_queries.go:338:    MATCH FOUND, added m5d.4xlarge to launch candidates list for instance i-00e3237b888ca98b3
SCE:2022-03-24T21:34:00 2022/03/24 21:34:53 instance_queries.go:338:    MATCH FOUND, added r5.2xlarge to launch candidates list for instance i-00e3237b888ca98b3
SCE:2022-03-24T21:34:00 2022/03/24 21:34:53 instance_queries.go:338:    MATCH FOUND, added r5a.2xlarge to launch candidates list for instance i-00e3237b888ca98b3
SCE:2022-03-24T21:34:00 2022/03/24 21:34:53 instance_queries.go:338:    MATCH FOUND, added r5a.4xlarge to launch candidates list for instance i-00e3237b888ca98b3
SCE:2022-03-24T21:34:00 2022/03/24 21:34:53 instance_queries.go:338:    MATCH FOUND, added r5ad.2xlarge to launch candidates list for instance i-00e3237b888ca98b3
SCE:2022-03-24T21:34:00 2022/03/24 21:34:53 instance_queries.go:338:    MATCH FOUND, added r5d.2xlarge to launch candidates list for instance i-00e3237b888ca98b3
SCE:2022-03-24T21:34:00 2022/03/24 21:34:53 instance_queries.go:338:    MATCH FOUND, added r5n.2xlarge to launch candidates list for instance i-00e3237b888ca98b3
SCE:2022-03-24T21:34:00 2022/03/24 21:34:54 autoscaling_configuration.go:486: Loaded default value (XX+1) from conf percentage.
SCE:2022-03-24T21:34:00 2022/03/24 21:34:54 autoscaling_configuration.go:195: Loaded MinOnDemand value to 0 from tag autospotting_min_on_demand_number
SCE:2022-03-24T21:34:00 2022/03/24 21:34:54 autoscaling_configuration.go:161: Loaded MinOnDemand value to 0.000000 from tag autospotting_min_on_demand_percentage
SCE:2022-03-24T21:34:00 2022/03/24 21:34:54 instance_queries.go:124: us-east-1 instace i-06d19bedb0d4c1900 belongs to enabled ASG ASG-TWO
SCE:2022-03-24T21:34:00 2022/03/24 21:34:54 autoscaling.go:693: ASG-TWO Counting already running on-demand instances
SCE:2022-03-24T21:34:00 2022/03/24 21:34:54 autoscaling.go:706: ASG-TWO Found (XX) on-demand instances running on a total of (YY)
SCE:2022-03-24T21:34:00 2022/03/24 21:34:54 autoscaling.go:132: Currently more than enough OnDemand instances running
SCE:2022-03-24T21:34:00 2022/03/24 21:34:54 autoscaling.go:712: Suspending processes on ASG ASG-TWO
SCE:2022-03-24T21:34:00 2022/03/24 21:34:54 instance_actions.go:105: Attaching spot instance i-02afc905985407021 to the group ASG-TWO
SCE:2022-03-24T21:34:00 2022/03/24 21:34:55 autoscaling.go:431: Waiting for instance i-02afc905985407021 to be in status InService [Pending]
SCE:2022-03-24T21:34:00 2022/03/24 21:34:57 action.go:48: Successfully launched spot instance i-0cfb8d3b5d31e0afa, exiting...
SCE:2022-03-24T21:34:00 2022/03/24 21:35:05 instance_actions.go:116: Terminating on-demand instance i-06d19bedb0d4c1900 from the group ASG-TWO
SCE:2022-03-24T21:34:00 2022/03/24 21:35:05 autoscaling.go:628: Issue while waiting for instance 0xc001f08328 to start: ResourceNotReady: failed waiting for successful resource state
SCE:2022-03-24T21:34:00 2022/03/24 21:35:05 autoscaling.go:438: Waiting for instance i-06d19bedb0d4c1900 to be in AutoScalingGroup with status InService
SCE:2022-03-24T21:34:00 2022/03/24 21:35:15 autoscaling.go:438: Waiting for instance i-06d19bedb0d4c1900 to be in AutoScalingGroup with status InService
SCE:2022-03-24T21:34:00 2022/03/24 21:35:23 autoscaling.go:438: Waiting for instance i-06d19bedb0d4c1900 to be in AutoScalingGroup with status InService
SCE:2022-03-24T21:34:00 2022/03/24 21:35:29 autoscaling.go:438: Waiting for instance i-06d19bedb0d4c1900 to be in AutoScalingGroup with status InService
SCE:2022-03-24T21:34:00 2022/03/24 21:35:33 autoscaling.go:438: Waiting for instance i-06d19bedb0d4c1900 to be in AutoScalingGroup with status InService
SCE:2022-03-24T21:34:00 2022/03/24 21:35:35 autoscaling.go:438: Waiting for instance i-06d19bedb0d4c1900 to be in AutoScalingGroup with status InService
SCE:2022-03-24T21:34:00 2022/03/24 21:35:36 autoscaling.go:413: Failed waiting instance i-06d19bedb0d4c1900 in status InService
SCE:2022-03-24T21:34:00 2022/03/24 21:35:36 autoscaling.go:633: Instance i-06d19bedb0d4c1900 is still not InService, trying to terminate it anyway.
SCE:2022-03-24T21:34:00 2022/03/24 21:35:36 autoscaling.go:638: us-east-1 ASG-TWO Terminating instance: i-06d19bedb0d4c1900
SCE:2022-03-24T21:34:00 2022/03/24 21:35:37 autoscaling.go:672: AccessDenied: User: arn:aws:sts::AWS-ACCOUNT:assumed-role/autospotting/autospotting is not authorized to perform: autoscaling:TerminateInstanceInAutoScalingGroup because no identity-based policy allows the autoscaling:TerminateInstanceInAutoScalingGroup action
status code: 403, request id: AWS-REQUEST-ID
SCE:2022-03-24T21:34:00 2022/03/24 21:35:37 instance_actions.go:119: On-demand instance i-06d19bedb0d4c1900 couldn't be terminated, re-trying...
SCE:2022-03-24T21:34:00 2022/03/24 21:35:37 autoscaling.go:726: Resuming processes on ASG ASG-TWO
SCE:2022-03-24T21:34:00 2022/03/24 21:35:37 autoscaling.go:327: us-east-1, couldn't perform spot replacement of i-02afc905985407021 
END RequestId: AWS-REQUEST-ID