Repository navigation
Releases: aws-samples/CloudInstanceOptimizer
Release list
v0.5.1
*Added option “useAllSubnets” (Default False) that will try to attach every possible subnet in your region to the CE for maximum EC2 availability. Note you might have to increase IP quotas in your account
*Added GPU benchmarking example
v0.5.0
*Added option to use EFA for multinode instances: "useEFA": true, note that using EFA requires placement groups which is automatically handled by CloudInstanceOptimizer, but if you submit many jobs, placement groups essentially will lock them all into a single subnet, which means you are limited by the ec2 capacity of a single subnet
*Add option to control how many ec2 types are assigned to a compute environment: “ec2_type_per_ce” : 1
*New input “new_ec2_batches” EC2 batches for noisy neighbor checks, this will send all replicates, then wait for all to queues and instances to terminate, then submit another batch, the intent is to ensure different hardware is used to add in potential variability from noisy neighbors
*Benchmark output now includes the available CPU, RAM, and GPU for each instance type and the N nodes used in job run
*Benchmark output now includes the time of execution of the ec2
*Fix error in pricing only working in us-east-1
*Fixed local zone error for subnet creation
*Add “ebs_volume” as in input to alter the EBS size for each ec2 instance, default is 1000GB
*Updated README
v0.4.0
*Add genetic algorithm option via pyGAD
*CDK will now use all available subnets
*CDK security group updated to enable internode communication when running tasks that use distributed computing.
*New json input option “ec2_multinode_count” that tells Batch to provision and hook up more ec2s for a given job
*Both gpu_count and ec2_multinode_count will accept lists for benchmarking a variety of node counts and gpu counts
*Always passing an environmental variable into the container now with number of expected nodes: EXPECTED_NODES=N
*Updated IAM so that jobs can review what is available in an s3 bucket before pulling
*Added exponential backoff to both job submission and job status querying
*SLURM job creation was setup and tested with ParallelCluster if we have a workflow that switches away from AWS Batch. We haven’t written instructions yet, so this isn’t fully exposed
*Added a reload button into the optimization streamlit
*Turned off Ray with awswrangler as this appears to have resulted in lock ups if actual jobs also wanted to use Ray for distributed computing
*When using a custom metric with ec2 instance type optimization, this now should appear in the streamlit drop down options
*Add a new -r command line argument that will reload the json info and then reprocess all of the log files for the ec2data.csv.
v0.3.0
v0.2.0
*added GPU hyperparameter optimization example
*fixed an SDK issue in which not all possible ec2 variations were pulled
*altered Batch execution to run in multiple CE to prevent any sort of backend clogging of CE configurations
*updated streamlit app to use plotly for barcharts to scale to very large numbers of ec2 instance types