PerfKitBenchmarker v1.15.1

v1.15.1

Breaking changes:

• Rename ycsb workloadd to workloadx to show that it is a custom PKB
  write-only workload.

New features:

• Add FairSeq Roberta Masked Multilingual LM benchmark for linux.
• Add bulk delete API operation to object storage API scripts.
• Catch preemptible instance's interrupt event.

Enhancements:

• Enable support for AWS's c6g and r6g families.
• Integrate edw_benchmark with new Client Interface and results aggregation
  framework.
• Update default stress-ng version from 5.23 to 9.25 (default on Ubuntu 1804).
• Added two new workloads(workloadw and workloadw_uniform) for testing mostly
  write use cases(read/write 5/95)

Bug fixes and maintenance updates:

• Parallelize entity deletion processing for a kind.
• Move coremark installation into a package.
• Fixed one object per stream run for object storage service benchmark.
• Benchmarking of AWS Athena now records client observed run time rather that
  Athena's reported run time.
• Use sub processes instead of threads for entity deletion processing.

PerfKitBenchmarker v1.15.0

Breaking changes:

• Remove ubuntu1404 and debian OS types.
  • Ubuntu 14.04 is now in LTS and has not been fully maintained since
    April 2019.
  • debian os_type alias referred to Ubuntu 14.04 on all clouds.
  • Ubuntu 16.04 is now the default.
• Remove the Runtime metric for Horovod.
• Change windows201X os_types to windows201X_core and windows201X_base
  to windows201X_desktop.
  • Windows Server Core is now referred to as windows201X_core.
    • This is recommended for most benchmarks, because PKB does not
      benefit from GUIs except for debugging.
  • Windows Server with Desktop Experience is now referred to as
    windows201X_desktop.
  • See
    the microsoft documentation
    for more information on the differences.
• Remove deprecated versionless windows os_type.
  • You will have to specify windows2012_core.
• Removed deprecated versionless rhel os_type.
  • You will have to specify rhel7 on most providers or amazonlinux1 on
    AWS.
• Deprecate Amazon Linux 1 amazonlinux1 os_type.
  • AL1 is EOL on 2021-01-01:
    https://aws.amazon.com/blogs/aws/update-on-amazon-linux-ami-end-of-life/
    at which point it will be removed from PKB.
  • You can use the recommended amazonlinux2 instead.
• Remove ping_also_run_using_external_ip from ping benchmark
  • You should now use the ip_addresses flag to specify whether to test
    with internal IPs, external IPs or both. This brings ping into alignment
    with how other network benchmarks function.
• The flag --emr-release-label is now deprecated in favor of setting
  dpb_service.version.
• In EMR DPB benchmarks setting: dpb_service.version is now Required.

New features:

• Add infiniband support in nccl benchmark.
• Added AwsVpcS3Endpoint for VPC connectivity to S3.
• Add --tcp_max_receive_buffer flag to set net.ipv4.tcp_rmem sysctl value
• Add --tcp_max_send_buffer flag to set net.ipv4.tcp_wmem sysctl value
• Add --rmem_max flag to set net.core.rmem_max sysctl value
• Add --wmem_max flag to set net.core.wmem_max sysctl value
• Add --os_type=debian9 support for AWS and Azure Providers.
• Add --os_type=debian10 support for GCP, AWS and Azure Providers.
• Add function to tune NCCL parameters.
• Add Coremark demo scripts at PerfKitBenchmarker/tools/demos/coremark.
• Add RHEL 8 to AWS, Azure and GCP providers
• Add CentOS 8 to Azure and GCP providers
• Add BERT models in Horovod benchmark
• Add timeline support in Horovod benchmark
• Add GceNfsService to support Filestore on GCP.
• Add maven package and use it in ycsb.
• Add replication cluster support to bigtable.
• Add AWS and Azure VPC peering support
• Add Snowflake Warehouse support
• Add large_scale_boot benchmark for linux and windows.
• Add Python client library for GCS alongside Boto connection library.
• Enabled OpenMPI version selection
• Adds support for running network benchmarks across a VPN.
  • Includes GCP implementation with static routes.
• Add a directory for PKB tutorials.
• Add two new tutorials: one for beginners and one for network dashboard.
• Update Boto client library for S3 to boto3.
• Add MultiStreamDelete to Object Storage API Tests.
• Add GCE Placement Group support. By default, placement groups are not
  created.
• Add ICMP ping support for Azure using EXTERNAL ip address.

Enhancements:

• Update Specsfs2014 to use the SP2 update rather than SP1.
• Update Multichase version to enable AARCH64 build support.
• Added flag --cassandra_maven_repo_url to use a maven repo when building
  cassandra, for example
  https://maven-central.storage-download.googleapis.com/maven2
• Optimize Azure Files settings.
• Migrate EKS cluster creation to use eksctl (requires a local eksctl).
  • Deprecate --eks_zones in favor of setting
    container_cluster.vm_spec.AWS.zone.
  • Make container_cluster.vm_spec.AWS.zone accept a region or comma
    separated list of zones.
  • Delete the obsolete --eks-verify-ssl
• Separate NVIDIA driver functionality from CUDA toolkit
• Added --runspec_tar option to SpecCPU (specifying installation tarball).
• Added generic support for using PreprovisionedData
  • Added the preprovision_ignore_checksum flag that can be used when
    downloading content that is not registered in PREPROVISIONED_DATA
  • Allow _InstallData in virtual_machine.py to catch the NotImplemented
    error, in order to proceed to downloading resources via their URL
• Implemented PreprovisionedData in the Maven package and make Maven more
  robust
  • Added maven_mirror_url to specify a custom Maven mirror
  • Added sha256 checksums for a few Maven versions
  • Made retrieving JAVA_HOME more robust
  • Fixed a java lib issue on CentOS
• Add OpenFOAM support for timing individual commands.
• Added --gce_subnet_name to specify existing GCE subnet to use.
• Run a Full Sweep (Create/Stat/Delete) on Mdtest when not dropping caches.
• Added version field to dpb_service configs. This corresponds to image
  version in Google Cloud Dataproc and release label in AWS EMR.
• Support delete timing on provider implementations of object_storage_api
  script
• Enable support for AWS's m6g family.
• Add SPEC CPU 2017 flag allowing to build but not run a suite.
• Enable support for specifying newer versions of GCC than what the OS
  supports by default.
• Sort AWS AMIs by CreationDate instead of name to get latest image.
• Add Spec17 configs compatible with v1.1.
• Better support for pytyping.
  • Make BaseVirtualMachine inherit from BaseOsMixin.
• Add connect_via_internal_ip flag for testing windows tests over the internal
  IP, similar to the ssh_via_internal_ip flag for linux.
  • Deprecate --ssh_via_internal_ip in favor of this new flag.

Bug fixes and maintenance updates:

• AWSBaseVirtualMachine subclasses require IMAGE_OWNER and IMAGE_NAME_FILTER.
  Fixes issue where some windows 2012 AMIs were selected from the wrong
  project.
• Retry yum install commands to bypass transient issues.
• Azure defaults to no placement group created.
• Add Sql Server support to managed relational DB
• Remove py27 tox tests.
• Update AKS cluster creation service principle handling, which was broken
• Update glibc version to 2.31.
• Propagate Azure credentials to AKS when running as a service principal. This
  is consistent with propagating VM credentials in azure_credentials.py.
  • This can be disabled by setting
    --bootstrap_azure_service_principal=False.
• Registered AWS VPC quota failures as quota failures.
• Do not try to use the GCP service account that is running PKB inside a GKE
  cluster if it obviously belongs to the wrong project.
• Upgrade psutil version to 5.6.6 from 3.0.0.
• Change default ior script to use 1GB sizes rather than 4GB.
• Remove nfs_gce_ip_range from filestore options.
• Upgrade YCSB to version 0.17.0.
• Do not pass a default --boot-disk-size or --boot-disk-type to GCE.
  • This allows GCE to set the preferred options for both.
  • Fixes an issue where CentOS and RHEL images grew beyond the default 10GB
    (to 20GB).
  • The current defaults leave other Linux OSes at 10GB and Windows OSes at
    50GB.
• Explicitly setting Netperf to Python 3.
• Update PKB cloud datastore ycsb benchmark:
  • change private_keyfile_dir from a constant value to a FLAG
  • delete all db entries as part of Cleanup
• Remove 'default' keyword from AWS and Azure boot_disk_size
• Check ycsb proportion explicitly against none.
• Fix issue in calculating Geometric means.
• Add support to PKB cloud datastore ycsb benchmark to use GCS files for
  datastore keyfiles.
• Add SSH keys to dpb EMR clusters for debuggability.
  • Security groups of the clusters will have to be manually edited to allow
    SSH.
• Cast sample values to floats, ensuring they conform to the specification
• Improve database deletion performance of Cloud datastore YCSB benchmark.
• After installing the python package tries to set default version of python
  if not set.
• Add check in Prepare stage of PKB cloud datastore ycsb benchmark to ensure
  db is empty before running.
• Categorize k8s cluster creation errors better.
• Improve database deletion performance of Cloud datastore YCSB benchmark.
• Change deletion batch size of Cloud datastore YCSB benchmark.

PerfKitBenchmarker 1.14.0

Breaking changes:

• Replaced gflags with absl-py. (GH-1518)
• Renamed GPU-related flags like so (old_flag_value -> new_flag_value):
  • gce_gpu_count -> gpu_count
  • gce_gpu_type -> gpu_type
• Replaced object_storage_objects_written_file* flags with prefix versions.
• Renamed tf_version flag to tf_pip_package flag to allow the user to specify
  the full tensorflow pip package name to be installed, instead of just being
  able to control the version.
• Changed default Tensorflow package when using CPUs to an Intel CPU-optimized
  build.
• Renamed cuda_toolkit_8 to cuda_toolkit
• Migrated cluster boot benchmark default machines to 'default_dual_core'.
• Changed metric name in mnist and inception3.
• Renamed the tf_batch_size flag in tensorflow_benchmark to tf_batch_sizes.
• Updated GCP sole tenancy support. Removed gcp_host_type added
  gcp_node_type.
• Fixed missing installation directory in gpu_pcie_bandwidth benchmark.
• Removed duplicated metrics from the HPCC benchmark. Metrics now use the name
  from the summary section of the benchmark-produced output.
• Benchmarks are expected to not modify FLAGS in any way. If FLAGS are modified
  and multiple run configurations are run in a single PKB invocation, benchmark
  configurations may be incorrect.
• Changed TF Serving benchmark to use ResNet instead of Inception.
• Renamed prepare_sleep_time flag to after_prepare_sleep_time.
• multichase_taskset_options flag changed to multichase_numactl_options.
• Deprecated CUDA version 8.
• Remove support for supplying CUDNN package path via the 'cudnn' flag.
• Raise IssueCommandError by default when vm_util.IssueCommand return code is
  non-zero. Previous behavior can be emulated by setting
  raise_on_failure=False.
• Fix AliCloud support by using the newer aliyun (instead of aliyuncli) cmd
  line utility and removing Paramiko dependency
• Changed support for reusing buckets for Azure blob service benchmarking. The
  storage account and resource group are now named based on the bucket name so
  that subsequent runs can use the same bucket.
• Changed Windows Server versions for GCP and Azure. Before this AWS used
  Windows Server Core (non-graphical), while GCP and Azure used Windows Server
  Base (graphical). All existing windows os_types now refer to the Core
  version and there is a new 'windows*_base' os_types for users who want to RDP
  in and use a full GUI.
• Changed --aws_user_name default from ubuntu to ec2-user. Individual
  OS types including Ubuntu should still override it appropriately.

New features:

• Windows benchmarks can now be run from linux controllers.
• MXNet benchmarks can now be run from linux controllers.
• Added initial support for preprovisioning benchmark binaries in the cloud,
  if binaries are not located in local /data directory.
• YCSB benchmark for Cloud Redis in GCP, Elasticache Redis in AWS, and
  Redis Cache in Azure.
• YCSB benchmark for DynamoDB in AWS.
• Added a flag, run_stage_iterations, which causes a benchmark's run stage to be
  called a specified number of times
• Added cuda_toolkit_version flag
• Added support for CUDA Toolkit 9.0
• Added MXNet support for CUDA Toolkit 9.0
• Added new version of HPCG, with CUDA 9 support
• Added latest gen 'default_dual_core' machine type configuration for AWS, Azure and GCP.
• Added memory as disk type for Linux only.
• Added support for publishing individual dstat samples with
  --dstat_publish_regex.
• Added Inception v3 benchmark that supports CPU, GPU and TPU. This benchmark is
  the same as TensorFlow with inception3 for CPU and GPU.
• Added container image building.
• Added netperf container benchmark.
• Added AWS container registry (ECR).
• Added Azure container registry (ACR).
• Added Google container registry (GCR) and added GKE autoscaling.
• Added create_time to VM metadata.
• Added new ycsb workload where each payload is 1mb versus the default 1kb.
• Added Tensorflow Serving benchmark which tests the throughput and latency of the
  standard model-server using a pre-trained inception3 model.
• Added AWS Fargate support.
• Added the ability to pass arbitrary parameters to tf_cnn_benchmarks.py in the
  Tensorflow benchmark, through the benchmark_args flag.
• Added hdrhistogram support to ycsb package.
• Added support for custom machine types on GKE.
• Added container_cluster_version flag to container_service.py.
• AWS EFS support via "disk_type: nfs"
• Added --disk_fill_size and --after_prepare_sleep_time flags
• Add timeout_minutes flag to assist with cleaning up stale resources
• All AWS resources and Azure resource groups are now tagged. (including timeout_minutes value).
• Added windows udp test using iperf3.
• Added timeout-decorator python package.
• Added timeout support for windows vm remote command call.
• Added IOR benchmark.
• Added mdtest to IOR benchmark.
• Added Elastic Container Service (EKS) as a container cluster type.
• Added ResNet benchmark.
• Added support for ACI (Azure Container Instances).
• Added spec cpu 2017 and feedback optimization for peak runs.
• Added glibc benchmark.
• Added lmbench benchmark.
• Added support to --disable_interrupt_moderation for Windows VMs on AWS.
• Added support to --disable_rss for Windows VMs on GCP.
• Added act benchmark.
• Added --gce_tags flag to add --tags when launching VMs on GCP.
• Added PKB support to publish samples immediately.
• Adding benchmarking of Memcached on cloud VMs using memtier benchmark.
• Adding support for different client/server machine types on memcached-memtier.
• Adding functionality in memtier benchmark to run a variable number of threads
  counts in each run.
• Adding support for AWS ARM machine types.
• Adding support for AWS ARM machines on SPECCPU.
• Added Horovod distributed Tensorflow traning benchmark.
• Added support for capacity reservations for VMs and added AWS implementation.
• Added support for adding additional flags to mount and /etc/fstab.
• Added support for Windows2012, 2016, and 2019 for AWS, Azure, and GCP.
• Terasort implementation on dpb backend.
• Added cluster boot benchmark implementation on dpb backend.
• Support multiple redis versions in cloud redis.
• Azure Files support via "disk_type: smb"
• Added MLPerf benchmark.
• Added stress-ng benchmark.
• Added flag --after_run_sleep_time which instructs PKB to sleep for the number
  of seconds specified after the run stage has completed.
• Added P5 cache type for Azure managed redis.
• Added AWS elasticache Memcached provider.
• Added additional options to the stress-ng benchmark.
• Added support for launching GCE VMs into the standard network tier.
• Added a demo under tools/.
• Added Nginx benchmark.
• Added support for measuring VM reboot time to the cluster boot benchmark.
• Added Cygwin support to WindowsVirtualMachine and enabled a Windows+Cygwin
  version of Coremark.
• Added sar trace utility to measure cpu steal time.
• Added ssh_via_internal_ip flag to use internal IP addresses when ssh'ing to
  runner VMs, will use external IP addrs if not set.
• Added support for launching GCP VM with shielded VM secure boot
• Added Amazon Linux 2 os type. --os_type=amazonlinux2
• Redis enterprise benchmark.
• Added flag --append_kernel_command_line, which allows appending to the
  kernel command line when using Linux os types.
• Added Spark SQL benchmark.
• Added aws_dynamodb_connectMax flag to vary the maximum number of connections
  to dynamodb.
• Added helpmatchmd flag to dump markdown formatted help strings
• Enable specifying source ip when creating a firewall rule.
• Added support for T4 GPUs on GCE.
• Added mpstat utility to measure cpu processor stats.
• Added a benchmark to run spark application on a cluster that computes an approximation to pi.
• Added a benchmark to run spark io workload on a cluster that provisions a database and queries it.
• Added flag --record_lscpu=True to record lscpu data as its own metric "lscpu"
• Added support for the Clear Linux OS type for AWS and static VMs (e.g. --os-type=clear)
• Added flag --placement_group_style to allow or disable different AWS placement groups options for ec2 VM benchmarks.
  Flag supports cluster, spread, and none placement group options.
• Added support for availability zones for Azure. Format for availability zone support is "location-availability_zone".
  Example: eastus2-1 specifies Azure location eastus2 with availability zone 1.
  A PKB zone can be either a Azure location or an Azure location with an availability zone.
• Added resource class and mapping for AWS Athena Service.
• Added support for Azure Dedicated Hosting. Simply add flag --dedicated_hosts=True to use.
• Records two new samples for /proc/cpuinfo data.
• Added support for specifying a number of vms per host for Azure & AWS Dedicated Hosting.
  Simply add flag num_vms_per_host=<#> to use. Benchmark fails if the # of vms per host specified exceeds the memory capacity of the host.
• Added support for skipping the creation of Azure availability sets with the --azure_availability_set=False flag (fixes #1863)
• Added support for AWS EFA networking with --aws_efa=True flag.
• Added NCCL benchmark for GPU networking.
• Added AWS DAX provider.
• Added Google Cloud Firestore ycsb benchmarks.
• Added support for un-managed data processing yarn cluster benchmarking.
• Added support for UDP_STREAM tests to netperf_benchmark
• Added flags to select stream size to netperf_benchmark
• Added placement group flag support for Azure. "cluster" will create proximity placement group.
  "spread" will create an availability set.
• Added a tcpdump collector with --tcpdump flag.
• Added ability to load datasets into BigQuery from csv files on GCS.
• Updated AzureVirtualNetwork to support location level virtual nets, which allows support for cross zone benchma...

v1.13.0: Release 1.13.0

Changes in this release:

Breaking changes:
- Removed gpus_per_node metadata from stencil2d and hpcg benchmarks
(GH-1455)
- Removed legacy disk metadata (use data_disk\* not scratch_disk\* and
removed
  num_striped_disks + aws_provisioned_iops metadata) (GH-1506)

New features:
- Added YCSB benchmark for Cloud Spanner (GH-1387 from @haih-g)
- Added CUDA-enabled HPCG benchmark (GH-1395)
- Added ManagedRelationalDatabase base classes and tests (GH-1405)
- Added gpus to GceVmSpec. This is now the only way to create VMs with
gpus on GCE due to a gcloud API change (GH-1406)
- Added flag_zip_defs which functions like flag_matrix_defs, but
performs a zip
  operation on the axes instead of a cross-product (GH-1414)
- Added TensorFlow Benchmarks. (GH-1420)
- Added --network_enable_BBR flag (GH-1515)
- Added pgbench benchmark for managed PostgreSQL (GH-1505)

Enhancements:
- Added basic auth support for Mesos provider. (GH-1390)
- Added --failed_run_samples_error_length flag to limit failed run error
length (GH-1391)
- Added `__eq__` and `__ne__` to IntegerList (GH-1395)
- Added total_free_memory_kb to VirtualMachine class and implemented it
for
  Linux vms (GH-1397)
- Created hpc_util for a place to share common HPC functions
- Added --runspec_estimate_spec flag to calculate an estimated spec
score (GH-1401)
- Added all HPCC metrics as samples when running HPCC (GH-1448)
- Added support for P100 gpus on Google Cloud Platform (GH-1450)
- Added gpu type to cuda_toolkit_8 metadata (GH-1453)
- Added num_gpus to cuda_toolkit_8 metadata (GH-1455)
- Added range mode to gpu_pcie_bandwidth which calculates average
bandwidth over
  a provided range of transfer sizes (GH-1482)
- Added size support to fio_benchmark scenarios (GH-1489)
- Added RemoteCommandWithReturnValue (GH-1516)

Bug fixes and maintenance updates:
- Fixed provision phase of memcached_ycsb benchmark for non-managed
memcached instances (GH-1384)
- Fixed GPU benchmarks on GCE to work with new GPU API (GH-1407)
- Changed cuda_toolkit_8 to use the gpu-specific base clock speed as the
default (GH-1453)
- Changed default AWS P2 region for gpu benchmarks (GH-1454)
- Fix multi-threading issue in fio.job filename (GH-1474)
- Decreased speccpu2006 default disk size to 50gb (GH-1484)
- Updated azure library version 1.0.3 for blob storage (GH-1491)
- Added azure host_caching to metadata (GH-1500)
- Fixed fio histograms when generated with numjobs>1 (GH-1501)

v1.12.0

Breaking changes:

• Azure CLI updated to 2.0 (GH-1359)
• Update gflags to version 3.1.1 (GH-1322)

New features:

• Azure container service (GH-1361)
• Add Reboot() method to BaseOsMixin (GH-1356)
• Add container cluster classes (GH-1355)
• Optional --completion_status_file flag to write the completion status of every config to a json file (GH-1349)
• Add --gcp_min_cpu_platform flag (GH-1336)
• Add GCP sole tenancy support (GH-1333)
• Add SHOC package (GH-1332)
• Add Stencil2D benchmark (GH-1332)
• Managed memcache service abstraction and a backend for AWS ElastiCache (GH-1235)
• Add Apache Beam benchmark (GH-1324)
• Add image_project to GCP YAML config options (GH-1319)
• Add support for AWS i3 (GH-1308)
• Optionally create failed run samples with --create_failed_run_samples (GH-1363)
• Add --storage_size flag for mysql_service benchmark (GH-1360 from @sspano)

Enhancements:

• Add optional step parameter to IntegerList (GH-1330)
• Upgrade to NVIDIA's latest version of CUDA Toolkit 8 (GH-1304)
• Add ID attribute to GCE VMs (GH-1318)
• Add ability to install netperf from local tar file (GH-1364)

Bug fixes and maintenance updates:

• Fix up README.md - incorrect command line example and added note on Amazon ES publisher (GH-1350 from @Lirt)
• Fix up README.md - remove outdated comments about GCP requirements (GH-1354 from @sspano)
• Fix creating instance with local SSD using NVMe interface (GH-1287)
• Make flag values consistent with gcloud CLI "SCSI" and "NVME" (GH-1287)
• Store os types in list instead of tuple so that it can be extended by third parties (GH-1348)
• Don't create host during VM Create if num_vms_per_host is specified (GH-1352)
• Fix up kubernetes provider (GH-1351)
• Add --region to describe/cancel spot-instance-request commands (GH-1347)
• Install sysbench05plus package in a separate path from sysbench (GH-1338)
• Change --openstack_additional_flags to be a list flag (GH-1340)
• Move gflags installation from netperf benchmark to a package (GH-1341)
• Move package installation of netperf from benchmark to a package (GH-1341)
• Use snake_case gflags APIs (GH-1328)
• Check if apt-update has been called in InstallPackages, and call it if it has not been (GH-1334)
• Fix broken regular expression in blazemark benchmark (GH-1311)
• Fix CloudSuite license page link (GH-1323)
• Update hadoop version (GH-1327)
• Use regional storage class for GCS buckets instead of DRA (GH-1326)
• Fix gflags FLAGS object initialization in tests (GH-1313)
• Fix bug with exception name in azure_network.py (GH-1314)
• Always set GPU clock speed in cuda toolkit installation (GH-1321)
• Specify image_project in gpu_pcie_bandwidth benchmark config (GH-1321)
• Move gpu clock speed logic to cuda_toolkit_8 module (GH-1279)
• Add timeout to netperf remote script (GH-1306)
• Don't assume that gcloud is in PATH in providers.gcp.util.GetDefaultProject (GH-1366)
• Make AWS S3 bucket deletion retryable (GH-1343)
• Empty bucket at the end of the Run stage of the object_storage_service benchmark (GH-1343)

v1.11.0: Release 1.11.0.

External contributions:
- Add ability to publish to InfluxDB (thanks @besanradwan GH-1278)
- Always set internal IP-Address on OpenStack VMs (thanks @yremmet
GH-1228)
- Add CloudSuite data-analytics benchmark (thanks @ivonindza GH-1285)
- Update README.md (thanks @ianmaddox GH-1291)
- Fix a small typo in the anti-affinity FLAG lookup (thanks @joelio
GH-1284)
- Deprecate flag openstack_volume_size (thanks @vhe182 GH-1220)
- Add ProfitBricks cloud api v3 support (thanks @aajdinov GH-1264)
- Enable OpenStack Cinder Volume Type Option (thanks @rossmartyn04
GH-1197)

New features:
- Add dpb_wordcount_benchmark (GH-1253)

Enhancements:
- Add helpmatch usage instructions (GH-1268)
- Add openStack_volume_type option decoder (GH-1267)
- Upgrade OpenStack CLI dependency to 3.6.0 (GH-1265)
- Add object storage service cold reads support, various improvements
(GH-1290, GH-1274)
- Add support for AWS Spot Instances (GH-1288)
- Bump fio version to v2.17 and enable histogram (GH-1300)
- Let publisher.py republish already-collected samples (GH 1299)
- Allow specifying temp dir (GH-1277)
- Add --json_write_mode (GH-1276)
- Allow override hpcc binaries (GH-1269)

Bug fixes and maintenance updates:
- Lock samples json file before writing to it (GH-1265)
- Fixed exception caused by publishers being None (GH-1302)
- Add error and warning log level options to reduce log size (GH-1301)
- Use force for DigitalOcean deletes (GH-1281)
- Print helpmatch usage instructions (GH-1280)
- Fix fio parser regex (GH-1275)
- Specify AWS region when calling describe-images (GH 1273)
- Make CheckPrerequisites more comprehensive (GH-1272)
- Use image_project flag in disk creation if set (GH-1271)
- Minor bug fixes and configuration improvements (GH-1262)

v1.10.0

v1.10.0

External contributions:

• Re-write flag_util_test.py so it's easier to read (thanks @yilei GH-1244)
• Fix bug that prevents running bonnie++ on Azure (thanks @rgodha GH-1218)
• Add kernel compile benchmark (thanks @maggiejiao GH-1215, GH-1255)
• Change the documentation to reflect that work should be on master (thanks
  @besanradwan GH-1209)
• Add flags to capture latency, bandwidth and IOPS logs in Fio benchmark (thanks @meteorfox GH-1205)
• Re-add 'vm_util' object to fix broken master branch (thanks @vhe182 GH-1196)
• Add EPEL Repo to Silo Installation (thanks @akrzos GH-1153)

New features:

• Add GPU PCIe bandwidth test (GH-1234, GH-1250, GH-1252, GH-1254)
• Add blazemark (GH-1145)
• Add simple memcached ycsb benchmark based on aerospike ycsb benchmark (GH-1199)
• Add scimark2 installation package (GH-1185)

Enhancements:

• Use FLAGS.aws_user_name if set for Rhel (GH-1249)
• Fio enhancements (GH-1246)
• Use aes128-ctr if aes128-cbc isn't available (GH-1242)
• Make aeropsike able to run with raw devices on static vms (GH-1243)
• Add some metadata to netperf samples (GH-1236)
• Remove unneeded Azure code and add availability sets (GH-1232)
• Add cassandra read concurrency flag (GH-1233)
• Add a metadata attribute 'ssd_count' to spark_service (GH-1229)
• Add a metadata attribute 'spark_svc_cloud' to spark_service (GH-1226)
• Add ability to specify boot_disk_size in AWS vm spec (GH-1231)
• Add a metadata attribute 'spark_svc_cloud' (GH-1236)
• Set a default zone for the spark test to allow for subnet creation (GH-1222)
• Update the location of the spark examples jar on the emr cluster (GH-1222)
• Persist package installation across reboots (GH-1185)
• Allow mongodb_ycsb to specify readahead settings (GH-1256)

Bug fixes and maintenance updates:

• Close Aersospike connection in 1 second (GH-1241)
• Fix bug with having multiple Azure data disks (GH-1223)
• Fix aws spark service (GH-1222)
• Add some git ignore entries (GH-1222)
• Fix gcs credential conflict (GH-1221)
• Fix bug with run_processes (GH-1212)
• Remove collector from RunBenchmarkTask args (GH-1210)
• Add the minimum tox version to the "tox is not installed" message (GH-1201)
• Move sysbench05plus path prefix logic into package (GH-1185)
• Fix bug with specifying Azure image (GH-1204)
• Fix two files that were failing boilerplate check (GH-1203)
• Update README.md (GH-1200)

v1.9.0: Release 1.9.0

Changes in this release.

New features:
- Add config file imports and allow for circular imports and add documentation (GH-1163)
- Add a way to run benchmarks in parallel (GH-1192)

Enhancements:
- ContainerizedDebianMixin: Ping to a fixed docker image and remove the sudo hack (GH-1171)
- Auto detect  openjdk, libsnappy package versions (GH-1181)
- Add flags to govern subnet creation and Optionally create only 1 subnet (GH-1182)
- Install docker images as packages (GH-1184)
- Call FLAGS.get_help if it's available (GH-1187)
- Publish boot time samples for all tests (GH-1156)

Bugfixes and maintenance updates:
- Update requirements.txt to set version of contextlib2 (GH-1164)
- Fix bug with config flags (GH-1165)
- Updated the cloudsuite web-serving benchmark (thanks @nooshin-mirzadeh, GH-1166)
- Fix load command parsing (GH-1168)
- openstack: Open all TCP and UDP ports for the internal network (GH-1169)
- Fix aerospike metadata (GH-1170)
- Adjust aerospike default replication factor (GH-1172)
- Try replacing FlagValues._flags instead of FlagValues.FlagDict first (GH-1175)
- Fix ycsb aggregator (GH-1176)
- Fix bug during cleanup phase of object_storage_service benchmark (GH-1178)
- Netperf thinktime fixes and support think time in nanoseconds instead of microseconds (GH-1179, GH-1186, GH-1188)
- Fix sysbench05plus installation on ubuntu16.04 (GH-1183)
- Fix race condition in Aerospike benchmark (GH-1190)
- Clean up some pickling, unpickling issues (GH-1191)
- Calculate free ram with more robust /proc/meminfo instead of `free`(GH-1194)
- Fix multithread netperf (GH-1149)
- Multiple fixes and refactoring of the linux package management on vms (GH-1152)
- Fix dstat metadata in Analyze (GH-1154)
- Obey flag overrides for static vm specs (GH-1155)
- Fix links in README file for OpenStack and Cloudstack setup steps (thanks @shakhat, GH-1157)
- Fix issue with GetConfig using flags (GH-1160)
- Remove scratch disk from netperf and object_storage_service benchmark (GH-1147, GH-1148)
- Netperf changes to add num_streams metadata to samples (GH-1177)
- Remove collector from RunBenchmarkTask args (GH-1210)

v1.8.1: Bugfixes and maintanence updates:

 - Fixed default machine type for Azure (GH-1161)

Release v1.8.0

New features:
- Multithreaded netperf benchmark (GH-1141, GH-1144)
- Terasort benchmark using spark service infrastructure (GH-1104)
- Support 10 or more disks on Azure (GH-1102)
- Add sysctl and set_files flags (GH-1096)
- Support DigitalOcean block storage (GH-1092)
- Add cloudsuite_data_serving_rec_count and cloudsuite_data_serving_op_count parameters to cloudsuite_data_serving benchmark (GH-1091)
- Add support for setting GCE boot disk size and type (GH-1106)
- Add flag to skip reload on ycsb benchmark (GH-1108)
- Support for AWS dedicated hosts (GH-1109)
- Object storage multistream write-only scenario (GH-1123)
- Allow multiple configs for the same benchmark in the same file (GH-1126)
- Add flat that allows run retries (GH-1129)
- Add fio_parameters flag (GH-1131)
- Add config matrices (GH-1134)
- Add events at the beginning and end of benchmarks (GH-1135)
- Add extra_zones flag to pkb (GH-1137)
- Add flag matrix filters (GH-1138)
- Support allowing port ranges on Firewall.AllowPort (GH-1139)
- Add static vm tags (GH-1142)

Breaking changes:
- Because of GH-1112, region names have changed.

Enhancements:
- Allow speccpu to run individual test (GH-1113)
- Use Azure resource management mode (GH-1112)
- GCP networks use subnets (GH-1087)
- Add top level flags key to configs (GH-1125)
- Precise resource usage for YCSB benchmarks (GH-1127)

Bugfixes and maintanence updates:
- Fix bugs with Terasort (GH-1136)
- Don't run container commands as sudo with ContainerizedDebianMixin (GH-1130)
- Automatically use amazon linux if os_type is rhel (GH-1124)
- Removed Panda as a dependency (GH-1118, GH-1119)
- Allow flag_util.ParseKeyValuePairs values to have colons in them (GH-1117)
- Add ycsb to data directory (GH-1116)
- Fix bug in mongodb_ycsb_benchmark cleanup (GH-1115)
- Fix bug where static VM disk specs were overriden (GH-1105)
- Use string formatting instead of addition to concatenate (GH-1103)
- Fixed regression with fio flag (GH-1097)
- Fix flags in static vm files (GH-1093)
- Fix wait_for_command.py: wait for status file to avoid race condition (GH-1090)