Skip to content

@bvliu bvliu released this Jun 25, 2020


Breaking changes:

  • Rename ycsb workloadd to workloadx to show that it is a custom PKB
    write-only workload.

New features:

  • Add FairSeq Roberta Masked Multilingual LM benchmark for linux.
  • Add bulk delete API operation to object storage API scripts.
  • Catch preemptible instance's interrupt event.


  • Enable support for AWS's c6g and r6g families.
  • Integrate edw_benchmark with new Client Interface and results aggregation
  • Update default stress-ng version from 5.23 to 9.25 (default on Ubuntu 1804).
  • Added two new workloads(workloadw and workloadw_uniform) for testing mostly
    write use cases(read/write 5/95)

Bug fixes and maintenance updates:

  • Parallelize entity deletion processing for a kind.
  • Move coremark installation into a package.
  • Fixed one object per stream run for object storage service benchmark.
  • Benchmarking of AWS Athena now records client observed run time rather that
    Athena's reported run time.
  • Use sub processes instead of threads for entity deletion processing.
Assets 2

@bvliu bvliu released this Jun 12, 2020

Breaking changes:

  • Remove ubuntu1404 and debian OS types.
    • Ubuntu 14.04 is now in LTS and has not been fully maintained since
      April 2019.
    • debian os_type alias referred to Ubuntu 14.04 on all clouds.
    • Ubuntu 16.04 is now the default.
  • Remove the Runtime metric for Horovod.
  • Change windows201X os_types to windows201X_core and windows201X_base
    to windows201X_desktop.
    • Windows Server Core is now referred to as windows201X_core.
      • This is recommended for most benchmarks, because PKB does not
        benefit from GUIs except for debugging.
    • Windows Server with Desktop Experience is now referred to as
    • See
      the microsoft documentation
      for more information on the differences.
  • Remove deprecated versionless windows os_type.
    • You will have to specify windows2012_core.
  • Removed deprecated versionless rhel os_type.
    • You will have to specify rhel7 on most providers or amazonlinux1 on
  • Deprecate Amazon Linux 1 amazonlinux1 os_type.
  • Remove ping_also_run_using_external_ip from ping benchmark
    • You should now use the ip_addresses flag to specify whether to test
      with internal IPs, external IPs or both. This brings ping into alignment
      with how other network benchmarks function.
  • The flag --emr-release-label is now deprecated in favor of setting
  • In EMR DPB benchmarks setting: dpb_service.version is now Required.

New features:

  • Add infiniband support in nccl benchmark.
  • Added AwsVpcS3Endpoint for VPC connectivity to S3.
  • Add --tcp_max_receive_buffer flag to set net.ipv4.tcp_rmem sysctl value
  • Add --tcp_max_send_buffer flag to set net.ipv4.tcp_wmem sysctl value
  • Add --rmem_max flag to set net.core.rmem_max sysctl value
  • Add --wmem_max flag to set net.core.wmem_max sysctl value
  • Add --os_type=debian9 support for AWS and Azure Providers.
  • Add --os_type=debian10 support for GCP, AWS and Azure Providers.
  • Add function to tune NCCL parameters.
  • Add Coremark demo scripts at PerfKitBenchmarker/tools/demos/coremark.
  • Add RHEL 8 to AWS, Azure and GCP providers
  • Add CentOS 8 to Azure and GCP providers
  • Add BERT models in Horovod benchmark
  • Add timeline support in Horovod benchmark
  • Add GceNfsService to support Filestore on GCP.
  • Add maven package and use it in ycsb.
  • Add replication cluster support to bigtable.
  • Add AWS and Azure VPC peering support
  • Add Snowflake Warehouse support
  • Add large_scale_boot benchmark for linux and windows.
  • Add Python client library for GCS alongside Boto connection library.
  • Enabled OpenMPI version selection
  • Adds support for running network benchmarks across a VPN.
    • Includes GCP implementation with static routes.
  • Add a directory for PKB tutorials.
  • Add two new tutorials: one for beginners and one for network dashboard.
  • Update Boto client library for S3 to boto3.
  • Add MultiStreamDelete to Object Storage API Tests.
  • Add GCE Placement Group support. By default, placement groups are not
  • Add ICMP ping support for Azure using EXTERNAL ip address.


  • Update Specsfs2014 to use the SP2 update rather than SP1.
  • Update Multichase version to enable AARCH64 build support.
  • Added flag --cassandra_maven_repo_url to use a maven repo when building
    cassandra, for example
  • Optimize Azure Files settings.
  • Migrate EKS cluster creation to use eksctl (requires a local eksctl).
    • Deprecate --eks_zones in favor of setting
    • Make accept a region or comma
      separated list of zones.
    • Delete the obsolete --eks-verify-ssl
  • Separate NVIDIA driver functionality from CUDA toolkit
  • Added --runspec_tar option to SpecCPU (specifying installation tarball).
  • Added generic support for using PreprovisionedData
    • Added the preprovision_ignore_checksum flag that can be used when
      downloading content that is not registered in PREPROVISIONED_DATA
    • Allow _InstallData in to catch the NotImplemented
      error, in order to proceed to downloading resources via their URL
  • Implemented PreprovisionedData in the Maven package and make Maven more
    • Added maven_mirror_url to specify a custom Maven mirror
    • Added sha256 checksums for a few Maven versions
    • Made retrieving JAVA_HOME more robust
    • Fixed a java lib issue on CentOS
  • Add OpenFOAM support for timing individual commands.
  • Added --gce_subnet_name to specify existing GCE subnet to use.
  • Run a Full Sweep (Create/Stat/Delete) on Mdtest when not dropping caches.
  • Added version field to dpb_service configs. This corresponds to image
    version in Google Cloud Dataproc and release label in AWS EMR.
  • Support delete timing on provider implementations of object_storage_api
  • Enable support for AWS's m6g family.
  • Add SPEC CPU 2017 flag allowing to build but not run a suite.
  • Enable support for specifying newer versions of GCC than what the OS
    supports by default.
  • Sort AWS AMIs by CreationDate instead of name to get latest image.
  • Add Spec17 configs compatible with v1.1.
  • Better support for pytyping.
    • Make BaseVirtualMachine inherit from BaseOsMixin.
  • Add connect_via_internal_ip flag for testing windows tests over the internal
    IP, similar to the ssh_via_internal_ip flag for linux.
    • Deprecate --ssh_via_internal_ip in favor of this new flag.

Bug fixes and maintenance updates:

  • AWSBaseVirtualMachine subclasses require IMAGE_OWNER and IMAGE_NAME_FILTER.
    Fixes issue where some windows 2012 AMIs were selected from the wrong
  • Retry yum install commands to bypass transient issues.
  • Azure defaults to no placement group created.
  • Add Sql Server support to managed relational DB
  • Remove py27 tox tests.
  • Update AKS cluster creation service principle handling, which was broken
  • Update glibc version to 2.31.
  • Propagate Azure credentials to AKS when running as a service principal. This
    is consistent with propagating VM credentials in
    • This can be disabled by setting
  • Registered AWS VPC quota failures as quota failures.
  • Do not try to use the GCP service account that is running PKB inside a GKE
    cluster if it obviously belongs to the wrong project.
  • Upgrade psutil version to 5.6.6 from 3.0.0.
  • Change default ior script to use 1GB sizes rather than 4GB.
  • Remove nfs_gce_ip_range from filestore options.
  • Upgrade YCSB to version 0.17.0.
  • Do not pass a default --boot-disk-size or --boot-disk-type to GCE.
    • This allows GCE to set the preferred options for both.
    • Fixes an issue where CentOS and RHEL images grew beyond the default 10GB
      (to 20GB).
    • The current defaults leave other Linux OSes at 10GB and Windows OSes at
  • Explicitly setting Netperf to Python 3.
  • Update PKB cloud datastore ycsb benchmark:
    • change private_keyfile_dir from a constant value to a FLAG
    • delete all db entries as part of Cleanup
  • Remove 'default' keyword from AWS and Azure boot_disk_size
  • Check ycsb proportion explicitly against none.
  • Fix issue in calculating Geometric means.
  • Add support to PKB cloud datastore ycsb benchmark to use GCS files for
    datastore keyfiles.
  • Add SSH keys to dpb EMR clusters for debuggability.
    • Security groups of the clusters will have to be manually edited to allow
  • Cast sample values to floats, ensuring they conform to the specification
  • Improve database deletion performance of Cloud datastore YCSB benchmark.
  • After installing the python package tries to set default version of python
    if not set.
  • Add check in Prepare stage of PKB cloud datastore ycsb benchmark to ensure
    db is empty before running.
  • Categorize k8s cluster creation errors better.
  • Improve database deletion performance of Cloud datastore YCSB benchmark.
  • Change deletion batch size of Cloud datastore YCSB benchmark.
Assets 2

@bvliu bvliu released this Jan 13, 2020

Breaking changes:

  • Replaced gflags with absl-py. (GH-1518)
  • Renamed GPU-related flags like so (old_flag_value -> new_flag_value):
    • gce_gpu_count -> gpu_count
    • gce_gpu_type -> gpu_type
  • Replaced object_storage_objects_written_file* flags with prefix versions.
  • Renamed tf_version flag to tf_pip_package flag to allow the user to specify
    the full tensorflow pip package name to be installed, instead of just being
    able to control the version.
  • Changed default Tensorflow package when using CPUs to an Intel CPU-optimized
  • Renamed cuda_toolkit_8 to cuda_toolkit
  • Migrated cluster boot benchmark default machines to 'default_dual_core'.
  • Changed metric name in mnist and inception3.
  • Renamed the tf_batch_size flag in tensorflow_benchmark to tf_batch_sizes.
  • Updated GCP sole tenancy support. Removed gcp_host_type added
  • Fixed missing installation directory in gpu_pcie_bandwidth benchmark.
  • Removed duplicated metrics from the HPCC benchmark. Metrics now use the name
    from the summary section of the benchmark-produced output.
  • Benchmarks are expected to not modify FLAGS in any way. If FLAGS are modified
    and multiple run configurations are run in a single PKB invocation, benchmark
    configurations may be incorrect.
  • Changed TF Serving benchmark to use ResNet instead of Inception.
  • Renamed prepare_sleep_time flag to after_prepare_sleep_time.
  • multichase_taskset_options flag changed to multichase_numactl_options.
  • Deprecated CUDA version 8.
  • Remove support for supplying CUDNN package path via the 'cudnn' flag.
  • Raise IssueCommandError by default when vm_util.IssueCommand return code is
    non-zero. Previous behavior can be emulated by setting
  • Fix AliCloud support by using the newer aliyun (instead of aliyuncli) cmd
    line utility and removing Paramiko dependency
  • Changed support for reusing buckets for Azure blob service benchmarking. The
    storage account and resource group are now named based on the bucket name so
    that subsequent runs can use the same bucket.
  • Changed Windows Server versions for GCP and Azure. Before this AWS used
    Windows Server Core (non-graphical), while GCP and Azure used Windows Server
    Base (graphical). All existing windows os_types now refer to the Core
    version and there is a new 'windows*_base' os_types for users who want to RDP
    in and use a full GUI.
  • Changed --aws_user_name default from ubuntu to ec2-user. Individual
    OS types including Ubuntu should still override it appropriately.

New features:

  • Windows benchmarks can now be run from linux controllers.
  • MXNet benchmarks can now be run from linux controllers.
  • Added initial support for preprovisioning benchmark binaries in the cloud,
    if binaries are not located in local /data directory.
  • YCSB benchmark for Cloud Redis in GCP, Elasticache Redis in AWS, and
    Redis Cache in Azure.
  • YCSB benchmark for DynamoDB in AWS.
  • Added a flag, run_stage_iterations, which causes a benchmark's run stage to be
    called a specified number of times
  • Added cuda_toolkit_version flag
  • Added support for CUDA Toolkit 9.0
  • Added MXNet support for CUDA Toolkit 9.0
  • Added new version of HPCG, with CUDA 9 support
  • Added latest gen 'default_dual_core' machine type configuration for AWS, Azure and GCP.
  • Added memory as disk type for Linux only.
  • Added support for publishing individual dstat samples with
  • Added Inception v3 benchmark that supports CPU, GPU and TPU. This benchmark is
    the same as TensorFlow with inception3 for CPU and GPU.
  • Added container image building.
  • Added netperf container benchmark.
  • Added AWS container registry (ECR).
  • Added Azure container registry (ACR).
  • Added Google container registry (GCR) and added GKE autoscaling.
  • Added create_time to VM metadata.
  • Added new ycsb workload where each payload is 1mb versus the default 1kb.
  • Added Tensorflow Serving benchmark which tests the throughput and latency of the
    standard model-server using a pre-trained inception3 model.
  • Added AWS Fargate support.
  • Added the ability to pass arbitrary parameters to in the
    Tensorflow benchmark, through the benchmark_args flag.
  • Added hdrhistogram support to ycsb package.
  • Added support for custom machine types on GKE.
  • Added container_cluster_version flag to
  • AWS EFS support via "disk_type: nfs"
  • Added --disk_fill_size and --after_prepare_sleep_time flags
  • Add timeout_minutes flag to assist with cleaning up stale resources
  • All AWS resources and Azure resource groups are now tagged. (including timeout_minutes value).
  • Added windows udp test using iperf3.
  • Added timeout-decorator python package.
  • Added timeout support for windows vm remote command call.
  • Added IOR benchmark.
  • Added mdtest to IOR benchmark.
  • Added Elastic Container Service (EKS) as a container cluster type.
  • Added ResNet benchmark.
  • Added support for ACI (Azure Container Instances).
  • Added spec cpu 2017 and feedback optimization for peak runs.
  • Added glibc benchmark.
  • Added lmbench benchmark.
  • Added support to --disable_interrupt_moderation for Windows VMs on AWS.
  • Added support to --disable_rss for Windows VMs on GCP.
  • Added act benchmark.
  • Added --gce_tags flag to add --tags when launching VMs on GCP.
  • Added PKB support to publish samples immediately.
  • Adding benchmarking of Memcached on cloud VMs using memtier benchmark.
  • Adding support for different client/server machine types on memcached-memtier.
  • Adding functionality in memtier benchmark to run a variable number of threads
    counts in each run.
  • Adding support for AWS ARM machine types.
  • Adding support for AWS ARM machines on SPECCPU.
  • Added Horovod distributed Tensorflow traning benchmark.
  • Added support for capacity reservations for VMs and added AWS implementation.
  • Added support for adding additional flags to mount and /etc/fstab.
  • Added support for Windows2012, 2016, and 2019 for AWS, Azure, and GCP.
  • Terasort implementation on dpb backend.
  • Added cluster boot benchmark implementation on dpb backend.
  • Support multiple redis versions in cloud redis.
  • Azure Files support via "disk_type: smb"
  • Added MLPerf benchmark.
  • Added stress-ng benchmark.
  • Added flag --after_run_sleep_time which instructs PKB to sleep for the number
    of seconds specified after the run stage has completed.
  • Added P5 cache type for Azure managed redis.
  • Added AWS elasticache Memcached provider.
  • Added additional options to the stress-ng benchmark.
  • Added support for launching GCE VMs into the standard network tier.
  • Added a demo under tools/.
  • Added Nginx benchmark.
  • Added support for measuring VM reboot time to the cluster boot benchmark.
  • Added Cygwin support to WindowsVirtualMachine and enabled a Windows+Cygwin
    version of Coremark.
  • Added sar trace utility to measure cpu steal time.
  • Added ssh_via_internal_ip flag to use internal IP addresses when ssh'ing to
    runner VMs, will use external IP addrs if not set.
  • Added support for launching GCP VM with shielded VM secure boot
  • Added Amazon Linux 2 os type. --os_type=amazonlinux2
  • Redis enterprise benchmark.
  • Added flag --append_kernel_command_line, which allows appending to the
    kernel command line when using Linux os types.
  • Added Spark SQL benchmark.
  • Added aws_dynamodb_connectMax flag to vary the maximum number of connections
    to dynamodb.
  • Added helpmatchmd flag to dump markdown formatted help strings
  • Enable specifying source ip when creating a firewall rule.
  • Added support for T4 GPUs on GCE.
  • Added mpstat utility to measure cpu processor stats.
  • Added a benchmark to run spark application on a cluster that computes an approximation to pi.
  • Added a benchmark to run spark io workload on a cluster that provisions a database and queries it.
  • Added flag --record_lscpu=True to record lscpu data as its own metric "lscpu"
  • Added support for the Clear Linux OS type for AWS and static VMs (e.g. --os-type=clear)
  • Added flag --placement_group_style to allow or disable different AWS placement groups options for ec2 VM benchmarks.
    Flag supports cluster, spread, and none placement group options.
  • Added support for availability zones for Azure. Format for availability zone support is "location-availability_zone".
    Example: eastus2-1 specifies Azure location eastus2 with availability zone 1.
    A PKB zone can be either a Azure location or an Azure location with an availability zone.
  • Added resource class and mapping for AWS Athena Service.
  • Added support for Azure Dedicated Hosting. Simply add flag --dedicated_hosts=True to use.
  • Records two new samples for /proc/cpuinfo data.
  • Added support for specifying a number of vms per host for Azure & AWS Dedicated Hosting.
    Simply add flag num_vms_per_host=<#> to use. Benchmark fails if the # of vms per host specified exceeds the memory capacity of the host.
  • Added support for skipping the creation of Azure availability sets with the --azure_availability_set=False flag (fixes #1863)
  • Added support for AWS EFA networking with --aws_efa=True flag.
  • Added NCCL benchmark for GPU networking.
  • Added AWS DAX provider.
  • Added Google Cloud Firestore ycsb benchmarks.
  • Added support for un-managed data processing yarn cluster benchmarking.
  • Added support for UDP_STREAM tests to netperf_benchmark
  • Added flags to select stream size to netperf_benchmark
  • Added placement group flag support for Azure. "cluster" will create proximity placement group.
    "spread" will create an availability set.
  • Added a tcpdump collector with --tcpdump flag.
  • Added ability to load datasets into BigQuery from csv files on GCS.
  • Updated AzureVirtualNetwork to support location level virtual nets, which allows support for cross zone benchmark runs.


  • Support for ProfitBricks API v4:
    • Add profitbricks_image_alias flag and support for image aliases
    • Add new location, us/ewr
  • Add aws_image_name_filter flag to ease specifying images.
  • Add c5/m5 support for NVME disks.
  • Add MNIST benchmark support for TPU, CPU and GPU
  • Created KubernetesPodSpec which allows the user to specify kubernetes resource
    requests and limits, including fractional CPUs.
  • Add skip_pending_runs_file flag and extension hooks to workaround SIGINT issues.
  • Add support for specsfs2014_load parameter as an integer list.
  • Publishers can be extended through external modules.
  • Add run_processes_delay flag to stagger parallel runs.
  • Add support for SPECspeed.
  • Add new os_types Centos7, Debian9, Ubuntu1404, Ubuntu1604, and Ubuntu1710.
  • Make it easier to RDP to PKB VMs
  • Add os_type support to KubernetesVirtualMachine.
  • Avoid setting up thread pool etc when run_processes is not set (to simplify
  • Added a sample benchmark for descriptive purposes.
  • Added GPU peer to peer topology information to metadata.
  • Added a flag, mpirun_allow_run_as_root which allows OpenMPI to run in the
    case that the user is root.
  • Modified KubernetesVirtualMachine to ensure that ssh is installed on the
  • Added container_cluster_num_vms flag.
  • Added the cluster's worker shape and worker count into the dpb service metadata.
  • Added azure_accelerated_networking flag for Azure SR-IOV support.
  • Added support for V100 GPUs on GCP.
  • Add support for distributed TensorFlow benchmark.
  • Added Azure Relational DB support for Postgres 9.6.
  • Added copy_benchmark_single_file_mb flag for single file support.
  • Record guest system information.
  • Support for static edw clusters.
  • Add more granularity to FAILED benchmarks with FailedSubstatus due to
    insufficient quota, cloud capacity, or known intermittent failures.
  • Update sysbench benchmark to version 1.0. (deprecate 0.4 and 0.5 versions)
  • Change GCP TPU command from alpha to beta.
  • Update configurable parameters for ycsb benchmarks.
  • Added tags to AWS kops instances and increased loadbalancer timeout.
  • Allow cassandra_ycsb to run on a single vm.
  • Adding support for using preprovisioned data for SPEC CPU 2006 benchmark.
  • Support negative numbers when parsing an integerlist.
  • Added float16 support for TensorFlow and MXNet Benchmarks.
  • Added flag --mx_key_value_store in MNXnet benchmark.
  • Added time_commands flag to enable diagnostic timing of commands
  • Added image processing speed in mnist.
  • Updated cloud TPU model link.
  • Updated AWS spot instance creation.
  • Added support for extending the failure sample with metadata if AWS spot VMs
    or GCP preemptible VMs are interrupted.
  • Added flags ycsb_version and ycsb_measurement_type to support
    user-specified ycsb versions and measurement types.
  • Added support to tensorflow_benchmark for running multiple batch sizes per run.
  • Added resnet152 in TensorFlow benchmark default models.
  • Added 50kb per payload ycsb workload.
  • Added num_cpus to virtual_machine published metadata.
  • Added a timeout to RobustRemoteCommand.
  • Added support for the gcp_min_cpu_platform flag on GKE clusters.
  • Preliminary support for NFS file systems
  • Added BigQuery provider implementation to extend the edw benchmarking in Perfkitbenchmarker.
  • Added support for authorizing gcloud to access Cloud Platform with a Google service account.
  • Added support for specification of resource specific ready timeouts.
  • Adding runner utilities to execute multiple sql scripts on edw clusters.
  • TPU checkpoints and summaries are now stored in GCS.
  • Updated cloud TPU models version.
  • Support for cloud NFS services (no implementation).
  • Added support for default batch sizes given a certain GPU type and model in
    the Tensorflow benchmark.
  • Added method to get the NfsService from the linux_virtual_machine.
  • Added support for fio job files in the data directory.
  • Added InvalidConfigurationError.
  • Added owner tag in metadata.
  • Added support for NVIDIA P4 GPUs.
  • Added YCSB timeseries parsing.
  • Added Intel MKL to HPCC benchmark.
  • Added flag support for enabling or disabling transparent hugepages on Linux
  • Add AWS MySql Aurora support
  • Abandon beta in TPU commands.
  • Update ycsb hdr histograms to output {bucket:count} data set for latencies, removing percentile:latency_value data set.
  • Added GPU test for ResNet benchmark.
  • Update ResNet testcases.
  • Remove unused lib in ResNet benchmark.
  • Added way to run several specsfs benchmarks in a single PKB run.
  • Add --preemptible in TPU creation command.
  • Add --skip_host_call in ResNet benchmark.
  • Update ResNet benchmark test for TensorFlow 1.9.
  • Removed the flag resnet_num_cores from the resnet benchmark and added the flag tpu_num_shards to MNIST, resnet, and inception3.
  • Update MNIST benchmark source path.
  • Replace old flags --tpu_name and --master with a new flag, --tpu.
  • Added support for EFS provisioned throughput.
  • Added support for --os_type ubuntu1804. Removed ubuntu1710 for GCP.
  • Added flag --iperf3_set_tcp_window_size to let the user avoid setting the
    TCP Window size in Iperf3 Windows performance tests.
  • Added flags --hbase_version and --hbase_use_stable to set the HBase
    version. Also upped hadoop to version 2.8.4.
  • Updated cuDNN installation methods.
  • Added support to schedule multiple TPUs.
  • Added GroupZonesIntoRegions support for
  • Added function to get the number of TPU cores.
  • Update spec17 copies per run to respect system available memory.
  • Add tensor2tensor benchmark.
  • Change train_steps to train_epochs in TPU test.
  • Add default tags like timeout_utc to GCE.
  • Add validation to all the TPU tests.
  • Add number of Train/Eval TPU shards in metadata.
  • The spark service resource supports a new flag (spark_service_log_level), to control the log level and generated output.
  • Updated openjdk 8 installation to support newer ycsb versions (after 0.13.0);
    Improved hdrhistogram result reporting in
  • Added flag --ntttcp_udp to allow the user to run UDP tests with ntttcp.
  • Added flag --ntttcp_packet_size to allow user to set the packet size in
    ntttcp tests.
  • Extract more data from the ntttcp results file and put into the metadata of
    the samples.
  • Updated the default of Cloud Bigtable client version to 1.4.0. Added
    --hbase_bin_url to allow bypassing GetHBaseURL().
  • Remove flag tf_benchmarks_commit_hash, and add tf_cnn_benchmarks_branch.
    Branch cnn_tf_vX.Y_compatible is compatible with TensorFlow version X.Y.
  • Added flags --ntttcp_sender_rb, --ntttcp_sender_sb,
    --ntttcp_receiver_rb, --ntttcp_receiver_sb to control the socket buffer
    sizes in ntttcp tests.
  • Move the ycsb_operation_count default from being set in the flag to being set in the workload file.
  • Introducing a new flag gcp_dataproc_subnet which allows a user to specify the subnet that a dataproc cluster will be part of.
  • Added support for running a subset of HPCC benchmarks.
  • Introducing a new multi string flag 'gcp_dataproc_property' which allows a user to modify many common configuration files when creating a Dataproc cluster.
  • Added configuration support for specifying type of the boot disk for a Dataproc cluster.
  • Added AddTag method to AzureResourceGroup.
  • Added some support for IPv6 (on static machines).
  • Added retransmit count to netperf metadata.
  • Added a common base class to PKB, PkbCommonTestCase,
    which uses absl.testing.flagsaver to save and restore
    flag values on test setUp / tearDown.
  • Remove all usage of mock_flags and replace with PkbCommonTestCase.
  • Upgraded memtier benchmark to version 1.2.15.
  • Add precision flag in Inception3 benchmark.
  • Support Hadoop 3.x.x in hadoop_terasort
  • Add z1 support for NVME disks.
  • Add default tags to dataproc.
  • Adding default run configurations for spec17 when none
    of os_type, runspec_config or is_partial_results flags is set.
  • Add a flag to control the number of worker threads used by the
    memcached server
  • Add default tags to GKE.
  • Add support for using real data from GCS and add support to download GCS data
    to vm in the Tensorflow benchmark.
  • Added flag nttcp_config_list to allow the user to supply a list of test
    configurations, all to be run in a single run phase.
  • Add support for --nouse_pkb_logging to use standard ABSL logging instead.
  • Improved support for booting more than 200 VMs with the cluster_boot benchmark.
  • Adding version support to redis server and setting permissions for newer redis versions.
  • Introduced app service metadata to indicate backend concurrency.
  • Added --ssh_reuse_connections to reuse SSH connections which speeds up benchmarks with lots of short commands.
  • Add abstract PreDelete method to the base resource class.
  • Add ability for linux vms to print the dmesg command before termination
    using the flag '--log_dmesg'.
  • TPU and its GCS bucket should be in the same region.
  • Adding field count and field length flag overrides to ycsb load.
  • Added --ssh_server_alive_interval and ssh_server_alive_count_max to adjust SSH server alive options.
  • Support for cloud SMB services (no implementation).
  • Added rwmixread option to fio scenarios. This option is ignored unless a split
    read/write workload is specified.
  • Added gce_local_ssd_count and gce_local_ssd_interface metadata to
  • Added option to use real training data to the Horovod benchmark.
  • Added support to record individual latency samples in the object_storage_service
    benchmark using the flag --record_individual_latency_samples.
  • Added --ssh_retries to adjust the number of times we retry for errors with a
    255 error code.
  • Added --num_benchmark_copies which controls the number of copies of each
    configuration to run.
  • Added --zone multistring flag as an additional way to specify zones. It can be
    used in conjunction with the --zones flag.
  • Added --before_cleanup_pause to ease debugging.
  • Added support for CUDA 10.1.
  • Added --skip_firewall_rules flag to help manage firewall rule economy.
  • Added Azure Premium Files support.
  • Added 'os_type' metadata to virtual machines for use in performance samples.
  • Storage specification related dpb terasort benchmark improvements.
  • Added support to run fio against multiple clients.
  • Added an optional flag to set a delay between boot tasks.
  • Added precision flag to Horovod.
  • Added a flag to Horovod that indicates that the VM is using a deep learning
  • Add support to set load upper bound for specsfs2014.
  • Add support for different workload sizes for stress-ng.
  • Implemented RobustRemoteCommand for Windows.
  • Extract GceVirtualMachine's GetNetwork into a method.
  • Updated redis memtier benchmark to pre-populate the redis db before testing.
  • Add Clear Linux support for AWS stripe disks.
  • Load memcached server before running benchmark.
  • Add support for xfs disk formatting.
  • Add gluster_fio benchmark.
  • Added support for static AWS VPCs with --aws_vpc and --aws_subnet flags.
  • Added support to append a region name to a bucket name for object storage
    benchmarking. For Azure, the region is appended to storage accounts and
    resource groups as well.
  • Added sysbench benchmarking for MySQL in a VM.
  • Added check that cuda_toolkit is installed when installing cudnn.
  • Added ability to set provider-specific MySQL flags for relational databases.
  • Add support for running t3 burstable VMs on AWS without unlimited mode.
  • Add support to run fio with a timeout on fio commands.
  • Added support for reporting bigtable cluster cpu utilization, gated by flag
  • Move vm_groups to inside the relational_db spec when there is one, to allow it
    to control what vms are used. Rename the old vm_spec and disk_spec under the
    relational_db to db_spec and db_disk_spec respectively, for added clarity.
  • Added ability in stress_ng benchmark to test --cpu-method's.
  • Updated cloud_spanner_ycsb_benchmark to add the support for GCP Go client
  • Make Kubernetes virtual machines not rebootable by default, to avoid changing
    things that require rebooting, since doing so is not ideal for containers.
  • Add Amd Blis support for HPCC.
  • Add Stress-ng support to run on different versions, 0.05.23 and 0.09.25, which
    are the defaults in Ubuntu1604 and Ubuntu1804.
  • Improved the usability of cloud bigtable ycsb benchmark by introducing flags
    --google_bigtable_enable_table_object_sharing, and --ycsb_skip_load_stage.
  • Added --cluster_boot_test_port_listening for cluster_boot benchmark. When
    set to True it will report the time until the remote command port (SSH
    port for Linux and WinRM port for Windows) in addition to the time until
    a remote command is successfully run.
  • Added flag --cluster_boot_test_rdp_port_listening to measure time until RDP
    port accepts a connection in Windows. It is True by default.
  • Add support from reading from BigQuery Storage API in the
    dpb_sparksql_benchmark. It uses Support is
    currently limited to GCP Dataproc provider, but it could run anywhere
    provided some auth is plumbed through.
  • Add support for generating CLI download commands for object_storage_service.
  • Added flag '--azure_low_priority_vms' to create Low Priority VMs in Azure.

Bug fixes and maintenance updates:

  • Moved GPU-related specs from GceVmSpec to BaseVmSpec
  • Fix ProfitBricks issue with extra / in the API url
  • Fix ProfitBricks volume availability zone issue
  • Bulk AllowPort restored.
  • Moved CustomMachineTypeSpec and related decoders to their own module
  • Updated GKE engine version to 1.9.6-gke.0 when using a GPU-accelerated cluster
  • Update python and pip package names used when installed with Yum.
  • Don't try to publish samples if there aren't any
  • Disallow overwriting of benchmarks using the same benchmark name.
  • Log exception thrown in RunBenchmark
  • Explicitly state python package names for RedHat on GCP
  • Always set password for 'postgres' user when creating PostgreSQL CloudSQL
  • Fixed EPEL package URL for Redhat 7.
  • Updated to latest version of NVIDIA CUDA Toolkit 8.
  • Updated Ubuntu image versions in Tensorflow benchmark.
  • Don't log wget output when installing CUDA Toolkit 8.
  • Install CUDA Toolkit 8 patch from NVIDIA.
  • Updated Ubuntu image versions in MNIST and MXNet benchmarks.
  • Allow SMB3 protocol for file transfer which is required for windows 2012 with encrypted transport
  • Updated Ubuntu image versions in HPCG, MNIST and MXNet benchmarks.
  • Fixed a bug getting the correct number of layers in MXNet.
  • Gcloud alpha version no longer required for GPU and minimum CPU platform.
  • Backfill image on GCE if empty.
  • Remove logging of empty environment variables in vm_util.IssueCommand.
  • Explicitly state python package names for RedHat and Centos in AWS and Azure.
  • Remove flag "tf_use_nccl" because the latest TensorFlow benchmark removed this
  • "nightly" now is not a valid argument in GCP TPU commands.
  • Make aerospike_ycsb runnable on Amazon AMI.
  • Fixed assumption that HOME was set.
  • Fixed issue with default K8s image setup.
  • Install the correct TensorFlow pip package based on whether VM has GPUs or
  • Change to root for command "lspci".
  • Install package pciutils when install cuda_toolkit.
  • Update TensorFlow benchmark arguments.
  • Isolate Darwin OS pkb runs to fix Issue # 1563
  • Fixed YCSB so it pushes workload once to each VM even if they are repeated.
  • Simplified resource and spec registration.
  • Fixed a bug in which the PKB recorded reboot time could be low for AWS because
    we would restart the creation timer for VMs that were reported as pending.
  • Increased timeout of GKE Cluster create command from 10 to 15 minutes.
  • Set ImageNet image shape as 3,224,224.
  • Updated MountDisk commands to use named parameters.
  • Calling repr(IntegerList) returns a readable string.
  • Support installing azure-cli on RedHat systems.
  • Fixed default behavior of using /usr/bin/time --quiet on all commands
  • Fixed ycsb failure when the same workload is ran more than once.
  • Fixed yum proxy config bug (GH-#1598 from @kopecpiotr)
  • Create destination directory for Azure blob storage downloads.
  • Made the pip package more robust by adding a symbolic link in /usr/bin if pip
    is installed in /usr/local/bin.
  • Make ~/.ssh/config only readable by the owner.
  • Increased timeout for Azure az vm create commands.
  • Increased timeout for GCP gcloud compute instances create commands.
  • Replace all underscores in the benchmark name with dashes when downloading
    preprovisioned benchmark data from Azure. This is because Azure blob storage
    container names cannot contain underscores.
  • Fixed the bug of running distributed TensorFlow on multiple batch sizes.
  • Updated gcloud compute networks create to use subnet-mode.
  • Changed default CUDA version from 8.0 to 9.0.
  • Updated default Tensorflow version to 1.7 when using GPUs.
  • Updated default Tensorflow tf_cnn_benchmarks version to a commit dated April
    2, 2018.
  • Fixed a bug in which boto could not be uninstalled on newer GCE images.
  • Update each ycsb workload to have its own recordcount configuration if
    ycsb_record_count is not set.
  • Fixed treatment of the boot time metric so that it is more like any other run
    stage metric.
  • Fixed bug of modifying the providers/aws/util.AWS_PREFIX value.
  • Made failures to create VMs on AWS, Azure, and GCP fail PKB quickly.
  • Fix Kubernetes StorageClass deletion
  • Added git installation to tensorflow_serving package.
  • MountDisk throws exception if mounting the disk fails.
  • Added support for preprovisioned benchmark data on KubernetesVirtualMachines.
  • Refactored speccpu2006 benchmark to use common elements for both
    speccpu2006 and speccpu2017.
  • Use flags['key'].parse(...) to set mocked flags in linux_virtual_machine_test
  • Cleanup some redundant logging and duplicate decoder statements.
  • Fixed build_tools package re-installation for speccpu2006.
  • Fixed fio job parsing, section parameters should always overrides global
    job parameters.
  • Refactored StaticVirtualMachines to use GetResourceClass() pattern.
  • Fixing the Redshift provider implementation to extend the edw benchmarking in pkb.
  • Support using --gcp_min_cpu_platform=none to clear --min-cpu-platform. This
    lets the flag override a benchmark spec.
  • Fix windows security protocol casting.
  • Added '--iteration' to MNIST benchmark. The default value of 50 is too small
    and can result in excessive communication overhead which negatively impacts
  • Added support for more AWS regions in the object storage benchmark.
  • Add ability to skip known failing scripts when running edw profiles.
  • Set the control port and data port for nuttcp benchmark.
  • Fix overwriting of bandwidth variable in nuttcp benchmark.
  • Fixed fio histogram parsing.
  • Refactored AwsKeyFileManager out of AwsVirtualMachine.
  • Added delay between server and client initiation to windows benchmarks.
  • Defaulted static machines to linux based.
  • Add time limit to windows fio benchmark.
  • Adding -w buffer parameter to windows iperf3 benchmark.
  • Fixed a bug in how we select images on AWS by introducing an additional
    regular expression to match against image names.
  • Terminate long running servers on windows benchmarks with timeouts.
  • Updated azure_cli package to match installation instructions.
  • Fix helpmatch for intergerlist.
  • Fix PSPing benchmark so that it runs on AWS and Azure.
  • Upgrade CPU pip package version in the Tensorflow benchmark to version 1.6.
  • Moved from ACS to AKS for Azure Kubernetes clusters.
  • Cleanup and fix Beam benchmark.
  • Sysbench failover tests added for GCP and AWS Aurora
  • Sysbench qps metric added
  • Fixed a bug of checking TPU exist.
  • Add GetMasterGrpcAddress method to CloudTpu.
  • Fix a bug of getting the number of TPU cores.
  • Set number of images in ResNet benchmark command so it can support other datasets.
  • Sysbench supports benchmarking MySQL 5.6
  • Update memcached server to install from a pre-built package provided by the operating system.
  • TensorFlow Serving benchmark now runs off master branch with optimized binaries
  • Updated HPCC benchmark to version 1.5.0.
  • Psping benchmark no longer report histogram for every sample metadata.
  • Specifies the number of threads to use in MXNet CPU test.
  • Fixed the way several unittests are using flags and mock_flags.
  • Fixed linter issues in several unittests.
  • Add python_pip_package_version as a class attribute of
    BaseVirtualMachine. Children classes that have the problem of
    pypa/pip#5247 can use an older pip package.
  • Update AWS Aurora Postgres default version from 9.6.2 to 9.6.8
  • Update speccpu17 to write profiles in different files for each benchmark.
  • Avoid installing unnecessary MySQL server for sysbench when client tools suffice.
  • Added support for running the object service benchmark with a GCP service
  • Add missing if name == 'main' stanza to some unittests, and fix those
    that were broken.
  • Consolidate FlagDictSubstitution and FlagsDecoder into a simpler context manager,
    OverrideFlags, which does not abuse the internals of FlagValues.
  • Changed GkeCluster so that it adds PKB metadata to all VMs that it creates.
  • Upgraded OpenMPI from 1.6.5 to 3.1.2.
  • Upgraded OpenBLAS from 0.2.15 to 0.3.3.
  • Add flag to control database machine type for managed relational databases.
  • Ensure randomly generated windows passwords start with a character.
  • Upgrade Tensorflow version to 1.12.
  • Install NCCL when installing Tensorflow with GPU support.
  • Update AzureBlobStorageService to persist for max of --timeout_minutes and --persistent_timeout_minutes.
  • Update S3Service to persist for max of --timeout_minutes and --persistent_timeout_minutes.
  • Add --project flag to GoogleCloudStorageService MakeBucket command.
  • Fix virtual_machine.GetResourceMetadata() so that it does not try to gather
    metadata from a VM that has not been provisioned yet.
  • TPU Pod doesn't support eval. Stop Evaluation in train phrase in MNIST benchmark.
  • Global steps was set incorrectly in Inception3 benchmark.
  • Add AWS T3 instances to list of non-placement group capable machine types.
  • Refactor managed redis resource base class into managed memory store resource base class
  • Remove flag tpu_zone because VM and TPU should be in the same zone.
  • Refactoring managed_memorystore to add GetIp, GetPort and GetPassword class methods for managed_memorystore base class.
  • Run individual stressors in stress-ng one by one.
  • Upgrade gcp cloud redis to use prod gcloud.
  • Upgrade glibc_benchmark's version of glibc to 2.29.
  • Add number of disks for AWS i3.metal instance to configuration
  • Fix bug in Azure disk letter assignment when attaching more than 24 disks.
  • Incremental fixes to support Python3.
  • Fix bug in Azure Windows VM names for long run URIs where the name could be
    longer than 15 characters.
  • Update aws dynamodb_ycsb_benchmark to use aws credentials from the runner vm.
  • Reboot immediately during call to ApplySysCtlPersient.
  • Set lower tcp keepalive thresholds on netperf vms.
  • Introduced NumCpusForBenchmark() which should be used instead of num_cpus for benchmark configuration.
  • Use stat -c %z /proc/ instead of uptime -s to find a boot timestamp since it
    is more universal.
  • Added support for regex_util.ExtractInt to extract an integer from a string.
  • Updated Coremark to v1.01 and improved its error handling and output parsing.
  • Added flag support to Coremark for choosing a parallelism method. Defaults to
    the existing mechanism of pthreads.
  • Make a temporary copy on VM's disk for scp on SMB.
  • Change the GCE Windows VM boot disk type to PD-STANDARD by default to better
    match the default for GCE Linux VMs.
  • Fixed azure credentials package to avoid pushing the entire .azure directory.
  • Update bigtable benchmark to make configurations configurable, and report them.
  • RobustRemoteCommand now retries on ssh connections refused.
  • Fixed a bug that was causing AWS capacity reservation creation to fail.
  • GceNetworkSpec's zone is equal to the VM's zone
  • Creates the bucket from PKB rather than from within the VM that PKB launches
    in TPU test.
  • Sets TPU zone MLPerf benchmark.
  • Compile Stress-ng version 0.05.23 rather than installing its package.
  • Added --gce_firewall_rules_clean_all option to network resource: clean up
    all the firewall rules that depend on the PKB-created network.
  • Fix AWS stripe disks assumption that the NVME index for the boot drive is always 0.
  • Increased default timeout for booting Windows VMs to 40 minutes.
  • Expose filesystem type and block size in vm metadata.
  • Fixed problem with AWS network creation in aws_dpb_emr, aws_nfs_service, and
  • Create the client_vm of the relational_db attribute of BenchmarkSpec from the
    default vm_group if there's no clients group. Fixes pgbench benchmark failure.
  • Fixed use of numpy.piecewise to be numpy > 1.13 compliant
  • Fixed the cluster boot benchmark config to use SSD boot disk types for GCP and
    Azure, adding support for specifying a boot_disk_type in Azure.
  • Fixed the default path of the spark example jar on an EMR cluster.
  • Check if an AWS default route exists before creation.
  • Fixed the broken reference to ycsb.YCSB_TAR_URL in the
    cloud_spanner_ycsb_benchmark. Without this, the benchmark is not runnable.
  • Fixed bug of query failing to find AWS internet gateway after Delete called.
  • Abort and log when container resources are exhausted.
  • Add a retry on gcloud commands when rate limited.
  • Check if an AWS firewall rule exists before creating one by querying AWS.
  • Add flag overrides for server and client vm groups inside relational_db.
  • Update requirements.txt for Python 3
  • Upgrade hadoop to version 3.2.1.
  • Install the netcat-openbsd and zlib.h debian package for aerospike_ycsb.
  • Updated to use the single source of truth of
    perfkitbenchmark_keyfile path for non-static VM, expanding the use case of
    stage-wise test, e.g., do prepare on machine A, do run stage on machine B.
  • Added --scp_connect_timeout and --ssh_connect_timeout flag to replace the
    hard-coded values at the call sites of vm_util.GetSshOptions.
  • Added support for Container Optimized OS in the GCP Provider. Support is
    currently limited to the cluster_boot benchmark.
  • Unifying metadata creation on cluster boot.
  • Added support for CoreOS Container Linux in GCP, AWS, and Azure Providers.
    Support is currently limited to the cluster_boot benchmark.
  • PkbCommonTestCase extends absltest.TestCase for py3 test backports.
  • Update tox python version.
  • Remove the remote file before pushing a local file to the remote place. This
    can avoid permission issue as the pushed file only allows user to read or
  • Add support for unmanaged NFS.
  • Fixed Cassandra branch name (GH-2025 from @marcomicera)
  • Add AWS T3 cpu credits argument to sample metadata.
  • Add OpenFOAM benchmark.
  • Removed RDP setup from Windows Startup script, because it was not needed by
    any tested provider.
  • Updated to handle the cases of read only and update only, where there
    is no result for certain hdr histogram group.
  • Add -m PEM to ssh-keygen to fix AWS.
  • Updated AWS Virtual Machine to handle shutting_down case when creating an instance.
  • Default Docker image for cluster_boot on Kubernetes (GH-2061 from @marcomicera)
  • Fixes to MySQL server and client settings to ensure enough data and temp
    directory space, and to prevent connection timeouts.
Assets 2

@s-deitz s-deitz released this Oct 16, 2017 · 1930 commits to master since this release

Changes in this release:

Breaking changes:
- Removed gpus_per_node metadata from stencil2d and hpcg benchmarks
- Removed legacy disk metadata (use data_disk\* not scratch_disk\* and
  num_striped_disks + aws_provisioned_iops metadata) (GH-1506)

New features:
- Added YCSB benchmark for Cloud Spanner (GH-1387 from @haih-g)
- Added CUDA-enabled HPCG benchmark (GH-1395)
- Added ManagedRelationalDatabase base classes and tests (GH-1405)
- Added gpus to GceVmSpec. This is now the only way to create VMs with
gpus on GCE due to a gcloud API change (GH-1406)
- Added flag_zip_defs which functions like flag_matrix_defs, but
performs a zip
  operation on the axes instead of a cross-product (GH-1414)
- Added TensorFlow Benchmarks. (GH-1420)
- Added --network_enable_BBR flag (GH-1515)
- Added pgbench benchmark for managed PostgreSQL (GH-1505)

- Added basic auth support for Mesos provider. (GH-1390)
- Added --failed_run_samples_error_length flag to limit failed run error
length (GH-1391)
- Added `__eq__` and `__ne__` to IntegerList (GH-1395)
- Added total_free_memory_kb to VirtualMachine class and implemented it
  Linux vms (GH-1397)
- Created hpc_util for a place to share common HPC functions
- Added --runspec_estimate_spec flag to calculate an estimated spec
score (GH-1401)
- Added all HPCC metrics as samples when running HPCC (GH-1448)
- Added support for P100 gpus on Google Cloud Platform (GH-1450)
- Added gpu type to cuda_toolkit_8 metadata (GH-1453)
- Added num_gpus to cuda_toolkit_8 metadata (GH-1455)
- Added range mode to gpu_pcie_bandwidth which calculates average
bandwidth over
  a provided range of transfer sizes (GH-1482)
- Added size support to fio_benchmark scenarios (GH-1489)
- Added RemoteCommandWithReturnValue (GH-1516)

Bug fixes and maintenance updates:
- Fixed provision phase of memcached_ycsb benchmark for non-managed
memcached instances (GH-1384)
- Fixed GPU benchmarks on GCE to work with new GPU API (GH-1407)
- Changed cuda_toolkit_8 to use the gpu-specific base clock speed as the
default (GH-1453)
- Changed default AWS P2 region for gpu benchmarks (GH-1454)
- Fix multi-threading issue in fio.job filename (GH-1474)
- Decreased speccpu2006 default disk size to 50gb (GH-1484)
- Updated azure library version 1.0.3 for blob storage (GH-1491)
- Added azure host_caching to metadata (GH-1500)
- Fixed fio histograms when generated with numjobs>1 (GH-1501)
Assets 2

@tedsta tedsta released this May 18, 2017 · 2085 commits to master since this release

Breaking changes:

  • Azure CLI updated to 2.0 (GH-1359)
  • Update gflags to version 3.1.1 (GH-1322)

New features:

  • Azure container service (GH-1361)
  • Add Reboot() method to BaseOsMixin (GH-1356)
  • Add container cluster classes (GH-1355)
  • Optional --completion_status_file flag to write the completion status of every config to a json file (GH-1349)
  • Add --gcp_min_cpu_platform flag (GH-1336)
  • Add GCP sole tenancy support (GH-1333)
  • Add SHOC package (GH-1332)
  • Add Stencil2D benchmark (GH-1332)
  • Managed memcache service abstraction and a backend for AWS ElastiCache (GH-1235)
  • Add Apache Beam benchmark (GH-1324)
  • Add image_project to GCP YAML config options (GH-1319)
  • Add support for AWS i3 (GH-1308)
  • Optionally create failed run samples with --create_failed_run_samples (GH-1363)
  • Add --storage_size flag for mysql_service benchmark (GH-1360 from @sspano)


  • Add optional step parameter to IntegerList (GH-1330)
  • Upgrade to NVIDIA's latest version of CUDA Toolkit 8 (GH-1304)
  • Add ID attribute to GCE VMs (GH-1318)
  • Add ability to install netperf from local tar file (GH-1364)

Bug fixes and maintenance updates:

  • Fix up - incorrect command line example and added note on Amazon ES publisher (GH-1350 from @Lirt)
  • Fix up - remove outdated comments about GCP requirements (GH-1354 from @sspano)
  • Fix creating instance with local SSD using NVMe interface (GH-1287)
  • Make flag values consistent with gcloud CLI "SCSI" and "NVME" (GH-1287)
  • Store os types in list instead of tuple so that it can be extended by third parties (GH-1348)
  • Don't create host during VM Create if num_vms_per_host is specified (GH-1352)
  • Fix up kubernetes provider (GH-1351)
  • Add --region to describe/cancel spot-instance-request commands (GH-1347)
  • Install sysbench05plus package in a separate path from sysbench (GH-1338)
  • Change --openstack_additional_flags to be a list flag (GH-1340)
  • Move gflags installation from netperf benchmark to a package (GH-1341)
  • Move package installation of netperf from benchmark to a package (GH-1341)
  • Use snake_case gflags APIs (GH-1328)
  • Check if apt-update has been called in InstallPackages, and call it if it has not been (GH-1334)
  • Fix broken regular expression in blazemark benchmark (GH-1311)
  • Fix CloudSuite license page link (GH-1323)
  • Update hadoop version (GH-1327)
  • Use regional storage class for GCS buckets instead of DRA (GH-1326)
  • Fix gflags FLAGS object initialization in tests (GH-1313)
  • Fix bug with exception name in (GH-1314)
  • Always set GPU clock speed in cuda toolkit installation (GH-1321)
  • Specify image_project in gpu_pcie_bandwidth benchmark config (GH-1321)
  • Move gpu clock speed logic to cuda_toolkit_8 module (GH-1279)
  • Add timeout to netperf remote script (GH-1306)
  • Don't assume that gcloud is in PATH in providers.gcp.util.GetDefaultProject (GH-1366)
  • Make AWS S3 bucket deletion retryable (GH-1343)
  • Empty bucket at the end of the Run stage of the object_storage_service benchmark (GH-1343)
Assets 2

@dlott dlott released this Mar 14, 2017 · 2161 commits to master since this release

External contributions:
- Add ability to publish to InfluxDB (thanks @besanradwan GH-1278)
- Always set internal IP-Address on OpenStack VMs (thanks @yremmet
- Add CloudSuite data-analytics benchmark (thanks @ivonindza GH-1285)
- Update (thanks @ianmaddox GH-1291)
- Fix a small typo in the anti-affinity FLAG lookup (thanks @joelio
- Deprecate flag openstack_volume_size (thanks @vhe182 GH-1220)
- Add ProfitBricks cloud api v3 support (thanks @aajdinov GH-1264)
- Enable OpenStack Cinder Volume Type Option (thanks @rossmartyn04

New features:
- Add dpb_wordcount_benchmark (GH-1253)

- Add helpmatch usage instructions (GH-1268)
- Add openStack_volume_type option decoder (GH-1267)
- Upgrade OpenStack CLI dependency to 3.6.0 (GH-1265)
- Add object storage service cold reads support, various improvements
(GH-1290, GH-1274)
- Add support for AWS Spot Instances (GH-1288)
- Bump fio version to v2.17 and enable histogram (GH-1300)
- Let republish already-collected samples (GH 1299)
- Allow specifying temp dir (GH-1277)
- Add --json_write_mode (GH-1276)
- Allow override hpcc binaries (GH-1269)

Bug fixes and maintenance updates:
- Lock samples json file before writing to it (GH-1265)
- Fixed exception caused by publishers being None (GH-1302)
- Add error and warning log level options to reduce log size (GH-1301)
- Use force for DigitalOcean deletes (GH-1281)
- Print helpmatch usage instructions (GH-1280)
- Fix fio parser regex (GH-1275)
- Specify AWS region when calling describe-images (GH 1273)
- Make CheckPrerequisites more comprehensive (GH-1272)
- Use image_project flag in disk creation if set (GH-1271)
- Minor bug fixes and configuration improvements (GH-1262)
Assets 2

@gareth-ferneyhough gareth-ferneyhough released this Jan 6, 2017 · 2225 commits to master since this release


External contributions:

New features:


  • Use FLAGS.aws_user_name if set for Rhel (GH-1249)
  • Fio enhancements (GH-1246)
  • Use aes128-ctr if aes128-cbc isn't available (GH-1242)
  • Make aeropsike able to run with raw devices on static vms (GH-1243)
  • Add some metadata to netperf samples (GH-1236)
  • Remove unneeded Azure code and add availability sets (GH-1232)
  • Add cassandra read concurrency flag (GH-1233)
  • Add a metadata attribute 'ssd_count' to spark_service (GH-1229)
  • Add a metadata attribute 'spark_svc_cloud' to spark_service (GH-1226)
  • Add ability to specify boot_disk_size in AWS vm spec (GH-1231)
  • Add a metadata attribute 'spark_svc_cloud' (GH-1236)
  • Set a default zone for the spark test to allow for subnet creation (GH-1222)
  • Update the location of the spark examples jar on the emr cluster (GH-1222)
  • Persist package installation across reboots (GH-1185)
  • Allow mongodb_ycsb to specify readahead settings (GH-1256)

Bug fixes and maintenance updates:

  • Close Aersospike connection in 1 second (GH-1241)
  • Fix bug with having multiple Azure data disks (GH-1223)
  • Fix aws spark service (GH-1222)
  • Add some git ignore entries (GH-1222)
  • Fix gcs credential conflict (GH-1221)
  • Fix bug with run_processes (GH-1212)
  • Remove collector from RunBenchmarkTask args (GH-1210)
  • Add the minimum tox version to the "tox is not installed" message (GH-1201)
  • Move sysbench05plus path prefix logic into package (GH-1185)
  • Fix bug with specifying Azure image (GH-1204)
  • Fix two files that were failing boilerplate check (GH-1203)
  • Update (GH-1200)
Assets 2

@asaksena asaksena released this Nov 18, 2016 · 2318 commits to master since this release

Changes in this release.

New features:
- Add config file imports and allow for circular imports and add documentation (GH-1163)
- Add a way to run benchmarks in parallel (GH-1192)

- ContainerizedDebianMixin: Ping to a fixed docker image and remove the sudo hack (GH-1171)
- Auto detect  openjdk, libsnappy package versions (GH-1181)
- Add flags to govern subnet creation and Optionally create only 1 subnet (GH-1182)
- Install docker images as packages (GH-1184)
- Call FLAGS.get_help if it's available (GH-1187)
- Publish boot time samples for all tests (GH-1156)

Bugfixes and maintenance updates:
- Update requirements.txt to set version of contextlib2 (GH-1164)
- Fix bug with config flags (GH-1165)
- Updated the cloudsuite web-serving benchmark (thanks @nooshin-mirzadeh, GH-1166)
- Fix load command parsing (GH-1168)
- openstack: Open all TCP and UDP ports for the internal network (GH-1169)
- Fix aerospike metadata (GH-1170)
- Adjust aerospike default replication factor (GH-1172)
- Try replacing FlagValues._flags instead of FlagValues.FlagDict first (GH-1175)
- Fix ycsb aggregator (GH-1176)
- Fix bug during cleanup phase of object_storage_service benchmark (GH-1178)
- Netperf thinktime fixes and support think time in nanoseconds instead of microseconds (GH-1179, GH-1186, GH-1188)
- Fix sysbench05plus installation on ubuntu16.04 (GH-1183)
- Fix race condition in Aerospike benchmark (GH-1190)
- Clean up some pickling, unpickling issues (GH-1191)
- Calculate free ram with more robust /proc/meminfo instead of `free`(GH-1194)
- Fix multithread netperf (GH-1149)
- Multiple fixes and refactoring of the linux package management on vms (GH-1152)
- Fix dstat metadata in Analyze (GH-1154)
- Obey flag overrides for static vm specs (GH-1155)
- Fix links in README file for OpenStack and Cloudstack setup steps (thanks @shakhat, GH-1157)
- Fix issue with GetConfig using flags (GH-1160)
- Remove scratch disk from netperf and object_storage_service benchmark (GH-1147, GH-1148)
- Netperf changes to add num_streams metadata to samples (GH-1177)
- Remove collector from RunBenchmarkTask args (GH-1210)
Assets 2

@tedsta tedsta released this Oct 21, 2016 · 2385 commits to master since this release

 - Fixed default machine type for Azure (GH-1161)
Assets 2

@tedsta tedsta released this Oct 19, 2016 · 2406 commits to master since this release

New features:
- Multithreaded netperf benchmark (GH-1141, GH-1144)
- Terasort benchmark using spark service infrastructure (GH-1104)
- Support 10 or more disks on Azure (GH-1102)
- Add sysctl and set_files flags (GH-1096)
- Support DigitalOcean block storage (GH-1092)
- Add cloudsuite_data_serving_rec_count and cloudsuite_data_serving_op_count parameters to cloudsuite_data_serving benchmark (GH-1091)
- Add support for setting GCE boot disk size and type (GH-1106)
- Add flag to skip reload on ycsb benchmark (GH-1108)
- Support for AWS dedicated hosts (GH-1109)
- Object storage multistream write-only scenario (GH-1123)
- Allow multiple configs for the same benchmark in the same file (GH-1126)
- Add flat that allows run retries (GH-1129)
- Add fio_parameters flag (GH-1131)
- Add config matrices (GH-1134)
- Add events at the beginning and end of benchmarks (GH-1135)
- Add extra_zones flag to pkb (GH-1137)
- Add flag matrix filters (GH-1138)
- Support allowing port ranges on Firewall.AllowPort (GH-1139)
- Add static vm tags (GH-1142)

Breaking changes:
- Because of GH-1112, region names have changed.

- Allow speccpu to run individual test (GH-1113)
- Use Azure resource management mode (GH-1112)
- GCP networks use subnets (GH-1087)
- Add top level flags key to configs (GH-1125)
- Precise resource usage for YCSB benchmarks (GH-1127)

Bugfixes and maintanence updates:
- Fix bugs with Terasort (GH-1136)
- Don't run container commands as sudo with ContainerizedDebianMixin (GH-1130)
- Automatically use amazon linux if os_type is rhel (GH-1124)
- Removed Panda as a dependency (GH-1118, GH-1119)
- Allow flag_util.ParseKeyValuePairs values to have colons in them (GH-1117)
- Add ycsb to data directory (GH-1116)
- Fix bug in mongodb_ycsb_benchmark cleanup (GH-1115)
- Fix bug where static VM disk specs were overriden (GH-1105)
- Use string formatting instead of addition to concatenate (GH-1103)
- Fixed regression with fio flag (GH-1097)
- Fix flags in static vm files (GH-1093)
- Fix wait for status file to avoid race condition (GH-1090)
Assets 2
You can’t perform that action at this time.