- #970: Add tooling to create a track from existing data
- #944: Allow to use significantly more clients
- #935: Add an asyncio-based load generator
- #875: Add 'Tracker' track-generation tool
- #988: Add experimental segment stats telemetry device
- #986: Allow to override GC log parameters
- #977: Migrate track param check to new IT infrastructure
- #972: Pass actor system env vars in integration test
- #971: Add thespian logs to ci
- #969: Add an it method for rally race
- #968: Add integration tests with eventdata
- #967: Raise an error when attempting to bulk-index without any corpora
- #966: Migrate more integration tests
- #963: Reduce mapped memory with many bulk clients
- #962: Increase bulk size to metrics store
- #959: Clarify the impact on refreshes for node-stats telemetry
- #958: Avoid duplicate URL parsing for async connections
- #954: Add flag to handle running processes automatically
- #953: Pass JAVA14_HOME in build
- #952: Allow to disable HTTP compression for all queries
- #951: Upgrade Thespian to 3.10.0
- #950: Add support for a RALLY_HOME env var everywhere
- #949: Upgrade Elasticsearch client to 7.6.0
- #947: Use pbzip2/pigz to decompress corpora if available
- #945: Add an override for the rally homedir
- #941: Improve response processing
- #938: Store units in results
- #934 (Breaking): Bump minimum Python version to 3.8
- #932: Improve layout of generated dashboards
- #931: Store Rally revision in metrics metadata
- #930: Update psutil to latest version
- #929: Allow index-stats to compare non-existing path
- #928: Don't invoke setup.py directly
- #925: Allow to specify an indices stats condition
- #923: Assume a TTY for interactive commands
- #919: Run integration tests with pytest
- #914: Improve release docs and IT prechecks
- #913: Normalize handling of --preserve-install
- #983: Assign worker-id globally
- #978: Remove os environment variables from env prep
- #965: Always shutdown worker thread pool
- #961: Determine most recent sample per client
- #960: Disable docstring linter check
- #956: Fix fallback implementation of ML-related runners
- #943: Fix request timeout handling
- #942: Honor response-compression-enabled parameter
- #927: Add unit for all query charts
- #918: Gather system metrics on Docker stop if possible
- #980: Change next Rally version in migration docs
- #955: Update docs to advise use of pbzip2 for new track corpus
- #948: Align track reference structure with track
- #939: Correct JMC download link
- #937: Fix port number in recipes docs
- #890: Speed up client-side bulk-handling
- #912: Honor only specified Python versions in prereq
- #909: Add custom probing URL
- #908: Validate docker-compose exists in it tests
- #907: Specify useful variables for Rally CI
- #906: Specify useful variables for Rally CI
- #904: Eliminate deprecation warnings in Python 3.7+
- #903: Show more user-friendly track loader errors
- #899 (Breaking): Use zeroes instead of whitespaces as padding bytes
- #895: Log race id on startup
- #887: Allow to override Python binary for make
- #886: Fix race condition with old ES processes in IT tests
- #884: Detach ES using Python standard mechanism
- #880: Show recent log output on integration test failure
- #879: Don't read from stdout/stderr on ES startup
- #873: Prevent installation with unsupported Python
- #872: Extend error msg to specify operation name for missing index
- #865: Updating force merge operation to force merge only track indices by default
- #901: Fix race in wait_for_pidfile.
- #894: Only set distribution-version if known
- #893: Only clear metrics store on benchmark end
- #891: Align exception handling in disk IO telemetry
- #885: Fix issue with fetching test-mode -1k document corpora.
- #883: Stop all locally running nodes
- #882: Make REST API check stricter
- #881: Wait for REST layer before version check
- #877: Don't write metrics to a closed metrics store
- #874: Store system metrics if race metadata are present
- #871: Skip setup phase in IT tests during docker release
- #868: Fix W0631
- #860: More resilient node shutdown
- #902: Add example how to determine actor system status
- #853: Allow to use the bundled JDK in Elasticsearch
- #830: Manage Elasticsearch nodes with dedicated subcommands
- #815: Add Python 3.8 in Rally tests
- #863: Allow piped stdin in run_subprocess_with_logging
- #862: Add support for excluded tasks in chart_generator
- #850: Allow to show track details
- #844: Add task exclude filter
- #840: Add make target to serve docs locally
- #836: Upgrade boto3
- #832: Only keep the most recent build log
- #829: Reduce usage of platform-specific code
- #818 (Breaking): Store build.log in log directory
- #816: Harmonize local pyenv versions with CI ones
- #805: Add lint + precommit make targets.
- #803: Calculate system metrics per node
- #802: Whitelist py.test in tox tests
- #798: Allow definition of body in restore-snapshot operation
- #793: Add ability to restore from a snapshot
- #789: Let the runner determine progress
- #788: Manage dev dependencies in setup.py
- #785: Don't attach telemetry devices for Docker
- #783: Upgrade pytest to 5.2.0
- #779 (Breaking): Gather cluster-level metrics in driver
- #778 (Breaking): Expose race-id as command line parameter
- #776: Add heapdump telemetry device
- #774: Raise an error if race is not found by id
- #773: Store race files always as race.json
- #771: Store track-related meta-data in results
- #767: Remove merge times from command line report
- #766: Ensure tox environment is clean
- #765: Start release process creating NOTICE
- #727: Allow function-level invocation of integration-test.sh.
- #861: Pass plugin params for all plugins
- #859: Detach Elasticsearch on startup
- #858: Use the venv pylint in the makefile
- #841: Store Disk I/O metrics if available
- #825: Change remaining it tests to port 19200
- #820: Upgrade psutil to work with newer linux kernels
- #814: Fix error handling for automatically derived version
- #804: Allow multiple nodes per IP
- #801: Initialize schedule lazily
- #800: Properly wait for recovery to finish
- #799: Don't set retries for restore snapshot
- #784: Attach telemetry device on Docker launch
- #781: Add support for OSNAME and ARCH variables in dist repo URLs.
- #768: Honor ingest-percentage for bulks
- #763 (Breaking): Run a task completely even without time-periods
- #834: Clarify uses of distribution-version
- #826: Add note about mustache template
- #821: Move logrotate comments to separate lines
- #817: Fix mistake in wait-for-recovery docs
- #810: Fix duplicated IP in distributed_load diagram
- #782: Update Elasticsearch version of targets and metricstore in CCR recipe
- #764 (Breaking): Remove MergeParts internal telemetry device
- #762: Check that origin points to main repo for release
- #761: Allow to retry until success
- #760: Improve logging of schedules
- #759: Show track and team revision when listing races
- #758: Improve error message on SSL errors
- #755: Add pull request template
- #748: Consistently render links to the documentation
- #746: Fixup ProcessLauncherTests issues
- #744: Change DiskIO callbacks to use attach/detach
- #739 (Breaking): Remove lap feature and all references.
- #737: Allow to attach telemetry devices without reprovisioning
- #735: Assume string type for params by default
- #731: Update DiskIo telemetry device to persist the counters
- #730: Be resilient upon startup
- #729 (Breaking): Drop 1.x support for cluster metadata
- #728: Allow to set distribution version as parameter
- #726: Don't check complete list of parameters in integration test
- #725: Capture team and track revisions in metrics metadata
- #723: Always exit on OOME
- #721: Update DiskIo telemetry device to persist the counters
- #720: Change filestore to be indexed by unique ID
- #719: ES as a Daemon (again)
- #716 (Breaking): Drop support for Elasticsearch 1.x
- #711: Change telemetry devices to rely on jvm.config instead of ES_JAVA_OPTS
- #701: Implement ES daemon-mode in process launcher
- #751: Option track-revision should work with track-repository
- #750: Retrieve timestamped commit hash separately
- #747: Log git output
- #702: Add Rally Docker image to release process
- #688: Fail Rally early if there are unused variables in track-params
- #713: Check tags in track and team repos
- #709 (Breaking): Drop support for Python 3.4
- #708: Align default request param extraction for queries
- #707: Upgrade pip for virtualenv
- #705: Provide default for datastore.secure in all cases
- #704: Add download subcommand
- #700: Allow stdout print when running in Docker
- #694: Measure execution time of bulk ingest pipeline
- #691: Remove node_count_per_host
- #710 (Breaking): Don't pass request_cache by default
- #706: Register pyenv Python versions
- #699: Fix home directory paths in provisioner_test.
- #698: Enter degraded mode on bootstrap failure
- #693: Assume UTC timezone if not specified
- #683: Store mean for response-related metrics
- #681: Use single node discovery type if suitable
- #679: Skip Python install if already existing
- #678: Upgrade Jinja to 2.10.1
- #674: [Experimental] Capture peak usage of JVM mem pools
- #671: Add ability to download from S3 buckets
- #670: Pass JAVA12_HOME in integration tests
- #669: Allow to override request timeout for force-merge
- #668: Chart generator segment memory and new track combinations
- #667: Add sleep operation
- #666: Introduce new command line parameter --track-revision
- #662: Add distribution flavor to metrics metadata
- #660: Add user tags when comparing races
- #659: Sort the track parameters / user tags when listing races
- #654: Allow to use ES 7.x as metrics store
- #649 (Breaking): Pass request-params as is in supported operations
- #648: Updates to support 7.x APIs
- #687: Fix release charts after #684
- #686: Honor track-params in chart-generator
- #684: Use license instead of dist flavor in charts
- #682: Provide a platform-independent Rally binary
- #675: Fix chart generator segment memory bug
- #673: Honor runtime JDK in provisioner
- #661: Small change in the venv-create
- #655: Add compatibility layer for existing metrics store
- #652: Properly authenticate at proxy server
- #650: Use --prune for all git fetch operations
- #647: Make types optional
- #646: Add node_name in node-stats docs for ...
- #644: Allow collection of jvm gc section in node-stats telemetry device
- #642: Allow passing any request parameter to the create index API
- #641: Temporarily disable Python 3.4 in build
- #639: Add recovery-stats telemetry device
- #637: Ignore JSON logs for merge parts analysis
- #633: Ensure Makefile install target includes all release dependencies
- #638: Correct recorder-based sampling interval
- #630: Improve error message on missing repo directory
- #627: Warn about skewed results when using node-stats telemetry device
- #625: Allow to specify a team revision
- #622: Include NOTICE.txt in release
- #620: Add license headers
- #617: Fix conflicting pipelines and distribution version
- #615: Add refresh/merge/flush totals in summary
- #612: Extract hits either as number or structured object
- #606: Improve release.sh script and prerequisites
- #604: Change logging level for esrally command line to DEBUG
- #613: Error can be a string, and shards not always present in response.
- #632: Update Release documentation for Rally
- #623: Update Developing Rally docs sections
- #621: Minor typo
- #610: Command fails with "=", ":" is needed.
- #599: Simplify development setup
- #594: Add make install as default target
- #587: Fix typos and inconsistencies in help documentation
- #576: Improved formatting of document count in list tracks output
- #574: Rename benchmark data directory to root directory
- #572: More fine-grained ML metrics
- #571: Reduce logging when loading tracks
- #570: Add ML API runners
- #563: Make Rally compatible with Python 3.7
- #586: Don't fail git clone is tty is not attached
- #579: Switch to official approach for HTTP compression
- #575: Always use remote branch for updates
- #598: Document --limit
- #569: Add shrink index runner
- #559: Write rendered track to a temporary file
- #558: Avoid throughput throttling in test mode
- #557: Adjust flight recorder flags for JDK 11
- #556: Upgrade to Thespian 3.9.3
- #555: Load only required corpora
- #549: Exponential back-off for retrying transport errors to metrics store
- #546: Update ccr stats endpoint
- #545: Use non-standard port for metrics store in tests
- #540: Simplify filtering node-stats-related documents
- #538: Retry metric store for more transport errors
- #565: Improve compatibility when loading invalid JSON
- #562: Defer startup of mechanic actor
- #560: Disable automatic log rotation
- #551: Fix HTTP TLS client certificate verification
- #544: Fix parsing of boolean values datastore.secure
- #542: Warn when cluster settings cannot be applied
- #537: make --test-mode work with uncompressed data #536
- #533: State clearly that only Unix is supported
- #529: Allow to control recency of ids in conflicts
- #523: Use a single shard for metrics indices
- #522: Fine-tune logging
- #521: Remove deprecated usage of parameter source
- #518: Derive JDK version at runtime
- #516: Add multi-cluster support for NodeStats telemetry device
- #515: Fold all stats per shard in the same doc for CCR
- #514: Report indexing times per shard
- #513: Allow to store custom metric document format
- #512: Make challenges optional
- #509: Remove post_launch bootstrap phase
- #508: Add node-stats-include-mem option to record jvm heap stats
- #507: Set retry-on-timeout=true for ES clients used by Telemetry devices
- #503: Use file-based logging configuration
- #497: Add record_process_stats() for process.* stats
- #526: Resolve artefact name based on download URL
- #524: Disable interpolation for config file
- #511: Don't fail if conflict-probability is zero
- #510: Allow conflict-probability value of 0
- #500: Convert exception to string in driver
- #493: Sample more node stats
- #490: Record "took" for bulk indexing
- #489: Get distribution download URL from car config
- #488: Add support for multiple clusters in custom runners
- #487: Add new Makefile target to run it tests inside Docker
- #485: Allow to benchmark Elasticsearch with and without x-pack
- #481: Add post_launch phase for Elasticsearch plugins
- #480: Add generic node-stats telemetry device
- #477: Improve simulation of bulk-indexing conflicts
- #475: Rename telemetry device that gathers GC stats
- #473: Don't pass type implicitly for queries
- #476: Create unique ids even for random conflicts
- #471: Enable actor message handlers to fail immediately
- #470: Allow to use track parameters in index definition
- #469: Align operation param names with runners
- #468 (Breaking): Require at least Rally 0.7.3 for config migration
- #466: Don't require git
- #464: esrally must not log clear text user passwords
- #462: Upgrade Elasticsearch client to 6.2.0
- #461 (Breaking): Remove 'index-settings' property
- #460: Report store and translog size
- #458: Add parameter support to telemetry devices
- #456 (Breaking): Remove automatic index management
- #454: Allow to ingest a subset of a document corpus
- #453 (Breaking): Remove operation type "index"
- #441: Allow to pass parameters via a file
- #440: Use curl retry features to workaround transient network problems
- #467: Return "pages" as unit for scrolls
- #465: Default to upgrading packages with setuptools
- #449: Fix venv detection with venv and Rally execution on non master
- #446: Use more specific name for mandatory plugin check selectively
- #437: Bump pytest* versions and beautify make test output
- #434: Remove Python faulthandler
- #431: Add a unique race id
- #429: Allow to use track parameters also in index / template definitions
- #426: Allow simpler configuration of cluster configuration
- #377: Index-append operation only indexing bulk-size * clients documents
- #423: esrally fails after initial config run if JDK9 is not installed
- #424: Clarify use of virtualenv in developing doc
- #420: Cannot benchmark multi-node cluster with benchmark-only pipeline
- #418: Measure Elasticsearch startup time
- #417: Set heap dump path
- #416: Improve logging when gathering index time related metrics
- #413: Cannot report to ES cluster with self-signed certificate or in-house certificate authority
- #411: Store Rally version in results meta-data
- #410: Allow parameter sources to indicate progress
- #409: Allow to override car variables
- #407: Upgrade to thespian 3.9.2
- #405: Improve internal message handling
- #404: Add a raw request runner
- #402: Enforce UTF-8 encoding for file I/O
- #392: Formatting for user-tags on rally-results
- #399: 0.9.0 - Unable to run race due to missing JDK9
- #398: Allow to override plugin variables
- #387: Require JDK 9 for source builds
- #384: Provide a specific error message if data file is present but wrong size
- #383: Abort current benchmark in case of attempted duplicate starts
- #376: Record indexing throttle time
- #366: Separate document corpus definition from indices
- #361: Don't measure every operation
- #359: Allow to define index settings per index
- #348: Prefer default data directory when --track-path is given
- #293: Allow users to explicitly define index management operations
- #396: Rally requires a team repo even if used as a load generator
- #394: Error in track preparation can hang benchmarks
- #397: Add a gentle reminder to update the Rally kernel with the important elasticsearch system configurations
- #389: Syntax error in elasticsearch_plugins documentation
- #388: Update rally daemon port requirements
- #355: Run In Kubernetes
- #375: Add refresh API
- #310: Introduce track parameters and allow to override them on the command line
- #371: Introduce put pipeline API
- #369: Prepare rename from operation-type "index" to "bulk"
- #363: Allow to retry operations (internally)
- #362: Make plugin remote URL optional
- #360: Show ES distribution version upon race start
- #358: Rally tables should always output the same rows
- #354: Be more lenient when custom parameter source does not provide a parameter
- #353: Make a few track properties optional
- #352: Allow to use a challenge element if there is only one challenge in a track
- #351 (Breaking): Ensure task names are unique
- #345: Allow to use an ES distribution for plugins that are built separately
- #326: Allow to define operations inline
- #356 (Breaking): Set a sane default socket timeout
- #350 (Breaking): Number of iterations should be treated per client, not "global"
- #364: Deprecate --cluster-health
- #333: Provide a clear error message for non-existing releases
- #331: Don't demand a compressed representation of the document corpus
- #226: Provide a Rally package with all dependencies for offline install
- #217: Add ability to continuously stream metrics
- #344: Client assignment can miss a (small) number of docs
- #338: Documents are not found if track is referenced via a file
- #337: Only map numeric HTTP status codes to request meta-data
- #334: Data partition algorithm failling with parallel indexing tasks
- #343: If python is required then python-dev should be as well
- #231: Simplify usage of Rally for offline-only use
- #332: Provide more information about errors
- #330: Provide better error message when Rally reads an incompatible track
- #329: Allow to skip cluster health check
- #323: Rally should not demand a local Java install
- #309: Allow to benchmark plugins from sources
- #292: Add a "simple track" mode
- #259: Prepare Rally for Java 9
- #328: Challenge-specific index settings are not applied to index template definitions
- #322: Add error-type to request meta-data
- #321: Don't log body when creating a new index
- #319: Upgrade to thespian 3.8.0
- #315: Simplify multi-node configuration
- #313: Allow car mixins
- #265: Have Rally detect and use more Java versions
- #318: Run tasks indefinitely
- #257: Distribute load-test driver
- #316: KeyError: 'vm_vendor' when running benchmark against single host cluster with 0.7.0
- #314: Allow to filter executed tasks
- #312 (Breaking): Move action-and-meta-data to (indexing) type definition
- #308: Unify implementation of track and team repositories
- #307: Check Python version on startup and fail if it does not meet requirements
- #304: Make distribution repositories configurable
- #296: Verify whether the number of documents in the track is correct
- #278: relative-time should be reset per task
- #228: Increased flexibility for termination of parallel tasks
- #299: Leave the host parsing on elasticsearch-py
- #227: Enhance flexibility of user tagging
- #196 (Breaking): Externalize car configuration
- #298: Rally froze at the end of a race, did not produce results
- #295: Provide number of query hits as metrics metadata
- #291: Show track size metrics when listing tracks
- #290: Allow to pass arbitrary request parameters for ES queries
- #286: Additional metrics for bulk requests
- #282 (Breaking): Remove list facts subcommand
- #258: Make 'race' self-contained
- #284 (Breaking): Disallow previously deprecated usage of track properties in meta block
- #283: Store race results in a format optimized for reporting
- #279 (Breaking): Separate race and metrics indices
- #276: Add affected index to meta-data returned by bulk index runner
- #275: Allow to define per-challenge cluster-settings
- #267: Provide a progress-indication for downloads
- #266: Make the gc telemetry device Java 9 compatible
- #246: Rally should print a warning if there are no measurement samples
- #223: Allow unlimited number of pages for scroll queries
- #222: Report number of hits, not just pages, for scroll queries
- #220 (Breaking): Allow monthly indices for metrics
- #138 (Breaking): Improve support Elasticsearch 5 as metrics store
- #285: Rally is stuck for benchmarks with a very large number of requests
- #280: The file-reader parameter source does not seem to pass additional parameters
- #288: Clarify usage of remote track repositories in docs
- #287: Document throughput semantics for each operation
- #274: Explain summary report output
- #271: Document how to implement custom runner classes
- #264: Documentation on settings and warning message when using a documents archive file
- #251: Support for non-deterministic distribution of operations
- #260: Have bulk parameter source provide the bulk size
- #249: Saving tournament report
- #155: Improve document count handling
- #263: Race condition when rolling log files
- #261: Unable to determine valid external socket address
- #253: Cannot determine CPU info for POWER8 chips
- #242: Rally does not detect the distribution version correctly for externally provisioned clusters
- #235: Allow Python files in a track directory that are unrelated to the track plugin
- #256: Document additional decompression options
- #247: Description of available operation types in the docs
- #241: Clearly document which Elasticsearch versions are supported
- #244: Show a nice error message when user tries to run a non-existing challenge on an existing track
- #245: Connection behind a proxy fails. has_internet_connection() returns False
- #234: Add request error rate to summary report
- #238: Simplify gathering of facts for benchmark results sharing
- #216: Add possibility to profile load driver internals
- #184: Allow to benchmark a single-machine cluster remotely
- #207: Improve response checks in bulk API runner
- #205: Make track configuration more modular
- #200: Allow root.dir to be set during configure
- #199: Let track authors decide on the default challenge
- #191: Support target-interval for operations
- #225: Final score report blank when metrics store specified
- #221: Scrolls fail against Elasticsearch master
- #203: Index time metrics are not aware of laps
- #202: Bulk index source reads data for all indices
- #224: Issue(s) with --user-tags
- #214: Improve documentation of "parallel"
- #213: Document how to support --test-mode in a track
- #208: Add FAQ item to explain latency, service time and their relation to "took"
The track format has changed a bit due a more flexible approach in how benchmarks are executed:
- Operations are defined in the
operationssection, execution details like number of warmup iterations, warmup time etc. are defined as part of theschedule. - Each query needs to be defined as a separate operation and referenced in the
schedule - You can (and in fact should) specify a
warmup-time-period(defined in sections) for bulk index operations. The warmup time period is specified in seconds.
For details please refer to the updated JSON schema for Rally tracks.
Hint: This is just relevant for you, if you have defined your own tracks. We already took care of updating the official Rally tracks.
We have separated the previously known "track setup" into two parts:
- Challenges: Which describe what happens during a benchmark (whether to index or search and with which parameters)
- Cars: Which describe the benchmark candidate settings (e.g. heap size, logging configuration etc.)
This influences the command line interface in a couple of ways:
- To list all known cars, we have added a new command
esrally list cars. To select a challenge, use now--challengeinstead of--track-setupand also specify a car now with--car. - Tournaments created by older versions of Rally are incompatible
- Rally must now be invoked with only one challenge and only one car (previously it was possible to specify multiple track setups)
We have also moved tracks to a dedicated repository. This allows you to support tracks for multiple versions of Elasticsearch but also requires that all users have git installed.
We have spent a lot of time to simplify first time setup of Rally. For starters, you are not required to setup your own metrics store if you don't need it.
However, you are then just able to run individual benchmarks but you cannot compare results or visualize anything in Kibana. If you don't need this, it is recommended that you
remove the configuration directory and run esrally configure. Rally will notify you on its first start of this change and guide you through the process.
Please raise a ticket in case of problems.
- Add a tournament mode. More information in the user docs
- External benchmarks can now specify target hosts and ports
- Ability to add a user-defined tag as metric meta-data
- Support .gzipped benchmark data contributed by @monk-ee. Thanks!
- Support for perf profiler
- Add a fulltext benchmark
Major changes:
- Rally can now benchmark a binary Elasticsearch distribution (starting with Elasticsearch 5.0.0-alpha1).
- Reporting improvements for query latency and indexing throughput on the command line.
- We store benchmark environment data alongside metrics.
- A new percolator track](elastic#74) contributed by Martijn van Groningen. Thanks!
Major changes:
- Added a JIT profiler. This allows to check warmup times but also in-depth inspection which optimizations were performed by the JIT compiler. If the HotSpot disassembler library is available, the logs will also contain the disassembled JIT compiler output which can be used for low-level analysis. We recommend to use JITWatch for analysis.
- Added pipeline support. Pipelines allow to define more flexibly which steps Rally executes during a benchmark. One of the use-cases for this is to run a benchmark based on a released build of Elasticsearch rather than building it ourselves.
Major changes:
-
Migrated the metrics data store from file-based to a dedicated Elasticsearch instance. Graphical reports can be created with Kibana (optional but recommended). It is necessary to setup an Elasticsearch cluster to store metrics data (a single node is sufficient). The cluster will be configured automatically by Rally. For details please see the README.
Related issues: #8, #21, #46,