20.2.0

What's Changed

Crossplatform

• Meet test history service! It tracks test results and time consumption for individual tests and allows other services to use this information to optimally schedule new test run. See more in doc (iOS, Android)
• There is a new timeWeighted test splitter that works hand in hand with test history service and provides the shortest possible time for entire test run. See more in doc (iOS, Android)

iOS

• Minimal supported target is macOS 12.0
• Get chrome tracing for the entire test run by using --trace option. See more in emceeFree runTests --help

Android

• Added readable Artifactory uploading error description
• Fixed possible crash in clients because of redundant internal checks
• Fixed possible zero test time bug
• Added check for desired sdk version before test run start
• Added fix for endless CLI process after test run
• Updated internal emulators version to 33.0.2. Other internal tools were also updated
• Added timeout for emulator boot complete
• Fixed wrong test result when test ended instantly

20.1.0

What's Changed

iOS

• By default buckets divide strategy is equally divided
• By default tests restart strategy is retry on worker
• By default queue is watching for workers and restart them if needed. See silentWorkerAutomaticRestartFeatureConfig
• Simulator setup before tests run is optimized. Now it requires less time
• Emcee now writes less logs by default. The log level can be configured by environment
• Improved plugins errors handling in workers. Memory consumption for plugin support is reduced
• Added Emcee metrics Grafana dashboard example

Android

• Improved test run result info output for clients (Gradle plugin and CLI)
• Added ability to set total tests execution timeout for clients (Gradle plugin and CLI)
• Worker now starts on 41001 port (was 80)
• Added Android metrics Grafana dashboard example
• Documentation was improved
• Android documentation now on the separate site
• Added ability to configure Queue analytics and metrics for containerized Queue
• Fixed bug with Allure report downloading on sdk level 29 and lower
• Fixed test start time event

20.0.0

What's Changed

Common:

• Added support for skipped tests on all platforms

iOS:

• There is new tests discoverer - runtimeXCTest. Use it for discovering dynamically added tests with frameworks like Quick, Kiwi, etc
• scheduleTests command dispatches tests to previously deployed queue. It's useful in systems where Emcee gets deployed with external platform operator
• default file verbosity has been changed to debug (was trace before change)
• better device management and improved cleaning routines that lead to reduced disk and memory consumption

Android:

• Another option to run tests - command line utility. Give it a try!
• Get video recorded during test run by using videoRecordStrategy
• Documentation refined

Cloud:

• Integrate Emcee test runner to your GitHub pipeline with Emcee.cloud github action
• Now Emcee offers alpha support for Appium cloud web server platform! Try it out - https://docs.emcee.cloud/cloud/appium/

Minor fixes:

• retryOnWorker tests retry mode does not cause excessive retries on queue

Announcement

Since version 20.0.0 Emcee we will change a minor version for ongoing regular releases. Major versions will be increased with significant changes or broad refactoring.

Emcee 19 — Android support, UX updates, and more

We are excited to announce that Emcee now supports the Android platform. Run Emcee queue and workers from container images in Docker or your Kubernetes cluster to shorten testing time. Keep calm, we continue to develop iOS part as well.

Android

Android workers, as well as their iOS counterparts, receive test jobs from queue. It means you get smart bucket slicing strategies, test retries, and better hardware utilization out of the box.

• Workers ship Android runtimes on board, on demand start and manage necessary devices with required sdk version. As for now we support sdk levels 24, 27, 29, 31 and will add more on a regular basis.
• It's possible to configure Android worker to send rich analytics. Use these insights when making decisions on equipment purchase or problem investigation.

Try out Android workers now!

iOS

• Xcode test plans are now supported. Build your project with test plan support xcodebuild build-for-testing -testPlan <name> and then feed the resulting xctestrun file to Emcee via --xctestrun. See more in Emcee runTests --help.
• Set http/https proxies through command line invocation or json config.
• Test discoverer now finds tests with swift concurrency.
• Minor improvements: fixed problems when Emcee might lose some analytics events, worker does better dynamic libraries lookup.

Emcee 18 — Resurrecting the Clones and Pluggables

18th release is concentrated on fixing bugs, speeding up things, and separating the functionality between features which become available after purchasing a licence and always-free version.

Emcee now has paid version

Starting from version 17, Emcee is distributed as-is as a closed source project and available in a form of prebuilt binaries. Version 18 goes beyond this and introduces two distinct products: free Emcee and paid Emcee.

Free version (available as a download on Github) allows you to test the product with up to 3 workers, however, you'll need to buy licenses for some richer functionality like:

• Allowing to use more than 3 workers
• For a technical support
• For EmceeAdmin GUI app
• For getting things done that you want to see in Emcee sooner than later.

Main functionality remains available in free version:

• For using plugins in your environment
• For using analytics, such as grafana, statsd, and kibana
• For a bit richer test discovery flows, such as discovering via runtime

Some notes about the current Github repository

The current source code has been frozen from April 2022, which was version 16. We will keep it available as is, and we're not planning on removing it. All made forks will still be present obviously as well, as we can't control them.

Version 17 is closed source, yet fully functional. You can use it without any limitations as well.

We haven't developed any plans on upstreaming changes from v17 into open source yet, however, we are considering it.

Automatic Job Deletion

This is huge one and helpful. Emcee now tracks if job is needed at all, and automatically deletes it if it is not needed to run anymore. This helps on CI: when you cancel the test run, job gets deleted automatically.
Tech note: previously Emcee client process issued a /jobDelete request which sometimes could get lost; now queue itself tracks all jobs.

Worker Resurrection

Emcee now attempts to restart its workers in case if they die. So if you decided to reboot your machine, or if it kernel panics, it will become a worker automatically.

Simulator Cloning

Emcee worker can now clone simulators instead of booting a new one. This speeds up creation quite a bit, and saves disk space. Emcee will clone any previously shut down simulator it booted. If Emcee never booted suitable simulator, it will create one instead. We advise to use smaller values of automaticSimulatorShutdown and automaticSimulatorDelete settings, like 120 seconds, to have better effect of simulator cloning.
As an example, booting a new simulator can take 1-2 minutes easily, but cloning and booting takes around 40 seconds.

Plugins are back!

You can use the new, a little simplified plugin SDK which is available here:
https://github.com/avito-tech/EmceePluginSupport

It's easier to wire up everything, as there are only two exposed targets now: EmceePlugin and EmceePluginModels.

Boilerplate code has been reduced just to:

let plugin = try Plugin { (event: PluginAppleTestEvent) in
    // process an event
}
plugin.join()

Bugs and small features and changes

• Emcee now uses socket API to talk to graphite and statsd instead of state of the art OutputStream APIs. Users complained about crashes related to graphite, and this change should help.
• You can query global queue state via /globalQueueState. Maybe this will be helpful for you to debug how queue is being formed when you submit your jobs.
• Worker ID is exposed as capability now, allowing you to pin your tests to a specific machine. Capability name is emcee.workerid and the value is actually worker ID.
• --device and --runtime arguments now accept only fully qualified Core Simulator IDs, e.g. com.apple.CoreSimulator.SimRuntime.iOS-16-4 and com.apple.CoreSimulator.SimDeviceType.iPhone-14. Please refer to xcrun simctl list for more examples.
• If you want to set environment variables for a test or app while using runTests command, set them in the calling environment with a EMCEE_ prefix.
• CPU architecture is now exposed as worker capability, allowing you to pin your tests to the machine with specific CPU arch. The key is emcee.cpu.architecture, the value is CPU arch returned by uname -a, e.g. arm64.
• testFinished event is now correctly delivered to plugins when tests times out.
• If plugin fails to start, error message will contain its standard output and error for debugging purposes.
• Emcee now deletes its temporary xcresult storage after test run finishes.
• You can now specify range of ports that workers can utilize. Queue server config has a new field workerPortRange which controls exactly that. You can also set queue port range via queuePortRange.
• New command discoverAppleTests lets you run test discovery much easier than before, without composing a test arg file
• Speaking of test arg files, they are gone from a free version of Emcee. We decided to keep them for paid release only, as they provide more flexibility.
• We are happy to say Version 18 supports macOS Ventura. Version 17 does not support Ventura.

Join our Telegram channels!

A decent place to discuss knowledge about Emcee and obtain a licence!

• Russian: https://t.me/emcee_ios

• English: https://t.me/emcee_ios_en

EmCeeResult

Preambula

This release brings long awaited feature - generating a resulting xcresult file which you can open in Xcode right after running your tests. Just use runTests command and provide --result-bundle <path> to generate your result bundle.

You can also control the lifetime of xcresult attachments via TestArgFile.entries.testAttachmentLifetime - it can be deleteOnSuccess, keepAlways, keepNever. Just like xcodebuild, huh!

Caveats

• This release DOES NOT SUPPORT PLUG-INS. You won't be able to compile them against Emcee v17. Please wait until we release further update for that!
• Worker sharing is not working between this Emcee version and previous versions. This is because we messed up key and value for HTTP field: it used to be application/json: Content-Type which swift-nio thing forcefully rejects; now it is set correctly to Content-Type: application/json.

Smaller Details

• Less memory leaks and more speed thanks to updating HTTP server to Vapor

• Less logs are generated! Again!

• Emcee now kills xcodebuild with more suitable signal SIGINT which leads to less result bundle corruptions.

• libssh freedom has arrived! Emcee does not require it to be installed on your system. Also, it is adviced to use key-based authentication.

• We got rid of duplicate errors from junit. We got rid of duplicate errors from junit.

• Help (e.g. Emcee runTests -h) got even better thanks to more information provided.

• You can now configure simulator count on each worker, locale, keyboard, language via runTests command. Please refer to Emcee runTests -h.

Join our Telegram channels!

A decent place to discuss and share your knowledge about Emcee!

• Russian: https://t.me/emcee_ios

• English: https://t.me/emcee_ios_en

Easy Peasy

We've conducted a bunch of interviews with teams who use Emcee. We were provided with nice feedback which we used to improve our product. In this release, we focus on ease of use and an easy first try experience.

Getting Started Guide with Pictures and Sample Project 🥩

It is available right at README.

Just Run These Tests 🤬

We introduce a new command - runTests. A very similar one to a famous runTestsOnRemoteQueue but it is dedicated to opening the world of simplicity and clear setup for most Emcee users. It has the following syntax and options:

$ Emcee runTests -h

• --queue: describes where should a queue be started. This argument may be repeated multiple times, in which case Emcee will use hosts one by one until it manages to start the queue. You must provide at least one value here.
• --worker: describes where worker should be started. This argument may be repeated multiple times. It is required to provide at least one value.
• --device: Device to run test on, e.g. iPhone X. Required. Note: iPhone SE generations might be tricky to use!
• --runtime: Runtime to run test on, e.g. 15.0. Required.
• --test-bundle: Location of an .xctest bundle which tests will be executed. Required.
• --app: Location of an .app bundle. Optional. Please refer to our guide to learn more about tests with host app.
• --runner: Location of an XX-Runner.app. Optional.
• --test: Test to execute, e.g. ClassName/testMethod. Optional. If not specified, Emcee will run all tests. You can specify multiple tests to execute, e.g. --test Class/test1 --test Class/test2
• --retries: How many retries to attempt to run each test successfully. Optional. By default it will retry tests once if failure happens.
• --test-timeout: Maximum test execution duration. Optional. Default test duration limit is 180 seconds.
• --junit: Path where the Junit report file should be created. Optional.
• --trace: Path where the Chrome trace file should be created. Optional.

Let's explain --queue and --worker arguments. These are URLs which describe how to connect to a machine. Currently only SSH is supported. Some examples:

• --queue ssh://emcee:pass@queue.example.com/Users/emcee/emceequeue.noindex: it means to run Emcee queue on a queue.example.com host, by authenticating with emcee username and pass password. The working directory will be located at /Users/emcee/emceequeue.noindex.
• --worker ssh://emcee@worker.example.com/Users/emcee/emceeworker.noindex?custom_rsa: it means the queue will start its worker on worker.example.com. Worker will use /Users/emcee/emceeworker.noindex as its working directory. Also, Emcee will use emcee as username and ~/.ssh/custom_rsa key to auth against a worker host. This key is expected to be present on queue host.
• --worker ssh://emcee@worker.example.com/Users/emcee/emceeworker.noindex#/absolute/path/to/custom_rsa: same as above, but queue will use a key from /absolute/path/to/custom_rsa.

Free HTTP Server for Everyone! 🌏

Emcee now contains a built-in HTTP server that will cover most of your needs. This significantly simplifies the flow and lowers the entry threshold.

You can pass local paths into runTests command. Provided local .app and .xctest bundles will be hosted via a built-in HTTP server transparently, allowing all workers to download artifacts in order to run tests.

You can also provide local paths inside test arg file which you pass into runTestsOnRemoteQueue, those will also be hosted via the built-in server. Handy!

And finally, you can provide a local queue server configuration file into runTestsOnRemoteQueue. Now there is no need to upload it to a HTTP server.

Other Goodies 🦚

Help me Dr. -h 🍡

You can now get help for any command by typing -h next to it, e.g. Emcee -h or Emcee runTests -h. Sounds like Emcee is ready for such kind of hi-tech.

Better errors 💢

Should you provide an incorrect test arg file or queue configuration JSON file, Emcee will now print human-readable errors. You will be amazed. Here is one error for your consideration:

Failed to decode value for key "entries" in "test arg file": key "xcTestBundle" not found at entries[0].buildArtifacts

That is a crystal clear explanation of what went wrong, is it not?

Optional jobId ☃️

It wasn't clear for some of our users that jobId field in a test arg file should be unique. Now you can omit it, Emcee will generate a random jobId for you.

Test Retry Mode 👯‍♀️

You can now specify how you'd like your tests to be retried if they fail. In your test arg file's entries there is a new field testRetryMode. Possible values:

• retryThroughQueue — if a test fails on a worker, it will be returned back to the queue. A queue will then retry this test on other workers, up to the specified number of retries. This is the default behavior. This allows the test to be executed on different hosts, potentially eliminating the problem of a broken environment (because every macOS installation in Universe is unique).

• retryOnWorker — if the test fails, the same worker will perform retry.

Simulator Patching is Better 📲

There was a bug (OMG) that resulted in redundant simulator patching (read: apply simulator settings). Now Emcee won't patch simulators if
everything is patched.

Cacheless Workers are Now Possible 💳

There is a way to disable worker cache. In queue server configuration, in worker specific settings, next to numberOfSimulators field there are two more fields:

• maximumCacheSize — maximum cache size in bytes.
• maximumCacheTTL — maximum TTL for any cached item, in seconds.

You can set 0 these fields, and it will disable cache entirely. This is handy if you want to debug something.

Prevent Emcee from Cleaning Runner Output 🏃

You can now stop workers from removing all test artifacts automatically after a test finishes and plugins terminate. In a test arg file's entries there is a new field runnerWasteCleanupPolicy. Possible values:

• clean — default value, it means everything will be deleted
• keep — all created files will be preserved.

Log Capturing Mode 📝

You can now control what logs should be captured during a test run. These logs are obtained from xcresult bundles after the tests finish. In the test arg file's entries there is a new field logCapturingMode. Possible values:

• allLogs — all log messages will be captured
• onlyCrashLogs — only logs that look like crash logs will be captures
• noLogs — no logs will be captured

Where without dyld

You can now insert dynamic libraries into a test. In a test arg file's entries there is a new field userInsertedLibraries. This is an array of paths. These paths will be passed as DYLD_INSERT_LIBRARIES env into the test. Combine it with state of the art macOS imaging features, with a fact that tests are being executed via xcodebuild (so it can expand __TESTBUNDLE__ and other strings - look at man xcodebuild.xctestrun), and this will provide you some additional flexibility for your test execution process.

Try it Today! 🧨

• Below is a universal binary (arm64 & x64) for your convenience! It even has built-in libssh, so you can use it with ease.
• If you download Emcee binary from GitHub, do not forget to remove quarantine attribute: xattr -c Emcee
• Please rebuild your plugins. v15 API is not compatible with v16.

---

Full Changelog: 15.0.0...16.0.0

Fifteen Releases Old

Today we celebrate a bit delayed Emcee v15 release 🥳

Data Centre Edition 🦚

Emcee queues can be run on multiple hosts now, allowing even more graceful degradation when necessary. For example, if you set your queue host on maintenance (turn it off, or if aliens destroy your machine), queue may now be started on a spare host(s), allowing you to keep running your tests. To use this, just provide multiple queue hosts in queue_config.json via queueServerDeploymentDestinations array. Emcee iterate over the hosts in provided order.

Simulator Settings 📱

• You can now import certificates 📜 into simulator keychain! Thanks to @EvgenyIv96 for implementing this feature.
• xcodebuild now deletes all attachments 📎 from resulting xcresult bundle if test was successful ✅, reducing a load on the workers. Thanks again to @EvgenyIv96 for implementing this.
• New iOS 15 keyboard gesture banner thing ⌨️ can be hidden by setting didShowContinuousPathIntroduction to true in your test arg file.

It Just Became Much Better Now 📈

• Emcee now parses xcresult files and extracts test failures from there in addition to parsing result stream. This improves test error capturing significantly. 🌡
• Progressive schedule strategy has been optimized for smaller worker count. 👷🏻‍♀️👩🏼‍🌾👩🏽‍🏭👩‍🚀
• New schedule strategy with fixed bucket size allows to (obvsly) set a fixed bucket size. This is useful for larger worker count. E.g. we use this strategy in Avito with our 80+ workers. 🖥
• New Emcee initTestArgFile command creates a sample test arg file which you can fill with your settings. Useful for quick-start! 🥩
• New Emcee initQueueServerConfig command creates a sample queue config. Again, useful for initial setup. 🌱
• Swift symbol demangling now uses dynamic buffer. This eliminates troubles with test discovery for some users of Emcee. 🦅
• Finally, as a reminder, Emcee does not require sudo. You can create a standard user with limited rights and make Emcee use it. 🔒 For this scenario, Emcee now checks if ~/L/D/CoreSimulator folder exists, and creates it if it is missing.

Meet prebuilt binary! 🤡

Just don't forget to install libssh2 via excellent package manager which uses chemistry, water supply, liquid manipulation, and drink-related domain terms to describe its all possible actions and processes. ☢️

Also, do not forget about clearly understandable fact that Apple Silicon chips have special destination path for installable things in that honorable package manager, and that it is different on Intel chips. Thus, it is unlikely you will be able to use Intel binary on Apple Silicon Mac, which is very due to extreme quality of any modern software. 💎 Please render this fine emoji in red in your mind.

Full Changelog: 14.0.0...15.0.0

Ahoy, matey!

Humaney

• If simulators fail to boot, there will be a better, a more human readable error about this. This came as a replacement for "xcrun process failed with exit code 69".

xcresult-eys into Plugins

• @venigreat implemented a feature that allows to access xcresult bundles from plugins, allowing to process them on the go. TestContext.environment now contains a new entry which points to a xcresult bundle. You can and should use XcodebuildTestRunnerConstants.envXcresultPath to get the name of that env.

• By the way, there is an open source Allure plugin for Emcee

Securitey

• @venigreat implemented the following feature: Emcee now allows to access SSH by using private key files. Previously it only used password based auth. You specify auth mechanism via queue configuration.

• @venigreat implemented a third feature: Emcee now can add arbitrary HTTP headers to its network requests. This is useful if your storage (e.g. artifactory, nexus, etc.) requires authentication via headers, or if your back-end wants to track its clients via some X-Fancy-Header value.

Grown Logs Man

In this release, we focused on adding support for distributed logging system, and we used Kibana for these purposes. Thus, we decided to document how Emcee logging system works, and provide in-depth explanation of what metrics Emcee reports.

Test Arg File Changes and More Flexible Analytics

Global Analytics

Emcee now respects two analytics configurations. One is a global one. It gets captured by the queue when it starts (queueServerConfiguration.globalAnalyticsConfiguration), and it is used to report some global events which do not relate to any specific job. Some examples are:

• Status of worker sharing feature
• Individual worker statuses, as seen by the queue (e.g. alive, silent, never started)
• Global number of enqueued and dequeued buckets and tests
• Global counter of current jobs on the queue
• Simulator operations metrics - allocation duration, action duration (these are durations of create, boot, shutdown and delete operations)

Specific Analytics

Another analytics configuration is a specific one. You provide a configuration for it via test arg file, and it is used to report job-specific analytic events. Some, if not all, examples:

• Job processing durations

• Job preparation duration - how long to takes to prepare job, i.e. discover all tests and enqueue them

• Bucket dequeue and enqueue events

• Test dequeue and enqueue events

• Bucket processing duration. Each job consists of tests, and tests are grouped into buckets. This metric is used to report the durations of these buckets.

• Stuck buckets in job

• Job queue states - number of enqueued and dequeued buckets, time to dequeue a bucket

• Test discovery events - test case count, test count, and aggregated test discovery duration

• Time to start a test metric

• Test preflight duration

• Test started event

• Time between tests

• Test finished event

• Concrete test duration

• Aggregated test duration

• Test postflight duration

• Useless test runner invocation duration metric

It is worth noting that specific analytics configuration does NOT inherit anything from global configuration. You must pass complete configurations via both queue configuration and via test arg file. One thing to keep in mind is that test arg file should contain a job specific analytics configuration, so at least analyticsConfiguration.persistentMetricsJobId can (and probably should) be different depending on your job (e.g. unit tests, ui tests, full regression suite, etc.).

As usual, it is better to refer to the unit tests to see all possible ways how test arg file can be configured. Tests for root test arg file structure are here, and tests for test arg file entries are here.

Kibana

Emcee now supports sending all its logs to a common Kibana end point(s). Kibana is configured via [global]AnalyticsConfiguration.kibanaConfiguration. Emcee will add the following values to logged messages:

• hostname — host which emitted a log message
• emceeVersion — version of Emcee
• emceeCommand — command being executed, e.g. dump, runTestsOnRemoteQueue, distWork, etc.
• workerId — worker which emitted a log message.
• processId and processName — pid and process name of Emcee itself, e.g. 3224 and EmceeWorker_ab33da.
• subprocessId and subprocessName — pid and process name of a subprocess which Emcee executed
• xcrunToolName — tool name which has been executed via xcrun, e.g. for xcrun xcodebuild this key will have xcodebuild value
• persistentMetricsJobId — a job id from test arg file
• subprocessPipe — standard output pipe name for messages logged by subprocesses, e.g. stdout or stderr. Currently Emcee logs subprocess messages during test discovery process, but more cases may be adopted in the future.

In addition to default keys that Emcee adds automatically, you may add your keys via [global]analyticsConfiguration.metadata. On our CI, for instance, we add the following keys to help us track down job-specific logs (so we append these values to specific configuration via testargfile): ciBranch, ciBuildNumber, ciPullRequestId.

Handy Pendy

• --temp-folder is now optional
• Emcee now removes test and test runner artifacts after executing a bucket. Plugins still have time to process test artifacts (see TestsWorkingDirectorySupport), but as soon as all plugins terminate, Emcee worker will clean up all files. This fixes an issue with a leaking disk space.
• Emcee parses xcodebuild -resultStreamPath file contents by default now and even attempts to add crash logs (if xcodebuild will emit them).
• Emcee now also collects the global test exceptions that happen before any test starts. Now, when test bundle crashes, Emcee will append these exceptions to all tests that were expected to run.
• xcrun simctl spawn calls are now limited by a hardcoded timeout of 30 seconds to prevent potential stalls. 30 seconds is more than enough, but if it is not, reach out to us.

Nice to Know

• Emcee now tracks test timeouts that may happen before any test starts, after the last test finishes, and in between two test runs. This timeout value is controlled by testArgFile.testTimeoutConfiguration.testRunnerMaximumSilenceDuration(https://github.com/avito-tech/Emcee/wiki/Test-Arg-File#testtimeou...