Custom runnerBuilder breaks logcat capture

[TWiStErRob]

Description

Using testInstrumentationRunnerArguments["runnerBuilder"] allows invalid characters in test names.

For more info see mannodermaus/android-junit5#263

A related causal problem on AGP: https://issuetracker.google.com/issues/223643506

Steps to Reproduce

1. Clone https://github.com/TWiStErRob/repros/tree/master/agp/junit5-logcat-filenames
2. Run gradlew :app:connectedCheck

Actual Results

logcat output with invalid characters, file name is truncated, file is empty.

Expected Results

logcat output in build folder and/or a failure when using the custom runner builder when it returns ":" in the test name.

AndroidX Test and Android OS Versions

3.4.0 / 1.4.0, any OS

Link to a public git repo demonstrating the problem:

https://github.com/TWiStErRob/repros/tree/master/agp/junit5-logcat-filenames

[brettchabot]

This sounds more like a AGP or junit5 bug . androidx.test doesn't control how logcat capture is done

Am I missing something?

[TWiStErRob]

Yeah, I kind of agree, that's why I just mentioned the effect, not the cause.

You provide the ability to use runnerBuilder so you could solve these problems at the root by not allowing these characters in test names. Although this is probably a breaking change and might not even worth it as the test names are really nice with these chars and reports don't really care.

What you might want to think about is if this kind of usage breaks orchestrator or test storage as they use IPC and disk. And/or provide some documentation on methods which interface with external world what kind of characters to expect (because normally test names are [A-Za-z.]+\(\)).

[brettchabot]

I'd suggest filing a bug against android gradle plugin/Studio as that is the tool that stores logcat and has these test name constraints.

From an androidx.test.runner perspective the only obvious constraint I can see is test names when used with orchestrator cannot have commas or hash, since those characters are used to parse the '-e class ' argument.