🚀 Feature: --file-reporter [name] [output file]

[wesleyolis]

Hi,

I would like to propose a new feature, while the current reporter is outputting to the console,
I would like another reporter or maybe set or reports to be outputting to file.
For each file reporter a output file name and the report to use must be specified.

Then an additional option of how the file should be open, in append mode or freshly create a new file.

What we would like to do, is have moch run our test and see the output to the screen, but at the same time have a back ground process interpret the output at the same time. This would allow us to have our functionality feel more integrated.

Ideally the file output would be interpret the test file output and in our case produce a list of tests that failed with a certain criteria. This list would then potentially in our case be used as regular expression used used as input into our next test run.

The motivation is that we have a dynamically configuration application, were if certain configuration is missing, then the operator doesn't have for what ever good reason and money, security, the ability to run those test in his environment. As a developer, we would like to run all the other tests for which their is configuration available, however, like to be pain less and little work.
We don't want to ahve to hoist and provide summaries of configuration dependancies for tests, that are already deep down in modules or even have to categorised the tests, extra. All of this would require man power attempting to keep config and test categorisation in sync, which is a tedious task and can be automated.

As of current the idea is to dump the output run to file and loose our cli display so that we can cache these tests that don't have permission or config to run. Then any future runs of test environment will be much faster, as we just skip the test instead of all the overhead of starting it up in the first place.

Mabye at a later stage, this could be improved that one can skip tests based on a file path and a relative call stack., which would also leaned itself to a much more efficient lookup algorithm speeding up the process at a later stage of the game.

Please let me know you thoughts, so I can potentially look at moving forward with and implementation.

[plroebuck]

Started out sounding like you wanted to reimplement tee command internally. Personally, I desire a simpler --output <file> option (with stdout as its default) and have the reporter autoselect its output stream based on its environment (file/stdout for CLI, console.log for browser).

However, your motivation paragraph makes your argument sound somewhat off-putting -- you don't want to keep your config and tests in sync. Write/run a capabilities function in your test startup (beforeAll), then write a gestalt-like function run in each test (before) to determine whether to continue or skip. Don't believe your concept of skipping future runs by analyzing past runs works since system capabilities can change outside your control.
Or did I misunderstand you?

[wesleyolis]

Tree would be perfect as long one want the same report output format to screen and file... :-)
Could be a compromise and a start for now for minimal viability.

Imagination having alot of test that exercise code, then code that is being exercise, sprinkled all over the place checks a config file for properties if they exist or not then uses then,basically check a json structure in mem for property. This can be improved, depends weather it is end to end testing or not. The idea we have had is that.. Is that when configuration is missing, then throw certain new Guid Error. We would then check a first run of our test for this GUID, knowing that these test are missing configuration. For reasons, that this developer many not have access rights to retrieve or use that configuration for that code. So instead of having to now continually have to categorize our tests to main list of who can run what at a code level and which test at which levels, which is a big task. It simpler to just relying on weather he was permitted to get certain configuration or not, if doesn't have configuration for those tests, then they won't run locally and have be run when process by a build CI process.

Their is a single point the configuration file, at which we can control everything from for a permissions and security basis. Our configuration file would basically be encrypted on the server and different sections with different encryption, one would require the correct encryption key to decrypt the configuration file for the settings you require after assume that role with you current amazon account. The server will then only return the section, to which your encryption key has granted you access to decrypt, which still be transmitted to you over an encryption, that will then be decrypted on your local machine directly into some secure memory area.

[JoshuaKGoldberg]

Per #5027, we don't plan on any new major features. This is implementable in userland with a custom reporter.

If the community ends up loving the idea and embracing a third party reporter that does it, we can consider adding this to core.

Cheers!