Rest API Spec tests runner

[Mpdreamz]

Opening this PR since this feature branch is becoming too big to manage.

what

This re-introduces the ability for us to run the rest api spec tests

Which are defined in yaml as a sequence of steps to run in the elasticsearch codebase itself.

Why

Prior to 2.0 we supported this by transforming the yaml into xunit csharp code that we checked into git. We decided to remove this as we had/have an extensive unit/integration test suite already and these tests became more an excersise in how well can we translate yaml into c# code (or more specifically anonymous C# classes).

We are reintroducing this for multiple reasons:

• Our integration test suite is vast but it mainly targets NEST. NEST uses Elasticsearch.Net but this gives us an opportunity to test Elasticsearch.Net and the more dynamic/freeform use cases.
• All Elasticsearch clients are running these tests, .NET was the odd one out. This gives us an indicator of conformity amongst the other clients

How

This time around we no longer translate the yaml test files to xunit C# project that we then compile and run.
Instead we created a runner that translates the yaml at run time and dispatches calls into an IElasticsearchClient instance.

In order to help reflection when we generate IElasticsearchClient it also injects private attributes on the method e.g [MapsApi("namespace.method", "path1,path2")] this takes the guessing out of the equation which method to call.

The runner itself lives under src/Tests/Tests.YamlRunner as an F# program.

It's main is hopefully self documents the tests we take to run these tests

let! locateResults = Commands.LocateTests namedSuite revision directory file
let readResults = Commands.ReadTests locateResults 
let! runResults = Commands.RunTests readResults client
let summary = Commands.ExportTests runResults outputFile
    
Commands.PrettyPrintResults outputFile
       
return summary.Failed + summary.Errors

Invoke

The runner builds with the rest of the projects and is not special in that sense.It's invocation is a little bit special though by design. We try hard to fully automate all our procedures through build/scripts/scripts.fsproj on any OS.

The runner however expects the elasticsearch endpoint to be spun up before invocation. To do this all the clients standardize on the same script (.ci/run-elasticsearch.sh) which only exists as a shell script. This ties this part to a POSIX shell.

The actual invocation of the runner however can still be done through our .build.sh rest-spec-test or .build.bat rest-spec-test

The readme.md under ci goes further into the setup.

if you have a POSIX shell (that can start bash) and docker installed locally you can call

ELASTICSEARCH_VERSION=8.0.0-SNAPSHOT ..ci/run-tests

which is exactly what Jenkins over at https://clients-ci.elastic.co will end up calling.

The .ci folder is mostly ready to be setup internally with our infra team, it makes sense for this to land in master before we submit a PR internally to get it up though.

This PR also contains a github actions workflow that i tinkered with but will leave disabled for now. It has a lot of potential to standardize our common actions steps across clients but lacks reporting tools for now. Interacting with Github Checks is hard and there is no tests visualizer just yet.

On Azure Devops you can log special messages that appear as github checks. We use this on our normal integration tests to surface reproduce commands without having to dig through logs for instance.

Given the close relation of Azure Devops and Github Actions I hope these are only a matter of time.

Whats next

Not all the tests actually pass sadly, this is in part because we don't have a block list for certain files in the same way other runners do and in part because some features are not yet implemented (transform_and_set / headers come to mind).

For now I treat these failures as a feature since the runner also exports an junit.xml file that we create manually. I would like to see this file visualized in Jenkins with failures to know my xml juggling produced something Jenkins understands.

Will follow up with a new meta ticket detailing what's outstanding post merge.

[russcam]

This is one serious PR! 🎉

I've had an initial pass-through and left some comments. I'd like to dig more thoroughly into the F# project

[russcam]

I've left some more comments. I think it would be good to have some unit tests for the F# project

[russcam]

Some XML comments (///) on types would help in navigating what each is used for. The majority of them look to directly map to YAML test runner concepts, are there URLs that can be linked to, that have definitions for each too?

[Mpdreamz]

F# editor support is pretty great but this is a bit of a weak spot, rider offers no help and Ionide generates DocFx comments.

Okay to follow up in a new PR?

[russcam]

++

[russcam]

Some XML comments to indicate what each type is used for?

[russcam]

I've added some PowerShell scripts to allow running the CI flow on Windows. I know that CI isn't going to run these, but it's useful to be able to just run these scripts locally to test

[Mpdreamz]

Thanks for the notes and PS script @russcam 👍

[russcam]

Thanks for the notes and PS script @russcam 👍

No worries! With Docker for Windows, will need to give permission to docker to be able to create files in the repo directory and descendents; may want to create a separate Windows account for this purpose, for docker to use.

[Mpdreamz]

@russcam thanks for the review, other than the xmldocs addresses your points. Can you give this the 👍 or 👎 , looking to get this in to iterate over in smaller PR's

[russcam]

This is looking good, @Mpdreamz 👍