Define test execution contracts to enable out of proc services

[sajayantony]

The goal of this item is to come up with a definition of how tests would communicate out of the xUnit runner. The proposed set of guiding principles here are

• Tests that need to make an network call usually need a well defined way of ensuring that the endpoint is running and ready.
• The endpoint/host is configurable and manageable through rest apis
• The endpoint could use the full framework to allow writing wcf services.
• xUnit can communicate with the host through well defined methods

As an example

PUT - http://localhost/initialize/test/ with body {name: "test name", parameters: "..."}
POST - http://localhost/shutdown
GET - http://localhost/log

Looping in folks for thoughts and elaborating further.
@StephenBonikowsky @roncain @iamjasonp

[roncain]

Currently each scenario test method sets itself up by getting a static property that returns the URL of the WCF service it will use.
Example:

        IRequestChannel channel = factory.CreateChannel(
              new EndpointAddress(Endpoints.HttpBaseAddress_Basic));

I think the simplest approach would be to continue to use this style, but make the static properties interact with a well-known REST application to acquire that URL (rather than returning constant strings as they do today). This would probably take the form of a static test helper issuing a Http GET to that REST application passing in the symbolic name of that endpoint (example: "HttpBasicEndpoint"). The REST service would be responsible for ensuring the requested WCF service was started and return a 200 response with the full URL of the endpoint the test requested. The test methods would then do their work against this URL. Actually this means the test methods as they exist today would not change.

This would also mean the REST service is responsible only for managing WCF services. It need not be aware of what tests will be run or what parameters they require. It also means the WCF services will be activated as they are needed. There is no need to try to decide up front which ones those are for a random set of test methods.

I think the REST service would also need to support management API's as you suggested, such as shutting down services, retrieving logs, etc.

One challenge will be to decide how that REST service is initially activated on every developer's machine. We can investigate that independently.

Some other interesting areas to explore include asking where the REST service runs and how it is configured. In the simplest case, it would run on the developer's machine and return localhost URL's. It could also be aware of WCF services running on other machines and return those remote URL's. Or the REST service itself could be running on a different machine.

I'd recommend walking before running and just get a localhost version of the REST service going.

[sajayantony]

@roncain @hongdai I've put together something we can use to discuss this a bit more.
I'll be updating the main comment to reflect the state of the discussion.
For the bootstrap toolset i've checked in something you can try out.

git clone https://github.com/SajayAntony/bifrost.git
cd bifrost
./build.ps1
./Artifacts/ensureBridge.ps1

The API definition is following the guidance here - http://sajayantony.github.io/bifrost/

@iamjasonp the idea here is that this would be the bootstrap for our test host and also for our cross platform api blueprint.

[roncain]

Nice! I cloned and debugged through bridge.exe (VS attach) to understand how this worked. And I experimented with changing ensureBridge.ps1 to invoke the Hostname resource (because I saw it was available). I also experimented with writing an HttpClient to call it, and that worked great:

static void Main(string[] args)
{
    HttpClient httpClient = new HttpClient();

    HttpRequestMessage request = new HttpRequestMessage(HttpMethod.Put, "http://localhost:8080/resource")
    request.Content = new StringContent("{name:'Bridge.Commands.WhoAmI'}");
    request.Content.Headers.ContentType = new MediaTypeHeaderValue("application/json");
    HttpResponseMessage response = httpClient.SendAsync(request).Result;
    if (response.StatusCode == HttpStatusCode.OK)
    {
        string responseContent = response.Content.ReadAsStringAsync().Result;
        Console.WriteLine("Response was '{0}'", responseContent);
    }
    else
    {
        Console.WriteLine("Unexpected status returned: {0}", response.StatusCode);
    }

    Console.ReadLine();
}

Question: I see that PUT is the natural Http method to say "create" (for us: start service) and DELETE is the one to say "destroy" (for us, stop service). What kinds of things were you thinking we would do with POST? Would that offer a way to "update" some aspects of the resource?

I think we have some interesting choices how to structure the routes and parameters. WebAPI would also allow parameters to be part of the route, or a query string (as well as the request body which was used here). But that is a secondary conversation as we go.

I like this approach.

[sajayantony]

Thanks for trying this out - @roncain as per the current api blueprint and as you correctly pointed out it would be a good idea to stick with the basic guiding principles of http verbs like PUT for create.

What kinds of things were you thinking we would do with POST? Would that offer a way to "update" some aspects of the resource?

Yes post in my mind was for update and it might include things like updating the state of a resource etc. This is open for the test to interpret as a configuration update. For e.g. maybe remove an endpoint or add state information like sending a key or data to the service etc. which might be dynamically generated on the client.

To be honest I even envision being able to centrally log operations and not have to do the painful text based logging. But for now I don't want to digress and over engineer a simple solution but keep options open and as @hongdai pointed out we should try to just solve the primary problem.

But from a project file perspective we should be able to call the script like ensureBridge.ps1 on windows which basically boils down to Invoke web request and checks again.

Invoke-RestMethod http://localhost:8080/resource/WhoAmI -Method PUT -Body "{name:'Bridge.Commands.WhoAmI'}" -ContentType application/json

Or OS X/Linux bash we don't want to get into cross machine launching and could resort to something simple like

curl --request PUT 'http://localhost:8080/resource' -H "Content-Type:application/json" -H "Accept: application/json" --data "{name:'Bridge.Commands.Hostname'}"

Design options

We have multiple options for using the body or parameters. But for consistency I personally feel the body makes more sense and is more flexible to pass in objects and other things to the constructor and if we pick this, we could just wrap it with simple helpers that standardize on the body semantics and make requests like this for e.g.

await Bridge.PutAsync(new {name : "Bridge.Commands.WhoAmI"}).StatusCode == OK

If we have many operations we could just make his a swagger and auto generate C# to get fancy for kicks ;)

Next up is to just launch our service using a the bridge.

Also if we use the Asp.net webapi client your code should condense to

using (HttpClient client = new HttpClient()) {
    client.BaseAddress = new Uri("http://localhost:8080");
    var result = client.PutAsJsonAsync("/resource/", 
            new { name = "Bridge.Commands.WhoAmI"}).Result;
    Assert.Equal(result.StatusCode , HttpStatusCode.OK);
}

I've added the tests to the bridge repo

[roncain]

Are you thinking this belongs in a different GitHub repo, or is WCF okay for it?

I recommend we take this in these stages (separate PR's):

1. Check this into WCF
2. Create friendly tests helpers that use HttpClient like you show. This includes agreeing on how the clients describe the request body and handle the response body.
3. Convert existing CMD to be launched by Bridge and change the Endpoint.cs to call the helpers
   I don't believe we should continue to maintain the current CMD approach or introduce new CMD's for other kinds of services.

[sajayantony]

@dmetzgar is currently looking at getting a crisp design for this in the build system.

[dmetzgar]

I looked into the bridge and have some design questions. The current system scans assemblies looking for types that have an Execute method, which is discovered through reflection. The assemblies would be loaded into the current appdomain. If I load the assemblies into a different appdomain, it's more difficult to do a reflective scan of types. Also, the current WcfService.exe relies on settings in its app.config. Creating a new appdomain per resource directory gives me the ability to set the app.config. One approach we can take is an interface shared between the bridge and any resources we want to load. But I think it's less intrusive if we just look for an .exe and .exe.config in the resource directory and use that as the appdomain's executing assembly. In that case, when you POST a new config resource directory, the bridge would create a new assembly and a new thread/task that does an execute assembly on whatever .exe it finds in the resource directory. That makes it easier to continue using the app.config for configuration of the WCF services.

If we take that approach, the resources API is not needed because the bridge is pretty dumb about what it's starting up. It probably wouldn't work if you want to eventually be able to query the bridge for the URLs to use in the test. But in that case, I would imagine you're not changing the port numbers and only need the server name/IP address, which the bridge could easily provide.

My thinking on breaking up the tests is that each test could be its own exe with its own config and would not have to implement any interfaces or follow any patterns in order to work with the bridge. The logging could be done by setting up a trace listener from the bridge. What do you guys think of this approach?

[mconnew]

I don't like the idea of launching separate processes for several reasons.

• Cleanup will be a lot harder as you can't guarantee the child processes will be killed when the bridge process ends
• Port sharing will be a lot harder with NetTcp so each test will have it's own port. This then makes firewall configuration harder
• Test suite will take longer to run as each test will require a full process launch
• Relying on .config files will dig a hole for us if/when the service code moves to core clr

The multiple app domain idea also won't be compatible with running on core clr as it doesn't support multiple app domains.

[hongdai]

Agree with mconnew. This approach has more disadvantages than advantages.

[dmetzgar]

Let me address Matt's comments.

• There will not be child processes. Everything is done in one process with different appdomains.
• If the nettcp tests are split up, I don't see how you get around port sharing or a port per servicehost, regardless of whether you're using different appdomains. Is there something you had in mind that would work better?
• Again, there are no child processes. Splitting up the tests would mean the first test suite execution would take longer regardless.
• I guess I'm confused about this last point. What it implies is that you will eventually implement servicehost on Core but will not implement config via the app.config. When that's done, you wouldn't need any of this. And I would imagine your test would be for both client and service. If that's the roadmap, how much time are we talking about here?

[hongdai]

@dmetzgar, could you clarify each test has it's own exe? I thought that meant each test will be built into a separate exe - this will significantly increase the build time of our tests. It would increase the running time because there are much more assemblies to load.

We recently change one test exe to several exes and there are noticeable performance degradation already.

[roncain]

I always like to get the functional requirements out into the light first and then design around that. I think these requirements are:

• Bridge either self-activates (IIS?) or is started only once during the build
• Bridge can be pinged to see if it is alive, and it can be ask to shutdown and/or restart.
• Bridge has no knowledge of WCF and is "configured" by giving it the location of an arbitrary collection of DLL's
• Though some convention (presence of Execute method? Main?) Bridge can find the available resources and knows how to interact with them.
• Bridge loads the resources into a separate AppDomain so that it has full control over its lifespan and can be reconfigured with a new or modified set of DLL's
• For each resource, it must be possible to start/restart it (PUT), alter its configuration (POST), retrieve data back from it (GET) and kill it (DELETE).

It is this final requirement -- the interaction model -- that makes me prefer an AppDomain over an EXE. I think the Reflection needs to move into the AppDomain too, so Bridge would likely have its own MBRO object that it creates in the new AppDomain and just delegates to it. I think each resource probably needs its own AppDomain (think about tearing down one resource but leaving the others running).

I used to think the PUT needed to return data, but now I think it only needs to return 200, or possibly some unique ID that can be re-used (imagine we had multiple instances of the same resource running someday). This requirement suggests we need more than an Execute() method. Maybe we literally need methods called PUT, POST, GET and DELETE inside each resource!

As a concrete example, WCF tests would do a PUT of the "WcfTestService" to get it started, and it would do a GET to retrieve the URL it could use to talk to it directly (details of GET parameters to be worked out -- think about getting the URL, getting current status, getting a log file, etc). For the service shutdown test, it could first start the service (PUT), get the URL (GET). ping it, then kill it(DELETE) and then expect the next ping to throw.

[mconnew]

@roncain, your fifth item is specifying an implementation and isn't talking about requirements. I would also suggest that the requirement of being reconfigured with a new or modified set of DLL's is not needed considering items 1 and 2. Unless you feel there's a need to change the set of DLL's mid test run, then changing the set of DLL's could be done by restarting the Bridge with a new set of DLL's.
Also can you clarify the last requirement. Do you mean forcefully or would cooperatively be sufficient? In other words, would asking the resource to shut itself down be sufficient, and trust that it's well behaving and able to do so?
@dmetzgar, although there are no definite plans, there have been some light (and very non-committal) discussions around what service side functionality in core clr would look like. If we bring service side functionality to core clr, then we will want to test the cross product of desktop clr, core clr, client code and service code which means we wouldn't be able to use a separate AppDomain for running the service code in the Bridge.

[roncain]

I think a key feature is getting something running soon and then exploring how it can be extended. I want to avoid getting paralyzed with options. Therefore the minimum necessary behavior to me seem to be:

• Bridge can be pinged and launched as needed (or self activate if we find a way).
• Bridge is expected to be always running and is not started and stopped per test.
• Bridge can be given a folder containing arbitrary files, including assemblies, scripts, etc. This is a simple form or dependency-injection. Bridge has no knowledge of WCF or tests.
• Based on some signature TBD, Bridge identifies the "resources" in those files. The signature is likely well-known method signatures in assemblies.
• Based on incoming URL, Bridge can invoke the corresponding methods in the specified resource (e.g. POST means START, DELETE means STOP)
• I believe the minimum set of commands a resource should support for now are START and STOP (or SETUP and TEARDOWN).
• The START command should be given the path to the folder originally given to the Bridge and also the original HttpRequestMessage passed to the Bridge (possibly to extract query parameters, body, etc.), and it should return a string, which Bridge retains. This string is probably a URL but it could be anything.
• The STOP command should be given the same string returned from START.
• The application can always do a GET to Bridge to obtain the string returned from START. This is probably the most frequent call made to Bridge, as tests fetch the URL of their service.
• Calling START is a no-op if the resource is already known running.
• Bridge does not know how to ping a running resource for a health check. It only knows how to START and STOP it. (In future we could add this if we need it).

I think separate AppDomains for each resource make the most sense for flexibility.
I believe concerns about possibly running Bridge in CoreCLR eventually are too far in the future to affect our v1 decisions. Good we discuss the future of server-side WCF in CoreCLR, but not for Bridge v1.

[mconnew]

We need to add support to the Bridge mechanism to allow for specifying a hostname to be localhost.fiddler to allow fiddler inspection of the requests when running on localhost. Currently there is a stopgap solution in #89 which will need to be reverted once this work is complete.

[roncain]

When I resolve #29 we will have a mechanism to pass MSBuild properties down into the built assembly which can be forwarded to the Bridge. We need a mechanism for the Bridge to accept a set of name value pairs when configured, and the fiddler prefix can be one of them.

[sajayantony]

Fixed by 3cadc82