Setup integration testing against real Azure services on the CI

[ppalaga]

We chatted about this with @agoncal

[agoncal]

@JonathanGiles mentioned that the tests are not executed against "real" Azure services (as tests could be unstable). They are run against proxies that have recorded specific Azure HTTP responses (this is my understanding, I'm sure @JonathanGiles will clarify this).

[ppalaga]

It would be great to hear more details.

[agoncal]

The Azure SDK uses a mechanism called "PlayBack Testing". Basically it records the network calls when invoking the Azure services, stores the requests/responses in JSon, and then runs the tests against these JSon files.

Here is the test class in AppConfiguration which is setting up the ConfigurationClient to handle either PLAYBACK or RECORD testing https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/appconfiguration/azure-data-appconfiguration/src/test/java/com/azure/data/appconfiguration/ConfigurationClientTest.java#L73 

When in PLAYBACK it will override the classpath configured HttpClient with PlaybackClient that is capable of matching up requests with the recorded responses. Or, when in RECORD it will add an HttpPipelinePolicy that will record network calls to create the playback record. 

And here is an example of a playback record: https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/appconfiguration/azure-data-appconfiguration/src/test/resources/session-records/ConfigurationAsyncClientTest.addConfigurationSettingConvenience%5B1%5D.json

Read https://github.com/Azure/azure-sdk-for-java/tree/main/sdk/core/azure-core-test

[radcortez]

We could probably try it with the config extension, since it is fairly simple.

[ppalaga]

Thanks for the explanation, @agoncal.

I think this concept of PlayBack testing could work well for testing on JVM.

For native, I have a bit of methodological concerns: Quarkus native integration tests consist of two substantial parts: 1. a test application 2. assertions. Out of those, only the test app is compiled into native executable. It is compiled together with all its transitive dependencies. I think it would not be correct to switch the production HTTP client for the PlayBack client, because that would result in a substantially different test app. We could miss important native mode issues in that way. I wonder how hard it would be to modify the concept so that the recorded responses would be sent over HTTP by a mock server?

We should definitely use this for JVM testing while we do not have anything better.

A technical question: How does the Azure SDK team handle to life cycle of the recorded responses? Are they updated regularly - time based, or perhaps on every deployment of the given Azure service?

[ppalaga]

cc @maxandersen

[agoncal]

@ppalaga Good question.

@alzimmermsft do you know how often the JSON requests/responses are updated?

[alzimmermsft]

Unfortunately, there isn't an exact answer to how often the playback record are re-recorded as this is left to the maintainer/maintainers of a given SDK. Generally, they should either be re-recorded or updated when new service features are supported by the SDK, but in many cases only new tests are recorded in these cases and minimal updates are applied to existing recordings (for example changing API versions in the JSON file with a find and replace mechanism). And for the SDKs this works as we also perform nightly, or manually triggered if needed, testing against live Azure resources.

I recently began discussions with our engineering systems team on ways we could make it easier, and safer, to record playback recordings by hooking into our existing live testing pipelines but at that time this was more a developer convenience, both internally and externally, and I didn't push too much on this. I can start those discussions up again if there will be a requirement to maintain "fresh" playback recordings. Azure/azure-sdk-tools#5138

I wonder how hard it would be to modify the concept so that the recorded responses would be sent over HTTP by a mock server?

Funny you mention this as this is something we are currently migrating to 😄 Azure/azure-sdk-for-java#24792

We're migrating to an out-of-process mock server instead of using the in-process mock HttpClient.

[agoncal]

@alzimmermsft so it is accurate to say that each time we bump the version of the Azure SDK we should re-record the JSON outputs?

[agoncal]

FYI I am starting experimenting this feature: https://github.com/quarkiverse/quarkus-azure-services/tree/agoncal/playback

[alzimmermsft]

@alzimmermsft so it is accurate to say that each time we bump the version of the Azure SDK we should re-record the JSON outputs?

I may have been a bit unclear on wording there, any time the SDK supports a new Azure service version the playback records should be re-recorded, not any time a new SDK version is released. So, say AppConfiguration releases a new service version that the SDK supports the playback recordings should be re-recorded, but we do new SDK releases every month and those don't require re-recording.

[ppalaga]

I wonder how hard it would be to modify the concept so that the recorded responses would be sent over HTTP by a mock server?

Funny you mention this as this is something we are currently migrating to smile Azure/azure-sdk-for-java#24792

We're migrating to an out-of-process mock server instead of using the in-process mock HttpClient.

Great news, thanks for the info! We will be able to use it for both JVM and native tests then.

[agoncal]

@alzimmermsft when I record my test again the real Azure App Configuration service, it generates a JSON file with empty calls and variables. What is do is just set the AZURE_TEST_MODE variable to RECORD and execute my test. Anything else I should be doing?

{
  "networkCallRecords" : [ ],
  "variables" : [ ]
}

When executing the test I do see some traces related to HTTP requests:

INFO  [com.azu.dat.app.imp.Con.listKeyValues] (main) {"az.sdk.message":"HTTP request","method":"GET","url":"https://appcs-quarkus-azure-app-conf.azconfig.io/kv?api-version=1.0","tryCount":"1","contentLength":0}
INFO  [com.azu.dat.app.imp.Con.listKeyValues] (reactor-http-nio-1) {"az.sdk.message":"HTTP response","statusCode":200,"url":"https://appcs-quarkus-azure-app-conf.azconfig.io/kv?api-version=1.0","durationMs":754}

See https://github.com/quarkiverse/quarkus-azure-services/blob/agoncal/playback/playback-tests/azure-app-configuration/src/test/resources/session-records/AzureAppConfigurationTest.azureAppConfiguration.json

[alzimmermsft]

@agoncal I don't know much about how the QuarkusTest annotation works, but does the test configure RecordNetworkPolicy in the service client builder when running in RECORD mode? Using ConfigurationClient and ConfigurationClientBuilder as an example:

public class TestExample extends TestBase {
    public ConfigurationClient setupTestClient() {
        ConfigurationClientBuilder builder = new ConfigurationClientBuilder()
            // In PLAYBACK use the PlaybackClient otherwise let the classpath determine the HttpClient.
            .httpClient(getTestMode() == TestMode.PLAYBACK ? interceptorManager.getPlaybackClient() : null) 
            .endpoint(endpoint)
            // TokenCredential needs to be mocked in PLAYBACK as it shouldn't require the network in PLAYBACK.
            .credential(getTestMode() == TestMode.PLAYBACK ? mockCredential : realCredential);
    
        if (getTestMode() == TestMode.RECORD) {
            // Add the RecordNetworkPolicy in RECORD mode to capture requests and responses.
            builder.addPolicy(interceptorManager.getRecordPolicy());
        }
        
        return builder.builderClient();
    }
}

[majguo]

Hello @ppalaga @agoncal Today I talked to @backwind1233 who is from Azure Spring Cloud team and has been working on developing spring starter on top of Azure Java SDK (which is quite similar Quarkus Azure service extension). He showed me that Azure Spring Cloud starters does run integration test against real azure services, for example, this file declares App Config service that will be deployed and tested during the integration testing.

As a result, I think we can circle back and also consider this direction which was the original request. I'll work on it next week and share more later.

[edburns]

I endorse @majguo 's approach to follow the existing practice for Spring Cloud Azure, which is to use actual service instances.

[agoncal]

Since I talked to @alzimmermsft, the Azure SDK team created this blog post with a video on there Test Proxy. It's really what we are looking for. Check it out https://devblogs.microsoft.com/azure-sdk/level-up-your-cloud-testing-game-with-the-azure-sdk-test-proxy.

[majguo]

@agoncal @ppalaga @edburns and All - Thank you all for valuable inputs. I just created a separate issue #110 to track the enhancement about setting up the integration tests using the Azure SDK test proxy. This issue (#37) will just serve its original request - Setup integration testing against real Azure services on the CI.