Add mock (unit) testing Java example

[t0yv0]

Hello!

• Vote on this issue by adding a 👍 reaction
• If you want to implement this feature, comment to let us know (we'll work with you on design, scheduling, etc.)

Issue details

This would illustrate how to do https://www.pulumi.com/docs/guides/testing/ for Java. See the Unit Testing section that mentions examples in other languages. We would first add an examples and then add Java reference to tag page. Note that APIs used for "unit" testing Pulumi programs need to become stable, so we might need to review these APIs before taking a further commitment.

@pawelprazak mentioned that mockito dependency is unfortunate, perhaps that can be avoided.

Affected area/feature

[pawelprazak]

Note to self: fix #419 after this one. actually it is possible to fix #419 first with a minimal subset of changes :) #591

[pawelprazak]

API that I'm working on:

PulumiTest.withOptions(new TestOptions(false))
  .monitorMocks(new MocksTest.MyMocks())
  .useMockRunner() // vs .useRealRunner()
  .build()

    /**
     * Run a Pulumi test stack callback asynchronously and return an exit code.
     *
     * @param stack the stack to run in Pulumi test runtime
     * @return a future exit code from Pulumi test runtime after running the stack
     */
    CompletableFuture<Integer> runAsync(Consumer<Context> stack);

    /**
     * Run a Pulumi test stack callback asynchronously and return {@link TestResult}.
     *
     * @param stack the stack to run in Pulumi test runtime
     * @return a future {@link TestResult} from Pulumi test runtime after running the stack
     */
    CompletableFuture<TestResult> runTestAsync(Consumer<Context> stack);

public class TestResult {

    private final int exitCode;
    private final List<Exception> exceptions;
    private final List<Resource> resources;
    private final List<String> errors;
    private final Stack stack;
    ...
}

There will also be PulumiTestInternal with a lot more API surface area for internal testing purposes, in fact, PulumiTest is just an interface that limits the internal one to the externally useful subset.

I'm also leaning towards having .useRealRunner() as the default and .useMockRunner() as an option, but I don't have a strong opinion on that one. Maybe we don't even need a mock runner.

The fact that we set a global singleton is an implementation detail, and it is also true for the Pulumi.run itself, and not specific to the test API, so we should document it but making is as part of API does not feel like thinking ahead, esp. since we may change this in the future.

[pawelprazak]

An advanced, internal test would look similar to (this was a work in progres version but close to what I have in mind): https://github.com/pulumi/pulumi-java/pull/590/files#diff-2fd8390afc1feee0a2d9b1064a0378ac8a486ca7c33716e8c42a0a39d803e804R39

    @Test
    void testTerminatesEarlyOnException() {
        var logger = defaultLogger();
        logger.setLevel(Level.OFF);
        var mock = PulumiTestInternal.withDefaults()
                .standardLogger(logger)
                .useRealRunner()
                .build();

        var result = mock.runTestAsync(ctx -> {
            Output.of(CompletableFuture.failedFuture(new RunException("Deliberate test error")));
            ctx.export("slowOutput", Output.of(
                    new CompletableFuture<Integer>()
                            .completeOnTimeout(1, 60, TimeUnit.SECONDS)
            ));
        }).join();

        assertThat(result.errors()).isNotNull();
        assertThat(result.errors()).isNotEmpty();
        assertThat(result.errors()).hasSize(1);
        assertThat(result.errors()).haveAtLeastOne(containsString("Deliberate test error"));

        assertThat(result.exceptions()).hasSize(2);
        assertThat(result.exceptions().get(0)).isExactlyInstanceOf(CompletionException.class);
        assertThat(result.exceptions().get(1)).isExactlyInstanceOf(RunException.class);
        assertThat(result.exceptions().get(1)).hasMessageContaining("Deliberate test error");

        assertThat(result.resources()).isNotNull();
        assertThat(result.resources()).isNotEmpty();
        assertThat(result.resources()).hasSize(1);
        var stack = result.stack();
        assertThat(Internal.of(stack.output("slowOutput", TypeShape.of(Object.class))).getDataAsync()).isNotCompleted();
        assertThat(Internal.of(stack.output("slowOutput", TypeShape.of(Object.class))).getValueNullable()).isNotCompleted();

        assertThat(result.exitCode()).isEqualTo(ProcessExitedAfterLoggingUserActionableMessage);
    }

[t0yv0]

And include ideas how to clean up the mock testing API.

[t0yv0]

See #591 (comment) with what's possible now and current pain-points.

[pawelprazak]

Thank you for providing usage for the current API.

We are really close to what I want to propose as the final state.

Current (as of writing):

public class AppTest {
    @Test
    public void testFoo() {
        // Pain: can we make this simpler?
        var mock = DeploymentTests.DeploymentMockBuilder.builder()
                .setOptions(new TestOptions(false))
                .setMocks(new MyMocks())
                .build(); // real runner and deployment are now the default, so this helps already

        var result = mock.runTestAsync(App::init).join();

        assertThat(result.exceptions).isEmpty();
        assertThat(result.resources.size()).isEqualTo(2);

        // Pain: finding resources is manual.
        Bucket bucket = (Bucket) result.resources.stream().filter(r -> r instanceof Bucket).findFirst().get();

        // Pain: awaiting outputs is non-trivial.
        var bucketData = Internal.of(bucket.bucket()).getDataAsync().join();
        assertThat(bucketData.isKnown()).isTrue();
        assertThat(bucketData.getValueNullable())
                .isEqualTo("my-bucket");

        var stackOut = Internal.of(result.stackOutput("bucketName", String.class)).getDataAsync().join();
        assertThat(stackOut.getValueNullable()).isEqualTo("my-bucket");
    }

    // the mock method signatures can be improved by using named typles
    public static class MyMocks implements Mocks {
        @Override
        public CompletableFuture<Tuples.Tuple2<Optional<String>, Object>> newResourceAsync(MockResourceArgs args) {
            Objects.requireNonNull(args.type);
            switch (args.type) {
                case "aws:s3/bucket:Bucket":
                    return CompletableFuture.completedFuture(
                            Tuples.of(Optional.of("i-1234567890abcdef0"),
                                    Map.of("bucket", "my-bucket"))
                    );
                default:
                    throw new IllegalArgumentException(String.format("Unknown resource '%s'", args.type));
            }
        }

        @Override
        public CompletableFuture<Map<String, Object>> callAsync(MockCallArgs args) {
            return CompletableFuture.completedFuture(null);
        }
    }
}       

Proposed changes:

        ...
        var test = PulumiTest.withOptions(TestOptions.preview())
                .mocks(new MyMocks()) // If omitted, EmptyMocks are provided with helpful exception if called
                .build();

        var result = test.runTestAsync(App::init).join();
        ...
        CompletableFuture<ResourceResult> newResourceAsync(ResourceArgs args) {...}
        ...
        CompletableFuture<Map<String, Object>> callAsync(CallArgs args) {...}
        ...

[t0yv0]

Curious, CallArgs and ResourceArgs are empty classes - I'm not sure actually we even use them or need them. Unifying them with MockCallArgs may be possible but we need to expose the structure that MockCallArgs exposes for the test logic for example. This may have implications for prod logic but may be workable.