feat: In-memory provider for e2e testing and minimal usage #546

liran2000 · 2023-08-07T15:51:41Z

This PR

Issue 523 - In-memory provider for e2e testing and minimal SDK usage

Signed-off-by: liran2000 <liran2000@gmail.com>

src/main/java/dev/openfeature/sdk/FlagEvaluationDetails.java

Kavindu-Dodan · 2023-08-07T21:40:10Z

src/test/java/dev/openfeature/sdk/testutils/Flag.java

+    private Flags.State state;
+    private Map<String, Object> variants;
+    private String defaultVariant;
+}


I propose adding a context evaluator backed by a user-provided callback

See go-sdk implementation for example - https://github.com/open-feature/go-sdk/blob/main/pkg/openfeature/memprovider/in_memory_provider.go#L167-L175

Trying to understand what is the purpose and value here with enhancing with additional testing capabilities. The in-memory provider is for testing purposes, how is adding a context evaluator helping with testing actual flow ? it will only test the testing provider ?

The current implementation resides as a test source. This mismatch with the requirement of the specification.

When we agreed on the in-memory provider through OFEP and added it to the specification ¹, the agreement was to make the in-memory an extra provider built into the SDK itself. And we agreed to support evaluation contexts through lambda/callback.

So the packaging of the implementation should be corrected first. I am proposing to move it to a package named memprovider or any better-named package.

Regarding the purpose, the main benefit of having evaluation context support is testing SDK. It allows us to write end-to-end tests (or to migrate existing ones based on flagd) for context evaluations and verify SDK correctness. Besides, end users can use the provider to prototype OpenFeature features.

Footnotes

https://github.com/open-feature/spec/blob/main/specification/appendix-A.md#in-memory-provider ↩

src/test/java/dev/openfeature/sdk/testutils/Flags.java

src/test/resources/features/testing-flags.json

Kavindu-Dodan · 2023-08-07T22:01:33Z

Overall great work. I would love to get this to mergeable status. Please check comments from my initial review and let me know if you have any concerns

Signed-off-by: liran2000 <liran2000@gmail.com>

Kavindu-Dodan · 2023-08-08T18:12:54Z

CONTRIBUTING.md

@@ -16,20 +16,6 @@ If you think we might be out of date with the spec, you can check that by invoki

 If you're adding tests to cover something in the spec, use the `@Specification` annotation like you see throughout the test suites.

-## End-to-End Tests


Rather than removing, IMO we should mention that the e2e tests use InMemoryProvider. Consider the GO SDK readme for reference - https://github.com/open-feature/go-sdk/blob/main/e2e/README.md

Kavindu-Dodan · 2023-08-08T18:14:14Z

src/test/java/dev/openfeature/sdk/e2e/StepDefinitions.java

+        File file = new File(classLoader.getResource("features/testing-flags.json").getFile());
+        Path resPath = file.toPath();
+        String conf = new String(java.nio.file.Files.readAllBytes(resPath), "UTF8");
+        Flags flags = Flags.builder().setConfigurationJson(conf).build();


This is unused and unnecessary 🤔 Was the intention here to fail fast with flag configuration validation?

Leftover, will remove it

Kavindu-Dodan · 2023-08-08T18:46:40Z

Some observations and suggestions for improvement

In the current implementation, flag configurations are loaded from a string. I think this is not ideal for general usage and diverge from the specification ¹

Provider must be initiated with a pre-defined set of flags provided to a constructor

I am suggesting changing the constructor to accept a known map of flags. The end user may use any mechanism to parse their flags and initialize the provider.

Further, this should allow us to simplify Flags class and make it a simple pojo (no builders and parsing)

https://github.com/open-feature/spec/blob/main/specification/appendix-A.md#sdk-end-to-end-testing ↩

- move to non-test package - add ContextEvaluator - update Flags structure - implement events provider - enrich tests - refactor Signed-off-by: liran2000 <liran2000@gmail.com>

Signed-off-by: liran2000 <liran2000@gmail.com>

Kavindu-Dodan · 2023-08-10T18:59:15Z

pom.xml

@@ -494,62 +486,6 @@
        </plugins>
      </build>
    </profile>
-


We do not need to remove this profile. It is still needed and works normally

Changes you made at StepDefinisions.java [1] is sufficient to fulfill the comment

[1] - https://github.com/open-feature/java-sdk/pull/546/files#diff-ca6f171ac07058b6e01935480aefe8e8a07379832fc06c334a9ad8887ae3881f

Without this section, we do not initialize the test-harness submodule. See the build result - https://github.com/open-feature/java-sdk/actions/runs/5824793560/job/15795407914?pr=546

By this comment it can be removed, if you still think it should remain update here and I can return it.

  

yes, we still need them. Tests are now independent from flagd but we need this profile to intialize git submodule and run gherkin tetsts

See the GitHub action - https://github.com/open-feature/java-sdk/blob/main/.github/workflows/pullrequest.yml#L42-L43

I was thinking we could just always pull the submodule and copy the gherkin tests, since now they don't require the external flagd process. I dont really mind if we keep these integration tests on a separate profile or not.

I think if we keep the profile though, we should add back the little bit of documentation in the contributing guide about it.

Kavindu-Dodan · 2023-08-10T19:43:27Z

src/main/java/dev/openfeature/sdk/providers/memory/InMemoryProvider.java

+            .build();
+    }
+
+    private static Value objectToValue(Object object) {


I think this should be exposed in Value class itself. This will help other provider implementations

Kavindu-Dodan · 2023-08-10T19:44:37Z

src/main/java/dev/openfeature/sdk/providers/memory/InMemoryProvider.java

+     * @param map map of objects
+     * @return a Structure object in the SDK format
+     */
+    public static Structure mapToStructure(Map<String, Object> map) {


This can be exposed from MutableStructure class so that it can be commonly used where needed

Kavindu-Dodan · 2023-08-10T19:45:36Z

src/main/java/dev/openfeature/sdk/providers/memory/InMemoryProvider.java

@@ -0,0 +1,184 @@
+package dev.openfeature.sdk.providers.memory;


I think this is a good package name and differentiate from contrib provider implemenations (dev.openfeature.contrib.providers.*)

Kavindu-Dodan · 2023-08-10T19:59:13Z

src/test/java/dev/openfeature/sdk/providers/memory/InMemoryProviderTest.java

+        provider.updateFlags(flags);
+    }
+
+    public static Map<String, Flag<?>> buildFlags() {


Given the public accessor status & multiple usages, I think this fits well into testutils - https://github.com/open-feature/java-sdk/tree/main/src/test/java/dev/openfeature/sdk/testutils

Probably with a class named TestFlags

Kavindu-Dodan · 2023-08-10T19:59:56Z

src/test/java/dev/openfeature/sdk/e2e/StepDefinitions.java

-import dev.openfeature.sdk.Reason;
-import dev.openfeature.sdk.Structure;
-import dev.openfeature.sdk.Value;
+import dev.openfeature.sdk.*;


Please consider avoiding wildcard imports (check other places too)

Signed-off-by: liran2000 <liran2000@gmail.com>

Kavindu-Dodan · 2023-08-11T18:18:49Z

src/test/java/dev/openfeature/sdk/testutils/TestFlagsUtils.java

+
+    /**
+     * Building flags for testing purposes.
+     * @return


Suggested change

* @return

suggesting the removal as return is not defined

Kavindu-Dodan · 2023-08-11T18:24:48Z

src/test/java/dev/openfeature/sdk/providers/memory/InMemoryProviderTest.java

+
+    @SneakyThrows
+    @Test
+    void mapToStructureTest() {


Let's move this to StructureTest

Kavindu-Dodan

Except for the latest reviews, I do not see any other concerns. Approval from my end :)

Signed-off-by: liran2000 <liran2000@gmail.com>

CONTRIBUTING.md

src/main/java/dev/openfeature/sdk/MutableStructure.java

src/main/java/dev/openfeature/sdk/providers/memory/InMemoryProvider.java

src/main/java/dev/openfeature/sdk/providers/memory/Flag.java

src/test/java/dev/openfeature/sdk/providers/memory/InMemoryProviderTest.java

toddbaert · 2023-08-14T15:07:01Z

src/test/java/dev/openfeature/sdk/testutils/TestFlagsUtils.java

+     * Building flags for testing purposes.
+     * @return map of flags
+     */
+    public static Map<String, Flag<?>> buildFlags() {


I think this API looks pretty solid. It's what I expected. Interested in @justinabrahms @thisthat 's takes.

toddbaert

Overall this looks really good, thanks @liran2000 !!

I have a few small comments, in particular this one (unless I'm missing something). Also watch out for the Javadoc stuff I mentioned to make sure people don't accidentally think some of these classes are for general SDK usage.

I think with a few small changes this can be merged!

Signed-off-by: liran2000 <liran2000@gmail.com>

src/test/java/dev/openfeature/sdk/providers/memory/InMemoryProviderTest.java

toddbaert · 2023-08-14T17:13:46Z

@liran2000 don't worry about the codecov... IMHO your coverage of new code is fine, but there's an issue with one test.

Signed-off-by: liran2000 <liran2000@gmail.com>

verify emit event called on the spied provider as it verifies the config changed event was emitted, which is the in-memory provider implementation. The event itself is a functionality of the events framework, thus not needed to be tested here, Signed-off-by: liran2000 <liran2000@gmail.com>

…o feat/523-in-memory-provider

toddbaert

Implementation looks good to me, and tests look stable now. Thanks again @liran2000

Approving. I will wait until EOD before merging to give others a chance to review.

sonarcloud · 2023-08-15T13:50:28Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
3 Code Smells

No Coverage information
0.0% Duplication

toddbaert · 2023-08-15T20:07:01Z

@liran2000 I opened a small PR to satisfy codecov: #561, and made some other minor changes I didn't notice here.

Add InMemoryProvider

c69b5a9

Signed-off-by: liran2000 <liran2000@gmail.com>

beeme1mr changed the title ~~In-memory provider for e2e testing and minimal SDK usage~~ feat: In-memory provider for e2e testing and minimal SDK usage Aug 7, 2023

Kavindu-Dodan reviewed Aug 7, 2023

View reviewed changes

src/main/java/dev/openfeature/sdk/FlagEvaluationDetails.java Show resolved Hide resolved

Kavindu-Dodan reviewed Aug 7, 2023

View reviewed changes

src/test/java/dev/openfeature/sdk/testutils/Flags.java Outdated Show resolved Hide resolved

Kavindu-Dodan reviewed Aug 7, 2023

View reviewed changes

src/test/resources/features/testing-flags.json Outdated Show resolved Hide resolved

liran2000 added 2 commits August 8, 2023 09:30

Remove flagd dependencies

29c99eb

Signed-off-by: liran2000 <liran2000@gmail.com>

handle PROVIDER_NOT_READY on evaluation

9af7d1d

Signed-off-by: liran2000 <liran2000@gmail.com>

Kavindu-Dodan reviewed Aug 8, 2023

View reviewed changes

liran2000 and others added 3 commits August 10, 2023 19:44

Merge branch 'open-feature:main' into feat/523-in-memory-provider

4f90f66

in-memory provider updates

47d8127

- move to non-test package - add ContextEvaluator - update Flags structure - implement events provider - enrich tests - refactor Signed-off-by: liran2000 <liran2000@gmail.com>

Remove Flags structure

542395e

Signed-off-by: liran2000 <liran2000@gmail.com>

liran2000 marked this pull request as ready for review August 10, 2023 17:23

liran2000 requested a review from a team as a code owner August 10, 2023 17:23

liran2000 requested a review from Kavindu-Dodan August 10, 2023 17:24

Remove leftover

4cf9ba4

Signed-off-by: liran2000 <liran2000@gmail.com>

Kavindu-Dodan reviewed Aug 10, 2023

View reviewed changes

Move common methods

f85cbb5

Signed-off-by: liran2000 <liran2000@gmail.com>

liran2000 requested a review from Kavindu-Dodan August 11, 2023 11:04

beeme1mr requested review from justinabrahms, lopitz and thomaspoignant August 11, 2023 18:04

Kavindu-Dodan reviewed Aug 11, 2023

View reviewed changes

Kavindu-Dodan approved these changes Aug 11, 2023

View reviewed changes

tests updates

1e508ca

Signed-off-by: liran2000 <liran2000@gmail.com>