prepare 5.9.1 release (#268)

* (5.0) add HTTP default headers method + some component refactoring * don't need to pass the whole config object to describeConfiguration() * simplify test logic for HTTP headers * (5.0) final test coverage improvements, for now, with enforcement * re-simplify DataBuilder * increase timeouts * misc fixes * rm unnecessary override * indents * update benchmark code for API change * support loading file data from a classpath resource * update metadata so Releaser knows about 4.x branch * minor test fixes * make class final * rm beta changelog items * test data source * more info about coverage in CONTRIBUTING.md * misc fixes/tests * use java-sdk-common 1.0.0 * use okhttp-eventsource 2.3.0 * use okhttp-eventsource 2.3.1 for thread fix * fix flaky tests due to change in EventSource error reporting * remove support for indirect put and indirect patch * fix typo in javadoc example code * clean up polling logic, fix status updating after an outage, don't reinit store unnecessarily (#256) * slightly change semantics of boolean setters, improve tests, misc cleanup * avoid NPEs if LDUser was deserialized by Gson (#257) * avoid NPEs if LDUser was deserialized by Gson * add test * fix release metadata * prepare 4.14.1 release (#200) * Releasing version 4.14.1 * exclude Kotlin metadata from jar + fix misc Gradle problems * update CI and Gradle to test with newer JDKs (#259) * update okhttp to 3.14.9 (fixes incompatibility with OpenJDK 8.0.252) * prepare 4.14.2 release (#205) * Releasing version 4.14.2 * update okhttp to 4.8.1 (fixes incompatibility with OpenJDK 8.0.252) * gitignore * Bump SnakeYAML from 1.19 to 1.26 to address CVE-2017-18640 * prepare 4.14.3 release (#209) * Releasing version 4.14.3 * comments * only log initialization message once in polling mode * [ch89935] Correct some logging call format strings (#264) Also adds debug logs for full exception information in a couple locations. * [ch90109] Remove outdated trackMetric comment from before service support. (#265) * Fix compatibility with Java 7. * Remove import that is no longer used. * add Java 7 build (#267) * prepare 4.14.4 release (#214) * Releasing version 4.14.4 * add and use getSocketFactory * alignment * add socketFactory to builder * test socket factory builder * preserve dummy CI config file when pushing to gh-pages (#271) * fix concatenation when base URI has a context path (#270) * fix shaded jar builds to exclude Jackson classes and not modify Jackson return types (#268) * add test httpClientCanUseCustomSocketFactory for DefaultFeatureRequestor * add httpClientCanUseCustomSocketFactory() test for DefaultEventSenderTest * add httpClientCanUseCustomSocketFactory() test to StreamProcessorTest * pass URI to in customSocketFactory event test * make test less ambiguous * copy rules to new FlagBuilder instances (#273) * Bump guava version (#274) * Removed the guides link * increment versions when loading file data, so FlagTracker will work (#275) * increment versions when loading file data, so FlagTracker will work * update doc comment about flag change events with file data * add ability to ignore duplicate keys in file data (#276) * add alias events (#278) * add alias events and function * update tests for new functionality * update javadoc strings * add validation of javadoc build to CI * update commons-codec to 1.15 (#279) * Add support for experiment rollouts * add tests and use seed for allocating user to partition * test serialization and add check for isExperiment * fix PollingProcessorTest test race condition + other test issues (#282) * use launchdarkly-java-sdk-common 1.1.0-alpha-expalloc.2 * Update src/test/java/com/launchdarkly/sdk/server/EvaluatorTest.java Co-authored-by: Sam Stokes <sstokes@launchdarkly.com> * Update src/test/java/com/launchdarkly/sdk/server/EvaluatorTest.java Co-authored-by: Sam Stokes <sstokes@launchdarkly.com> * Update src/test/java/com/launchdarkly/sdk/server/EvaluatorTest.java Co-authored-by: Sam Stokes <sstokes@launchdarkly.com> * Update src/test/java/com/launchdarkly/sdk/server/EvaluatorTest.java Co-authored-by: Sam Stokes <sstokes@launchdarkly.com> * changes per code review comments * Please enter the commit message for your changes. Lines starting * fix null pointer exception * address code review comments * address more comments * missed a ! for isUntracked() * fix default boolean for json * make untracked FALSE by default * refactoring of bucketing logic to remove the need for an extra result object (#283) * add comment to enum * various JSON fixes, update common-sdk (#284) * simlpify the logic and make it match node/.Net sdks * Update src/main/java/com/launchdarkly/sdk/server/EventFactory.java Co-authored-by: Sam Stokes <sstokes@launchdarkly.com> * add the same comment as the Node SDK * Remove outdated/meaningless doc comment. (#286) * protect against NPEs if flag/segment JSON contains a null value * use java-sdk-common 1.2.0 * fix Jackson-related build issues (again) (#288) * update to okhttp-eventsource patch for stream retry bug, improve tests (#289) * update to okhttp-eventsource patch for stream retry bug, improve test * add test for appropriate stream retry * add public builder for FeatureFlagsState (#290) * add public builder for FeatureFlagsState * javadoc fixes * clarify FileData doc comment to say you shouldn't use offline mode (#291) * improve validation of SDK key so we won't throw an exception that contains the key (#293) * fix javadoc link in FileData comment (#294) * fix PollingProcessor 401 behavior and use new HTTP test helpers (#292) * re-fix metadata to remove Jackson dependencies, also remove Class-Path from manifest (#295) * make FeatureFlagsState.Builder.build() public (#297) * clean up tests using java-test-helpers 1.1.0 (#296) * use Releaser v2 config + newer CI images (#298) * [ch123129] Fix `PollingDataSourceBuilder` example. (#299) * Updates docs URLs * always use US locale when parsing HTTP dates * use Gson 2.8.9 * don't try to send more diagnostic events after an unrecoverable HTTP error * ensure module-info file isn't copied into our jars during build * use Gradle 7 * update build for benchmarks * more Gradle 7 compatibility changes for benchmark job * test with Java 17 in CI (#307) * test with Java 17 in CI * also test in Java 17 for Windows * fix choco install command * do date comparisons as absolute times, regardless of time zone (#310) * fix suppression of nulls in JSON representations (#311) * fix suppression of nulls in JSON representations * distinguish between situations where we do or do not want to suppress nulls * fix identify/track null user key check, also don't create index event for alias * use latest java-sdk-common * fix setting of trackEvents/trackReason in allFlagsState data when there's an experiment * implement contract tests (#314) * Merge Big Segments feature branch for 5.7.0 release (#316) Includes Big Segments implementation and contract test support for the new behavior. * Fix for pom including SDK common library as a dependency. (#317) * Upload JUnit XML to CircleCI on failure (#320) Fix a bug in the CircleCI config that was only uploading JUnit XML on _success_, not failure. * Add application tag support (#319) * Enforce 64 character limit on application tag values (#323) * fix "wrong type" logic in evaluations when default value is null * Rename master to main in .ldrelease/config.yml (#325) * Simpler way of setting base URIs in Java (#322) Now supports the `ServiceEndpoints` config for setting custom URIs for endpoints in a single place * make BigSegmentStoreWrapper.pollingDetectsStaleStatus test less timing-sensitive * make LDEndToEndClientTest.test____SpecialHttpConfigurations less timing-sensitive * make data source status tests less timing-sensitive * use streaming JSON parsing for incoming LD data * fix tests * rm unused * rm unused * use okhttp-eventsource 2.6.0 * update eventsource to 2.6.1 to fix pom/manifest problem * increase efficiency of summary event data structures (#335) Co-authored-by: Eli Bishop <eli@launchdarkly.com> Co-authored-by: LaunchDarklyCI <dev@launchdarkly.com> Co-authored-by: LaunchDarklyCI <LaunchDarklyCI@users.noreply.github.com> Co-authored-by: Gavin Whelan <gwhelan@launchdarkly.com> Co-authored-by: ssrm <ssrm@users.noreply.github.com> Co-authored-by: Harpo Roeder <hroeder@launchdarkly.com> Co-authored-by: Ben Woskow <48036130+bwoskow-ld@users.noreply.github.com> Co-authored-by: Elliot <35050275+Apache-HB@users.noreply.github.com> Co-authored-by: Robert J. Neal <rneal@launchdarkly.com> Co-authored-by: Robert J. Neal <robertjneal@users.noreply.github.com> Co-authored-by: Sam Stokes <sstokes@launchdarkly.com> Co-authored-by: LaunchDarklyReleaseBot <launchdarklyreleasebot@launchdarkly.com> Co-authored-by: Ember Stevens <ember.stevens@launchdarkly.com> Co-authored-by: ember-stevens <79482775+ember-stevens@users.noreply.github.com> Co-authored-by: Alex Engelberg <alex.benjamin.engelberg@gmail.com> Co-authored-by: Alex Engelberg <aengelberg@launchdarkly.com>
launchdarkly · Jun 30, 2022 · 1392dea · 1392dea
1 parent 04422de
commit 1392dea
Show file tree

Hide file tree

Showing 19 changed files with 1,057 additions and 409 deletions.
diff --git a/build.gradle b/build.gradle
@@ -74,7 +74,7 @@ ext.versions = [
     "jackson": "2.11.2", 
     "launchdarklyJavaSdkCommon": "1.3.0",
     "okhttp": "4.8.1", // specify this for the SDK build instead of relying on the transitive dependency from okhttp-eventsource
-    "okhttpEventsource": "2.3.2",
+    "okhttpEventsource": "2.6.1",
     "slf4j": "1.7.21",
     "snakeyaml": "1.26",
     "jedis": "2.9.0"

diff --git a/src/main/java/com/launchdarkly/sdk/server/DataModelSerialization.java b/src/main/java/com/launchdarkly/sdk/server/DataModelSerialization.java
@@ -0,0 +1,141 @@
+package com.launchdarkly.sdk.server;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import com.google.gson.JsonElement;
+import com.google.gson.stream.JsonReader;
+import com.google.gson.stream.JsonToken;
+import com.launchdarkly.sdk.server.DataModel.FeatureFlag;
+import com.launchdarkly.sdk.server.DataModel.Segment;
+import com.launchdarkly.sdk.server.DataModel.VersionedData;
+import com.launchdarkly.sdk.server.interfaces.DataStoreTypes.DataKind;
+import com.launchdarkly.sdk.server.interfaces.DataStoreTypes.FullDataSet;
+import com.launchdarkly.sdk.server.interfaces.DataStoreTypes.ItemDescriptor;
+import com.launchdarkly.sdk.server.interfaces.DataStoreTypes.KeyedItems;
+import com.launchdarkly.sdk.server.interfaces.SerializationException;
+
+import java.io.IOException;
+import java.util.AbstractMap;
+import java.util.Map;
+
+import static com.launchdarkly.sdk.server.DataModel.FEATURES;
+import static com.launchdarkly.sdk.server.DataModel.SEGMENTS;
+import static com.launchdarkly.sdk.server.JsonHelpers.gsonInstance;
+
+/**
+ * JSON conversion logic specifically for our data model types.
+ * <p>
+ * More general JSON helpers are in JsonHelpers.
+ */
+abstract class DataModelSerialization {
+  /**
+   * Deserializes a data model object from JSON that was already parsed by Gson.
+   * <p>
+   * For built-in data model classes, our usual abstraction for deserializing from a string is inefficient in
+   * this case, because Gson has already parsed the original JSON and then we would have to convert the
+   * JsonElement back into a string and parse it again. So it's best to call Gson directly instead of going
+   * through our abstraction in that case, but it's also best to implement that special-casing just once here
+   * instead of scattered throughout the SDK.
+   * 
+   * @param kind the data kind
+   * @param parsedJson the parsed JSON
+   * @return the deserialized item
+   */
+  static VersionedData deserializeFromParsedJson(DataKind kind, JsonElement parsedJson) throws SerializationException {
+    VersionedData item;
+    try {
+      if (kind == FEATURES) {
+        item = gsonInstance().fromJson(parsedJson, FeatureFlag.class);
+      } else if (kind == SEGMENTS) {
+        item = gsonInstance().fromJson(parsedJson, Segment.class);
+      } else {
+        // This shouldn't happen since we only use this method internally with our predefined data kinds
+        throw new IllegalArgumentException("unknown data kind");
+      }
+    } catch (RuntimeException e) {
+      // A variety of unchecked exceptions can be thrown from JSON parsing; treat them all the same
+      throw new SerializationException(e);
+    }
+    return item;
+  }
+
+  /**
+   * Deserializes a data model object from a Gson reader.
+   * 
+   * @param kind the data kind
+   * @param jr the JSON reader
+   * @return the deserialized item
+   */
+  static VersionedData deserializeFromJsonReader(DataKind kind, JsonReader jr) throws SerializationException {
+    VersionedData item;
+    try {
+      if (kind == FEATURES) {
+        item = gsonInstance().fromJson(jr, FeatureFlag.class);
+      } else if (kind == SEGMENTS) {
+        item = gsonInstance().fromJson(jr, Segment.class);
+      } else {
+        // This shouldn't happen since we only use this method internally with our predefined data kinds
+        throw new IllegalArgumentException("unknown data kind");
+      }
+    } catch (RuntimeException e) {
+      // A variety of unchecked exceptions can be thrown from JSON parsing; treat them all the same
+      throw new SerializationException(e);
+    }
+    return item;
+  }
+
+  /**
+   * Deserializes a full set of flag/segment data from a standard JSON object representation
+   * in the form {"flags": ..., "segments": ...} (which is used in both streaming and polling
+   * responses).
+   * 
+   * @param jr the JSON reader
+   * @return the deserialized data
+   */
+  static FullDataSet<ItemDescriptor> parseFullDataSet(JsonReader jr) throws SerializationException {
+    ImmutableList.Builder<Map.Entry<String, ItemDescriptor>> flags = ImmutableList.builder();
+    ImmutableList.Builder<Map.Entry<String, ItemDescriptor>> segments = ImmutableList.builder();
+
+    try {
+      jr.beginObject();
+      while (jr.peek() != JsonToken.END_OBJECT) {
+        String kindName = jr.nextName();
+        Class<?> itemClass;
+        ImmutableList.Builder<Map.Entry<String, ItemDescriptor>> listBuilder;
+        switch (kindName) {
+        case "flags":
+          itemClass = DataModel.FeatureFlag.class;
+          listBuilder = flags;
+          break;
+        case "segments":
+          itemClass = DataModel.Segment.class;
+          listBuilder = segments;
+          break;
+        default:
+          jr.skipValue();
+          continue;
+        }
+        jr.beginObject();
+        while (jr.peek() != JsonToken.END_OBJECT) {
+          String key = jr.nextName();
+          @SuppressWarnings("unchecked")
+          Object item = JsonHelpers.deserialize(jr, (Class<Object>)itemClass);
+          listBuilder.add(new AbstractMap.SimpleEntry<>(key,
+              new ItemDescriptor(((VersionedData)item).getVersion(), item)));
+        }
+        jr.endObject();
+      }
+      jr.endObject();
+
+      return new FullDataSet<ItemDescriptor>(ImmutableMap.of(
+          FEATURES, new KeyedItems<>(flags.build()),
+          SEGMENTS, new KeyedItems<>(segments.build())
+          ).entrySet());
+    } catch (IOException e) {
+      throw new SerializationException(e);
+    } catch (RuntimeException e) {
+      // A variety of unchecked exceptions can be thrown from JSON parsing; treat them all the same
+      throw new SerializationException(e);
+    }
+  }
+}
diff --git a/src/main/java/com/launchdarkly/sdk/server/DefaultEventProcessor.java b/src/main/java/com/launchdarkly/sdk/server/DefaultEventProcessor.java
@@ -460,6 +460,7 @@ private void triggerFlush(EventBuffer outbox, BlockingQueue<FlushPayload> payloa
       } else {
         logger.debug("Skipped flushing because all workers are busy");
         // All the workers are busy so we can't flush now; keep the events in our state
+        outbox.summarizer.restoreTo(payload.summary);
         synchronized(busyFlushWorkersCount) {
           busyFlushWorkersCount.decrementAndGet();
           busyFlushWorkersCount.notify();
@@ -506,7 +507,7 @@ void addToSummary(Event e) {
     }
 
     boolean isEmpty() {
-      return events.isEmpty() && summarizer.snapshot().isEmpty();
+      return events.isEmpty() && summarizer.isEmpty();
     }
 
     long getAndClearDroppedCount() {
@@ -517,7 +518,7 @@ long getAndClearDroppedCount() {
 
     FlushPayload getPayload() {
       Event[] eventsOut = events.toArray(new Event[events.size()]);
-      EventSummarizer.EventSummary summary = summarizer.snapshot();
+      EventSummarizer.EventSummary summary = summarizer.getSummaryAndReset();
       return new FlushPayload(eventsOut, summary);
     }
 

diff --git a/src/main/java/com/launchdarkly/sdk/server/DefaultFeatureRequestor.java b/src/main/java/com/launchdarkly/sdk/server/DefaultFeatureRequestor.java
@@ -1,6 +1,9 @@
 package com.launchdarkly.sdk.server;
 
 import com.google.common.annotations.VisibleForTesting;
+import com.google.gson.stream.JsonReader;
+import com.launchdarkly.sdk.server.interfaces.DataStoreTypes.FullDataSet;
+import com.launchdarkly.sdk.server.interfaces.DataStoreTypes.ItemDescriptor;
 import com.launchdarkly.sdk.server.interfaces.HttpConfiguration;
 import com.launchdarkly.sdk.server.interfaces.SerializationException;
 
@@ -11,6 +14,7 @@
 import java.nio.file.Files;
 import java.nio.file.Path;
 
+import static com.launchdarkly.sdk.server.DataModelSerialization.parseFullDataSet;
 import static com.launchdarkly.sdk.server.Util.concatenateUriPath;
 import static com.launchdarkly.sdk.server.Util.configureHttpClientBuilder;
 import static com.launchdarkly.sdk.server.Util.getHeadersBuilderFor;
@@ -23,7 +27,7 @@
 import okhttp3.Response;
 
 /**
- * Implementation of getting flag data via a polling request. Used by both streaming and polling components.
+ * Implementation of getting flag data via a polling request.
  */
 final class DefaultFeatureRequestor implements FeatureRequestor {
   private static final Logger logger = Loggers.DATA_SOURCE;
@@ -59,7 +63,8 @@ public void close() {
     Util.deleteDirectory(cacheDir);
   }
 
-  public AllData getAllData(boolean returnDataEvenIfCached) throws IOException, HttpErrorException, SerializationException {
+  public FullDataSet<ItemDescriptor> getAllData(boolean returnDataEvenIfCached)
+      throws IOException, HttpErrorException, SerializationException {
     Request request = new Request.Builder()
         .url(pollingUri.toURL())
         .headers(headers)
@@ -75,18 +80,18 @@ public AllData getAllData(boolean returnDataEvenIfCached) throws IOException, Ht
         logger.debug("Cache hit count: " + httpClient.cache().hitCount() + " Cache network Count: " + httpClient.cache().networkCount());
         return null;
       }
-
-      String body = response.body().string();
 
-      if (!response.isSuccessful()) {
-        throw new HttpErrorException(response.code());
-      }
-      logger.debug("Get flag(s) response: " + response.toString() + " with body: " + body);
+      logger.debug("Get flag(s) response: " + response.toString());
       logger.debug("Network response: " + response.networkResponse());
       logger.debug("Cache hit count: " + httpClient.cache().hitCount() + " Cache network Count: " + httpClient.cache().networkCount());
       logger.debug("Cache response: " + response.cacheResponse());
-
-      return JsonHelpers.deserialize(body, AllData.class);
+
+      if (!response.isSuccessful()) {
+        throw new HttpErrorException(response.code());
+      }
+
+      JsonReader jr = new JsonReader(response.body().charStream());
+      return parseFullDataSet(jr);
     }
   }
 }
diff --git a/src/main/java/com/launchdarkly/sdk/server/EventOutputFormatter.java b/src/main/java/com/launchdarkly/sdk/server/EventOutputFormatter.java
@@ -5,12 +5,14 @@
 import com.launchdarkly.sdk.EvaluationReason;
 import com.launchdarkly.sdk.LDUser;
 import com.launchdarkly.sdk.LDValue;
-import com.launchdarkly.sdk.server.EventSummarizer.CounterKey;
 import com.launchdarkly.sdk.server.EventSummarizer.CounterValue;
+import com.launchdarkly.sdk.server.EventSummarizer.FlagInfo;
+import com.launchdarkly.sdk.server.EventSummarizer.SimpleIntKeyedMap;
 import com.launchdarkly.sdk.server.interfaces.Event;
 
 import java.io.IOException;
 import java.io.Writer;
+import java.util.Map;
 
 /**
  * Transforms analytics events and summary data into the JSON format that we send to LaunchDarkly.
@@ -28,6 +30,7 @@ final class EventOutputFormatter {
     this.gson = JsonHelpers.gsonInstanceForEventsSerialization(config);
   }
 
+  @SuppressWarnings("resource")
   final int writeOutputEvents(Event[] events, EventSummarizer.EventSummary summary, Writer writer) throws IOException {
     int count = events.length;    
     try (JsonWriter jsonWriter = new JsonWriter(writer)) {
@@ -112,59 +115,48 @@ private final void writeSummaryEvent(EventSummarizer.EventSummary summary, JsonW
     jw.name("features");
     jw.beginObject();
 
-    CounterKey[] unprocessedKeys = summary.counters.keySet().toArray(new CounterKey[summary.counters.size()]);
-    for (int i = 0; i < unprocessedKeys.length; i++) {
-      if (unprocessedKeys[i] == null) {
-        continue;
-      }
-      CounterKey key = unprocessedKeys[i];
-      String flagKey = key.key;
-      CounterValue firstValue = summary.counters.get(key);
+    for (Map.Entry<String, FlagInfo> flag: summary.counters.entrySet()) {
+      String flagKey = flag.getKey();
+      FlagInfo flagInfo = flag.getValue();
 
       jw.name(flagKey);
       jw.beginObject();
 
-      writeLDValue("default", firstValue.defaultVal, jw);
+      writeLDValue("default", flagInfo.defaultVal, jw);
 
       jw.name("counters");
       jw.beginArray();
 
-      for (int j = i; j < unprocessedKeys.length; j++) {
-        CounterKey keyForThisFlag = unprocessedKeys[j];
-        if (j != i && (keyForThisFlag == null || !keyForThisFlag.key.equals(flagKey))) {
-          continue;
-        }
-        CounterValue value = keyForThisFlag == key ? firstValue : summary.counters.get(keyForThisFlag);
-        unprocessedKeys[j] = null;
-
-        jw.beginObject();
-
-        if (keyForThisFlag.variation >= 0) {
-          jw.name("variation");
-          jw.value(keyForThisFlag.variation);
+      for (int i = 0; i < flagInfo.versionsAndVariations.size(); i++) {
+        int version = flagInfo.versionsAndVariations.keyAt(i);
+        SimpleIntKeyedMap<CounterValue> variations = flagInfo.versionsAndVariations.valueAt(i);
+        for (int j = 0; j < variations.size(); j++) {
+          int variation = variations.keyAt(j);
+          CounterValue counter = variations.valueAt(j);
+
+          jw.beginObject();
+
+          if (variation >= 0) {
+            jw.name("variation").value(variation);
+          }
+          if (version >= 0) {
+            jw.name("version").value(version);
+          } else {
+            jw.name("unknown").value(true);
+          }
+          writeLDValue("value", counter.flagValue, jw);
+          jw.name("count").value(counter.count);
+
+          jw.endObject();
         }
-        if (keyForThisFlag.version >= 0) {
-          jw.name("version");
-          jw.value(keyForThisFlag.version);
-        } else {
-          jw.name("unknown");
-          jw.value(true);
-        }
-        writeLDValue("value", value.flagValue, jw);
-        jw.name("count");
-        jw.value(value.count);
-
-        jw.endObject(); // end of this counter
       }
-      
+
       jw.endArray(); // end of "counters" array
-
       jw.endObject(); // end of this flag
     }
 
     jw.endObject(); // end of "features"
-
-    jw.endObject();
+    jw.endObject(); // end of summary event object
   }
 
   private final void startEvent(Event event, String kind, String key, JsonWriter jw) throws IOException {