From 9f10eea90bf274ce64e5f0277bd4461fcf8f893a Mon Sep 17 00:00:00 2001 From: Russ Cam Date: Tue, 14 Apr 2020 08:48:43 +1000 Subject: [PATCH 1/3] Update custom serialization docs This commit updates the custom serialization docs that discuss using JsonNetSerializer. 1. Remove the implication that the internal serializer is based on Json.NET. 2. Include recommendation to hook up JsonNetSerializer when using Newtonsoft.Json.Linq types like JObject --- .../HighLevel/Serialization/CustomSerialization.doc.cs | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/tests/Tests/ClientConcepts/HighLevel/Serialization/CustomSerialization.doc.cs b/tests/Tests/ClientConcepts/HighLevel/Serialization/CustomSerialization.doc.cs index 4cdad9b5068..3a1a3ee81ee 100644 --- a/tests/Tests/ClientConcepts/HighLevel/Serialization/CustomSerialization.doc.cs +++ b/tests/Tests/ClientConcepts/HighLevel/Serialization/CustomSerialization.doc.cs @@ -114,8 +114,10 @@ public class MyPercolationDocument * * We ship a separate {nuget}/NEST.JsonNetSerializer[NEST.JsonNetSerializer] package that helps in composing a custom `SourceSerializer` * using `Json.NET`, that is smart enough to delegate the serialization of known NEST types back to the built-in - * `RequestResponseSerializer`. This package is also useful if you want to control how your documents and values are stored - * and retrieved from Elasticsearch using `Json.NET`, without interfering with the way NEST uses `Json.NET` internally. + * `RequestResponseSerializer`. This package is also useful if + * + * . You want to control how your documents and values are stored and retrieved from Elasticsearch using `Json.NET` + * . You want to use `Newtonsoft.Json.Linq` types such as `JObject` within your documents * * The easiest way to hook this custom source serializer up is as follows */ From 513b637313d080ecc46ae466fc77375d382c4bf9 Mon Sep 17 00:00:00 2001 From: Russ Cam Date: Tue, 14 Apr 2020 16:30:17 +1000 Subject: [PATCH 2/3] Update docs to utf8json --- .../Serialization/CustomSerialization.doc.cs | 44 +++++++++++-------- 1 file changed, 25 insertions(+), 19 deletions(-) diff --git a/tests/Tests/ClientConcepts/HighLevel/Serialization/CustomSerialization.doc.cs b/tests/Tests/ClientConcepts/HighLevel/Serialization/CustomSerialization.doc.cs index 3a1a3ee81ee..b45fe630ac7 100644 --- a/tests/Tests/ClientConcepts/HighLevel/Serialization/CustomSerialization.doc.cs +++ b/tests/Tests/ClientConcepts/HighLevel/Serialization/CustomSerialization.doc.cs @@ -21,34 +21,40 @@ namespace Tests.ClientConcepts.HighLevel.Serialization /**[[custom-serialization]] * === Custom Serialization * - * NEST 6.x ships with a _shaded_ Json.NET dependency, meaning that all of Json.NET's types are - * internalized and IL merged into the NEST assembly, and their namespace has been changed - * from `Newtonsoft.Json` to `Nest.Json`. + * After internalizing the serialization routines, and IL-merging the Newtonsoft.Json package in 6.x, we are pleased to + * announce that the next stage of serialization improvements have been completed in 7.0. * - * Why would we do this, you may ask? Well, NEST has always isolated its dependency on Json.NET as best as it could, - * but this meant that we had to mandate how certain things were in the client. For instance, - * NEST heavily relied on the fact that the `IContractResolver` used by the configured serializer was - * an instance of `ElasticContractResolver`, so if you wanted to deserialize your `_source` or `_fields` - * using your own resolver, you were out of luck. In addition, continued improvements to NEST's serialization pipeline - * was stymied by this dependency and as client authors, we wanted to unhinder ourselves from this in order to explore the myriad - * of exciting developments happening in the .NET ecosystem around performance with the likes of `Span`, - * `ArrayPool` and a more compact UTF-8 string representation. + * Both SimpleJson and Newtonsoft.Json have been completely removed and replaced with an implementation of Utf8Json, a fast + * serializer that works directly with UTF-8 binary. * - * So what did we do in 6.x and how does it affect you? + * With the move to Utf8Json, we have removed some features that were available in the previous JSON libraries that have + * proven too onerous to carry forward at this stage. * - * The `NEST` nuget package from 6.0.0 onwards will use the internal Json.NET serializer and will in effect, behave the same - * as it did in previous releases. If you previously relied on custom Json.NET serialization and configuration with custom - * `JsonSerializerSettings` and `JsonConverter` types for example, things have changed a bit. The following section explains - * how to continue working with these with NEST 6.x. + * - JSON in the request is never indented, even if SerializationFormatting.Indented is specified. The serialization + * routines generated by Utf8Json never generate an IJsonFormatter that will indent JSON, for performance reasons. + * We are considering options for exposing indented JSON for development and debugging purposes. + * + * - NEST types cannot be extended by inheritance. With NEST 6.x, additional properties can be included for a type by deriving from + * that type and annotating these new properties. With the current implementation of serialization with Utf8Json, this approach will not work. + * + * - Serializer uses Reflection.Emit. Utf8Json uses Reflection.Emit to generate efficient formatters for serializing types that it sees. + * Reflection.Emit is not supported on all platforms, for example, UWP, Xamarin.iOS, and Xamarin.Android. + * + * - Elasticsearch.Net.DynamicResponse deserializes JSON arrays to List. SimpleJson deserialized JSON arrays to object[], + * but Utf8Json deserializes them to List. This change is preferred for allocation and performance reasons. + * + * - Utf8Json is much stricter when deserializing JSON object field names to C# POCO properties. With the internal Json.NET serializer + * in 6.x, JSON object field names would attempt to be matched with C# POCO property names first by an exact match, falling back to a + * case insensitive match. With Utf8Json in 7.x however, JSON object field names must match exactly the name configured for the + * C# POCO property name. */ public class GettingStarted { /**[float] * ==== Injecting a new serializer * - * Starting with NEST 6.x you can inject a serializer that is isolated to only be called - * for the (de)serialization of `_source`, `_fields`, or wherever a user provided value is expected - * to be written and returned. + * You can inject a serializer that is isolated to only be called for the (de)serialization of `_source`, `_fields`, or + * wherever a user provided value is expected to be written and returned. * * Within NEST, we refer to this serializer as the `SourceSerializer`. * From ee7f97f4c5edac9ec99124bce49c43ac42b8de4d Mon Sep 17 00:00:00 2001 From: Russ Cam Date: Wed, 15 Apr 2020 10:23:51 +1000 Subject: [PATCH 3/3] update docs --- .../custom-serialization.asciidoc | 57 +++++++++++-------- docs/query-dsl.asciidoc | 28 ++++----- 2 files changed, 47 insertions(+), 38 deletions(-) diff --git a/docs/client-concepts/high-level/serialization/custom-serialization.asciidoc b/docs/client-concepts/high-level/serialization/custom-serialization.asciidoc index 59ca6695413..6c2502ef5a8 100644 --- a/docs/client-concepts/high-level/serialization/custom-serialization.asciidoc +++ b/docs/client-concepts/high-level/serialization/custom-serialization.asciidoc @@ -15,32 +15,38 @@ please modify the original csharp file found at the link and submit the PR with [[custom-serialization]] === Custom Serialization -NEST 6.x ships with a _shaded_ Json.NET dependency, meaning that all of Json.NET's types are -internalized and IL merged into the NEST assembly, and their namespace has been changed -from `Newtonsoft.Json` to `Nest.Json`. - -Why would we do this, you may ask? Well, NEST has always isolated its dependency on Json.NET as best as it could, -but this meant that we had to mandate how certain things were in the client. For instance, -NEST heavily relied on the fact that the `IContractResolver` used by the configured serializer was -an instance of `ElasticContractResolver`, so if you wanted to deserialize your `_source` or `_fields` -using your own resolver, you were out of luck. In addition, continued improvements to NEST's serialization pipeline -was stymied by this dependency and as client authors, we wanted to unhinder ourselves from this in order to explore the myriad -of exciting developments happening in the .NET ecosystem around performance with the likes of `Span`, -`ArrayPool` and a more compact UTF-8 string representation. - -So what did we do in 6.x and how does it affect you? - -The `NEST` nuget package from 6.0.0 onwards will use the internal Json.NET serializer and will in effect, behave the same -as it did in previous releases. If you previously relied on custom Json.NET serialization and configuration with custom -`JsonSerializerSettings` and `JsonConverter` types for example, things have changed a bit. The following section explains -how to continue working with these with NEST 6.x. +After internalizing the serialization routines, and IL-merging the Newtonsoft.Json package in 6.x, we are pleased to +announce that the next stage of serialization improvements have been completed in 7.0. + +Both SimpleJson and Newtonsoft.Json have been completely removed and replaced with an implementation of Utf8Json, a fast +serializer that works directly with UTF-8 binary. + +With the move to Utf8Json, we have removed some features that were available in the previous JSON libraries that have +proven too onerous to carry forward at this stage. + +* JSON in the request is never indented, even if SerializationFormatting.Indented is specified. The serialization +routines generated by Utf8Json never generate an IJsonFormatter that will indent JSON, for performance reasons. +We are considering options for exposing indented JSON for development and debugging purposes. + +* NEST types cannot be extended by inheritance. With NEST 6.x, additional properties can be included for a type by deriving from +that type and annotating these new properties. With the current implementation of serialization with Utf8Json, this approach will not work. + +* Serializer uses Reflection.Emit. Utf8Json uses Reflection.Emit to generate efficient formatters for serializing types that it sees. +Reflection.Emit is not supported on all platforms, for example, UWP, Xamarin.iOS, and Xamarin.Android. + +* Elasticsearch.Net.DynamicResponse deserializes JSON arrays to List. SimpleJson deserialized JSON arrays to object[], +but Utf8Json deserializes them to List. This change is preferred for allocation and performance reasons. + +* Utf8Json is much stricter when deserializing JSON object field names to C# POCO properties. With the internal Json.NET serializer +in 6.x, JSON object field names would attempt to be matched with C# POCO property names first by an exact match, falling back to a +case insensitive match. With Utf8Json in 7.x however, JSON object field names must match exactly the name configured for the +C# POCO property name. [float] ==== Injecting a new serializer -Starting with NEST 6.x you can inject a serializer that is isolated to only be called -for the (de)serialization of `_source`, `_fields`, or wherever a user provided value is expected -to be written and returned. +You can inject a serializer that is isolated to only be called for the (de)serialization of `_source`, `_fields`, or +wherever a user provided value is expected to be written and returned. Within NEST, we refer to this serializer as the `SourceSerializer`. @@ -109,8 +115,11 @@ back to NEST's built-in serializer. We ship a separate {nuget}/NEST.JsonNetSerializer[NEST.JsonNetSerializer] package that helps in composing a custom `SourceSerializer` using `Json.NET`, that is smart enough to delegate the serialization of known NEST types back to the built-in -`RequestResponseSerializer`. This package is also useful if you want to control how your documents and values are stored -and retrieved from Elasticsearch using `Json.NET`, without interfering with the way NEST uses `Json.NET` internally. +`RequestResponseSerializer`. This package is also useful if + +. You want to control how your documents and values are stored and retrieved from Elasticsearch using `Json.NET` + +. You want to use `Newtonsoft.Json.Linq` types such as `JObject` within your documents The easiest way to hook this custom source serializer up is as follows diff --git a/docs/query-dsl.asciidoc b/docs/query-dsl.asciidoc index be9da05bd0e..a9a75ac229d 100644 --- a/docs/query-dsl.asciidoc +++ b/docs/query-dsl.asciidoc @@ -49,13 +49,13 @@ NEST exposes all of the full text queries available in Elasticsearch * <> -* <> +* <> -* <> +* <> * <> -* <> +* <> * <> @@ -71,13 +71,13 @@ include::query-dsl/full-text/common-terms/common-terms-usage.asciidoc[] include::query-dsl/full-text/intervals/intervals-usage.asciidoc[] -include::query-dsl/full-text/match-bool-prefix/match-bool-prefix-usage.asciidoc[] +include::query-dsl/full-text/match/match-usage.asciidoc[] -include::query-dsl/full-text/match-phrase-prefix/match-phrase-prefix-usage.asciidoc[] +include::query-dsl/full-text/match-bool-prefix/match-bool-prefix-usage.asciidoc[] include::query-dsl/full-text/match-phrase/match-phrase-usage.asciidoc[] -include::query-dsl/full-text/match/match-usage.asciidoc[] +include::query-dsl/full-text/match-phrase-prefix/match-phrase-prefix-usage.asciidoc[] include::query-dsl/full-text/multi-match/multi-match-usage.asciidoc[] @@ -122,14 +122,14 @@ NEST exposes all of the term queries available in Elasticsearch * <> -* <> - * <> * <> * <> +* <> + * <> See the Elasticsearch documentation on {ref_current}/term-level-queries.html[Term level queries] for more details. @@ -160,14 +160,14 @@ include::query-dsl/term-level/regexp/regexp-query-usage.asciidoc[] include::query-dsl/term-level/term/term-query-usage.asciidoc[] -include::query-dsl/term-level/terms-set/terms-set-query-usage.asciidoc[] - include::query-dsl/term-level/terms/terms-list-query-usage.asciidoc[] include::query-dsl/term-level/terms/terms-lookup-query-usage.asciidoc[] include::query-dsl/term-level/terms/terms-query-usage.asciidoc[] +include::query-dsl/term-level/terms-set/terms-set-query-usage.asciidoc[] + include::query-dsl/term-level/wildcard/wildcard-query-usage.asciidoc[] [[compound-queries]] @@ -283,10 +283,10 @@ Specialized types of queries that do not fit into other groups * <> -* <> - * <> +* <> + * <> See the Elasticsearch documentation on {ref_current}/specialized-queries.html[Specialized queries] for more details. @@ -305,10 +305,10 @@ include::query-dsl/specialized/pinned/pinned-query-usage.asciidoc[] include::query-dsl/specialized/rank-feature/rank-feature-query-usage.asciidoc[] -include::query-dsl/specialized/script-score/script-score-query-usage.asciidoc[] - include::query-dsl/specialized/script/script-query-usage.asciidoc[] +include::query-dsl/specialized/script-score/script-score-query-usage.asciidoc[] + include::query-dsl/specialized/shape/shape-query-usage.asciidoc[] [[span-queries]]