Skip to content

Update custom serialization docs #4580

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Apr 15, 2020
Merged

Update custom serialization docs #4580

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
9f10eea
Update custom serialization docs
russcam Apr 13, 2020
513b637
Update docs to utf8json
russcam Apr 14, 2020
ee7f97f
update docs
russcam Apr 15, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 33 additions & 24 deletions docs/client-concepts/high-level/serialization/custom-serialization.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,32 +15,38 @@ please modify the original csharp file found at the link and submit the PR with
[[custom-serialization]]
=== Custom Serialization

NEST 6.x ships with a _shaded_ Json.NET dependency, meaning that all of Json.NET's types are
internalized and IL merged into the NEST assembly, and their namespace has been changed
from `Newtonsoft.Json` to `Nest.Json`.

Why would we do this, you may ask? Well, NEST has always isolated its dependency on Json.NET as best as it could,
but this meant that we had to mandate how certain things were in the client. For instance,
NEST heavily relied on the fact that the `IContractResolver` used by the configured serializer was
an instance of `ElasticContractResolver`, so if you wanted to deserialize your `_source` or `_fields`
using your own resolver, you were out of luck. In addition, continued improvements to NEST's serialization pipeline
was stymied by this dependency and as client authors, we wanted to unhinder ourselves from this in order to explore the myriad
of exciting developments happening in the .NET ecosystem around performance with the likes of `Span<T>`,
`ArrayPool<T>` and a more compact UTF-8 string representation.

So what did we do in 6.x and how does it affect you?

The `NEST` nuget package from 6.0.0 onwards will use the internal Json.NET serializer and will in effect, behave the same
as it did in previous releases. If you previously relied on custom Json.NET serialization and configuration with custom
`JsonSerializerSettings` and `JsonConverter` types for example, things have changed a bit. The following section explains
how to continue working with these with NEST 6.x.
After internalizing the serialization routines, and IL-merging the Newtonsoft.Json package in 6.x, we are pleased to
announce that the next stage of serialization improvements have been completed in 7.0.

Both SimpleJson and Newtonsoft.Json have been completely removed and replaced with an implementation of Utf8Json, a fast
serializer that works directly with UTF-8 binary.

With the move to Utf8Json, we have removed some features that were available in the previous JSON libraries that have
proven too onerous to carry forward at this stage.

* JSON in the request is never indented, even if SerializationFormatting.Indented is specified. The serialization
routines generated by Utf8Json never generate an IJsonFormatter<T> that will indent JSON, for performance reasons.
We are considering options for exposing indented JSON for development and debugging purposes.

* NEST types cannot be extended by inheritance. With NEST 6.x, additional properties can be included for a type by deriving from
that type and annotating these new properties. With the current implementation of serialization with Utf8Json, this approach will not work.

* Serializer uses Reflection.Emit. Utf8Json uses Reflection.Emit to generate efficient formatters for serializing types that it sees.
Reflection.Emit is not supported on all platforms, for example, UWP, Xamarin.iOS, and Xamarin.Android.

* Elasticsearch.Net.DynamicResponse deserializes JSON arrays to List<object>. SimpleJson deserialized JSON arrays to object[],
but Utf8Json deserializes them to List<object>. This change is preferred for allocation and performance reasons.

* Utf8Json is much stricter when deserializing JSON object field names to C# POCO properties. With the internal Json.NET serializer
in 6.x, JSON object field names would attempt to be matched with C# POCO property names first by an exact match, falling back to a
case insensitive match. With Utf8Json in 7.x however, JSON object field names must match exactly the name configured for the
C# POCO property name.

[float]
==== Injecting a new serializer

Starting with NEST 6.x you can inject a serializer that is isolated to only be called
for the (de)serialization of `_source`, `_fields`, or wherever a user provided value is expected
to be written and returned.
You can inject a serializer that is isolated to only be called for the (de)serialization of `_source`, `_fields`, or
wherever a user provided value is expected to be written and returned.

Within NEST, we refer to this serializer as the `SourceSerializer`.

Expand Down Expand Up @@ -109,8 +115,11 @@ back to NEST's built-in serializer.

We ship a separate {nuget}/NEST.JsonNetSerializer[NEST.JsonNetSerializer] package that helps in composing a custom `SourceSerializer`
using `Json.NET`, that is smart enough to delegate the serialization of known NEST types back to the built-in
`RequestResponseSerializer`. This package is also useful if you want to control how your documents and values are stored
and retrieved from Elasticsearch using `Json.NET`, without interfering with the way NEST uses `Json.NET` internally.
`RequestResponseSerializer`. This package is also useful if

. You want to control how your documents and values are stored and retrieved from Elasticsearch using `Json.NET`

. You want to use `Newtonsoft.Json.Linq` types such as `JObject` within your documents

The easiest way to hook this custom source serializer up is as follows

Expand Down
28 changes: 14 additions & 14 deletions docs/query-dsl.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,13 +49,13 @@ NEST exposes all of the full text queries available in Elasticsearch

* <<intervals-usage,Intervals Usage>>

* <<match-bool-prefix-usage,Match Bool Prefix Usage>>
* <<match-usage,Match Usage>>

* <<match-phrase-prefix-usage,Match Phrase Prefix Usage>>
* <<match-bool-prefix-usage,Match Bool Prefix Usage>>

* <<match-phrase-usage,Match Phrase Usage>>

* <<match-usage,Match Usage>>
* <<match-phrase-prefix-usage,Match Phrase Prefix Usage>>

* <<multi-match-usage,Multi Match Usage>>

Expand All @@ -71,13 +71,13 @@ include::query-dsl/full-text/common-terms/common-terms-usage.asciidoc[]

include::query-dsl/full-text/intervals/intervals-usage.asciidoc[]

include::query-dsl/full-text/match-bool-prefix/match-bool-prefix-usage.asciidoc[]
include::query-dsl/full-text/match/match-usage.asciidoc[]

include::query-dsl/full-text/match-phrase-prefix/match-phrase-prefix-usage.asciidoc[]
include::query-dsl/full-text/match-bool-prefix/match-bool-prefix-usage.asciidoc[]

include::query-dsl/full-text/match-phrase/match-phrase-usage.asciidoc[]

include::query-dsl/full-text/match/match-usage.asciidoc[]
include::query-dsl/full-text/match-phrase-prefix/match-phrase-prefix-usage.asciidoc[]

include::query-dsl/full-text/multi-match/multi-match-usage.asciidoc[]

Expand Down Expand Up @@ -122,14 +122,14 @@ NEST exposes all of the term queries available in Elasticsearch

* <<term-query-usage,Term Query Usage>>

* <<terms-set-query-usage,Terms Set Query Usage>>

* <<terms-list-query-usage,Terms List Query Usage>>

* <<terms-lookup-query-usage,Terms Lookup Query Usage>>

* <<terms-query-usage,Terms Query Usage>>

* <<terms-set-query-usage,Terms Set Query Usage>>

* <<wildcard-query-usage,Wildcard Query Usage>>

See the Elasticsearch documentation on {ref_current}/term-level-queries.html[Term level queries] for more details.
Expand Down Expand Up @@ -160,14 +160,14 @@ include::query-dsl/term-level/regexp/regexp-query-usage.asciidoc[]

include::query-dsl/term-level/term/term-query-usage.asciidoc[]

include::query-dsl/term-level/terms-set/terms-set-query-usage.asciidoc[]

include::query-dsl/term-level/terms/terms-list-query-usage.asciidoc[]

include::query-dsl/term-level/terms/terms-lookup-query-usage.asciidoc[]

include::query-dsl/term-level/terms/terms-query-usage.asciidoc[]

include::query-dsl/term-level/terms-set/terms-set-query-usage.asciidoc[]

include::query-dsl/term-level/wildcard/wildcard-query-usage.asciidoc[]

[[compound-queries]]
Expand Down Expand Up @@ -283,10 +283,10 @@ Specialized types of queries that do not fit into other groups

* <<rank-feature-query-usage,Rank Feature Query Usage>>

* <<script-score-query-usage,Script Score Query Usage>>

* <<script-query-usage,Script Query Usage>>

* <<script-score-query-usage,Script Score Query Usage>>

* <<shape-query-usage,Shape Query Usage>>

See the Elasticsearch documentation on {ref_current}/specialized-queries.html[Specialized queries] for more details.
Expand All @@ -305,10 +305,10 @@ include::query-dsl/specialized/pinned/pinned-query-usage.asciidoc[]

include::query-dsl/specialized/rank-feature/rank-feature-query-usage.asciidoc[]

include::query-dsl/specialized/script-score/script-score-query-usage.asciidoc[]

include::query-dsl/specialized/script/script-query-usage.asciidoc[]

include::query-dsl/specialized/script-score/script-score-query-usage.asciidoc[]

include::query-dsl/specialized/shape/shape-query-usage.asciidoc[]

[[span-queries]]
Expand Down
50 changes: 29 additions & 21 deletions tests/Tests/ClientConcepts/HighLevel/Serialization/CustomSerialization.doc.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,34 +21,40 @@ namespace Tests.ClientConcepts.HighLevel.Serialization
/**[[custom-serialization]]
* === Custom Serialization
*
* NEST 6.x ships with a _shaded_ Json.NET dependency, meaning that all of Json.NET's types are
* internalized and IL merged into the NEST assembly, and their namespace has been changed
* from `Newtonsoft.Json` to `Nest.Json`.
* After internalizing the serialization routines, and IL-merging the Newtonsoft.Json package in 6.x, we are pleased to
* announce that the next stage of serialization improvements have been completed in 7.0.
*
* Why would we do this, you may ask? Well, NEST has always isolated its dependency on Json.NET as best as it could,
* but this meant that we had to mandate how certain things were in the client. For instance,
* NEST heavily relied on the fact that the `IContractResolver` used by the configured serializer was
* an instance of `ElasticContractResolver`, so if you wanted to deserialize your `_source` or `_fields`
* using your own resolver, you were out of luck. In addition, continued improvements to NEST's serialization pipeline
* was stymied by this dependency and as client authors, we wanted to unhinder ourselves from this in order to explore the myriad
* of exciting developments happening in the .NET ecosystem around performance with the likes of `Span<T>`,
* `ArrayPool<T>` and a more compact UTF-8 string representation.
* Both SimpleJson and Newtonsoft.Json have been completely removed and replaced with an implementation of Utf8Json, a fast
* serializer that works directly with UTF-8 binary.
*
* So what did we do in 6.x and how does it affect you?
* With the move to Utf8Json, we have removed some features that were available in the previous JSON libraries that have
* proven too onerous to carry forward at this stage.
*
* The `NEST` nuget package from 6.0.0 onwards will use the internal Json.NET serializer and will in effect, behave the same
* as it did in previous releases. If you previously relied on custom Json.NET serialization and configuration with custom
* `JsonSerializerSettings` and `JsonConverter` types for example, things have changed a bit. The following section explains
* how to continue working with these with NEST 6.x.
* - JSON in the request is never indented, even if SerializationFormatting.Indented is specified. The serialization
* routines generated by Utf8Json never generate an IJsonFormatter<T> that will indent JSON, for performance reasons.
* We are considering options for exposing indented JSON for development and debugging purposes.
*
* - NEST types cannot be extended by inheritance. With NEST 6.x, additional properties can be included for a type by deriving from
* that type and annotating these new properties. With the current implementation of serialization with Utf8Json, this approach will not work.
*
* - Serializer uses Reflection.Emit. Utf8Json uses Reflection.Emit to generate efficient formatters for serializing types that it sees.
* Reflection.Emit is not supported on all platforms, for example, UWP, Xamarin.iOS, and Xamarin.Android.
*
* - Elasticsearch.Net.DynamicResponse deserializes JSON arrays to List<object>. SimpleJson deserialized JSON arrays to object[],
* but Utf8Json deserializes them to List<object>. This change is preferred for allocation and performance reasons.
*
* - Utf8Json is much stricter when deserializing JSON object field names to C# POCO properties. With the internal Json.NET serializer
* in 6.x, JSON object field names would attempt to be matched with C# POCO property names first by an exact match, falling back to a
* case insensitive match. With Utf8Json in 7.x however, JSON object field names must match exactly the name configured for the
* C# POCO property name.
*/
public class GettingStarted
{
/**[float]
* ==== Injecting a new serializer
*
* Starting with NEST 6.x you can inject a serializer that is isolated to only be called
* for the (de)serialization of `_source`, `_fields`, or wherever a user provided value is expected
* to be written and returned.
* You can inject a serializer that is isolated to only be called for the (de)serialization of `_source`, `_fields`, or
* wherever a user provided value is expected to be written and returned.
*
* Within NEST, we refer to this serializer as the `SourceSerializer`.
*
Expand Down Expand Up @@ -114,8 +120,10 @@ public class MyPercolationDocument
*
* We ship a separate {nuget}/NEST.JsonNetSerializer[NEST.JsonNetSerializer] package that helps in composing a custom `SourceSerializer`
* using `Json.NET`, that is smart enough to delegate the serialization of known NEST types back to the built-in
* `RequestResponseSerializer`. This package is also useful if you want to control how your documents and values are stored
* and retrieved from Elasticsearch using `Json.NET`, without interfering with the way NEST uses `Json.NET` internally.
* `RequestResponseSerializer`. This package is also useful if
*
* . You want to control how your documents and values are stored and retrieved from Elasticsearch using `Json.NET`
* . You want to use `Newtonsoft.Json.Linq` types such as `JObject` within your documents
*
* The easiest way to hook this custom source serializer up is as follows
*/
Expand Down