Having some troubles... and documentation seems inaccurate to current version (v1.7.0)

[thunderstumpges]

First let me preface this with the comment that I really like the potential of this library, and love the fluent interface approach. I am sorry if I sound frustrated, but with Elasticsearch being a steep learning curve anyway, it is tough if the documentation for the library is incomplete or inaccurate. I can't say for sure if I am running into bugs or just undocumented "features" :)

I am using documentation provided at http://nest.azurewebsites.net/

Starting at the beginning. I need to create a mapping.

1. Starting with attribute mapping on my type, the documentation shows many ElasticType attribute properties, but on mine I only see two: Name and important one for me IdProperty which I'm guessing (and hoping) is to tell Nest what property should be mapped to the _id for the type. It is not shown nor discussed in the documentation.
2. Documentation for Map<T>() shows an empty method signature:

var response = this.ConnectedClient.Map<ElasticSearchProject>();

However in code, it takes a Func<PutMappingDescriptor<T>,PutMappingDescriptor<T>>

While I have seen references in other places in the documentation of use of this with m => m.MapFromAttributes() I'm not 100% certain what this does (though I guessed that it should map my type from the attributes I specified.

I have (I believe) successfully created my mapping (though I don't see any reference to the id field when I get the mapping back using a raw REST call)

1. Attempting to use the BulkAsync call was failing for me using almost exactly the code example in the Bulk documentation. Documentation shows:

var descriptor = new BulkDescriptor();

foreach (var i in Enumerable.Range(0, 1000))
{
    descriptor.Index<ElasticSearchProject>(op => op
        .Document(new ElasticSearchProject {Id = i})
    );
}
var result = client.Bulk(descriptor);

With that code, and my attribute annotated type, the result from the Bulk call were validation errors on every operation saying "missing index, missing type". SO I then found undocumented... an option for "Index" and "Type" on the descriptor.Index call:

bulk.Index<MyType>(op => op.Index("myindex").Type("mytype").Document(objMyType));

This didn't help :(

Only when I finally tried the also undocumented FixedPath option on the BulkDescriptor object did it actually specify the index and type in the request (and I'm guessing it would have to be the same index and type for all documents in the BulkDescriptor request..?

bulk.FixedPath("myindex","mytype");

1. After finally getting a bunch of documents indexed, I notice it is NOT using my id field as specified in the ElasticType(IdProperty="id") attribute. Instead I get internally generated IDs for all of my documents.

Am I just missing some more documentation somewhere? Is there a bug with the code? I can't tell for sure. Thanks in advance.

[thunderstumpges]

Well to follow up a little more, I think I have indeed run into at least one bug with the Bulk API. I have attempted to explicitly set an Id field, and the request still doesn't specify anything in the index action json (I was able to locate the request sent in debugger)

bulk.Index<GeoLocation>(op => op
   .Document(location)
   .Id(location.id)
   .Index("geo")
   .Type("geolocation"));

As mentioned earlier, if I don't use "FixedPath" I get the errors about "missing index, missing type".
Here is the request generated from above code:

`http://es:9200/geolocation/geolocation/_bulk,
Request:

{ "index" :  {} }
{"id":"kec.1234","location":{"lat":41.75979,"lon":69.38798},...}

If however I call directly the Index method, I DO get back the proper id, and it generates the proper request (inferring the index, type, and id all correctly)

var res = elasticClient.Index(location);

results in:
Url: http://es:9200/geolocation/geolocation/kec.1234
Request:

{
  "id": "kec.1234",  "location": {    "lat": 41.75979,    "lon": 69.38798  },...
}

[Mpdreamz]

Hey @thunderstumpges

I can only echo your frustration with the out of date documentation, we are taking a radical different approach for NEST 2.0 combining unit/integration tests and roslyn to create something we call literate tests to drive our documentation. This means that any code in the documentation will be code thats always compiled and run on each commit.

Our .Map() method translates to the elasticsearch Put Mapping API which is used to update an existing index's mapping. You can only introduce new fields and not update existing mappings. Its usually a better approach to describe the mapping when you create the index using the CreateIndex() api.

var x = this.Client.CreateIndex(index, s => s
    .AddMapping<ElasticsearchProject>(m => m
        .MapFromAttributes()
        .Properties(pp=>pp
            .String(pps=>pps.Name(p=>p.Country).Index(FieldIndexOption.Analyzed))
        )
    )
);

Here we create index and add a mapping for ElasticsearchProject and explicitly tell it to infer the mapping from the CLR properties and associated attributes using MapFromAttributes() using .Properties() we can override what it infers with a more explicit mapping for the Country property.

On your bulk problem, this seems strange and would love to get to the bottom of whats going on. Our unit test that test .Document() on an unbounded bulk request (e.g not calling Fixed*) still translates properly to an operation with an inferred index, type and id, see here

[Mpdreamz]

I have some thoughts on what goes on, have a look at the tests trying to reproduce this:

https://github.com/elastic/elasticsearch-net/blob/develop/src/Tests/Nest.Tests.Unit/Reproduce/Reproduce1528Tests.cs#L26

When no default index is set on index settings the bulk throws an NRE which is bad.

But when it is the json that is generated: https://github.com/elastic/elasticsearch-net/blob/develop/src/Tests/Nest.Tests.Unit/Reproduce/Issue1528.json does include index and type and id.

[Mpdreamz]

Still trying to capture the behavior you are seeing of completely empty bulk operation headers:

{ "index" :  {} }

Can you share a bit more how your POCO looks like?

[thunderstumpges]

Thanks for the quick turn-around. Your tests look just about like my setup. I am setting the default index on connection settings, and it is the index for this operation. A couple of differences I see are that I am not specifying ElasticProperty.Name but rather letting the inference determine it. Also I have a significantly more complicated structure with a couple of Nested objects. Guessing the difference must be somewhere in there.

Here is my complete poco. The fields are organized and named this way as it is the format of an existing json data source, so while I can change the attributes, I cannot change the structure or property names.

    // ReSharper disable InconsistentNaming
    [ElasticType(IdProperty = "id")]
    public class GeoLocation
    {
        [ElasticProperty(Type = FieldType.Nested)]
        public GeoLocationCountry country { get; set; }

        [ElasticProperty(Index = FieldIndexOption.NotAnalyzed)]
        public string id { get; set; }

        [ElasticProperty(Type = FieldType.GeoPoint)]
        public GeoLocationLatLon location { get; set; }

        [ElasticProperty(Type = FieldType.Nested)]
        public List<GeoLocationName> names { get; set; }

        public string type { get; set; }

        public class GeoLocationCountry
        {
            [ElasticProperty(Index = FieldIndexOption.NotAnalyzed)]
            public string code { get; set; }
            [ElasticProperty(Index = FieldIndexOption.NotAnalyzed)]
            public string id { get; set; }

            public string name { get; set; }
        }

        public class GeoLocationLatLon
        {
            public double lat { get; set; }
            public double lon { get; set; }
        }

        public class GeoLocationName
        {
            [ElasticProperty(Index = FieldIndexOption.NotAnalyzed)]
            public string language { get; set; }

            public string name { get; set; }

            [ElasticProperty(Index = FieldIndexOption.NotAnalyzed)]
            public string script { get; set; }
        }
    }
    // ReSharper restore InconsistentNaming

[thunderstumpges]

Our .Map() method translates to the elasticsearch Put Mapping API which is used to update an existing index's mapping. You can only introduce new fields and not update existing mappings. Its usually a better approach to describe the mapping when you create the index using the CreateIndex() api.

So, I used the .Map<T>(...) to create my mapping, and even tested several times deleting the entire index and letting my code recreate it. Are you saying I should not do that? The intent was to create it if it did not exist, and update it should there be an additional field down the road.

Another thought, is that I don't get any metadata in my mapping definition when echoing it back form elasticsearch. According to the book I'm reading (Elasticsearch: The Definitive Guide chapter 10), after putting the mapping (especially after explicitly setting the id "Path")

// update the mapping.
var mapRes = await elasticClient.MapAsync<GeoLocation>(m => m.MapFromAttributes().IdField(md=>md.Path("id")));

I should have an _id metadata field: (excerpt below is from the book example)

PUT /my_index
{
    "mappings": {
        "my_type": {
            "_id": {
                "path": "doc_id" 1
            },
            "properties": {
                "doc_id": {
                    "type":   "string",
                    "index":  "not_analyzed"
}  }  }  }}

[thunderstumpges]

Thanks for setting up that unit test. I have cloned the repo, built the test, and I will work on updating it to reproduce the issue I have. Likely won't get around to it again until Monday morning though.

[thunderstumpges]

Umm... so actually two of the four tests in that file fail on tip. Both of the "Explicit" tests fail for me:

Issue1528_ExplicitTypeAndIndex Failed
  { "index" :  {"_index":"myindex","_type":"sometype","_id":"my_id_0"} }
  Expected: True
  But was:  False

I think the assertion json file is incorrect for the "explicit" methods, it still has 'default-index':

{ "index" :  {"_index":"default-index", "_type":"mytype","_id":"my_id_0"} }
{"name":"my_id_0"}

I am modifying the json file locally to continue testing.

[Mpdreamz]

So, I used the .Map(...) to create my mapping, and even tested several times deleting the entire
index and letting my code recreate it. Are you saying I should not do that? The intent was to create it if
it did not exist, and update it should there be an additional field down the road.

This is fine as long as you are aware of its append only nature 👍 Changing properties won't be updated. I personally typically go down the road of reindexing into a time based index e.g myindex-[timestamp] and using aliases to point myindex to the latest version. If your dataset is too big for this pattern and your domain is not volatile put mapping with its caveats can work just as fine for you :)

This is related to the id path behaviour you've witnessed, given a completely empty index with no mappings in place.

var mapRes = await client.MapAsync<GeoLocation>(m => m.MapFromAttributes());
var mapRes2 = await client.MapAsync<GeoLocation>(m => m.MapFromAttributes().IdField(md=>md.Path("id")));

The call to IdField() will only succeed if its done as part of mapRes as part of mapRes2 it will fail since a mapping is already in place for GeoLocation.

side note: id field paths support has been completely removed from Elasticsearch 2.0.

[Mpdreamz]

On the topic of the weird bulk output, your POCO does not seem weird at all, will try to make the tests fail too later today.

Umm... so actually two of the four tests in that file fail on tip. Both of the "Explicit" tests fail for me:

argh I thought I pushed those up before running out Sunday, sorry for that noise. Just pushed them back up.

[thunderstumpges]

Ok, I think I figured it out. Based on documentation on connecting it said we could use SetJsonSerializerSettingsModifier to:

update the internal Json.NET serializer settings to your liking

I used this to change the ContractResolver to our own internal contract resolver which globally ignores some property names (from Thrift and Avro serialilzation of objects). I didn't really need it in this scenario, but looking at the code now that I have it, you guys have a very important ContractResolver, and it seems that we should really not change it out. (I may have to inherit another one from yours to ignore our properties sometime later, but not needed for this particular case).

After removing the setting of the ContractResolver, everything worked great.

Maybe good to mention the pitfalls of changing different things in SetJsonSerializerSettingsModifier. what things are actually alright to change? Our internal ContractResolver is actually inherited from CamelCasePropertyNamesContractResolver... any chance we could use something like this for Pocos that don't use camel-case property names? Is that already handled by the ElasticContractResolver ?

By the way, thanks so much for the very quick and thorough follow up. I appreciate the time taken to explain reasoning and caveats to different things. Everything is working great now!

[Mpdreamz]

Thank you @thunderstumpges for not rage-quiting 😄 your feedback is very very valuable.

You are right if you set your own ContractResolver through SetJsonSerializerSettingsModifier it needs to inherit from ElasticContractResolver, i will make it throw a hard exception if it does not!

If all you want to do is change how property names are handled globally you can also modify this by calling:

DefaultPropertyNameInferrer(p=>p.ToLowerInvariant()) on ConnectionSettings if you want to leave the property names verbatim call DefaultPropertyNameInferrer(p=>p)