Golang struct to elastic mapping

[yuva29]

Is there any way to convert a golang struct to elastic mapping probably using some custom tags or something?

[olivere]

There is no support for that in Elastic, but you could of course do that with something like github.com/fatih/structs.

However, I have doubts about this approach. The problem is that mapping can get rather complex, and you need to put all that subtleties into a struct field. Therefore, I typically take a different approach where I prefill all the different types of fields into a dynamic mapping/template, and use suffixes to make ES automatically pick up the correct type. Let me know if you need more information about that approach.

[kashid16]

Any solution for this? @olivere Will you please elaborate this - 'dynamic mapping/template'

[olivere]

@kashid16 I don't see a necessity for doing this. In all but the very basic mappings, the Go structs for the results are typically very different to the mapping. Your mapping is typically rather complex, uses one field with different types, different analyzers, is configured to copy the content to some other field etc. In most of my use cases there's not even a 1:1 mapping between the field in the result set and the field in the mapping. ES is not a database.

It's rather convenient to simply pass the mapping into the raw body, maybe configured by a Go template, like illustrated here.

[ablegao]

有什么解决办法吗？@olivere请您详细说明一下 - '动态映射/模板'

func EsStructToMapping(in interface{}) interface{} {

	// "settings": map[string]interface{}{
	// 	"refresh_interval":   "2s",
	// 	"number_of_shards":   1,
	// 	"number_of_replicas": 1,
	// },
	mappings := map[string]map[string]interface{}{}
	types := reflect.TypeOf(in).Elem()
	fieldLen := types.NumField()
	for i := 0; i < fieldLen; i++ {
		es_tag := types.Field(i).Tag.Get("es")
		if es_tag == "-" {
			continue
		}
		estags := strings.Split(es_tag, ",")
		name := types.Field(i).Tag.Get("json")
		names := strings.Split(name, ",")
		name = names[0]
		mappings[name] = map[string]interface{}{}
		for _, v := range estags {
			field := strings.Split(v, ":")
			switch field[1] {
			case "true":
				mappings[name][field[0]] = true
			case "false":
				mappings[name][field[0]] = false
			default:
				mappings[name][field[0]] = field[1]
			}
		}
	}
	// outmap["mappings"] = map[string]interface{}{
	// 	"properties": mappings,
	// }
	return mappings
}

type Demo struct {
	APIV       string    `json:"apiv" es:"type:keyword,index:true"`
	ReqType    string    `json:"log_type" es:"type:keyword,index:true"`
	PullId     string    `json:"pull_id" es:"type:keyword,index:true"`
	ReqId      string    `json:"s_req_id" es:"type:keyword,index:true"`
	CReqId     string    `json:"reqid"  es:"type:keyword,index:True"`
	SlotID     int       `json:"slot_id" es:"type:long,index:true"`
	AndroidID  string    `json:"android_id" es:"type:keyword,index:true"`
	Gaid       string    `json:"gaid" es:"type:keyword,index:true"`
	Ifv        string    `json:"ifv" es:"type:keyword,index:true"`
}

[olivere]

@ablegao Reflection-based mapping breaks down as soon as you don't have a simple 1:1 mapping. E.g. how do you map a field that is just a tiny bit more complex, like:

"name": {
  "type": "text",
  "fields": {
    "keyword": {
      "type": "keyword",
      "ignore_above": 100
    }
  }
}

I still think the best way today is to have a "handwritten" mapping that uses Go templates, like I used here.

On top of that, please see how to specify a list of re-usable fields with dynamic_templates to make your indices reusable. Dynamic templates with index patterns will help you a lot in managing your indices over time.

[YaroslavPodorvanov]

Solution https://github.com/YaroslavPodorvanov/golang-struct-to-elastic-mapping