Field [elastiknn_dense_float_vector] is defined twice.

[zeppelinen]

Hi!

First of all thank you for great job.

I fail to create an index with two elastiknn_dense_float_vector fields. I get the following error:

{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Field [elastiknn_dense_float_vector] is defined twice."}],"type":"illegal_argument_exception","reason":"Field [elastiknn_dense_float_vector] is defined twice."},"status":400}

Index with single elastiknn_dense_float_vector field is created with no problems.

Relevant piece of my index mappings:

      "name_vector" : {
        "type" : "elastiknn_dense_float_vector",
        "elastiknn": {
          "dims": 512,
          "model": "lsh",
          "similarity": "l2",
          "L": 99,
          "k": 1,
          "w": 3
        }
      },
      "summary_vector": {
        "type": "elastiknn_dense_float_vector",
        "elastiknn": {
          "dims": 512,
          "model": "lsh",
          "similarity": "l2",
          "L": 99,
          "k": 1,
          "w": 3
        }
      }

[alexklibisz]

@zeppelinen Coincidentally I also ran into a closely related issue when updating to es 7.9.2. I'll keep you posted. I should have some to look into it more this weekend and early next week.

[alexklibisz]

@zeppelinen Which version are you using? I was trying to reproduce the issue but was actually able to make an index with the following mapping:

{
  "issue-176" : {
    "mappings" : {
      "properties" : {
        "df0" : {
          "type" : "elastiknn_dense_float_vector",
          "similarity" : "boolean",
          "elastiknn" : {
            "dims" : 10
          }
        },
        "df1" : {
          "type" : "elastiknn_dense_float_vector",
          "similarity" : "boolean",
          "elastiknn" : {
            "model" : "lsh",
            "similarity" : "angular",
            "dims" : 10,
            "L" : 10,
            "k" : 1
          }
        },
        "df2" : {
          "type" : "elastiknn_dense_float_vector",
          "similarity" : "boolean",
          "elastiknn" : {
            "model" : "lsh",
            "similarity" : "angular",
            "dims" : 10,
            "L" : 10,
            "k" : 1
          }
        },
        "df3" : {
          "type" : "elastiknn_dense_float_vector",
          "similarity" : "boolean",
          "elastiknn" : {
            "model" : "lsh",
            "similarity" : "l2",
            "dims" : 10,
            "L" : 21,
            "k" : 2,
            "w" : 3
          }
        },
        "df4" : {
          "type" : "elastiknn_dense_float_vector",
          "similarity" : "boolean",
          "elastiknn" : {
            "model" : "permutation_lsh",
            "dims" : 10,
            "k" : 22,
            "repeating" : false
          }
        },
        "id" : {
          "type" : "keyword",
          "store" : true
        },
        "sb0" : {
          "type" : "elastiknn_sparse_bool_vector",
          "similarity" : "boolean",
          "elastiknn" : {
            "dims" : 10
          }
        },
        "sb1" : {
          "type" : "elastiknn_sparse_bool_vector",
          "similarity" : "boolean",
          "elastiknn" : {
            "model" : "sparse_indexed",
            "dims" : 10
          }
        },
        "sb2" : {
          "type" : "elastiknn_sparse_bool_vector",
          "similarity" : "boolean",
          "elastiknn" : {
            "model" : "lsh",
            "similarity" : "jaccard",
            "dims" : 10,
            "L" : 10,
            "k" : 2
          }
        },
        "sb3" : {
          "type" : "elastiknn_sparse_bool_vector",
          "similarity" : "boolean",
          "elastiknn" : {
            "model" : "lsh",
            "similarity" : "jaccard",
            "dims" : 10,
            "L" : 10,
            "k" : 2
          }
        },
        "sb4" : {
          "type" : "elastiknn_sparse_bool_vector",
          "similarity" : "boolean",
          "elastiknn" : {
            "model" : "lsh",
            "similarity" : "hamming",
            "dims" : 10,
            "L" : 10,
            "k" : 3
          }
        }
      }
    }
  }
}

[zeppelinen]

Interesting. I was using 7.6.2 with plugin compiled from sources using openjdk-13. And it looks like I could mess up branches, as I was testing builds against different elasticsearch versions. I check environment and update you later.

…

On Fri, Oct 23, 2020, 11:12 PM Alex Klibisz ***@***.***> wrote: @zeppelinen <https://github.com/zeppelinen> Which version are you using? I was trying to reproduce the issue but was actually able to make an index with the following mapping: { "issue-176" : { "mappings" : { "properties" : { "df0" : { "type" : "elastiknn_dense_float_vector", "similarity" : "boolean", "elastiknn" : { "dims" : 10 } }, "df1" : { "type" : "elastiknn_dense_float_vector", "similarity" : "boolean", "elastiknn" : { "model" : "lsh", "similarity" : "angular", "dims" : 10, "L" : 10, "k" : 1 } }, "df2" : { "type" : "elastiknn_dense_float_vector", "similarity" : "boolean", "elastiknn" : { "model" : "lsh", "similarity" : "angular", "dims" : 10, "L" : 10, "k" : 1 } }, "df3" : { "type" : "elastiknn_dense_float_vector", "similarity" : "boolean", "elastiknn" : { "model" : "lsh", "similarity" : "l2", "dims" : 10, "L" : 21, "k" : 2, "w" : 3 } }, "df4" : { "type" : "elastiknn_dense_float_vector", "similarity" : "boolean", "elastiknn" : { "model" : "permutation_lsh", "dims" : 10, "k" : 22, "repeating" : false } }, "id" : { "type" : "keyword", "store" : true }, "sb0" : { "type" : "elastiknn_sparse_bool_vector", "similarity" : "boolean", "elastiknn" : { "dims" : 10 } }, "sb1" : { "type" : "elastiknn_sparse_bool_vector", "similarity" : "boolean", "elastiknn" : { "model" : "sparse_indexed", "dims" : 10 } }, "sb2" : { "type" : "elastiknn_sparse_bool_vector", "similarity" : "boolean", "elastiknn" : { "model" : "lsh", "similarity" : "jaccard", "dims" : 10, "L" : 10, "k" : 2 } }, "sb3" : { "type" : "elastiknn_sparse_bool_vector", "similarity" : "boolean", "elastiknn" : { "model" : "lsh", "similarity" : "jaccard", "dims" : 10, "L" : 10, "k" : 2 } }, "sb4" : { "type" : "elastiknn_sparse_bool_vector", "similarity" : "boolean", "elastiknn" : { "model" : "lsh", "similarity" : "hamming", "dims" : 10, "L" : 10, "k" : 3 } } } } } } — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#176 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AG4UK5KF4K4WWS5VVCBTX23SMHWS7ANCNFSM4S4PACNA> .

[alexklibisz]

I added a test to make sure it's working in #178 . The test creates a mapping like the one above, indexes 100 docs with this mapping, runs a count request to count the number of fields with each field, makes sure the result for each request is 100, then runs a query targeting each of the vector fields and makes sure it returns some neighbors. It's possible there's still some edge case that I hadn't thought of.

[zeppelinen]

I tested same configuration again using plugin from official project releases page and it works just fine.
Looks like my environment is broken. I'd blame adoptopenjdk docker container used for build.
Thank you @alexklibisz for your great work. I continue testing, hope will be back with more useful feedback.