Ensure Elasticsearch documents have an _id and track content hash for partial updates

[Mpdreamz]

This uses Url as our _id

This allows us to do direct GET request based on urls.

GET /semantic-docs-dev/_doc/%2Fdocs%2Freference%2Fintegrations%2Fsonicwall_firewall

And we store a hash of the contents.

This allows us to conditionally update a document only if the hash has changed

POST /my-index/_update/1
{
  "scripted_upsert": true,
  "script": {
    "source": """
      if (ctx.op != 'create') {
        if (ctx._source.hash == params.hash ) {
            ctx.op = "noop"
        }
        else {
            ctx._source = params.doc
        }
      }
    """,
    "params": {
      "hash": "SOME-HASH",
      "doc": {
        "hash": "SOME-HASH",
        "semantic": "TEXT"
      }
    }
  }
}

However because we use semantic fields the equivalent is not allowed in _bulk operations

POST /semantic-docs-dev/_bulk
{"update":{"_id":"1"}}
{ "scripted_upsert": true, "script": { "source": "if (ctx.op != 'create') { if (ctx._source.hash == params.hash ) { ctx.op = 'noop' } else { ctx._source = params.doc } }", "params": { "hash": "SOME-HASH", "doc": { "hash": "SOME-HASH-2", "semantic": "DIFFERENT TEXT" } } } }

See https://www.elastic.co/docs/reference/elasticsearch/mapping-reference/semantic-text#semantic-text-updates for the rules that prevent it currently. Opened elastic/elasticsearch#136074 to discuss with Elasticsearch team.

The only option i see now is to first index into an index without semantic fields then search for updates through scroll and feed it to bulk index updates into the index with semantic fields.

Side note

The mapping for url is updated to use the path hierarchy tokenizer so its easier for us to constrain searches to specific locations e.g

GET /semantic-docs-dev/_search
{
    "query": {
        "term": {
          "url.prefix": {
            "value": "/docs/reference/integrations"
          }
        }
    }
}