Adds the fields API for `dense_vector` field mapper.

Adds a `DenseVector` interface for the value type.

Implemented by:
 * `KnnDenseVector` which wraps a decoded float array from `VectorValues`
 * `BinaryDenseVector` which lazily decodes a `BytesRef` from `BinaryDocValues`

The vector operations have moved into those implements from `BinaryDenseVectorScriptDocValues.java` and  `KnnDenseVectorScriptDocValues.java`, respectively.

The `DenseVector` API is:
```
float getMagnitude();
double dotProduct(float[] | List);
double l1Norm(float[] | List);
double l2Norm(float[] | List);
float[] getVector();
int dims();

boolean isEmpty(); // does the value exist
int size();        // 0 if isEmpty(), 1 otherwise
Iterator<Float> iterator()
```

`dotProduct`, `l1Norm` and `l2Norm` take a `float[]` or a `List` via the
a delegating `default` method on the `DenseVector` interface.

The `DenseVectorDocValuesField` abstract class contains two getter APIS.
It is implemented by  `KnnDenseVectorDocValuesField` and
`BinaryDenseVectorDocValuesField`.

```
DenseVector get()
DenseVector get(DenseVector defaultValue)
```

The `get()` method is included because there isn't a good default dense vector,
so that API returns an empty `DenseVector` which throws an
`IllegalArgumentException` for all method calls other than `isEmpty()`,
`size()` and `iterator()`.

The empty dense vector will always be `DenseVector.EMPTY` in case users want
to use equality checks.

Refs: elastic#79105

If rolling upgrade was used from version prior GeoIPv2 (<`7.14`) then
geoip downloader wouldn't be started so no new databases were
downloaded. This is especially troubling in `8.x` as we no longer
provide default databases inside ES so after upgrade no geoip enrichment
can take place until downloader is started with workaround (setting
`ingest.geoip.downloader.enabled` to `false` and `true` again). This is
because logic that was used to lower number of requests / cluster update
listeners at the startup was too optimistic about order of actions / who
can be elected master at what time.  This change fixes that and also
cleans up logs when there are some ignorable errors and adds debug
logging on start and stop of the task to ease up troubleshooting. It
also adds rolling upgrade test to make sure the fix works.

ES 8+ requires JDK 17 to be built

bug Introduced by elastic#83835

This switches back our token tagging to take into account the tokens position when reconstituting and tagging tokens for NER.

This PR unmutes some tests that were muted for backporting elastic#83290

This adds a warning-level deprecation if a user has set the node.attr.data setting, since it is a sign that they are
trying to create a hot/warm setup in the way that is no longer supported.
Closes elastic#83800

…lastic#83728)

This PR adds a new global privilege which can be used to restrict writes
for user profile data. The privilege is configurable for the names of
the top level keys in the profile data maps (`data` and `access`), which
by convetion are "application" names. Lastly it adds such a privilege,
for the `kibana-*` application namespace, to the `kibana_system`
built-in role.

Eg:

```
{
  "global": {
    "application": {
      "manage": {
        "applications": [...]
      }
    },
    "profile": {
      "write": {
          "applications": [...]
        }
      }
    }
}
```

Notes: * for every role there can be only one list of application names
for the write profile privilege, and the list does not support excludes
(and it supports wildcards) * there is no validation that the privilege
refers to valid application names (eg empty application name)

Removes the following methods from the SecurityClient component
of the High Level Rest Client

- putUser
- deleteUser
- changePassword
- authenticate

As part of this change, I renamed the SecurityClientTestHelper class
to TestSecurityClient and made it a real object rather than a set of
utility methods.

This was needed because different tests need different RequestOptions
objects, but passing it into every method made it cumbersome.
The code is clearer if we use a field in the test client itself. 

Relates: elastic#83423

…lastic#83814)

This commit corrects the snapshot creation and restoration docs to
describe the usage of `"none"` to restore no feature states. Previously,
they incorrectly stated that using an empty array would accomplish this,
but specifying an empty array results in the default behavior (rather
than preventing feature state snapshot/restoration).

Authentication headers are persisted as part of a task definition including ML
jobs, CCR following etc. The persistence process store them into either an
index or the cluster state. In both cases, the headers are retrieved from
ThreadContext as a string which is the serialised form of the Authentication
object. This string is always serialised with the node's version.

The problem is: In a mixed cluster, the task can be created in a newer node and
persisted into an index but then needs to be loaded by a older node. The older
node does not understand the newer format of the serialised Authentication
object and hence error out on reading it.

This PR adds additional logic in places where the headers are persisted. It
compares the Authentication version with minNodeVersion and rewrites it if the
minNodeVersion is older. Since we already filter security headers in places
where headers are persisted, the new logic is hooked into the same places and
essentially another enhancement on how to handle security headers for persisted
tasks.

Resolves: elastic#83567

We have recently seen a number of failures in ActiveDirectorySessionFactoryTests
where we fail to get a response from the Samba Server we use in
the default time frame of 5 sec. The fixture seems to be up and
running successfully so there is the case that it's just too slow
to respond within the 5 sec. This commit bumps the timeout to 15s

Adjust some of the random values that are out of bounds.

…lastic#84051)

This commit removes the exception throwing and assumes that when the area is zero, the polygon has 
the right orientation. The exception will be thrown at indexing time when the polygon is invalid .

This test was failing in rare cases when there are only few operations
and maxSeqNoOfUpdatesOrDeletes is greater then maxSeqNoOfUpdatesOrDeletesBeforeRollback

Today the `ResultDeduplicator` may complete a collection of listeners in
contexts different from the ones in which they were submitted. This
commit makes sure that the context is preserved in the listener.

rewrites checkpointing as internal actions, reducing several sub-calls to
only 1 per data node that has at least 1 primary shard of the indexes of
interest.

Robustness: The current checkpointing sends a request to every shard
 - primary and replica - and collects the results. If 1 request fails, even 
for a replica, checkpointing fails. See elastic#75780 for details.

Performance: The current checkpointing is wasteful, it uses get index 
and get index stats which results in a lot more calls and executes a 
lot more code which produces results we are not interested in.

Number of requests before and after:
before: 1 + #shards * #indices * (#replicas + 1)
after: #data_nodes_holding_gt1_shard

Fixes elastic#75780

Adds an API that can be used to find out how much memory ML
is permitted to use and is currently using on each node, both
within the JVM heap, and natively, outside of the JVM.

…3457)

We have two implementations of XContentParser that both delegate all of its methods to a delegate, either an inner parser provided at construction (FilterXContentParser) or a more dynamic variant that is returned by overriding the delegate method (DelegatingXContentParser).

Effectively the two classes do exactly the same, the only difference being how the delegate parser is provided. While these two are two separate implementations, they could inherit from each other.

With this change we make FilterXContentParser be the previous DelegatingXContentParser, that allows to override the delegate method, and we introduce a new FilterXContentParserWrapper that takes the fixed delegate as a constructor argument.

Additionally, XContentSubParser is rewritten to extend FilterXContentParserWrapper.

Closes elastic#84036

In a large cluster we expect most nodes not to have a copy of most
shards, but today during replica shard allocation we create a new (and
nontrivial) object for each node that has no copy of a shard. With this
commit we check at deserialization time whether the response is empty
and, if so, avoid the unnecessary instantiation.

Relates elastic#77466

In elastic#84038 we added a dependency on having a valid `threadContext` in a
repository, but some tests use mocking and may end up with a `null`
here. This seems not to be a problem in recent branches but causes
failures in 8.0. With this commit we ensure that we always have a valid
`threadContext` to avoid any problems.

Moving to Lucene 9 slightly changed the base url for the analysis components
which this PR fixes.

…lastic#84096)

This reverts commit a9cdbf4.

The role name change does not play well with API key creation.

The current `ignore_unavailable` definition is a bit misleading. The parameter primarily determines if a request that targets a missing or closed index returns an error.

…tic#83843)

A regular data stream can be migrated to a tsdb data stream if in template that created the data stream, the `index_mode` field is set to `time_series` and the data stream's `index_mode` property is either not specified or set to `standard`. Then on the next rollover the data stream is migrated to be a tsdb data stream.

When that happens the data stream's `index_mode` property is set to `time_series` and the new backing index's `index.mode` index setting is also set to `time_series`.

Closes elastic#83520

…#84053)

The bulk API response with a `200 OK` HTTP status always returns an entry for each action in the request. Partial responses aren't applicable.

…_dimension (elastic#83920)

At the moment we really don't know what configuring a
`time_series_dimension` should *do* when there are nested documents.
So, for now, we're going to disable it. One day when someone has a good
idea of how it should work we can build that. But for now we don't want
to guess wrong and then lock us into some annoying behavior that no one
needs but we have to support for backwards compatibility reasons.

Closes: elastic#83915

This removes the defaults from the slow log for the remaining queries in
the `parent-join` module. So it should be easier to read the slow log
when it contains these queries.

Relates to elastic#76515

This commit utilizes the index mapping hash to share the fields-caps for 
indices with the same index mapping to reduce the memory usage and the
size of transport messages.

Closes elastic#78665
Closes elastic#82879

…tic#84025)

Clarifies that the `orientation` mapping parameter only applies to WKT polygons. GeoJSON polygons use a default orientation of `RIGHT`, regardless of the mapping parameter.

Also notes that the document-level `orientation` parameter overrides the default orientation for both WKT and GeoJSON polygons.

Closes elastic#84009.

…ic#83918)

This PR addressed the behavior described in elastic#83917, in which Feature migrations
which have encountered errors are not re-run in some cases. As of this PR, Features
which have encountered errors during migration are treated the same as Features 
requiring migration.

This PR also adds a test which artificially replicates elastic#83917.

Replaces some funny code formatting created when we applied spotless to
the code globally. Spotless does a fine job, but this was a little funky
looking.

…GeoShapeCellValues

…earchableSnapshot

* Add a note that the http_ca.crt certificate that is generated and
stored in config/certs can be used to configure any client to trust
the certificate that elasticsearch uses for TLS on the HTTP layer
* Add a note that the elasticsearch-create-enrollment-token CLI
tool can only be used with auto-configured TLS settings.

* Add Maps.newLinkedHashMapWithExpectedSize
* Return LinkedHashMap
* Don't cast LinkedHashMap

…hase (elastic#84184)

…lastic#84200)

This test would fail if we introduce the network partition while the
master is still publishing a cluster state update and hasn't received
the ack from the victim node. In this case the default publish timeout
means that the master will wait for 30s before completing the stalled
publication and moving on to the `node-left` one, but
`ensureStableCluster` also times out after 30s which leaves not much
time for the master to remove the victim node.

This commit reduces the publish timeout to 10s so that the master
recovers well before `ensureStableCluster` times out.

Closes elastic#84172

Lucene issues that resulted in elasticsearch changes:

LUCENE-9820 Separate logic for reading the BKD index from logic to intersecting it.
LUCENE-10377: Replace 'sortPos' with 'enableSkipping' in SortField.getComparator()
LUCENE-10301: make the test-framework a proper module by moving all test
classes to org.apache.lucene.tests
LUCENE-10300: rewrite how resources are read in ukrainian morfologik analyzer:
LUCENE-10054 Make HnswGraph hierarchical

Relates to PR#83667

The cluster allocation explain API includes a top-level status
indicating to the user whether the shard can be assigned/rebalanced/etc
or not. Today this status is fairly terse and experience shows that
users sometimes struggle to understand how to interpret it and to decide
on follow-up actions.

This commit makes the top-level explanation more detailed and
actionable. For instance, in the cases like `THROTTLED` where the status
is transient we instruct the user to wait; if a shard is lost we say to
restore it from a snapshot; if a shard cannot be assigned we say to
choose a specific node where its assignment is expected and to address
the obstacles.

Co-authored-by: James Rodewig <james.rodewig@elastic.co>

We have found that ContentPath.pathAsText() ends up being a very hot path
in indexing, and that it is specifically called a lot when checking that the current
path does not lead to a metadata mapper. The metadata mappers that accept
input only exist at the root, however, so instead of building the full path we can
instead check that we are on the root mapper and then just pass the current
field to the lookup.

…84209)

Adds a dedicated section for doc-value-only fields to the docs that can be linked to.

This commit updates readMap to use helper method that presizes map
correctly. It guarantees no resizing when adding given numbers of
entries to a hashmap / linkedhashmap.

Using the generic pool for these all the activities surrounding cluster coordination
seems broken. They all get blocked on the same mutex in the `Coordinator` eventually
and run effectively sequentially.
Particularly in clusters with larger node counts this could lead to situations where
lots of generic threads are needlessly spun up only for the purpose of waiting on the mutex.
Since we at times also lock on the mutex in the coordinator on transport threads, it is
particularly unforuntate when there's lots of generic threads waiting on it.

=> fix this by using a single threaded pool for coordination work.

Relates to elastic#76515

The `MasterService` executes batches of tasks which compute changes to
the `ClusterState`. After executing each batch the `MasterService`
publishes the updated cluster state and may notify tasks in the batch
when the nodes have acked the state (or failed, or timed out). Many
tasks compute some kind of result during their execution which needs to
be made available to the ack completion handler for subsequent
activities.

Today there's no good general way to pass anything to the ack completion
handler other than the fact that the update was acked or not. Some tasks
work around this by storing their result in the `ClusterState` itself.
Others use the executor to capture the result and pass it through.
Neither solution works well with batching: later tasks in a batch may
overwrite the part of the `ClusterState` containing the results of
earlier tasks, and batching executors are re-used across batches.

This commit adjusts the `ClusterStateTaskExecutor` interface so that now
implementations that wish to listen for acks must supply a listener for
each task they successfully execute. The `MasterService` collects the
listeners for the batch and notifies them as acks are received. This
gives the executor control over the ack handler of each task which lets
it pass in any extra data needed.

Effectively this is the same as elastic#83562 but for ack listeners instead of
publish listeners.

Removes the extra "allow_bwc_indices" flag on the repository to access older snapshots.

Make it clearer that security configuration happens on installation
and not on first start of Elasticsearch.

…ers (elastic#84236)

Because of the way Filebeat parses CSV files the text structure finder
needs to generate a regular expression that will ignore the header row
of the CSV file.

It does this by concatenating the column names separated by the delimiter
with optional quoting. However, if there are hundreds of columns this can
lead to a very long regular expression, potentially one that cannot be
evaluated by some programming languages.

This change limits the length of the regular expression to 1000 characters
by only including elements for the first few columns when there are many.
Matching 1000 characters of header should be sufficient to reliably
identify the header row even when it is much longer. It is extremely
unlikely that there would be a data row where the first 1000 characters
exactly matched the header but then subsequent fields diverged.

Fixes elastic#83434

In elastic#83290 we added an assertion that Elasticsearch returns the product
header in every REST response. Unfortunately this isn't always the case,
we found bugs in a couple of released versions and fixed them in elastic#84038
and elastic#84089. With this commit we skip the new assertion in the
known-buggy versions.

Closes elastic#84036 again.

… (elastic#84234)

Security audit ignore filter policies work on the audit event's action field.

Co-authored-by: Johannes Mahne <johannes.mahne@elastic.co>

…tic#84244)

A bug introduced in elastic#72423 could cause initial scaling from 0->1 to be much too high for the given tier.

This only occurs if the autoscaling capacity API was called numerous times in a row while there were no machine learning nodes, but autoscaling was enabled.

…tic#84192)

This PR properly registers `NamedXContentRegistry` entries for `SystemIndexMigrationTaskParams` and `SystemIndexMigrationTaskState`. It also adds tests for the XContent de/serialization for those classes, 
and fixes a bug revealed by these tests in `SystemIndexMigrationTaskState`'s parser.

Finally, it adds an integration test which simulates the conditions in which elastic#84115 occurs: A node restart 
while the migration is in progress. This ensures that we have fixed that particular bug.

Co-authored-by: David Turner <david.turner@elastic.co>
Co-authored-by: Gordon Brown <gordon.brown@elastic.co>

We want more information on the write latencies. Lets trace log
the timing on uploading data blobs which involve almost no CPU during
uploading to gather some data.

Co-authored-by: James Rodewig <james.rodewig@elastic.co>
Co-authored-by: Ioannis Kakavas <ikakavas@protonmail.com>

Ensures that we have enough documents in each shard to trigger the cancellation
check. Fixes the issue with uneven document distribution due to bad routing and
increases number of documents to be higher than CHECK_CANCELLED_SCORER_INTERVAL.

Fixes elastic#84095

This flag has already been removed by ES on Oct 21, 2019 after they created their own allocator: elastic@547e399
However, in parallel, on Oct 28, 2019 all command line flags were moved from jvm.options file into the launcher, and this option has been re-introduced: elastic@c4fbda3

See original commit from Oct 21 for the reasoning on why this flag is not needed anymore.

This PR introduces the lookup runtime fields which are used to retrieve 
data from the related indices. The below search request enriches its
search hits with the location of each IP address from the `ip_location`
index.

```
POST logs/_search
{
  "runtime_mappings": {
    "location": {
      "type": "lookup",
      "lookup_index": "ip_location",
      "query_type": "term",
      "query_input_field": "ip",
      "query_target_field": "_id",
      "fetch_fields": [
        "country",
        "city"
      ]
    }
  },
  "fields": [
    "timestamp",
    "message",
    "location"
  ]
}
```

Response:

```
{
  "hits": {
    "hits": [
      {
        "_index": "logs",
        "_id": "1",
        "fields": {
          "location": [
            {
              "city": [ "Montreal" ],
              "country": [ "Canada" ]
            }
          ],
          "message": [ "the first message" ]
        }
      }
    ]
  }
}
```

The test is failing now because in elastic#84051, we are not failing in polygons when we can compute the signed 
area and the error has been moved to the indexer.

…tic#84194)

If a remote cluster has the skip_unavailable option set and is
unavailable during a datafeed search then we should retry rather
than risk using incomplete search results. Using incomplete search
results could lead to spurious anomalies being found that do not
make sense once the remote cluster is available once again.

Fixes elastic#83838

Due to elastic#84268

…ic#84183)

This commit adds readMapFromList to StreamInput. This method allows to
efficiently read list from the stream and create a presized map from it
without having to allocate intermediate collection.

…entations where possible (elastic#81681)

- Rework task implementations to avoid project usage at execution time
- In general usages of getProject() within a task should be avoided as it is not compatible
with gradle configuration cache. Related to elastic#57918

…stic#84242)

When an NLP model is deployed, or a DFA/Anomaly job is assigned, we have historically relied only on the xpack.ml.max_lazy_ml_nodes to determine if scaling is possible. But, in certain scenarios, it may be that scaling is available when xpack.ml.max_lazy_ml_nodes is fully satisfied.

xpack.ml.max_ml_node_size is now checked to see if the current ML nodes exceed this size. If not, we assume vertical scaling is possible and allow the tasks to be created.

closes elastic#84198

…n mixed node environments (elastic#83833)

Resolves elastic#83581
Resolves elastic#83788

SQL REST requests using `format=txt` stand out from the other formats because the cursor needs to carry the formatter state from the initial request to subsequent scroll requests. The state is needed to be able to format the subsequent pages with column widths that match the widths in the first page.

Currently, this is solved by wrapping cursor objects together with the formatter state in `TextFormatterCursor`. Hence, a query might return a `ListCursor` and `TextFormat.PLAIN_TEXT` adds the necessary state when formatting the output. This approach is handy because a `TextFormatterCursor` is a `Cursor` and delegates calls to the wrapped cursor. Unfortunately, it also has some downsides that have been revealed when looking into elastic#83581 and elastic#83788:

- MediaType formatting is a concern of the REST layer of the plugin that does not have access to the serialization logic usually accessible through the `NamedWriteablesRegistry`. Because the formatting required to (de)serialize the cursors for the wrapping, this meant that the `Cursor` deserialization could only use a subset of `NamedWriteable`s specific to Cursors (namely the ones returned by `Cursors.getNamedWriteables`). This subset of writeables is not enough to deserialize `CompositeAggCursor` which lead to the surprising design that `CompositeAggCursor` has a `nextQuery` member consisting of the serialized `SearchSourceBuilder` whose deserialization is suspended until the call to `Cursor.nextPage`.
- `TextFormat.PLAIN_TEXT` had to deserialize the complete cursor to do it's work, not just the information relevant to the formatter. Because of the `min_compatible_version` redirects that occur during rolling upgrades, it can happen though that an upgraded node needs to deserialize a cursor from a redirected SQL query written by an older node. Because we do not offer any form of bwc for cursors, this causes an error and leads to elastic#83581.

In this PR, I propose to change the design such that the REST layer can add state to the cursor strings without having to read or write instances of `Cursor`. This allows to address the issues described above and also made fixing elastic#83788 very straightforward.

Attaching state to the cursor is achieved by extending the format of encoded cursors. Previously, an base64 encoded SQL cursor had the following wire format `<version><zoneId><cursor>`. Now, there are two different cursor types:

* `<version><zoneId><CursorType.NO_STATE><cursor>` for most cases
* `<version><ZoneOffset.UTC><CursorType.WITH_STATE><formatterState><base64encodedCursor>` for txt format cursors. Note, `<ZoneOffset.UTC>` could be any zone id because consumers will use the one encoded in `<base64encodedCursor>`.

This serialization format ensures that every version of ES can read and check the cursor version and produce an according error in case of a mismatch.

…ade (elastic#84284)

…yword (elastic#83943)

Resolves elastic#81939 (part of
elastic#70192)

Deprecates the `index_include_frozen` REST parameter in the `/_sql`
endpoint and all the syntax including the `FROZEN` keyword.

When running in CI multiple test suites run in parallel which causes
significant resource contention for heavy tests like the one that tests
datafeeds with CCS.

Increasing the time to wait for the datafeed to complete should avoid
spurious failures.

Fixes elastic#84290

This commit adds an explanation for the relation between `allow_partial_search_results` and `skip_unavailable` in CCS requests.

Relates to elastic#33915

Closes elastic#82407

Co-authored-by: James Rodewig <james.rodewig@elastic.co>

(cherry picked from commit 297b69e)

# Conflicts:
#	docs/reference/release-notes/8.1.asciidoc

Co-authored-by: Ievgen Degtiarenko <ievgen.degtiarenko@elastic.co>

elastic/kibana#101216 adds a new ECS mapper feature to the Ingest Pipelines UI. This updates the ES docs to cover the new feature.

The current upgrade docs contain some language that's only applicable to 8.0. This updates the docs so they're usable in other 8.x branches.

Hopefully fixes elastic#84204. When we build an expect script in the packaging
tests, we insert `expect eof` after supplying the password. This makes
expect wait until the output of ES is closed, which isn't actaully what
we want, as we then go on to expect on a number of other outputs. So,
remove this `eof`, and let the other expectations do their work.

Update docs for v8.0.1-BC2 release

(cherry picked from commit 7809c8e)

# Conflicts:
#	docs/reference/release-notes.asciidoc

Co-authored-by: Salvatore Campagna <93581129+salvatore-campagna@users.noreply.github.com>

…4296)

Under rare circumstances the 10 documents the test indexes might land on only
one shard, resulting in a different output for the validation query than
expected. By adding fixed doc ids we can avoid this and also make the test more
reproducable.

Closes elastic#84218

This is a forward-port of elastic#82487, elastic#83544, elastic#83601, elastic#84145, and elastic#84246, but given that the branches had diverged so much they were not a straightforward cherry-picks. It required modifying the interface of the NodeDeprecationChecks to include ClusterState as we do in 7.x.