Adding best_compression #49974

PR elastic#50333 has been merged.

DFRSimilarity -> DFR similarity

The Analysis docs mention including a default analyzer in the index settings. However, no example snippet is included. This adds an example snippet that users can easily copy and adjust.

Currently, if an updateable synonym filter is included in a multiplexer filter, it is not reloaded via the _reload_search_analyzers because the multiplexer itself doesn't pass on the analysis mode of the filters it contains, so its not recognized as "updateable" in itself. Instead we can check and merge the AnalysisMode settings of all filters in the multiplexer and use the resulting mode (e.g. search-time only) for the multiplexer itself, thus making any synonym filters contained in it reloadable. This, of course, will also make the analyzers using the multiplexer be usable at search-time only. Closes elastic#50554

testFailureToAndFromXContentWithDetails had incorporated that guessRootCauses returned the wrapper when the underlying exception was not an ElasticsearchException, fixed. Relates elastic#50525

Related to elastic#50353

Signed-off-by: Mark Vieira <portugee@gmail.com>

Adds a 'text analysis overview' page to the analysis topic docs. The goals of this page are: * Concisely summarize the analysis process while avoiding in-depth concepts, tutorials, or API examples * Explain why analysis is important, largely through highlighting problems with full-text searches missing analysis * Highlight how analysis can be used to improve search results

This commits makes the "init" ILM step retryable. It also adds a test where an index is created with a non-parsable index name and then fails. Related to elastic#48183

Currently we reuse the same test connection for all connection attempts in the testConcurrentConnectsAndDisconnects test. This means that if the connection fails due to a pre-existing connection, the connection will be closed impacting the state of all connection attempts. This commit fixes the test, by returning a unique connection for each attempt. Fixes elastic#49903.

* SQL: Optimisation fixes for conjunction merges This commit fixes the following issues around the way comparisions are merged with ranges in conjunctions: * the decision to include the equality of the lower limit is corrected; * the selection of the upper limit is corrected to use the upper bound of the range; * the list of terms in the conjunction is sorted to have the ranges at the bottom; this allows subsequent binary comarisions to find compatible ranges and potentially be merged away. The end guarantee being that the optimisation takes place irrespective of the order of the conjunction terms in the statement. Some comments are also corrected. * adress review observation on anon. comparator Replace anonymous comparator of split AND Expressions with a lambda.

Signed-off-by: Mark Vieira <portugee@gmail.com>

Previously, the following situation would throw an error: * A search contains a `collapse` on a particular field. * The search spans multiple indices, and in one index the field is mapped as a concrete field, but in another it is a field alias. The error occurs when we attempt to merge `CollapseTopFieldDocs` across shards. When merging, we validate that the name of the collapse field is the same across shards. But the name has already been resolved to the concrete field name, so it will be different on shards where the field was mapped as an alias vs. shards where it was a concrete field. This PR updates the collapse field name in `CollapseTopFieldDocs` to the original requested field, so that it will always be consistent across shards. Note that in elastic#32648, we already made a fix around collapsing on field aliases. However, we didn't test this specific scenario where the field was mapped as an alias in only one of the indices being searched.

This commit removes the UBI-based Docker images as we are not using this work for now.

Signed-off-by: Mark Vieira <portugee@gmail.com>

A very large number of recursive calls can cause a stack overflow exception. This commit forks the recursive calls for non-async processors. Once forked, each thread will handle at most 10 recursive calls to help keep the stack size and thread count down to a reasonable size.

Follow-up to elastic#50722, which was also backported to 7.5 and 7.6.

This change add new ILM action to wait for SLM policy execution to ensure that index has snapshot before deletion. Closes elastic#45067

This test replaces the watch index after watcher got started. This triggers watches being reloaded and while this happens the trigger engine is paused, which disallows watches from being triggered. At this time there are no watches in the .watches index and I think this is just unlucky timing. Reloading of watches happens in the background and the watch state can be started when that happens. For normal schedule trigger engines this is not an issue, because watches that are meant to be triggered are triggered when the engine triggers the next time. However for the mock scheduled trigger engine this is different, because watches are triggered programatically and there is no retry in this test. I think just adding `timeWarp().trigger("mywatch");` inside a `assertBusy(...)`` is the right fix here. If it fails because the mock schedule trigger engine is paused then the test will try again. In the mean time the the watches can be reloaded, which then resumes the mock scheduled trigger engine. Closes elastic#50658

Co-Authored-By: Lisa Cawley <lcawley@elastic.co>

It's impossible to tell why elastic#50754 fails without this change. We're failing to close the `exchange` somewhere and there is no write timeout in the GCS SDK (something to look into separately) only a read timeout on the socket so if we're failing on an assertion without reading the full request body (at least into the read-buffer) we're locking up waiting forever on `write0`. This change ensure the `exchange` is closed in the tests where we could lock up on a write and logs the failure so we can find out what broke elastic#50754.

…emoves notes in GET job stats API docs. (elastic#50728)

… API docs (elastic#50732) Co-Authored-By: Lisa Cawley <lcawley@elastic.co>

Co-authored-by: Lisa Cawley <lcawley@elastic.co>

We are only using the snapshot tool with JDK8 and it only works until and excluding 7.4.0. There is no point in continuing to have it in `master`. We should maintain it in `7.x` if it ever needs any changes but its unused and just dead code in `master`.

Co-authored-by: Lisa Cawley <lcawley@elastic.co>

This one is failing a lot => mute Relates elastic#50781

This adds the necessary named XContent classes to the HLRC for the lang ident model. This is so the HLRC can call `GET _ml/inference/lang_ident_model_1?include_definition=true` without XContent parsing errors. The constructors are package private as since this classes are used exclusively within the pre-packaged model (and require the specific weights, etc. to be of any use).

) This PR introduces test infrastructure for downgrading a cluster while interacting with a given repository. It fixes the fact that repository metadata in the new format could be written while there's still older snapshots in the repository that require the old-format metadata to be restorable.

If a pipeline referenced by a transform does not exist, we should not allow the transform to be created. We do allow the pipeline existence check to be skipped with defer_validations, but if the pipeline still does not exist on `_start`, the pipeline will fail to start. relates: elastic#50135

…50799)

This commit removes the type parameter from `CreateIndexRequest.mapping(type, object...)`, and the associated delegating method on `CreateIndexRequestBuilder`. To make migration simpler, the method on `CreateIndexRequest` is renamed to `simpleMapping`, and on `CreateIndexRequestBuilder` to `setMapping`; this should help the compiler catch all necessary changes on upgrades. Relates to elastic#41059

This replaces the hand rolled parsing code for scripted metric with `ObjectParser` which is simpler to work with because it is declarative.

Replaces the "funny" `Function<String, ConstructingObjectParser<T, Void>>` with a much simpler `ConstructingObjectParser<T, String>`. This makes pretty much all of our object parsers static.

Today we periodically check the indexing buffer memory every 5 seconds or after we have used 1/30 of the configured memory. If the total used memory is over the threshold, then we refresh the "largest" shards. If refreshing takes longer these intervals (i.e., 5s or 1/30 buffer), then we continue to enqueue refreshes to these shards. This leads to two issues: - The refresh thread pool can be exhausted and other shards can't refresh - Execute too many refreshes for the "largest" shards With this change, we only refresh the largest shards if they are not refreshing. Here we rely on the periodic check to trigger another refresh if needed. We can harden this by making the ongoing refresh triggers the memory check when it's completed. I opted out this option in this PR for simplicity. See: https://discuss.elastic.co/t/write-queue-continue-to-rise/213652/

…ic#50824)

@jrodewig

* [DOCS] Move snapshot-restore docs out of modules. * [DOCS] Incorporates comments from @jrodewig. * [DOCS] Fix snippet tests

The `template` field was deprecated in 6.0 in favor of `index_patterns`, and can now be removed. Relates to elastic#21009.

* Fix Snapshot Shard Status Request Deduplication The request deduplication didn't actually work for these requests since they had no `equals` and `hashCode` so the deduplicator wouldn't actually recognize equal requests.

elastic#50842) Muting the test while I work on fixing its flakiness. Relates elastic#50781

This commit removes validation logic of source and dest indices for data frame analytics and replaces it with using the common `SourceDestValidator` class which is already used by transforms. This way the validations and their messages become consistent while we reduce code. This means that where these validations fail the error messages will be slightly different for data frame analytics.

Extract reusable functionality from SQL into its own dedicated project QL. Implemented as a plugin, it provides common components across SQL and the upcoming EQL. While this commit is fairly large, for the most part it's just a big file move from sql package to the newly introduced ql.

We need the same fix we did in `7.x` (elastic#50797) and only get snapshot status for the current version or older. Otherwise these tests break for e.g.`7.0.1` due to the same index metadata incompatibility. Closes elastic#50819

The packed-refs support was using the original .git path, changed to use the real .git directory after reference from worktree has been followed. Relates elastic#47464

One of the example snippets in the range query docs was missing a required 'T' in the `date` format. This adds the required 'T'.

…elastic#50844) Mappings built by this method should all be wrapped with _doc, so there's no need to pass the type any more. This also renames the method to simpleMapping, in line with CreateIndexRequest, to help migration by causing compilation errors; and changes the signature to take a String... rather than an Object.... Relates to elastic#41059

…lastic#50858)

The test checked queue size and active count, however, ThreadPoolExecutor pulls out the request from the queue before marking the worker active, risking that we think all tasks are done when they are not. Now check on completed-tasks metric instead, which is guaranteed to be monotonic. Relates elastic#50769

This change fixes the upgrade of index metadata that contain a custom similarity with options that are not compatible with BM25. The upgrade doesn't need a real similarity service so we fake one that resolves all custom similarity to BM25 but this logic fails because the BM25 provider checks that all options are compatible. This commit removes the verification step as it is not needed during the upgrade (the verification is done when the index is restored/opened). Closes elastic#50763

When I implemented elastic#50609 I suppressed a few backwards compatibility checks until I finished the backport. I've now finished the backport so this enables the tests.

It can be lowered to 7.5.99 now that the functionality was backported in elastic#48972.

When a composite aggregation is reduced using the results from an index that has one of the fields unmapped we were throwing away the formatter. This is mildly annoying, except in the case of IP addresses which were coming out as non-utf-8-characters. And tripping assertions. This carefully preserves the formatter from the working bucket. Closes elastic#50600

* Fix SLM check for restore in progress This commit fixes the check in SLM where the `RestoreInProgress` metadata was checked for existence. Rather than check existence we should instead check the `isEmpty` method. Prior to this, a successful restore for a repository that used SLM retention would prevent SLM retention from running in subsequent invocations, due to SLM thinking that a restore was still running.

…c#49451) Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

…0881)

Signed-off-by: Mark Vieira <portugee@gmail.com>

This adds the `PUT` API for creating trained models that support our format. This includes * HLRC change for the API * API creation * Validations of model format and call

Today we make multiple attempts to corrupt the translog header in `TranslogHeaderTests#testCurrentHeaderVersion`, but if we are extraordinarily unlucky then this sequence of corruptions may restore the file to its original state. This change adjusts the test to only corrupt the file once, which is certain not to leave the file in its original state.

The Document Level Security BitSet cache stores a secondary "lookup map" so that it can determine which cache entries to invalidate when a Lucene index is closed (merged, etc). There was a memory leak because this secondary map was not cleared when entries were naturally evicted from the cache (due to size/ttl limits). This has been solved by adding a cache removal listener and processing those removal events asyncronously.

When creating a role, we do not check if the exceptions for the field permissions are a subset of granted fields. If such a role is assigned to a user then that user's authentication fails for this reason. We added a check to validate role query in elastic#46275 and on the same lines, this commit adds check if the exceptions for the field permissions is a subset of granted fields when parsing the index privileges from the role descriptor. Co-authored-by: Yogesh Gaikwad <bizybot@users.noreply.github.com>

The Document Level Security BitSet Cache (see elastic#43669) had a default configuration of "small size, long lifetime". However, this is not a very useful default as the cache is most valuable for BitSets that take a long time to construct, which is (generally speaking) the same ones that operate over a large number of documents and contain many bytes. This commit changes the cache to be "large size, short lifetime" so that it can hold bitsets representing billions of documents, but releases memory quickly. The new defaults are 10% of heap, and 2 hours. This also adds some logging when a single BitSet exceeds the size of the cache and when the cache is full. Resolves: elastic#49260

The enterprise license type must has "max_resource_units" and may not have "max_nodes". This change adds support for this new field, validation that the field is present if-and-only-if the license is enterprise and bumps the license version number to reflect the new field.

Hide the `.async-search-*` in Security by making it a restricted index namespace. The namespace is hard-coded. To grant privileges on restricted indices, one must explicitly toggle the `allow_restricted_indices` flag in the indices permission in the role definition. As is the case with any other index, if a certain user lacks all permissions for an index, that index is effectively nonexistent for that user.

Relates a620525

…astic#50890) The system created and models we provide now use the `_xpack` user for uniformity with our other features The `PUT` action is now an admin cluster action And XPackClient class now references the action instance.

…lastic#50853) Follow up to elastic#50692 that starts writing a `min_version` field to the `RepositoryData` so that pre-7.6 ES versions can not read it (and potentially corrupt it if they attempt to modify the repo contents) after the repository moved to the new metadata format.

This adds a new "http" sub-command to the certutil CLI tool. The http command generates certificates/CSRs for use on the http interface of an elasticsearch node/cluster. It is designed to be a guided tool that provides explanations and sugestions for each of the configuration options. The generated zip file output includes extensive "readme" documentation and sample configuration files for core Elastic products.

Today we split the on-disk cluster metadata across many files: one file for the metadata of each index, plus one file for the global metadata and another for the manifest. Most metadata updates only touch a few of these files, but some must write them all. If a node holds a large number of indices then it's possible its disks are not fast enough to process a complete metadata update before timing out. In severe cases affecting master-eligible nodes this can prevent an election from succeeding. This commit uses Lucene as a metadata storage for the cluster state, and is a squashed version of the following PRs that were targeting a feature branch: * Introduce Lucene-based metadata persistence (elastic#48733) This commit introduces `LucenePersistedState` which master-eligible nodes can use to persist the cluster metadata in a Lucene index rather than in many separate files. Relates elastic#48701 * Remove per-index metadata without assigned shards (elastic#49234) Today on master-eligible nodes we maintain per-index metadata files for every index. However, we also keep this metadata in the `LucenePersistedState`, and only use the per-index metadata files for importing dangling indices. However there is no point in importing a dangling index without any shard data, so we do not need to maintain these extra files any more. This commit removes per-index metadata files from nodes which do not hold any shards of those indices. Relates elastic#48701 * Use Lucene exclusively for metadata storage (elastic#50144) This moves metadata persistence to Lucene for all node types. It also reenables BWC and adds an interoperability layer for upgrades from prior versions. This commit disables a number of tests related to dangling indices and command-line tools. Those will be addressed in follow-ups. Relates elastic#48701 * Add command-line tool support for Lucene-based metadata storage (elastic#50179) Adds command-line tool support (unsafe-bootstrap, detach-cluster, repurpose, & shard commands) for the Lucene-based metadata storage. Relates elastic#48701 * Use single directory for metadata (elastic#50639) Earlier PRs for elastic#48701 introduced a separate directory for the cluster state. This is not needed though, and introduces an additional unnecessary cognitive burden to the users. Co-Authored-By: David Turner <david.turner@elastic.co> * Add async dangling indices support (elastic#50642) Adds support for writing out dangling indices in an asynchronous way. Also provides an option to avoid writing out dangling indices at all. Relates elastic#48701 * Fold node metadata into new node storage (elastic#50741) Moves node metadata to uses the new storage mechanism (see elastic#48701) as the authoritative source. * Write CS asynchronously on data-only nodes (elastic#50782) Writes cluster states out asynchronously on data-only nodes. The main reason for writing out the cluster state at all is so that the data-only nodes can snap into a cluster, that they can do a bit of bootstrap validation and so that the shard recovery tools work. Cluster states that are written asynchronously have their voting configuration adapted to a non existing configuration so that these nodes cannot mistakenly become master even if their node role is changed back and forth. Relates elastic#48701 * Remove persistent cluster settings tool (elastic#50694) Adds the elasticsearch-node remove-settings tool to remove persistent settings from the on disk cluster state in case where it contains incompatible settings that prevent the cluster from forming. Relates elastic#48701 * Make cluster state writer resilient to disk issues (elastic#50805) Adds handling to make the cluster state writer resilient to disk issues. Relates to elastic#48701 * Omit writing global metadata if no change (elastic#50901) Uses the same optimization for the new cluster state storage layer as the old one, writing global metadata only when changed. Avoids writing out the global metadata if none of the persistent fields changed. Speeds up server:integTest by ~10%. Relates elastic#48701 * DanglingIndicesIT should ensure node removed first (elastic#50896) These tests occasionally failed because the deletion was submitted before the restarting node was removed from the cluster, causing the deletion not to be fully acked. This commit fixes this by checking the restarting node has been removed from the cluster. Co-authored-by: David Turner <david.turner@elastic.co>

…50780)

Password changes are only allowed when the user is currently authenticated by a realm (that permits the password to be changed) and not when authenticated by a bearer token or an API key.

This commit improves the performance of warning value extraction in the low-level REST client, and is similar to the approach taken in elastic#24114. There are some differences since the low-level REST client might be connected to Elasticsearch through a proxy that injects its own warnings.

Currently, the connection manager is configured with a default profile for both the sniff and proxy connection stratgies. This profile correctly reflects the expected number of connection (6 for sniff, 18 for proxy). This commit removes the proxy strategy usages of the per connection attempt profile configuration. Additionally, it refactors other unnecessary code around the connection manager. The connection manager now can always be built inside the remote connection.

Signed-off-by: Mark Vieira <portugee@gmail.com>

This commit adds retries for windows cleanup after tests, which may fail due to file locks not being immediately released after a windows process exits. closes elastic#50825

This commit moves the packaging tests for elasticsearch-certgen to java from bats. Although certgen is deprecated, the tests are moved rather than just deleted, and the tests themselves should be easily adaptable to certutil. One note is that the test is simplified to use a single node, rather than the two node test from bats, which was problematic given how the newer distro tests only operate with a single distribution. relates elastic#46005

…tic#50820) * Refresh cached phase policy definition if possible on new policy There are some cases when updating a policy does not change the structure in a significant way. In these cases, we can reread the policy definition for any indices using the updated policy. This commit adds this refreshing to the `TransportPutLifecycleAction` to allow this. It allows us to do things like change the configuration values for a particular step, even when on that step (for example, changing the rollover criteria while on the `check-rollover-ready` step). There are more cases where the phase definition can be reread that just the ones checked here (for example, removing an action that has already been passed), and those will be added in subsequent work. Relates to elastic#48431

Currently, proxy mode allows a remote cluster connection to be setup by expecting all open connections to be routed through an intermediate proxy. The proxy must use some logic to ensure that the connections end up on the correct remote cluster. One mechanism provided is that the default distribution TLS implementations will forward the host component of the configured address to the remote connection using the SNI extension. This is limiting as it requires that the proxy be configured in a way that always uses a valid hostname as the proxy address. Instead, this commit adds an additional setting to allow the server_name to be configured independently. This allows the proxy address to be specified as a IP literal, but the server_name specified as an arbitrary string which still must be a valid hostname. It also decouples the server_name from the requirement of being a DNS resolvable domain.

…sform (elastic#50777)

… and testWaitForSnapshotSlmExecutedBefore test (elastic#50818) * Fix flaky TimeSeriesLifecycleActionsIT#testWaitForSnapshot test This change adds some randomness and cleanup step to TimeSeriesLifecycleActionsIT#testWaitForSnapshot and testWaitForSnapshotSlmExecutedBefore tests in attempt to make them stable. Reletes to elastic#50781 * Formatting changes * Longer timeout

A normal flush has the same effect as a synced flush on Elasticsearch 7.6 or later. It's deprecated in 7.6 and will be removed in 8.0. Relates elastic#50776

With retention leases, users do not need to set index.soft_deletes.retention.operations. This change removes it from the requirements of CCR

Previously custom realms were limited in what services and components they had easy access to. It was possible to work around this because a security extension is packaged within a Plugin, so there were ways to store this components in static/SetOnce variables and access them from the realm, but those techniques were fragile, undocumented and difficult to discover. This change includes key services as an argument to most of the methods on SecurityExtension so that custom realm / role provider authors can have easy access to them. Resolves: elastic#48369

…#46456) This change adds a new `kibana_admin` role, and deprecates the old `kibana_user` and`kibana_dashboard_only_user`roles. The deprecation is implemented via a new reserved metadata attribute, which can be consumed from the API and also triggers deprecation logging when used (by a user authenticating to Elasticsearch). Some docs have been updated to avoid references to these deprecated roles. Co-authored-by: Tim Vernum <tim@adjective.org> Co-authored-by: Larry Gregory <legrego@users.noreply.github.com>

…0965) Relates elastic#50964

Relates to elastic#50611 and elastic#50612

…stic#50918)

…ic#50903) uses remote client(s) to correctly retrieve index checkpoints from remote clusters

…assi… (elastic#50914) Adds a new parameter to regression and classification that enables computation of importance for the top most important features. The computation of the importance is based on SHAP (SHapley Additive exPlanations) method.

Has the new cluster state storage layer emit warnings in case metadata performance is very slow. Relates elastic#48701

The logic in AsyncLucenePersistedState was flawed, unexpectedly queuing up two update tasks in parallel.

Generally speaking, deprecated analysis components in elasticsearch will issue deprecation warnings when they are first used. However, this means that no warnings are emitted when indexes are created with deprecated components, and users have to actually index a document to see warnings. This makes it much harder to see these warnings and act on them at appropriate times. This is worse in the case where components throw exceptions on upgrade. In this case, users will not be aware of a problem until a document is indexed, instead of at index creation time. This commit adds a new check that pushes an empty string through all user-defined analyzers and normalizers when an IndexAnalyzers object is built for each index; deprecation warnings and exceptions are now emitted when indexes are created or opened. Fixes elastic#42349

Until elastic#50914 is backported to 7.x

@hackaholic

Updates a snippet to use the `s` query string parameter rather than piping the output to a separate `sort` command. This ensures the snippet is tested and available in clients other than curl (Kibana console, etc.). Issue was originally raised by @hackaholic in elastic#40926.

There is a JVM bug causing `Thread#suspend` calls to randomly take multiple seconds breaking these tests that call the method numerous times in a loop. Increasing the timeout would will not work since we may call `suspend` tens if not hundreds of times and even a small number of them experiencing the blocking will lead to multiple minutes of waiting. This PR detects the specific issue by timing the `Thread#suspend` calls and skips the remainder of the test if it timed out because of the JVM bug. Closes elastic#50047

We added a new rounding in elastic#50609 that handles offsets to the start and end of the rounding so that we could support `offset` in the `composite` aggregation. This starts moving `date_histogram` to that new offset.

…stic#50802) In classes where the client is used directly rather than through a call to executeAsyncWithOrigin explicitly require the client to be OriginSettingClient rather than using the Client interface. Also remove calls to deprecated ClientHelper.clientWithOrigin() method.

We seem to have settled on the `ContextParser` interface for parsing stuff, mostly because `ObjectParser` implements it. We don't really *need* the old `Aggregator.Parser` interface any more because it duplicates `ContextParser` but with the arguments reversed. This adds support to `AggregationSpec` to declare aggregation parsers using `ContextParser`. This should integrate cleanly with `ObjectParser`. It doesn't drop support for `Aggregator.Parser` or change the plugin intrface at all so it *should* be safe to backport to 7.x. And we can remove `Aggregator.Parser` in a follow up which is only targeted to 8.0.

) This commit begins the process of removing types from the document parsing infrastructure. Initially, we just ignore the user-supplied type after it has been removed from the mapping json structure, and always supply _doc as the name of the root parser. The production code change is very small here, and most of the changeset consists of alterations to Mapper test code that was passing in non-standard type names and checking serialization. Relates to elastic#41059

…50914 (elastic#50979)

Check it out: ``` $ curl -u elastic:password -HContent-Type:application/json -XPOST localhost:9200/test/_update/foo?pretty -d'{ "dac": {} }' { "error" : { "root_cause" : [ { "type" : "x_content_parse_exception", "reason" : "[2:3] [UpdateRequest] unknown field [dac] did you mean [doc]?" } ], "type" : "x_content_parse_exception", "reason" : "[2:3] [UpdateRequest] unknown field [dac] did you mean [doc]?" }, "status" : 400 } ``` The tricky thing about implementing this is that x-content doesn't depend on Lucene. So this works by creating an extension point for the error message using SPI. Elasticsearch's server module provides the "spell checking" implementation.

…8917)" (elastic#50945) This reverts commit 77a8bc3.

Add tracking of snapshot versions to RepositoryData to make BwC logic more efficient. Follow up to elastic#50853

…ic#50862) We already deprecated and removed the camel-case versions of the nGram and edgeNGram filters a while ago and we should do the same with the nGram and edgeNGram tokenizers. This PR deprecates the use of these names in favour of ngram and edge_ngram in 7 and disallows usage in new indices starting with 8. Closes elastic#50561

…c#50002) It is fairly common to filter the geo point candidates in geohash_grid and geotile_grid aggregations according to some viewable bounding box. This change introduces the option of specifying this filter directly in the tiling aggregation. This is even more relevant to `geo_shape` where the bounds will restrict the shape to be within the bounds this optional `bounds` parameter is parsed in an equivalent fashion to the bounds specified in the geo_bounding_box query.

…stic#50874) * [ML][Inference] Adding classification_weights to ensemble models classification_weights are a way to allow models to prefer specific classification results over others this might be advantageous if classification value probabilities are a known quantity and can improve model error rates.

* Extend the optimizations for equalities This commit supplements the optimisations of equalities in conjunctions and disjunctions: * for conjunctions, the existing optimizations with ranges are extended with not-equalities and inequalities; these lead to a fast resolution, the conjunction either being evaluate to a FALSE, or the non-equality conditions being dropped as superfluous; * optimisations for disjunctions are added to be applied against ranges, inequalities and not-equalities; these lead to disjunction either becoming TRUE or the equality being dropped, either as superfluous or merged into a range/inequality. * Adress review notes * Fix the bug around wrongly optimizing 'a=2 OR a!=?', which only yields TRUE for same values in equality and inequality. * Var renamings, code style adjustments, comments corrections. * Address further review comments. Extend optim. - fix a few code comments; - extend the Equals OR NotEquals optimitsation (a=2 OR a!=5 -> a!=5); - extend the Equals OR Range optimisation on limits equality (a=2 OR 2<=a<5 -> 2<=a<5); - in case an equality is being removed in a conjunction, the rest of possible optimisations to test is now skipped. * rename one var for better legiblity - s/rmEqual/removeEquals

This commit allows the plugin installer to install multiple plugins in a single invocation. The installation will be treated as a transaction, so that all of the plugins are install successfully, or none of the plugins are installed.

Adds a command-line tool to remove broken custom metadata from the cluster state. Relates to elastic#48701

Bad interaction with other recently merged PR.

Signed-off-by: Mark Vieira <portugee@gmail.com>

…fter elastic#50996 (elastic#50997) after elastic#50996 (backport of elastic#50002) merged, the version guard on geo-grid `bounds` parameter can be updated to 7.6 re-enables bwc tests

The existing package private visibility of the serialization ctor isn't enough when copying example code to oder packages or projects when bootstrapping a new project from it.

When installing multiple plugins at once, this commit changes the behavior to report installed plugins as we go. In the case of failure, we emit a message that we are rolling back any plugins that were installed successfully, and also that they were successfully rolled back. In the case a plugin is not successfully rolled back, we report this clearly too, alerting the user that there might still be state on disk they would have to clean up.

…stic#50509) Documents the `breakers`, `script`, and `discovery` parameters returned by the `_nodes/stats` API.

This commit migrates the simple test of the cron eval tool from bats to java packaging tests. relates elastic#46005

When building IntelliJ generates several source files related to the benchmarks. This commit adds that path to the gitignore so these don't get accidentally committed.

The chown utility for packaging tests works on windows when the given path is a directory, but would fail if the path was a single file. This commit fixes it to handle both cases. relates elastic#50825

Now that we've backported elastic#50938 to 7.x it should be safe to run its test against BWC clusters that include that branch.

…maly detection docs (elastic#50959) Co-Authored-By: David Roberts <dave.roberts@elastic.co>

* Fix generics usages. * Sealed match processor class.

There have been occasional failures, presumably due to too many tests running in parallel, caused by jobs taking around 15 seconds to open. (You can see the job open successfully during the cleanup phase shortly after the failure of the test in these cases.) This change increases the wait time from 10 seconds to 20 seconds to reduce the risk of this happening.

As the test classes are spread across more than one project, the Gradle classpath contains not just folders but also jars. This commit allows the test class to explore the archive content and load matching classes from said source.

Literal constructor makes an implicit conversion for each value given which turns out has some subtle side-effects. Improve MathProcessors to preserve numeric type where possible Fix bug on issue compatibility between date and intervals Preserve the source when folding inside the Optimizer

The current formatting config allows some long generic declarations to break the 140 character limit. Tweak the config to wrap such lines.

Opt-in buildSrc for automatic formatting. This required a config tweak in order to pick up all the Java sources, and as a result more files are now found in the Enrich plugin, that were previously missed. Following on from the formatting, improve some code's layout, most notably by replacing a number of String concatentations with String.format().

Ensures that there are not too many concurrent dynamic mapping updates going out from the data nodes to the master. Closes elastic#50670

This commit adds the name of the current pipeline to ingest metadata. This pipeline name is accessible under the following key: '_ingest.pipeline'. Example usage in pipeline: PUT /_ingest/pipeline/2 { "processors": [ { "set": { "field": "pipeline_name", "value": "{{_ingest.pipeline}}" } } ] } Closes elastic#42106

Removed superfluous `=`.

It's possible that the index could return no settings and thus throw a `NullPointerException`. I wasn't able to reproduce the original issue, but this should guard against in the future. Resolves elastic#50646

These injected actions are harmless and safe to ignore for non-CCR indices. Resolves elastic#50548

…51031) Knowing about used analysis components and mapping types would be incredibly useful in order to know which ones may be deprecated or should get more love. Some field types also act as a proxy to know about feature usage of some APIs like the `percolator` or `completion` fields types for percolation and the completion suggester, respectively.

…tic#51055) This test failed a couple of different ways, related to timing, as well as concurrent snapshots, and also naming. This commit splits the giant `assertBusy` into separate parts so that we don't perform ~5 different requests and tests in the same loop. It also gives each test a unique repository so that no other test can accidentally re-use snapshots. Resolves elastic#50358 (hopefully!)

…1016) In these tests there was a very small chance that keys could collide, which causes test failures. Resolves elastic#49307

Fix NPE in PinnedQuery call to DisjunctionMaxScorer. Added test and fix that tests for score type. Closes elastic#51034

This moves the testing of custom significance heuristic plugins from an `ESIntegTestCase` to an example plugin. This is *much* more "real" and can be used as an example for anyone that needs to actually build such a plugin. The old test had testing concerns and the example all jumbled together.

Adds a note to the 'Community Contributed Clients' page explicitly stating that Elastic does not support or endorse the clients.

Replace DES with AES to align with modern encryption standards Resolves: elastic#50843

Fixes HttpCertificateCommandTests.testTextFileSubstitutions on Windows Resolves: elastic#50964

Also adds `assertWatchCount(0)` snippet inside assertBusy(...) based on the findings in: elastic#32299 (comment) Relates to elastic#32299

Relates to elastic#30777

…c#51056)

) Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

The PreConfiguredTokenFilter#singletonWithVersion uses the version internaly for the token filter factories but it registers only one instance in the cahce and not one instance per version. This can lead to exceptions like the one described in elastic#50734 since the singleton is created and cached using the version created of the first index that is processed. Remove the singletonWithVersion() methods and use the elasticsearchVersion() methods instead. Fixes: elastic#50734

The short alias for `unassigned.for` is `uf`, not 'ua'.

* Fix Infinite Retry Loop in loading RepositoryData We were running into an infinite loop when trying to load corrupted (or otherwise un-loadable) repository data for a repo that uses best effort consistency (e.g. that was just freshly mounted as done in the test) because we kepy resetting to `-1` on `IOException`, listing and finding the broken generation `N` and then interpreted the subsequent reset to `-1` as a concurrent change to the repository.

When we receive a request with an Authorization header that contains a Bearer token that is not generated by us or that is malformed in some way, attempting to decode it as one of our own might cause a number of exceptions that are not IOExceptions. This commit ensures that we catch and log these too and call onResponse with `null, so that we can return 401 instead of 500. Resolves: elastic#50497

Updates several example snippets in the Cluster Allocation Explain API docs to consistently use the `my_index` index. Previously, the snippets switches from `my_index` to `idx`, which could confuse users. Co-authored-by: Emmanuel DEMEY <demey.emmanuel@gmail.com> Co-authored-by: Emmanuel DEMEY <demey.emmanuel@gmail.com>

Closes elastic#49653. When using _FILE environment variables to supply values to Elasticsearch, following symlinks when checking that file permissions are secure.

With elastic#35848, users can now retrieve total hits as an integer when the `rest_total_hits_as_int` query parameter is `true`. This is the default value. This updates several snippet examples in the Watcher docs that used a workaround to get a total hits integer.

Synced flush was a brilliant idea. It supports instant recoveries with a quite small implementation. However, with the presence of sequence numbers and retention leases, it is no longer needed. This change removes it from 8.0. Relates elastic#5077

Tracking issue: elastic#51103

Updates snippet to consistently use 2-space indentation. The snippet previously used a mix of tab/5-space and 2-space indents. Co-authored-by: Peter Johnson <wiz@wiz.co.nz> Co-authored-by: Peter Johnson <peter@geocode.earth>

See elastic#32299

elastic#51111)

* Changes titles to sentence case. * Appends pages with 'reference' to differentiate their content from conceptual overviews. * Moves the 'Normalizers' page to end of the Analysis topic pages.

This helps the topic better match the structure of our machine learning docs, e.g. https://www.elastic.co/guide/en/machine-learning/7.5/ml-concepts.html This PR only includes the 'Anatomy of an analyzer' page as a 'Concepts' child page, but I plan to add other concepts, such as 'Index time vs. search time', with later PRs.

Adds a 'Configure text analysis' page to house tutorial content for the analysis topic. Also relocates the following pages as children as this new page: * 'Test an analyzer' * 'Configuring built-in analyzers' * 'Create a custom analyzer' I plan to add a tutorial for specifying index-time and search-time analyzers to this section as part of a future PR.

elastic/docs#1687 added support for the `[%collapsible]` Asciidoc attribute, which creates collapsible sections in the HTML output. This PR makes two related changes to the nodes stats API documentation: * Makes the response parameter sections collapsible. This allows users to more easily navigate the page without long walls of text. * Reorders the response parameter sections to match the default order returned by the API. Relates to elastic#47524.

If some internal `.watcher` or so index gets created during these tests then the shard counts on snapshot restores and creates won't match up with expectations. Fixed by only creating the snapshot for the test index Closes elastic#50819

…c#51072) If a transform config got lost (e.g. because the internal index disappeared) tasks could not be stopped using transform API. This change makes it possible to stop transforms without a config, meaning to remove the background task. In order to do so force must be set to true.

There is no longer any point in restricting the types for a Watcher search template. The types parameter was deprecated in 7x, and can be removed from 8x. Relates to elastic#41059

We have a few references left in master to the AllFieldMapper, which was deprecated in the 6x line and only kept in 7x for backwards compatibility. These references can be removed entirely in master.

Object fields cannot be used as features. At the moment _explain API includes them and even worse it allows it does not error when an object field is excluded. This creates the expectation to the user that all children fields will also be excluded while it's not the case. This commit omits object fields from the _explain API and also adds an error if an object field is included or excluded.

increase timeout of DocsClientYamlTestSuiteIT to 35 minutes, temporary solution for issue elastic#49753

This commit changes our behavior so that when we receive a request with an invalid/expired/wrong access token or API Key we do not fallback to authenticating as the anonymous user even if anonymous access is enabled for Elasticsearch.

When you declare an ObjectParser with top level named objects like we do with `significant_terms` we didn't support "did you mean". This fixes that. Relates elastic#50938

This commit upgrades the OWASP HTML sanitizer used by watcher to the latest version and also upgrades guava, which it depends on. The guava upgrade also requires the addition of a new dependency that guava itself requires as of version 27.0. The sanitizer's behavior has changed to re-write these templated values with a comment that results in this output `{{ctx.metadata.name}}`. This would be an issue if we attempted to sanitize the template, but the code that uses the sanitizer runs the rendered string through the sanitizer, which means that the templated values have been replaced already. Relates elastic#50395

This change introduces a new feature for indices so that they can be hidden from wildcard expansion. The feature is referred to as hidden indices. An index can be marked hidden through the use of an index setting, `index.hidden`, at creation time. One primary use case for this feature is to have a construct that fits indices that are created by the stack that contain data used for display to the user and/or intended for querying by the user. The desire to keep them hidden is to avoid confusing users when searching all of the data they have indexed and getting results returned from indices created by the system. Hidden indices have the following properties: * API calls for all indices (empty indices array, _all, or *) will not return hidden indices by default. * Wildcard expansion will not return hidden indices by default unless the wildcard pattern begins with a `.`. This behavior is similar to shell expansion of wildcards. * REST API calls can enable the expansion of wildcards to hidden indices with the `expand_wildcards` parameter. To expand wildcards to hidden indices, use the value `hidden` in conjunction with `open` and/or `closed`. * Creation of a hidden index will ignore global index templates. A global index template is one with a match-all pattern. * Index templates can make an index hidden, with the exception of a global index template. * Accessing a hidden index directly requires no additional parameters. Relates elastic#50251

…c#51146) If 1000 different category definitions are created for a job in the first 100 buckets it processes then an audit warning will now be created. (This will cause a yellow warning triangle in the ML UI's jobs list.) Such a large number of categories suggests that the field that categorization is working on is not well suited to the ML categorization functionality.

Tests in BuildPluginIT copy the workspace but exclude the build directories based on whether the directory string representation includes `/build/` or not. This check is problematic if the directory of the project has a parent directory also named `build`. The change in this commit checks to see if the path relative to the project directory has any path parts equal to `build`.

…#51145)

We added tracking of index feature usage in elastic#51031 but due to some copy and paste errors the test fails on some seeds. This fixes those errors.

Now that we've backported elastic#50869 we can update the skip config for its test.

…1075) These policies store statistics, but since stats updating is asynchronous, it's possible for the update from one test to bleed into a separate one. This change switches the tests to use separate policy ids so that their stats are tracked independently. It also relaxes the checking constraint in one of the tests. Hopefully this: Resolves elastic#48531 Resolves elastic#48017

This commit updates the BWC version we use in wire serialization for the hidden indices to V_7_7_0 after the backport of the feature to the 7.x branch. Relates elastic#50452

This commit merely adds the skeleton for the autoscaling project, adding the basics to include the autoscaling module in the default distribution, opt-in to code formatting, and a placeholder for the docs.

The tests, when creating broken serialized blobs could randomly create a sequence of bytes that is partially readable by the deserializer and then not throw `IOException` but instead `ElasticsearchParseException`. We should just handle these unexpected exceptions downstream properly and pass them wrapped as `RepositoryException` to the listener to fix the test and keep the API consistent.

* Deprecates _upgrade API Ref elastic#47678 * Move deprecation flags to path section. Add deprecation warning tests for _upgrade API. Ref elastic#47678

Creating indices with soft deletes disabled is no longer supported in 8.0.

The docs test suite is still timing out on CI at 35 minutes, so pushing it to 40 minutes while we determine the cause of the slowdown. Relates: elastic#49753

This test was recently un-muted, but is still failing Relates: elastic#50781

…tic#51151)

Small updates to grammar, syntax, and unclear wordings.

Add cool down period after snapshot finalization and delete to prevent eventually consistent AWS S3 from corrupting shard level metadata as long as the repository is using the old format metadata on the shard level.

Co-authored-by: debadair <debadair@elastic.co>

Take into account of number of unmapped fields when calculating against limit. Closes elastic#49002

…c#50823) The hierarchy of fields/sub-fields under a field that is of an unsupported data type will be marked as unsupported as well. Until this change, the behavior was to set the unsupported data type field's hierarchy as empty. Example, considering the following hierarchy of fields/subfields a -> b -> c -> d, if b would be of type "foo", then b, c and d will be marked as unsupported.

…astic#50960)" (elastic#51214) Reverts elastic#50960 This commit has been causing test failures during upgrade tests: specifically, an upgraded node becomes master and sends a cluster state update to a 7.x node; this node sees that the mapping version of its .tasks index is the same as the master, so asserts that the serialized mappings are the same; however, because the master has rewritten the mapping to use _docinstead oftasks`, we get an assertion failure. The logical fix is for the master to increment its mapping version when it rewrites the mapping, but there isn't a simple way to do that currently. This reverts commit 774bfb5.

check custom meta data to avoid NPE, fixes a problem introduced in elastic#51072 fixes elastic#51153

…tic#51029) * REST PreparedStatement-like query parameters are now supported in the form of an array of non-object, non-array values where ES SQL parser will try to infer the data type of the value being passed as parameter.

Currently, we do not exclude soft-deleted documents when opening index reader in the FrozenEngine. Relates elastic#50775

New index templates can only contain a single mapping, as we no longer support multiple types. This commit changes the internal classes used to represent a PutIndexTemplateRequest from holding a map of types to mappings, to instead holding a single mapping string. Relates to elastic#41059

We no longer need to explicitly enable soft-deletes in CCR tests. Relates elastic#50775

Since autoscaling is currently only under development, this commit causes the autoscaling docs to be excluded any time that release docs are being built.

Relates elastic#51127

…ew flow

…arch

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding best_compression #49974

Adding best_compression #49974

Commits on Jan 21, 2020

Commits on Jan 23, 2020

Commits on Jan 29, 2020

Commits on Jan 30, 2020

Commits on Jan 31, 2020

Commits on Feb 1, 2020