Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compat/mapping #20

Open
wants to merge 139 commits into
base: compat/delete_update
Choose a base branch
from
Open

Compat/mapping #20

wants to merge 139 commits into from

Conversation

pgomulka
Copy link
Owner

@pgomulka pgomulka commented Jul 8, 2020

  • Have you signed the contributor license agreement?
  • Have you followed the contributor guidelines?
  • If submitting code, have you built your formula locally prior to submission with gradle check?
  • If submitting code, is your pull request against master? Unless there is a good reason otherwise, we prefer pull requests against master and will backport as needed.
  • If submitting code, have you checked that your submission is for an OS and architecture that we support?
  • If you are submitting this code for a class then read our policy for that.

pgomulka and others added 30 commits June 17, 2020 16:06
however some tests needs overrides
Add to the list of pipeline aggregations
This is a test to ensure we do not run into a regression like we did in
elastic#59031
Today we empty the searchable snapshots cache when cleanly closing a
shard, but leak cache files in some cases involving an unclean shutdown.
Such leaks are not permanent, they are cleaned up on shard relocation or
deletion, but they still might last for arbitrarily long until that
happens. This commit introduces a cleanup process that runs during node
startup to catch such leaks sooner.

Also, today we permit searchable snapshots to be held on custom data
paths, and store the corresponding cache files within the custom
location. Supporting this feature would make the cleanup process
significantly more complicated since it would require each node to parse
the index metadata for the shards it held before shutdown. Yet, this
feature is undocumented and offers minimal benefits to searchable
snapshots. Therefore with this commit we forbid custom data paths for
searchable snapshot shards.
…9230)

Updates docs and snippets for changes made to the get data stream API with
PR elastic#59128.
* Adding page for get snapshot API.

* Adding values for state and cleaning up some other formatting.

* Adding missing forward slash to GET request.

* Updating values for start_time and end_time in TESTRESPONSE.

* Swap "return" for "retrieve"

* Swap "return" for "retrieve" 2

* Change .snapshot to .response

* Adding response parameters and incorporating edits from review.

* Update response example to include repository info

* Change dash to underscore

* Add data type for snapshot in response

* Incorporating review comments and adding missing response definitions.

* Minor rewording in description.
Changes:
* Documents the `size` default as `10`.
* Updates `size` param def to note its relation to pipes.
* Updates the `head` and `tail` pipe docs to modify sequences.
* Documents the `fetch_size` parameter.

Relates to elastic#59014 and elastic#59063
We need to use a concurrent collection to keep track of the shipped operations
as they can arrive concurrently since elastic#58018.

Relates elastic#58018
Waiting `INIT` here is dead code in newer versions that don't use `INIT`
any longer and leads to nothing being written to the repository in older versions
if the snapshot is cancelled at the `INIT` step which then breaks repo consistency
checks.
Since we have other tests ensuring that snapshot abort works properly we can just remove
the wait for `INIT` here and backport this down to 7.8 to fix tests.

relates elastic#59140
This commit adds rejections when the indexing memory limits are
exceeded for primary or coordinating operations. The amount of bytes
allow for indexing is controlled by a new setting
indexing_limits.memory.limit.
We are leaking a FileChannel in elastic#39585 if we release a safe commit with 
CancellableThreads. Although it is a bug in Lucene where we do not close
a FileChannel if we failed to create a NIOFSIndexInput, I think it's
safer if we release a safe commit using the generic thread pool instead.

Closes elastic#39585
Relates elastic#45409
This modifies the `variable_width_histogram`'s distant bucket handling
to:
1. Properly handle integer overflows
2. Recalculate the average distance when new buckets are added on the
   ends. This should slow down the rate at which we build extra buckets
   as we build more of them.
jrodewig and others added 30 commits July 13, 2020 12:30
This commit moves the modules REST tests to the
newly introduced yamlRestTest source set. A few
tests have also been re-named to include the correct
IT suffix. Without changing the names, the testing
conventions task would fail since now that the YAML
tests are no longer present pacify the convention.
These tests have moved to the internalClusterTest
source set.

related: elastic#56841
We have recently added internal metrics to monitor the amount of
indexing occurring on a node. These metrics introduce back pressure to
indexing when memory utilization is too high. This commit exposes these
stats through the node stats API.
This commit adds data stream info to the `/_xpack` and `/_xpack/usage` APIs. Currently the usage is
pretty minimal, returning only the number of data streams and the number of indices currently
abstracted by a data stream:

```
  ...
  "data_streams" : {
    "available" : true,
    "enabled" : true,
    "data_streams" : 3,
    "indices_count" : 17
  }
  ...
```
…ser (elastic#59449)

Updates the 8.0 breaking changes to clarify that passwords for the removed
`kibana` user are not preserved for the replacement `kibana_system` users.

Closes elastic#59353
I've always been confused by the strange behavior that I saw when
working on elastic#57304. Specifically, I saw switching from a bimorphic
invocation to a monomorphic invocation to give us a 7%-15% performance
bump. This felt *bonkers* to me. And, it also made me wonder whether
it'd be worth looking into doing it everywhere.

It turns out that, no, it isn't needed everywhere. This benchmark shows
that a bimorphic invocation like:
```
LongKeyedBucketOrds ords = new LongKeyedBucketOrds.ForSingle();
ords.add(0, 0); <------ this line
```

is 19% slower than a monomorphic invocation like:
```
LongKeyedBucketOrds.ForSingle ords = new LongKeyedBucketOrds.ForSingle();
ords.add(0, 0); <------ this line
```

But *only* when the reference is mutable. In the example above, if
`ords` is never changed then both perform the same. But if the `ords`
reference is assigned twice then we start to see the difference:
```
immutable bimorphic    avgt   10   6.468 ± 0.045  ns/op
immutable monomorphic  avgt   10   6.756 ± 0.026  ns/op
mutable   bimorphic    avgt   10   9.741 ± 0.073  ns/op
mutable   monomorphic  avgt   10   8.190 ± 0.016  ns/op
```

So the conclusion from all this is that we've done the right thing:
`auto_date_histogram` is the only aggregation in which `ords` isn't final
and it is the only aggregation that forces monomorphic invocations. All
other aggregations use an immutable bimorphic invocation. Which is fine.

Relates to elastic#56487
* Adds hard_bounds to histogram aggregations

Adds a hard_bounds parameter to explicitly limit the buckets that a histogram
can generate. This is especially useful in case of open ended ranges that can
produce a very large number of buckets.

Co-authored-by: Mark Tozzi <mark.tozzi@gmail.com>
Since we most likely going to have 7.10 we should update version
in tests skips to 7.99.99.
TIP block was missing due to the lack of line break prior to the "TIP"

Co-authored-by: Leaf-Lin <39002973+Leaf-Lin@users.noreply.github.com>
This commit updates the node stats version constants to reflect the fact
that index pressure stats were backported to 7.9. It also reenables BWC
tests.
…stic#59375)

The primary shards of follower indices during the bootstrap need to be
on nodes with the remote cluster client role as those nodes reach out to
the corresponding leader shards on the remote cluster to copy Lucene
segment files and renew the retention leases. This commit introduces a
new allocation decider that ensures bootstrapping follower primaries are
allocated to nodes with the remote cluster client role.

Relates elastic#54146
Relates elastic#53924
Closes elastic#58534

Co-authored-by: Jason Tedor <jason@tedor.me>
API keys can be created without names using grant API key action. This is considered as a bug (elastic#59484). Since the feature has already been released, we need to accomodate existing keys that are created with null names. This PR relaxes the parser logic so that a null name is accepted.
…PIs (elastic#54572)

This adds a compatible APIs for typed endpoints removed in elastic#41640

202 failing tests
These 2 tests are explicitly fixed by this PR
CompatRestIT. test {yaml=mtermvectors/21_deprecated_with_types/Deprecated camel case and _ parameters should fail in Term Vectors query}
CompatRestIT. test {yaml=mtermvectors/30_mix_typeless_typeful/mtermvectors without types on an index that has types}

I think more yml tests should be added to 7.x to cover these endpoints

17th june
1306tests | 197failures | 16ignored | 10m56.91sduration
- Fix duplicate path deprecation by removing duplicate test resources
- fix deprecated non annotated input property in LazyPropertyList
- fix deprecated usage of AbstractArchiveTask.version
- Resolve correct test resources
Validating if registered compatible rest handlers are overriding
compatibleWith method and specify the right compatible version
("197 błędów / +197")
based on compat/search branch

fixed tests from delete and update directories.
Delete test is still not fixed
CompatRestIT. test {yaml=delete/70_mix_typeless_typeful/DELETE with typeless API on an index that has types}

current state
1306tests | 174failures

previously
1306tests | 197failures

relates elastic#54160
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.