Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sorting: Use first non-empty result to detect if sort is required #8236

wants to merge 1,895 commits into from


None yet
Copy link

s1monw commented Oct 27, 2014

Closes #8226

rjernst and others added some commits Jul 15, 2014

Share numeric data analyzer instances between mappings
use similar mechanism that shares numeric analyzers for long/double/... for dates as well. This has nice memory save properties with many date fields mapping case, as well as analysis saves (thread local resources)
closes #6843
Releasable XContentBuilder
make the builder releasable (auto closeable), and use it in shards state
also make XContentParser releasable (AutoCloseable) and not closeable since it doesn't throw an IOException
closes #6869
[AGGS] Pass current docid being processed to scripts.
Scripts may internally cache based on docid (as expressions do). This
change makes numeric aggregations using scripts pass the docid when
it changes.
Store: Only send shard exists requests if shards exist locally on dis…
…k and are not allocated on that node according to the cluster state.

Closes #6870
[Infra] re-send failed shard messages when receiving a cluster state …
…that still refers to them

In rare cases we may fail to send a shard failure event to the master, or there is no known master when the shard has failed (ex. a couple of node leave the cluster canceling recoveries and causing a master to step down at the same time). When that happens and a cluster state arrives from the (new) master we should resend the shard failure in order for the master to remove the shard from this node.

Closes #6881
[Recovery] don't start a gateway recovery if source node is not found
Due to change introduced in #6825, we now start a local gateway recovery for replicas, if the source node can not be found. The recovery then fails because we never recover replicas from disk.

Closes #6879
[DOCS] : Indexed scripts/templates
These are the docs for the indexed scripts/templates feature.
Also moved the namespace for the REST endpoints.

Closes #6851

Aggregations: Fixed Histogram key_as_string bug
The key as string field in the response for the histogram aggregation will now only show if format is specified on the request.

Closes #6655
Threadpool Info: Allow to serialize negative thread pool sizes
As a SizeValue is used for serializing the thread pool size, a negative number
resulted in throwing an exception when deserializing (using -ea an assertionerror
was thrown).

This fixes a check for changing the serialization logic, so that negative numbers are read correctly, by adding an internal UNBOUNDED value.

Closes #6325
Closes #5357

This comment has been minimized.

Copy link

spalger commented on rest-api-spec/api/indexed_script.create.json in 1465edb Jul 16, 2014

I think this file (and the related files) should be renamed to reflect the updated method names.

This comment has been minimized.

Copy link

clintongormley replied Jul 16, 2014

@spenceralger already pushed

clintongormley and others added some commits Jul 16, 2014

REST: Renamed indexed_script and indexed_template specs
The file name of the REST specs should be the same as
the endpoint which it documents.
REST: Fixed indexed script/template tests
The clients don't have access to the error message
for matching purposes.  Rewritten tests to work
with the clients
Serialization: Fix bwc issue by falling back to old threadpool serial…

This fixes an issue introduced by the serialization changes in #6486
which are not needed at all. Node that the serialization itself is not broken
but the TransportClient uses its own version on initial connect and getting
the NodeInfos.
bin/plugin removes itself
If you call `bin/plugin --remove es-plugin` the plugin got removed but the file `bin/plugin` itself was also deleted.

We now don't allow the following plugin names:

* elasticsearch
* plugin
* elasticsearch.bat
* plugin.bat
* service.bat

Closes #6745

(cherry picked from commit 248ea31)
[Infra] remove indicesLifecycle.Listener from IndexingMemoryController
The IndexingMemoryController determines the amount of indexing buffer size and translog buffer size each shard should have. It takes memory from inactive shards (indexing wise) and assigns it to other shards. To do so it needs to know about the addition and closing of shards. The current implementation hooks into the indicesService.indicesLifecycle() mechanism to receive call backs, such shard entered the POST_RECOVERY state. Those call backs are typically run on the thread that actually made the change. A mutex was used to synchronize those callbacks with IndexingMemoryController's background thread, which updates the internal engines memory usage on a regular interval. This introduced a dependency between those threads and the locks of the internal engines hosted on the node. In a *very* rare situation (two tests runs locally) this can cause recovery time outs where two nodes are recovering replicas from each other.

 This commit introduces a a lock free approach that updates the internal data structures during iterations in the background thread.

Closes #6892
[Store] delete unallocated shards under a cluster state task
This is to prevent a rare racing condition where the very same shard gets allocated to the node after our sanity check that the cluster state didn't check and the actual deletion of the files.

Closes #6902
[Engine] `index.fail_on_corruption` is not updateable
The `index.fail_on_corruption` was not updateable via the index settings
API. This commit also fixed the setting prefix to be consistent with other
setting on the engine. Yet, this feature is unreleased so this won't break anything.

Closes #6941
[RESTORE] Fail restore if snapshot is corrupted
today if a snapshot is corrupted the restore operation
never terminates. Yet, if the snapshot is corrupted there
is no way to restore it anyway. If such a snapshot is restored
today the only way to cancle it is to delete the entire index which
might cause dataloss. This commit also fixes an issue in InternalEngine
where a deadlock can occur if a corruption is detected during flush
since the InternalEngine#snapshotIndex aqcuires a topLevel read lock
which prevents closing the engine.

Closes #6938
[TEST] Stress test for update and delete concurrency.
This test deletes and updates using upserts documents over several threads in a
tight loop. It counts the number of responses and verifies that the versions at
the end are correct.

clintongormley and others added some commits Oct 20, 2014

Update plugins.asciidoc
Added Vietnamese Analyser to plugins page

Closes #6647
Docs: Update minimum-should-match.asciidoc
Add %-sign to examle in the last section

Closes #8157
Docs: Update filtered-query.asciidoc
Fix mistyping

Closes #8167
Docs: Provide example of deleting a repository
Example of deleting a repository with explanation that snapshots themselves are left untouched.

Closes #8172
Docs: missing quote
fix missing quote

Closes #8176
Parent/child: Check if there is a search context, otherwise throw a q…
…uery parse exception.

Also added a bwc test that runs a delete by query with a has_child query and verifies that only that operation is ignored when recovering from disk during a upgrade.

Closes #8031
Closes #8177
[CORE] Don't handle FNF exceptions when reading snapshot
We used to handle FNF exceptions in the store when reading a snapshot.
For instance if we can't open a segments file for a given commit point
we just return an empty metadata object and tracelog the even. This can
cause shards to be false marked as corrupted if a shard is forcefully
removed while a recovery started at the same time. We should in general
bubble up these exceptions and let the caller decided how to handle the
[SEARCH] Passing fieddata_fields as a non array causes OOM
If `fielddata_fields` are passed as a simple value instead of an array
we end up in an infinite loop createing parsed elements with null
This commit validates the incoming token

Closes #8203
Enable ClusterInfoService by default
Since we enabled the disk threshold decider by default, we need to
enable the cluster info service so that disk usages and shard sizes can
be gathered also.

Adds a test that checks that we are gathering information by default.
[TESTS] Change test node watermarks for DiskThresholdDecider
Ensures that we can still run tests if the machine running them does not
have enough free disk space to be below the high watermark

Update warmers.asciidoc
It is kind of implied right now.  This is just to clarify that warmers will also be run on node startup since we get asked specifically about that.
Update core-types.asciidoc
The doc mentions the ability to disable norms starting in 1.2.0 but does not provide a specific example.  Added an example of the syntax to do so.
[STORE] Don't catch FNF/NSF exception when reading metadata
When reading metadata we do catch FileNotFound and NoSuchFileExceptions
today, log the even and return an empty metadata object. Yet, in some cases
this might be the wrong thing todo ie. if a commit point is provided these
situations are actually an error and should be rethrown. This commit
pushes the responsiblity to the caller to handle this exception.

Closes #8207
Docs: rolling upgrade process seems incorrect
When reading the [rolling upgrade process](, you can see that we wrote:

* disable allocation
* upgrade node1
* upgrade node2
* upgrade node3
* ...
* enable allocation

That won't work as after a node has been removed and restarted, no shard will be allocated anymore.
So closing node2 and remaining nodes, won't help to serve index and search request anymore.

We should write:

* disable allocation
* upgrade node1
* enable allocation
* wait for shards being recovered on node1
* disable allocation
* upgrade node2
* enable allocation
* wait for shards being recovered on node2
* disable allocation
* upgrade node3
* enable allocation
* wait for shards being recovered on node3
* disable allocation
* ...
* enable allocation

I think this documentation update should go in 1.3, 1.4, 1.x and master branches.

Closes #8218
Closes #7973.
[TEST] `Scope.SUITE` is not reproducible due to late cluster initiali…

The cluster for `Scope.SUITE` tests must be initialize in a static manner
before the first test runs otherwise the random context used to initialize
the cluster is taken from tests randomness rather than the suites randomness.
This means test clusters will have different setups if only a single test is
executed or even the test might have a entirely different random sequence.

This comment has been minimized.

Copy link
Contributor Author

s1monw commented Oct 27, 2014

I marked this as 1.3.5 since the test for it doesn't fail on the other branches. Yet, I think the difference here is only in terms of parsing and I wonder if we should add the new method in master too just to make sure we don't depend on shards putting in the right TopDocs instance even if results are emtpy


This comment has been minimized.

Copy link

jpountz commented Oct 27, 2014


@s1monw s1monw force-pushed the s1monw:issues/8826 branch Oct 27, 2014

@s1monw s1monw force-pushed the s1monw:issues/8826 branch to 8981e58 Oct 27, 2014


This comment has been minimized.

Copy link
Contributor Author

s1monw commented Oct 27, 2014

pushed to 1.3

@s1monw s1monw closed this Oct 27, 2014

@s1monw s1monw deleted the s1monw:issues/8826 branch Oct 27, 2014

@clintongormley clintongormley changed the title [SORT] Use first non-empty result to detect if sort is required Sorting: Use first non-empty result to detect if sort is required Nov 3, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.