Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ML] Merge the Jindex master feature branch #36702

Merged
merged 71 commits into from Dec 18, 2018

Conversation

Projects
None yet
@davidkyle
Copy link
Member

commented Dec 17, 2018

For #32905

davidkyle and others added some commits Aug 9, 2018

[ML] Job and datafeed mappings with index template (#32719)
Index mappings for the configuration documents
[ML] Job in index: delete filter action (#34642)
This changes the delete filter action to search
for jobs using the filter to be deleted in the index
rather than the cluster state.
[ML] Job in Index: Enable integ tests (#34851)
Enables the ml integration tests excluding the rolling upgrade tests and a lot of fixes to
make the tests pass again.
Merge branch 'master' into feature-jindex-master
Conflicts due to the deprecation of AbstractComponent and removal of settings
[ML] Reimplement established model memory (#35500)
This is the 7.0 implementation of a master node service to
keep track of the native process memory requirement of each ML
job with an associated native process.

The new ML memory tracker service works when the whole cluster
is upgraded to at least version 6.6. For mixed version clusters
the old mechanism of established model memory stored on the job
in cluster state was used. This means that the old (and complex)
code to keep established model memory up to date on the job object
has been removed in 7.0.

Forward port of #35263
[ML] Need to wait for shards to replicate in distributed test (#35541)
Because the cluster was expanded from 1 node to 3 indices would
initially start off with 0 replicas.  If the original node was
killed before auto-expansion to 1 replica was complete then
the test would fail because the indices would be unavailable.
Merge branch 'master' into feature-jindex-master
Resolve conflicts due to fields removed from AnalysisConfig

javanna and others added some commits Dec 17, 2018

[TEST] fix float comparison in RandomObjects#getExpectedParsedValue
This commit fixes a test bug introduced with #36597. This caused some
test failure as stored field values comparisons would not work when CBOR
xcontent type was used.

Closes #29080
[Geo] Integrate Lucene's LatLonShape (BKD Backed GeoShapes) as defaul…
…t `geo_shape` indexing approach (#35320)

This commit  exposes lucene's LatLonShape field as the
default type in GeoShapeFieldMapper. To use the new 
indexing approach, simply set "type" : "geo_shape" in 
the mappings without setting any of the strategy, precision, 
tree_levels, or distance_error_pct parameters. Note the 
following when using the new indexing approach:

* geo_shape query does not support querying by 
MULTIPOINT.
* LINESTRING and MULTILINESTRING queries do not 
yet support WITHIN relation.
* CONTAINS relation is not yet supported.
The tree, precision, tree_levels, distance_error_pct, 
and points_only parameters are deprecated.
ingest: support default pipelines + bulk upserts (#36618)
This commit adds support to enable bulk upserts to use an index's
default pipeline. Bulk upsert, doc_as_upsert, and script_as_upsert
are all supported.

However, bulk script_as_upsert has slightly surprising behavior since
the pipeline is executed _before_ the script is evaluated. This means
that the pipeline only has access the data found in the upsert field
of the script_as_upsert. The non-bulk script_as_upsert (existing behavior)
runs the pipeline _after_ the script is executed. This commit
does _not_ attempt to consolidate the bulk and non-bulk behavior for
script_as_upsert.

This commit also adds additional testing for the non-bulk behavior,
which remains unchanged with this commit.

fixes #36219
Fix duplicate phrase in shrink/split error message (#36734)
This commit removes a duplicate "must be a" from the shrink/split error
messages.
Deprecate types in get_source and exist_source (#36426)
This change adds a new untyped endpoint `{index}/_source/{id}` for both the
GET and the HEAD methods to get the source of a document or check for its
existance. It also adds deprecation warnings to RestGetSourceAction that emit
a warning when the old deprecated "type" parameter is still used. Also updating
documentation and tests where appropriate.

Relates to #35190
Revert "[Geo] Integrate Lucene's LatLonShape (BKD Backed GeoShapes) a…
…s default `geo_shape` indexing approach (#35320)"

This reverts commit 5bc7822.
Enhance Invalidate Token API (#35388)
This change:

- Adds functionality to invalidate all (refresh+access) tokens for all users of a realm
- Adds functionality to invalidate all (refresh+access)tokens for a user in all realms
- Adds functionality to invalidate all (refresh+access) tokens for a user in a specific realm
- Changes the response format for the invalidate token API to contain information about the 
   number of the invalidated tokens and possible errors that were encountered.
- Updates the API Documentation

After back-porting to 6.x, the `created` field will be removed from master as a field in the 
response

Resolves: #35115
Relates: #34556
Add raw sort values to SearchSortValues transport serialization (#36617)
In order for CCS alternate execution mode (see #32125) to be able to do the final reduction step on the CCS coordinating node, we need to serialize additional info in the transport layer as part of each `SearchHit`. Sort values are already present but they are formatted according to the provided `DocValueFormat` provided. The CCS node needs to be able to reconstruct the lucene `FieldDoc` to include in the `TopFieldDocs` and `CollapseTopFieldDocs` which will feed the `mergeTopDocs` method used to reduce multiple search responses (one per cluster) into one.

This commit adds such information to the `SearchSortValues` and exposes it through a new getter method added to `SearchHit` for retrieval. This info is only serialized at transport and never printed out at REST.
Watcher: Ensure all internal search requests count hits (#36697)
In previous commits only the stored toXContent version of a search
request was using the old format. However an executed search request was
already disabling hit counts. In 7.0 hit counts will stay enabled by
default to allow for proper migration.

Closes #36177
Ensure MapperService#getAllMetaFields elements order is deterministic (
…#36739)

MapperService#getAllMetaFields returns an array, which is created out of
an `ObjectHashSet`. Such set does not guarantee deterministic hash
ordering. The array returned by its toArray may be sorted differently
at each run. This caused some repeatability issues in our tests (see #29080)
as we pick random fields from the array of possible metadata fields,
but that won't be repeatable if the input array is sorted differently at
every run. Once setting the tests seed, hppc picks that up and the sorting is
deterministic, but failures don't repeat with the seed that gets printed out
originally (as a seed was not originally set).
See also https://issues.carrot2.org/projects/HPPC/issues/HPPC-173.

With this commit, we simply create a static sorted array that is used for
`getAllMetaFields`. The change is in production code but really affects
only testing as the only production usage of this method was to iterate
through all values when parsing fields in the high-level REST client code.
Anyways, this seems like a good change as returning an array would imply
that it's deterministically sorted.
@davidkyle

This comment has been minimized.

Copy link
Member Author

commented Dec 18, 2018

run gradle build tests 2

@droberts195
Copy link
Contributor

left a comment

LGTM on the basis these changes have been previously reviewed in the feature branch PRs

@droberts195 droberts195 added the v7.0.0 label Dec 18, 2018

@davidkyle davidkyle merged commit e294056 into master Dec 18, 2018

8 checks passed

CLA Commit author has signed the CLA
Details
elasticsearch-ci-1 Build finished.
Details
elasticsearch-ci-2 Build finished.
Details
elasticsearch-ci/default-distro Build finished.
Details
elasticsearch-ci/docbldesx Build finished.
Details
elasticsearch-ci/docs-check Build finished.
Details
elasticsearch-ci/oss-distro-docs Build finished.
Details
elasticsearch-ci/packaging-sample Build finished.
Details

jasontedor added a commit to jasontedor/elasticsearch that referenced this pull request Dec 18, 2018

Merge remote-tracking branch 'elastic/master' into lazier-ingest-geoip
* elastic/master: (31 commits)
  enable bwc tests and switch transport serialization version to 6.6.0 for CAS features
  [DOCs] Adds ml-cpp PRs to alpha release notes (elastic#36790)
  Synchronize WriteReplicaResult callbacks (elastic#36770)
  Add CcrRestoreSourceService to track sessions (elastic#36578)
  [Painless] Add tests for boxed return types (elastic#36747)
  Internal: Remove originalSettings from Node (elastic#36569)
  [ILM][DOCS] Update ILM API authorization docs (elastic#36749)
  Core: Deprecate use of scientific notation in epoch time parsing (elastic#36691)
  [ML] Merge the Jindex master feature branch (elastic#36702)
  Tests: Mute SnapshotDisruptionIT.testDisruptionOnSnapshotInitialization
  Update versions in SearchSortValues transport serialization
  Update version in SearchHits transport serialization
  [Geo] Integrate Lucene's LatLonShape (BKD Backed GeoShapes) as default `geo_shape` indexing approach (elastic#36751)
  [Docs] Fix error in Common Grams Token Filter (elastic#36774)
  Fix rollup search statistics (elastic#36674)
  SQL: Fix wrong appliance of StackOverflow limit for IN (elastic#36724)
  [TEST] Added more logging
  Invalidate Token API enhancements - HLRC (elastic#36362)
  Deprecate types in index API (elastic#36575)
  Disable bwc tests until elastic#36555 backport is complete (elastic#36737)
  ...

jasontedor added a commit to jasontedor/elasticsearch that referenced this pull request Dec 18, 2018

Merge remote-tracking branch 'elastic/master' into ingest-geoip-module
* elastic/master: (61 commits)
  enable bwc tests and switch transport serialization version to 6.6.0 for CAS features
  [DOCs] Adds ml-cpp PRs to alpha release notes (elastic#36790)
  Synchronize WriteReplicaResult callbacks (elastic#36770)
  Add CcrRestoreSourceService to track sessions (elastic#36578)
  [Painless] Add tests for boxed return types (elastic#36747)
  Internal: Remove originalSettings from Node (elastic#36569)
  [ILM][DOCS] Update ILM API authorization docs (elastic#36749)
  Core: Deprecate use of scientific notation in epoch time parsing (elastic#36691)
  [ML] Merge the Jindex master feature branch (elastic#36702)
  Tests: Mute SnapshotDisruptionIT.testDisruptionOnSnapshotInitialization
  Update versions in SearchSortValues transport serialization
  Update version in SearchHits transport serialization
  [Geo] Integrate Lucene's LatLonShape (BKD Backed GeoShapes) as default `geo_shape` indexing approach (elastic#36751)
  [Docs] Fix error in Common Grams Token Filter (elastic#36774)
  Fix rollup search statistics (elastic#36674)
  SQL: Fix wrong appliance of StackOverflow limit for IN (elastic#36724)
  [TEST] Added more logging
  Invalidate Token API enhancements - HLRC (elastic#36362)
  Deprecate types in index API (elastic#36575)
  Disable bwc tests until elastic#36555 backport is complete (elastic#36737)
  ...

hendrikmuhs added a commit that referenced this pull request Dec 31, 2018

[ML] fix x-pack usage regression caused by index migration (#36936)
Changes the feature usage retrieval to use the job manager rather than
directly talking to the cluster state, because jobs can now be either in
cluster state or stored in an index

This is a follow-up of #36702 / #36698

hendrikmuhs added a commit that referenced this pull request Dec 31, 2018

[ML] fix x-pack usage regression caused by index migration (#36936)
Changes the feature usage retrieval to use the job manager rather than
directly talking to the cluster state, because jobs can now be either in
cluster state or stored in an index

This is a follow-up of #36702 / #36698

hendrikmuhs added a commit that referenced this pull request Dec 31, 2018

[ML] fix x-pack usage regression caused by index migration (#36936)
Changes the feature usage retrieval to use the job manager rather than
directly talking to the cluster state, because jobs can now be either in
cluster state or stored in an index

This is a follow-up of #36702 / #36698

@droberts195 droberts195 deleted the feature-jindex-master branch Jan 8, 2019

@colings86 colings86 added v7.0.0-beta1 and removed v7.0.0 labels Feb 7, 2019

@jimczi jimczi added the >enhancement label Feb 11, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.