New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ML] Store ml job configurations in the new .ml-config index #36698

Merged
merged 63 commits into from Dec 18, 2018

Conversation

Projects
None yet
5 participants
@davidkyle
Copy link
Contributor

davidkyle commented Dec 17, 2018

Store ml anomaly detector jobs and datafeed configurations in the .ml-config index.
Prior to this change job configs were stored in the clusterstate, with this change all new jobs will stored in the new .ml-config index. Version 7 will expect all ml configurations to be stored in the index rather than clusterstate as a step towards that goal all closed job configs will be migrated from the clusterstate to the config index after upgrading.

This commit merges work undertaken on a feature branch.
See #32905

davidkyle and others added some commits Aug 9, 2018

[ML] Job and datafeed mappings with index template (#32719)
Index mappings for the configuration documents
[ML] Job in index: delete filter action (#34642)
This changes the delete filter action to search
for jobs using the filter to be deleted in the index
rather than the cluster state.
[ML] Job in Index: Enable integ tests (#34851)
Enables the ml integration tests excluding the rolling upgrade tests and a lot of fixes to 
make the tests pass again.
Merge branch '6.x' into feature-jindex-6x
Conflicts due to the deprecation of AbstractComponent and removal of settings
[ML] Reimplement established model memory (#35263)
This is the 6.6/6.7 implementation of a master node service to
keep track of the native process memory requirement of each ML
job with an associated native process.

The new ML memory tracker service works when the whole cluster
is upgraded to at least version 6.6.  For mixed version clusters
the old mechanism of established model memory stored on the job
in cluster state is used.  This means that the old (and complex)
code to keep established model memory up to date on the job object
cannot yet be removed.  When this change is forward ported to 7.0
the old way of keeping established model memory updated will be
removed.
[ML] Job In Index: Enable GET APIS in mixed state (#35344)
This enables calls to the job and datafeed APIs in a mixed cluster state
before jobs have been migrated
[ML] Need to wait for shards to replicate in distributed test (#35541)
Because the cluster was expanded from 1 node to 3 indices would
initially start off with 0 replicas.  If the original node was
killed before auto-expansion to 1 replica was complete then
the test would fail because the indices would be unavailable.
[ML] Job in index: Restore ability to update cluster state jobs (#35539)
Job updates can apply to cluster state or index jobs this includes 
reverting a model snapshot and finalizing the job
[ML] Job in index: Enable get and update actions for clusterstate jobs (
#35598)

Small fixes to read from all locations and added index mappings for DelayedDataCheckConfig

droberts195 and others added some commits Dec 14, 2018

[ML] Adapt to periodic persistent task refresh (#36494)
After #36069 the
approach for reallocating ML persistent tasks after refreshing
job memory requirements can be simplified.
Fix TooManyJobsIT.testMultipleNodes
Two problems:

1. Stack overflow during async iteration when lots of
   jobs on same machine
2. Not effectively setting search size in all cases
[ML] Remove unreliable test assertions
The same cases are covered in MlMigrationIT
Use execute() instead of submit() in MlMemoryTracker
We don't need a Future to wait for completion

@davidkyle davidkyle added the :ml label Dec 17, 2018

@elasticmachine

This comment has been minimized.

Copy link

elasticmachine commented Dec 17, 2018

davidkyle and others added some commits Dec 17, 2018

[FEATURE][ML] Add cluster setting that enables/disables config migrat…
…ion (#36700)

This commit adds a cluster settings called `xpack.ml.enable_config_migration`.
The setting is `true` by default. When set to `false`, no config migration will
be attempted and non-migrated resources (e.g. jobs, datafeeds) will be able
to be updated normally.

Relates #32905
@davidkyle

This comment has been minimized.

Copy link
Contributor Author

davidkyle commented Dec 17, 2018

retest this please

dimitris-athanasiou and others added some commits Dec 17, 2018

@davidkyle

This comment has been minimized.

Copy link
Contributor Author

davidkyle commented Dec 18, 2018

run gradle build tests 2

@droberts195
Copy link
Contributor

droberts195 left a comment

LGTM on the basis these changes have been previously reviewed in the feature branch PRs

@davidkyle davidkyle merged commit 66a582a into 6.x Dec 18, 2018

7 checks passed

CLA Commit author has signed the CLA
Details
elasticsearch-ci-1 Build finished.
Details
elasticsearch-ci-2 Build finished.
Details
elasticsearch-ci/default-distro Build finished.
Details
elasticsearch-ci/docbldesx Build finished.
Details
elasticsearch-ci/oss-distro-docs Build finished.
Details
elasticsearch-ci/packaging-sample Build finished.
Details

hendrikmuhs added a commit that referenced this pull request Dec 31, 2018

[ML] fix x-pack usage regression caused by index migration (#36936)
Changes the feature usage retrieval to use the job manager rather than
directly talking to the cluster state, because jobs can now be either in
cluster state or stored in an index

This is a follow-up of #36702 / #36698

hendrikmuhs added a commit that referenced this pull request Dec 31, 2018

[ML] fix x-pack usage regression caused by index migration (#36936)
Changes the feature usage retrieval to use the job manager rather than
directly talking to the cluster state, because jobs can now be either in
cluster state or stored in an index

This is a follow-up of #36702 / #36698

hendrikmuhs added a commit that referenced this pull request Dec 31, 2018

[ML] fix x-pack usage regression caused by index migration (#36936)
Changes the feature usage retrieval to use the job manager rather than
directly talking to the cluster state, because jobs can now be either in
cluster state or stored in an index

This is a follow-up of #36702 / #36698

@droberts195 droberts195 deleted the feature-jindex-6x branch Jan 8, 2019

@davidkyle davidkyle added the v6.6.0 label Jan 18, 2019

@davidkyle davidkyle changed the title [ML] Merge the Jindex 6x feature branch [ML] Store ml job configurations in the new .ml-config index Jan 18, 2019

@lcawl lcawl added the >feature label Jan 18, 2019

@codebrain codebrain referenced this pull request Jan 25, 2019

Open

[meta] 6.6.0 Release #3552

11 of 106 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment