Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ML] Improve model_memory_limit UX for data frame analytics jobs #44699

Closed
6 tasks done
droberts195 opened this issue Jul 22, 2019 · 1 comment
Closed
6 tasks done

[ML] Improve model_memory_limit UX for data frame analytics jobs #44699

droberts195 opened this issue Jul 22, 2019 · 1 comment
Assignees
Labels

Comments

@droberts195
Copy link
Contributor

droberts195 commented Jul 22, 2019

The current workflow for memory limits is:

  1. User specifies a memory limit up front with no guidance as to what is sensible/realistic
  2. They start the analysis
  3. Analysis chugs away for a while reindexing the source index into the destination index
  4. C++ process is started and checks whether the memory limit is sufficient to do the analysis
  5. If it's not then the process exits, reporting how much memory was required in the error message

Two possible ways to solve it are:

  1. We duplicate that logic that calculates the memory requirement from the C++ code into the Java and UI code
  2. We add a mode of operation to the C++ process where you just supply the spec and instead of actually doing the analysis it just tells you:
    i. What you'd have to set the memory limit to to do the analysis entirely in RAM
    ii. What the minimum memory limit is that will enable the analysis to run at all (using disk)

Although option 2 is a lot of work, the memory calculations done by the C++ are now so complex that it is impractical to duplicate them. Therefore we should implement option 2. The work will be something along the lines of:

@elasticmachine
Copy link
Collaborator

Pinging @elastic/ml-core

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

5 participants