Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update cpu and memory requests after cluster creation #88

Closed
jacobtomlinson opened this issue Jul 20, 2018 · 15 comments
Closed

Update cpu and memory requests after cluster creation #88

jacobtomlinson opened this issue Jul 20, 2018 · 15 comments
Labels
enhancement

Comments

@jacobtomlinson
Copy link
Member

On our Pangeo deployment users get a default worker-template.yaml which allows them to create clusters by simply running cluster = KubeCluster() without having to worry about what a kubernetes even is.

However in some occasions people want to be able to update the memory and cpu ratios of their workers depending on what they are running. The current workflow for this is to either specify the whole template as a dict or to copy the default worker-template.yaml, understand it, update the values and then user KubeCluster.from_yaml().

Personally I ended up writing a couple of helper functions in my notebook which look like this:

def update_worker_memory(cluster, new_limit):
    cluster.pod_template.spec.containers[0].resources.limits["memory"] = new_limit
    cluster.pod_template.spec.containers[0].resources.requests["memory"] = new_limit
    if '--memory-limit' in cluster.pod_template.spec.containers[0].args:
        index = cluster.pod_template.spec.containers[0].args.index('--memory-limit')
        cluster.pod_template.spec.containers[0].args[index + 1] = new_limit
    return cluster

def update_worker_cpu(cluster, new_limit):
    cluster.pod_template.spec.containers[0].resources.limits["cpu"] = new_limit
    cluster.pod_template.spec.containers[0].resources.requests["cpu"] = new_limit
    if '--nthreads' in cluster.pod_template.spec.containers[0].args:
        index = cluster.pod_template.spec.containers[0].args.index('--nthreads')
        cluster.pod_template.spec.containers[0].args[index + 1] = new_limit
    return cluster

This allows me to adjust the worker template after the cluster has been created and all new workers will follow the updated values.

I'm considering how to add this functionality into the core project. I'm inspired by the dask-jobqueue SLURMCluster which allows you to specify cores and memory as kwargs. Therefore perhaps @mrocklin, @jhamman or @guillaumeeb have thoughts.

Before I go charging in to raise a PR I would like to discuss options.

  • Would it be useful to add methods to the KubeCluster object to update sizes after creation as I am above?
  • Should we add kwargs to the cluster init and if so should they create the cluster and use the helpers or update the config before creation?
  • Are there any other ways of specifying memory and cpu that I haven't captured in the examples above?
@mrocklin
Copy link
Member

mrocklin commented Jul 20, 2018 via email

@guillaumeeb
Copy link
Member

I'd be happy to help here, but I don't really know how to... I've never used KubeCluster yet, but I'm planning to give it a go in the next few months. From my limited point of view, I'd like to be able to easily specify my worker size when initializing a cluster, so if this is not easily possible yet, an improvement would be welcome.

One thought: does or may this affect Cloud VM flavor also? Are these hard coded in another place when defining the Kubernetes cluster? It will be good if by specifying cpu, mem, and perhaps flavor, the automatically created VM are adapted to those.

@jacobtomlinson
Copy link
Member Author

jacobtomlinson commented Aug 1, 2018 •
edited

@guillaumeeb you currently specify the whole worker config when you initialise the cluster and can configure everything from resources to docker image.

The admin of the cluster can also specify a default config to use if the user doesn't specify one. This issue is discussing how to make tweaks to resources in the default config on the fly without having to specify a whole new config.

@guillaumeeb
Copy link
Member

@jacobtomlinson does Kubernetes chose by itself the machine type to use according to what is specified in the pod spec? Looking at http://dask-kubernetes.readthedocs.io/en/latest/#quickstart, I don't see anything like n1-standard-2 or another flavor.

Sorry if this is a dumb question, I still need to learn how Kubernetes is working.

@jacobtomlinson
Copy link
Member Author

jacobtomlinson commented Aug 1, 2018 via email

@guillaumeeb
Copy link
Member

So looking at core.py, currently the easiest way to do what you say is to use the following method:

pod_spec = make_pod_spec(image='daskdev/dask:latest',
                         memory_limit='4G', memory_request='4G',
                         cpu_limit=1, cpu_request=1,
                         env={'EXTRA_PIP_PACKAGES': 'fastparquet git+https://github.com/dask/distributed'})

Which implies that you understand all the rest.

Again I've only a limited view of all this, but it sounds like it would be a welcome functionality here. The way I would do that is to add your update methods or an equivalent, and call that in the __init__ with associated kwargs, just after pod_template is secured, somewhere here https://github.com/dask/dask-kubernetes/blob/master/dask_kubernetes/core.py#L174, or below.

Updating after creation looks like and edge case to me, this is not what we are doing in dask-jobqueue. But I can understand in some situation you might want to do this.

@jacobtomlinson
Copy link
Member Author

jacobtomlinson commented Aug 2, 2018 •
edited

Yes I agree that updating it after creating is an edge case. Perhaps a better way would be to allow users to call make_pod_spec with missing kwargs that get filled in from the defaults.

E.g

pod_spec = make_pod_spec(memory_limit='4G', memory_request='4G',
                         cpu_limit=1, cpu_request=1)

In this example the image and extra packages would come from the default. There is no reason why a scientist should even know what a docker image is.

@jhamman jhamman mentioned this issue Aug 7, 2018
Closed
@mturok
Copy link

mturok commented Aug 14, 2018 •
edited

Not sure it's such an edge case. It's possible to think about running a single dask scheduler for multiple tasks, each of which may have a slightly different set of arguments/variables.

@jacobtomlinson
Copy link
Member Author

@mturok interesting point. However the example I put above would modify the scheduler which would not be great for multi-use clusters.

@yuvipanda
Copy link
Collaborator

As long as the pods get created with matching values, there should be no pathological cases.

@mamoit
Copy link

mamoit commented Nov 27, 2018

Quick question, that may be related to this issue.
Would it be possible to use multiple pod specs and scale them according to what the client requests?
Like the use case described here for the pure dask distributed.

client.submit(process, d, resources={'GPU': 1})

The main use case is to have some instances with GPUs and others without, and use them as needed.

I guess this would require another scheduling layer to fit the work in the smallest pool possible, plus some logic for when a pool is maxed out but the other one is not.

@jacobtomlinson
Copy link
Member Author

That is an interesting idea but would involve upstream changes in dask and distributed. For now I would just create multiple clusters.

@mrocklin I'm sure you are very busy at the moment do you have any thoughts on this?

@guillaumeeb
Copy link
Member

We are thinking about this in dask/distributed#2118 and dask/distributed#2208 (comment).

But we didn't talk about adaptive part yet ("scale them according to what the client requests"), which would probably need modifications too.

@mrocklin
Copy link
Member

I think that this seems like a reasonable request. I think that it would require additional logic to the Adaptive class that looked at resources when making requests. It's non-trivial work, but seems reasonably in-scope.

@jacobtomlinson
Copy link
Member Author

As work is ongoing in the issues that @guillaumeeb mentioned and much of the scaling logic in dask-kubernetes has been replaced with SpecCluster from distributed I'm going to close this in favour of dask/distributed#2118 in particular.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@yuvipanda @mrocklin @jacobtomlinson @mamoit @mturok @guillaumeeb