Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[tune] Add Fractional GPU example/docs #3169

Merged
merged 3 commits into from
Nov 1, 2018
Merged

[tune] Add Fractional GPU example/docs #3169

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
6f84ab3
Add example for fractional GPU support
richardliaw Oct 30, 2018
e84fb56
Update tune_mnist_keras.py
richardliaw Oct 31, 2018
eec4292
Update doc/source/tune-usage.rst
richardliaw Oct 31, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions doc/source/tune-usage.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -193,9 +193,12 @@ Using GPUs (Resource Allocation)

Tune will allocate the specified GPU and CPU ``trial_resources`` to each individual trial (defaulting to 1 CPU per trial). Under the hood, Tune runs each trial as a Ray actor, using Ray's resource handling to allocate resources and place actors. A trial will not be scheduled unless at least that amount of resources is available in the cluster, preventing the cluster from being overloaded.

Fractional values are also supported, (i.e., ``"gpu": 0.2``). You can find an example of this in the `Keras MNIST example <https://github.com/ray-project/ray/blob/master/python/ray/tune/examples/tune_mnist_keras.py>`__.

If GPU resources are not requested, the ``CUDA_VISIBLE_DEVICES`` environment variable will be set as empty, disallowing GPU access.
Otherwise, it will be set to the GPUs in the list (this is managed by Ray).


If your trainable function / class creates further Ray actors or tasks that also consume CPU / GPU resources, you will also want to set ``extra_cpu`` or ``extra_gpu`` to reserve extra resource slots for the actors you will create. For example, if a trainable class requires 1 GPU itself, but will launch 4 actors each using another GPU, then it should set ``"gpu": 1, "extra_gpu": 4``.

.. code-block:: python
Expand Down
10 changes: 8 additions & 2 deletions python/ray/tune/examples/tune_mnist_keras.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,8 @@ def create_parser():
parser = argparse.ArgumentParser(description='Keras MNIST Example')
parser.add_argument(
"--smoke-test", action="store_true", help="Finish quickly for testing")
parser.add_argument(
"--use-gpu", action="store_true", help="Use GPU in training.")
parser.add_argument(
'--jobs',
type=int,
Expand All @@ -113,8 +115,8 @@ def create_parser():
parser.add_argument(
'--threads',
type=int,
default=None,
help='threads used in operations (default: all)')
default=2,
help='threads used in operations (default: 2)')
parser.add_argument(
'--steps',
type=float,
Expand Down Expand Up @@ -185,6 +187,10 @@ def create_parser():
},
"run": "train_mnist",
"num_samples": 1 if args.smoke_test else 10,
"trial_resources": {
"cpu": args.threads,
"gpu": 0.5 if args.use_gpu else 0
},
"config": {
"lr": lambda spec: np.random.uniform(0.001, 0.1),
"momentum": lambda spec: np.random.uniform(0.1, 0.9),
Expand Down