[raysgd] Cleanup User API #7384

richardliaw · 2020-02-29T23:22:10Z

Why are these changes needed?

improves usability of the trainer with the following:

data_creator returns iterables/loaders rather than datasets.
automatically sets "sampler" when returning a DataLoader.
remove data_loader_config
remove batch_size (becomes hyperparameter)
rename "replica" to "worker"
adds timing profiles to the stats
automatically averages custom stats
Make loss function optional (Only if passing in a custom operator)
Forces all Trainer args to be named

TODO:

Add UnifiedLogger to Trainer by default

Related issue number

Checks

I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://ray.readthedocs.io/en/latest/.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failure rates at https://ray-travis-tracker.herokuapp.com/.

…o fp16_pytorch

Co-Authored-By: Edward Oakes <ed.nmi.oakes@gmail.com>

AmplabJenkins · 2020-03-05T22:07:01Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/22782/
Test FAILed.

AmplabJenkins · 2020-03-05T22:14:51Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/22783/
Test FAILed.

AmplabJenkins · 2020-03-06T00:53:18Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/22789/
Test FAILed.

AmplabJenkins · 2020-03-07T08:37:09Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/22843/
Test FAILed.

AmplabJenkins · 2020-03-07T19:58:11Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/22852/
Test FAILed.

AmplabJenkins · 2020-03-07T20:52:18Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/22853/
Test FAILed.

edoakes

LGTM but mostly trusting @maximsmol because I'm not too familiar

edoakes · 2020-03-09T23:30:14Z

python/ray/util/sgd/tests/test_torch.py

+            model_creator=model_creator,
+            data_creator=single_loader,
+            optimizer_creator=optimizer_creator,
+            config=dict(batch_size=100000),


Suggested change

config=dict(batch_size=100000),

config={batch_size: 100000},

To match others

same with below

edoakes · 2020-03-09T23:30:51Z

python/ray/util/sgd/torch/torch_runner.py

+                return loaders
+            else:
+                raise ValueError(
+                    "loaders must be <= 2. Got {}".format(loaders))


Suggested change

"loaders must be <= 2. Got {}".format(loaders))

"number of loaders must be <= 2. Got {}".format(loaders))

edoakes · 2020-03-09T23:31:57Z

python/ray/util/sgd/torch/torch_runner.py

+        logger.debug("Instantiating dataloaders.")
+        # When creating loaders, a filelock will be used to ensure no
+        # race conditions in data downloading among different workers.
+        with FileLock(os.path.expanduser("~/.ray_data.lock")):


Should this be in a tmpfile to avoid issues with persistence (e.g., if the machine crashes)? Not sure, not familiar with FileLock

AmplabJenkins · 2020-03-10T01:36:51Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/22922/
Test FAILed.

AmplabJenkins · 2020-03-10T05:22:36Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/22927/
Test FAILed.

AmplabJenkins · 2020-03-10T06:16:09Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/22931/
Test FAILed.

AmplabJenkins · 2020-03-10T07:33:59Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/22936/
Test FAILed.

AmplabJenkins · 2020-03-10T08:57:02Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/22941/
Test FAILed.

richardliaw and others added 30 commits February 5, 2020 09:24

Init fp16

df57735

fp16 and schedulers

2342e76

scheduler linking and fp16

49abded

to fp16

cc4465f

loss scaling and documentation

00f9926

more documentation

bee750a

add tests, refactor config

80e43a8

moredocs

9a89b89

more docs

1d20075

fix logo, add test mode, add fp16 flag

3e98f6d

fix tests

168b813

fix scheduler

9a77e49

fix apex

fa1eaaa

improve safety

bea67f6

fix tests

420a2f2

fix tests

f53e9e6

remove pin memory default

fce88a5

rm

39f5328

fix

f8caf4b

Update doc/examples/doc_code/raysgd_torch_signatures.py

9d76e21

fix

b7ca982

Merge branch 'fp16_pytorch' of https://github.com/richardliaw/ray int…

c278e9b

…o fp16_pytorch

migrate changes from other PR

dff9521

ok thanks

20428a7

pass

62ccf6d

signatures

0c388fa

lint'

beeae3c

Update python/ray/experimental/sgd/pytorch/utils.py

748a553

Apply suggestions from code review

acacee0

Co-Authored-By: Edward Oakes <ed.nmi.oakes@gmail.com>

should address most comments

daf5794

richardliaw added 2 commits March 5, 2020 16:15

nitdocs

11ab4f2

loss

1946701

richardliaw marked this pull request as ready for review March 6, 2020 04:47

richardliaw changed the title ~~[raysgd] Cleanups~~ [raysgd] Cleanup User API Mar 6, 2020

comments

82cb45a

richardliaw mentioned this pull request Mar 6, 2020

[raysgd] Utility Function for dataloaders #7495

Closed

Merge branch 'master' into benchmark

680db9b

richardliaw added 2 commits March 7, 2020 12:06

fix

4bb08c7

fix

92860ea

richardliaw assigned edoakes Mar 7, 2020

edoakes approved these changes Mar 9, 2020

View reviewed changes

runner_tests

cda4bdf

richardliaw added 2 commits March 9, 2020 21:45

codes

b8576f2

example

03eda70

fix_test

6be5237

fix

9604309

tests

6b44ece

richardliaw merged commit d192ef0 into ray-project:master Mar 10, 2020

richardliaw deleted the benchmark branch March 10, 2020 15:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[raysgd] Cleanup User API #7384

[raysgd] Cleanup User API #7384

richardliaw commented Feb 29, 2020 •

edited

Loading

AmplabJenkins commented Mar 5, 2020

AmplabJenkins commented Mar 5, 2020

AmplabJenkins commented Mar 6, 2020

AmplabJenkins commented Mar 7, 2020

AmplabJenkins commented Mar 7, 2020

AmplabJenkins commented Mar 7, 2020

edoakes left a comment

edoakes Mar 9, 2020

edoakes Mar 9, 2020

edoakes Mar 9, 2020

edoakes Mar 9, 2020

AmplabJenkins commented Mar 10, 2020

AmplabJenkins commented Mar 10, 2020

AmplabJenkins commented Mar 10, 2020

AmplabJenkins commented Mar 10, 2020

AmplabJenkins commented Mar 10, 2020

	"loaders must be <= 2. Got {}".format(loaders))
	"number of loaders must be <= 2. Got {}".format(loaders))

[raysgd] Cleanup User API #7384

[raysgd] Cleanup User API #7384

Conversation

richardliaw commented Feb 29, 2020 • edited Loading

Why are these changes needed?

Related issue number

Checks

AmplabJenkins commented Mar 5, 2020

AmplabJenkins commented Mar 5, 2020

AmplabJenkins commented Mar 6, 2020

AmplabJenkins commented Mar 7, 2020

AmplabJenkins commented Mar 7, 2020

AmplabJenkins commented Mar 7, 2020

edoakes left a comment

Choose a reason for hiding this comment

edoakes Mar 9, 2020

Choose a reason for hiding this comment

edoakes Mar 9, 2020

Choose a reason for hiding this comment

edoakes Mar 9, 2020

Choose a reason for hiding this comment

edoakes Mar 9, 2020

Choose a reason for hiding this comment

AmplabJenkins commented Mar 10, 2020

AmplabJenkins commented Mar 10, 2020

AmplabJenkins commented Mar 10, 2020

AmplabJenkins commented Mar 10, 2020

AmplabJenkins commented Mar 10, 2020

richardliaw commented Feb 29, 2020 •

edited

Loading