write tutorials to specify the standard of Batch #142

youkaichao · 2020-07-16T11:58:52Z

write tutorials to specify the standard of Batch
bugfix for shape of Batch(a=1), shape property do not raise exception
type conversions are now consistent for all setters (constructor, setattr, and setitem)
scalars are converted to numpy scalars automatically

docs/tutorials/batch.rst

docs/index.rst

docs/tutorials/batch.rst

tianshou/data/batch.py

youkaichao · 2020-07-17T15:47:03Z

@duburcqa ready for review now, it is done. I think the explanation of Aggregation of Heterogeneous Batches is crystally clear now :)

tianshou/data/batch.py

youkaichao · 2020-07-18T01:11:14Z

Modifications to batch.py is a byproduct of this PR. The focus is to write that tutorial. So please take a look at batch.rst file if possible 💗 @Trinkle23897 @duburcqa

If you do not have time to compile the tutorial, I have prepared a pdf version here :)

Understand Batch — Tianshou 0.2.4 documentation.pdf

codecov-commenter · 2020-07-18T01:13:37Z

Codecov Report

Merging #142 into dev will increase coverage by 0.08%.
The diff coverage is 94.23%.

@@            Coverage Diff             @@
##              dev     #142      +/-   ##
==========================================
+ Coverage   88.69%   88.77%   +0.08%     
==========================================
  Files          31       31              
  Lines        2026     2023       -3     
==========================================
- Hits         1797     1796       -1     
+ Misses        229      227       -2

Flag	Coverage Δ
#unittests	`88.77% <94.23%> (+0.08%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
tianshou/data/buffer.py	`92.17% <ø> (ø)`
tianshou/data/batch.py	`94.87% <94.23%> (+0.76%)`	⬆️
tianshou/data/utils.py	`83.33% <0.00%> (-2.78%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f8ad6df...39a393e. Read the comment docs.

docs/tutorials/batch.rst

duburcqa · 2020-07-18T07:33:19Z

Modifications to batch.py is a byproduct of this PR. The focus is to write that tutorial. So please take a look at batch.rst file if possible heartpulse @Trinkle23897 @duburcqa

I'm afraid that maybe it is good, but it is way too long to read it for now. 5 or 6 pages is the maximum. No way someone new with the lib will spend so much time reading this whereas it is NOT relevant for most user. Basic user does not care how Batch works, only the learning algorithms.

Start with pictures and short statement as must as possible, and the pictures are too big.
Use the minimum number of examples, and introduce them sometime before the explanations to draw the attention.
Remove everything that is not necessary to understand how it works and what it does. There are many "note" that you could remove. Find another place to put such details or simply provide an example.
Delete things that I don't necessary to understand how it works, such as length and shape, and maybe aggregation.

youkaichao · 2020-07-18T14:14:31Z

7.5 pages in 316914b : 3 pages for basic and 4.5 pages for advanced. Most advice is taken. Aggregation of Heterogeneous Batches is kept because I do not know how to be more concise :(

Understand Batch — Tianshou 0.2.4 documentation.pdf

duburcqa · 2020-07-18T14:31:57Z

Updated comments: Understand.Batch.Tianshou.0.2.4.documentation.pdf

youkaichao · 2020-07-18T15:37:23Z

8 pages now: 3 for basic and 5 for advanced. The layout in pdf is malformed, but it is ok in html pages.

Move figure to the top; Trim the description of aggregation; add a figure for aggregation.

Understand Batch — Tianshou 0.2.4 documentation.pdf

youkaichao · 2020-07-18T17:14:40Z

I find many inconsistent type conversions and delegate all of them to Batch.__init__ to both eliminate duplicate code and keep type consistency in 67068dc and b000381 .

tianshou/data/batch.py

youkaichao · 2020-07-19T00:10:51Z

Any further comments for thhe tutorial?

Trinkle23897 · 2020-07-19T02:30:18Z

Any further comments for thhe tutorial?

New version, please have a look

Understand Batch — Tianshou 0.2.4 documentation.pdf

I have no other comments

tianshou/data/batch.py

* add doc for len exceptions * doc move; unify is_scalar_value function * remove some issubclass check * bugfix for shape of Batch(a=1) * keep moving doc * keep writing batch tutorial * draft version of Batch tutorial done * improving doc * keep improving doc * batch tutorial done * rename _is_number * rename _is_scalar * shape property do not raise exception * restore some doc string * grammarly [ci skip] * grammarly + fix warning of building docs * polish docs * trim and re-arrange batch tutorial * go straight to the point * minor fix for batch doc * add shape / len in basic usage * keep improving tutorial * unify _to_array_with_correct_type to remove duplicate code * delegate type convertion to Batch.__init__ * further delegate type convertion to Batch.__init__ * bugfix for setattr * add a _parse_value function * remove dummy function call * polish docs Co-authored-by: Trinkle23897 <463003665@qq.com>

* make fileds with empty Batch rather than None after reset * dummy code * remove dummy * add reward_length argument for collector * Improve Batch (#126) * make sure the key type of Batch is string, and add unit tests * add is_empty() function and unit tests * enable cat of mixing dict and Batch, just like stack * bugfix for reward_length * add get_final_reward_fn argument to collector to deal with marl * minor polish * remove multibuf * minor polish * improve and implement Batch.cat_ * bugfix for buffer.sample with field impt_weight * restore the usage of a.cat_(b) * fix 2 bugs in batch and add corresponding unittest * code fix for update * update is_empty to recognize empty over empty; bugfix for len * bugfix for update and add testcase * add testcase of update * make fileds with empty Batch rather than None after reset * dummy code * remove dummy * add reward_length argument for collector * bugfix for reward_length * add get_final_reward_fn argument to collector to deal with marl * make sure the key type of Batch is string, and add unit tests * add is_empty() function and unit tests * enable cat of mixing dict and Batch, just like stack * dummy code * remove dummy * add multi-agent example: tic-tac-toe * move TicTacToeEnv to a separate file * remove dummy MANet * code refactor * move tic-tac-toe example to test * update doc with marl-example * fix docs * reduce the threshold * revert * update player id to start from 1 and change player to agent; keep coding * add reward_length argument for collector * Improve Batch (#128) * minor polish * improve and implement Batch.cat_ * bugfix for buffer.sample with field impt_weight * restore the usage of a.cat_(b) * fix 2 bugs in batch and add corresponding unittest * code fix for update * update is_empty to recognize empty over empty; bugfix for len * bugfix for update and add testcase * add testcase of update * fix docs * fix docs * fix docs [ci skip] * fix docs [ci skip] Co-authored-by: Trinkle23897 <463003665@qq.com> * refact * re-implement Batch.stack and add testcases * add doc for Batch.stack * reward_metric * modify flag * minor fix * reuse _create_values and refactor stack_ & cat_ * fix pep8 * fix reward stat in collector * fix stat of collector, simplify test/base/env.py * fix docs * minor fix * raise exception for stacking with partial keys and axis!=0 * minor fix * minor fix * minor fix * marl-examples * add condense; bugfix for torch.Tensor; code refactor * marl example can run now * enable tic tac toe with larger board size and win-size * add test dependency * Fix padding of inconsistent keys with Batch.stack and Batch.cat (#130) * re-implement Batch.stack and add testcases * add doc for Batch.stack * reuse _create_values and refactor stack_ & cat_ * fix pep8 * fix docs * raise exception for stacking with partial keys and axis!=0 * minor fix * minor fix Co-authored-by: Trinkle23897 <463003665@qq.com> * stash * let agent learn to play as agent 2 which is harder * code refactor * Improve collector (#125) * remove multibuf * reward_metric * make fileds with empty Batch rather than None after reset * many fixes and refactor Co-authored-by: Trinkle23897 <463003665@qq.com> * marl for tic-tac-toe and general gomoku * update default gamma to 0.1 for tic tac toe to win earlier * fix name typo; change default game config; add rew_norm option * fix pep8 * test commit * mv test dir name * add rew flag * fix torch.optim import error and madqn rew_norm * remove useless kwargs * Vector env enable select worker (#132) * Enable selecting worker for vector env step method. * Update collector to match new vecenv selective worker behavior. * Bug fix. * Fix rebase Co-authored-by: Alexis Duburcq <alexis.duburcq@wandercraft.eu> * show the last move of tictactoe by capital letters * add multi-agent tutorial * fix link * Standardized behavior of Batch.cat and misc code refactor (#137) * code refactor; remove unused kwargs; add reward_normalization for dqn * bugfix for __setitem__ with torch.Tensor; add Batch.condense * minor fix * support cat with empty Batch * remove the dependency of is_empty on len; specify the semantic of empty Batch by test cases * support stack with empty Batch * remove condense * refactor code to reflect the shared / partial / reserved categories of keys * add is_empty(recursive=False) * doc fix * docfix and bugfix for _is_batch_set * add doc for key reservation * bugfix for algebra operators * fix cat with lens hint * code refactor * bugfix for storing None * use ValueError instead of exception * hide lens away from users * add comment for __cat * move the computation of the initial value of lens in cat_ itself. * change the place of doc string * doc fix for Batch doc string * change recursive to recurse * doc string fix * minor fix for batch doc * write tutorials to specify the standard of Batch (#142) * add doc for len exceptions * doc move; unify is_scalar_value function * remove some issubclass check * bugfix for shape of Batch(a=1) * keep moving doc * keep writing batch tutorial * draft version of Batch tutorial done * improving doc * keep improving doc * batch tutorial done * rename _is_number * rename _is_scalar * shape property do not raise exception * restore some doc string * grammarly [ci skip] * grammarly + fix warning of building docs * polish docs * trim and re-arrange batch tutorial * go straight to the point * minor fix for batch doc * add shape / len in basic usage * keep improving tutorial * unify _to_array_with_correct_type to remove duplicate code * delegate type convertion to Batch.__init__ * further delegate type convertion to Batch.__init__ * bugfix for setattr * add a _parse_value function * remove dummy function call * polish docs Co-authored-by: Trinkle23897 <463003665@qq.com> * bugfix for mapolicy * pretty code * remove debug code; remove condense * doc fix * check before get_agents in tutorials/tictactoe * tutorial * fix * minor fix for batch doc * minor polish * faster test_ttt * improve tic-tac-toe environment * change default epoch and step-per-epoch for tic-tac-toe * fix mapolicy * minor polish for mapolicy * 90% to 80% (need to change the tutorial) * win rate * show step number at board * simplify mapolicy * minor polish for mapolicy * remove MADQN * fix pep8 * change legal_actions to mask (need to update docs) * simplify maenv * fix typo * move basevecenv to single file * separate RandomAgent * update docs * grammarly * fix pep8 * win rate typo * format in cheatsheet * use bool mask directly * update doc for boolean mask Co-authored-by: Trinkle23897 <463003665@qq.com> Co-authored-by: Alexis DUBURCQ <alexis.duburcq@gmail.com> Co-authored-by: Alexis Duburcq <alexis.duburcq@wandercraft.eu>

* add doc for len exceptions * doc move; unify is_scalar_value function * remove some issubclass check * bugfix for shape of Batch(a=1) * keep moving doc * keep writing batch tutorial * draft version of Batch tutorial done * improving doc * keep improving doc * batch tutorial done * rename _is_number * rename _is_scalar * shape property do not raise exception * restore some doc string * grammarly [ci skip] * grammarly + fix warning of building docs * polish docs * trim and re-arrange batch tutorial * go straight to the point * minor fix for batch doc * add shape / len in basic usage * keep improving tutorial * unify _to_array_with_correct_type to remove duplicate code * delegate type convertion to Batch.__init__ * further delegate type convertion to Batch.__init__ * bugfix for setattr * add a _parse_value function * remove dummy function call * polish docs Co-authored-by: Trinkle23897 <463003665@qq.com>

* make fileds with empty Batch rather than None after reset * dummy code * remove dummy * add reward_length argument for collector * Improve Batch (thu-ml#126) * make sure the key type of Batch is string, and add unit tests * add is_empty() function and unit tests * enable cat of mixing dict and Batch, just like stack * bugfix for reward_length * add get_final_reward_fn argument to collector to deal with marl * minor polish * remove multibuf * minor polish * improve and implement Batch.cat_ * bugfix for buffer.sample with field impt_weight * restore the usage of a.cat_(b) * fix 2 bugs in batch and add corresponding unittest * code fix for update * update is_empty to recognize empty over empty; bugfix for len * bugfix for update and add testcase * add testcase of update * make fileds with empty Batch rather than None after reset * dummy code * remove dummy * add reward_length argument for collector * bugfix for reward_length * add get_final_reward_fn argument to collector to deal with marl * make sure the key type of Batch is string, and add unit tests * add is_empty() function and unit tests * enable cat of mixing dict and Batch, just like stack * dummy code * remove dummy * add multi-agent example: tic-tac-toe * move TicTacToeEnv to a separate file * remove dummy MANet * code refactor * move tic-tac-toe example to test * update doc with marl-example * fix docs * reduce the threshold * revert * update player id to start from 1 and change player to agent; keep coding * add reward_length argument for collector * Improve Batch (thu-ml#128) * minor polish * improve and implement Batch.cat_ * bugfix for buffer.sample with field impt_weight * restore the usage of a.cat_(b) * fix 2 bugs in batch and add corresponding unittest * code fix for update * update is_empty to recognize empty over empty; bugfix for len * bugfix for update and add testcase * add testcase of update * fix docs * fix docs * fix docs [ci skip] * fix docs [ci skip] Co-authored-by: Trinkle23897 <463003665@qq.com> * refact * re-implement Batch.stack and add testcases * add doc for Batch.stack * reward_metric * modify flag * minor fix * reuse _create_values and refactor stack_ & cat_ * fix pep8 * fix reward stat in collector * fix stat of collector, simplify test/base/env.py * fix docs * minor fix * raise exception for stacking with partial keys and axis!=0 * minor fix * minor fix * minor fix * marl-examples * add condense; bugfix for torch.Tensor; code refactor * marl example can run now * enable tic tac toe with larger board size and win-size * add test dependency * Fix padding of inconsistent keys with Batch.stack and Batch.cat (thu-ml#130) * re-implement Batch.stack and add testcases * add doc for Batch.stack * reuse _create_values and refactor stack_ & cat_ * fix pep8 * fix docs * raise exception for stacking with partial keys and axis!=0 * minor fix * minor fix Co-authored-by: Trinkle23897 <463003665@qq.com> * stash * let agent learn to play as agent 2 which is harder * code refactor * Improve collector (thu-ml#125) * remove multibuf * reward_metric * make fileds with empty Batch rather than None after reset * many fixes and refactor Co-authored-by: Trinkle23897 <463003665@qq.com> * marl for tic-tac-toe and general gomoku * update default gamma to 0.1 for tic tac toe to win earlier * fix name typo; change default game config; add rew_norm option * fix pep8 * test commit * mv test dir name * add rew flag * fix torch.optim import error and madqn rew_norm * remove useless kwargs * Vector env enable select worker (thu-ml#132) * Enable selecting worker for vector env step method. * Update collector to match new vecenv selective worker behavior. * Bug fix. * Fix rebase Co-authored-by: Alexis Duburcq <alexis.duburcq@wandercraft.eu> * show the last move of tictactoe by capital letters * add multi-agent tutorial * fix link * Standardized behavior of Batch.cat and misc code refactor (thu-ml#137) * code refactor; remove unused kwargs; add reward_normalization for dqn * bugfix for __setitem__ with torch.Tensor; add Batch.condense * minor fix * support cat with empty Batch * remove the dependency of is_empty on len; specify the semantic of empty Batch by test cases * support stack with empty Batch * remove condense * refactor code to reflect the shared / partial / reserved categories of keys * add is_empty(recursive=False) * doc fix * docfix and bugfix for _is_batch_set * add doc for key reservation * bugfix for algebra operators * fix cat with lens hint * code refactor * bugfix for storing None * use ValueError instead of exception * hide lens away from users * add comment for __cat * move the computation of the initial value of lens in cat_ itself. * change the place of doc string * doc fix for Batch doc string * change recursive to recurse * doc string fix * minor fix for batch doc * write tutorials to specify the standard of Batch (thu-ml#142) * add doc for len exceptions * doc move; unify is_scalar_value function * remove some issubclass check * bugfix for shape of Batch(a=1) * keep moving doc * keep writing batch tutorial * draft version of Batch tutorial done * improving doc * keep improving doc * batch tutorial done * rename _is_number * rename _is_scalar * shape property do not raise exception * restore some doc string * grammarly [ci skip] * grammarly + fix warning of building docs * polish docs * trim and re-arrange batch tutorial * go straight to the point * minor fix for batch doc * add shape / len in basic usage * keep improving tutorial * unify _to_array_with_correct_type to remove duplicate code * delegate type convertion to Batch.__init__ * further delegate type convertion to Batch.__init__ * bugfix for setattr * add a _parse_value function * remove dummy function call * polish docs Co-authored-by: Trinkle23897 <463003665@qq.com> * bugfix for mapolicy * pretty code * remove debug code; remove condense * doc fix * check before get_agents in tutorials/tictactoe * tutorial * fix * minor fix for batch doc * minor polish * faster test_ttt * improve tic-tac-toe environment * change default epoch and step-per-epoch for tic-tac-toe * fix mapolicy * minor polish for mapolicy * 90% to 80% (need to change the tutorial) * win rate * show step number at board * simplify mapolicy * minor polish for mapolicy * remove MADQN * fix pep8 * change legal_actions to mask (need to update docs) * simplify maenv * fix typo * move basevecenv to single file * separate RandomAgent * update docs * grammarly * fix pep8 * win rate typo * format in cheatsheet * use bool mask directly * update doc for boolean mask Co-authored-by: Trinkle23897 <463003665@qq.com> Co-authored-by: Alexis DUBURCQ <alexis.duburcq@gmail.com> Co-authored-by: Alexis Duburcq <alexis.duburcq@wandercraft.eu>

youkaichao added 7 commits July 16, 2020 19:57

add doc for len exceptions

3a565b1

doc move; unify is_scalar_value function

1f7964b

remove some issubclass check

1598a01

bugfix for shape of Batch(a=1)

62fda2f

keep moving doc

32ce479

keep writing batch tutorial

9ae8211

draft version of Batch tutorial done

548fdfb

Trinkle23897 reviewed Jul 17, 2020

View reviewed changes

improving doc

35827bc

youkaichao requested a review from duburcqa July 17, 2020 15:44

keep improving doc

8775e98

youkaichao changed the title ~~WIP: write tutorials to specify the standard of Batch~~ write tutorials to specify the standard of Batch Jul 17, 2020

batch tutorial done

679d966

youkaichao mentioned this pull request Jul 17, 2020

The length of empty Batch #138

Closed

duburcqa reviewed Jul 17, 2020

View reviewed changes

tianshou/data/batch.py Outdated Show resolved Hide resolved

duburcqa reviewed Jul 17, 2020

View reviewed changes

tianshou/data/batch.py Outdated Show resolved Hide resolved

duburcqa reviewed Jul 17, 2020

View reviewed changes

tianshou/data/batch.py Outdated Show resolved Hide resolved

tianshou/data/batch.py Outdated Show resolved Hide resolved

tianshou/data/batch.py Outdated Show resolved Hide resolved

duburcqa reviewed Jul 17, 2020

View reviewed changes

tianshou/data/batch.py Outdated Show resolved Hide resolved

youkaichao added 4 commits July 18, 2020 01:18

rename _is_number

44034b6

rename _is_scalar

94f69b1

shape property do not raise exception

6ec46c1

restore some doc string

17d907a

Trinkle23897 added 3 commits July 18, 2020 12:36

grammarly [ci skip]

1b2b087

grammarly + fix warning of building docs

80fa524

polish docs

d112f5a

Trinkle23897 reviewed Jul 18, 2020

View reviewed changes

docs/tutorials/batch.rst Outdated Show resolved Hide resolved

go straight to the point

316914b

Trinkle23897 and others added 2 commits July 18, 2020 22:41

minor fix for batch doc

c5755ee

add shape / len in basic usage

40660a9

youkaichao added 4 commits July 18, 2020 23:38

keep improving tutorial

c70a66f

unify _to_array_with_correct_type to remove duplicate code

38980fd

delegate type convertion to Batch.__init__

b000381

further delegate type convertion to Batch.__init__

67068dc

bugfix for setattr

736023a

duburcqa reviewed Jul 18, 2020

View reviewed changes

tianshou/data/batch.py Outdated Show resolved Hide resolved

tianshou/data/batch.py Outdated Show resolved Hide resolved

tianshou/data/batch.py Show resolved Hide resolved

youkaichao added 2 commits July 19, 2020 08:01

add a _parse_value function

c7ed637

remove dummy function call

39a393e

polish docs

67aee9e

duburcqa approved these changes Jul 19, 2020

View reviewed changes

duburcqa reviewed Jul 19, 2020

View reviewed changes

tianshou/data/batch.py Show resolved Hide resolved

tianshou/data/batch.py Show resolved Hide resolved

Trinkle23897 approved these changes Jul 19, 2020

View reviewed changes

Trinkle23897 merged commit fa542f8 into thu-ml:dev Jul 19, 2020

youkaichao deleted the batch_tutorial branch July 19, 2020 07:53

This was referenced Jul 24, 2020

Asynchronous sampling vector environment #134

Merged

assignment with heterogeneous batches #163

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

write tutorials to specify the standard of Batch #142

write tutorials to specify the standard of Batch #142

youkaichao commented Jul 16, 2020 •

edited

youkaichao commented Jul 17, 2020 •

edited

youkaichao commented Jul 18, 2020

codecov-commenter commented Jul 18, 2020 •

edited

duburcqa commented Jul 18, 2020

youkaichao commented Jul 18, 2020 •

edited

duburcqa commented Jul 18, 2020 •

edited

youkaichao commented Jul 18, 2020 •

edited

youkaichao commented Jul 18, 2020

youkaichao commented Jul 19, 2020

Trinkle23897 commented Jul 19, 2020 •

edited

write tutorials to specify the standard of Batch #142

write tutorials to specify the standard of Batch #142

Conversation

youkaichao commented Jul 16, 2020 • edited

youkaichao commented Jul 17, 2020 • edited

youkaichao commented Jul 18, 2020

codecov-commenter commented Jul 18, 2020 • edited

Codecov Report

duburcqa commented Jul 18, 2020

youkaichao commented Jul 18, 2020 • edited

duburcqa commented Jul 18, 2020 • edited

youkaichao commented Jul 18, 2020 • edited

youkaichao commented Jul 18, 2020

youkaichao commented Jul 19, 2020

Trinkle23897 commented Jul 19, 2020 • edited

youkaichao commented Jul 16, 2020 •

edited

youkaichao commented Jul 17, 2020 •

edited

codecov-commenter commented Jul 18, 2020 •

edited

youkaichao commented Jul 18, 2020 •

edited

duburcqa commented Jul 18, 2020 •

edited

youkaichao commented Jul 18, 2020 •

edited

Trinkle23897 commented Jul 19, 2020 •

edited