Fix padding of inconsistent keys with Batch.stack and Batch.cat #130

youkaichao · 2020-07-11T14:25:21Z

For keys that are not shared across all batches, batches that do not have these keys will be padded by zeros. E.g.

>>> a = Batch(a=np.zeros([4, 4]), common=Batch(c=np.zeros([4, 5])))
>>> b = Batch(b=np.zeros([4, 6]), common=Batch(c=np.zeros([4, 5])))
>>> c = Batch.stack([a, b])
>>> c.a.shape
(8, 4)
>>> c.b.shape
(8, 6)
>>> c.common.c.shape
(8, 5)

simplify Batch.cat_
raise exception for stacking with partial keys and axis!=0

tianshou/data/batch.py

duburcqa · 2020-07-11T15:21:18Z

tianshou/data/batch.py

-                        self.__dict__[k] = \
-                            _create_value(val, len(batches))
-                        self.__dict__[k][i] = val
+                    if isinstance(val, (dict, Batch)):


Why replacing something working and efficient by such a complicated nested if/else code ?

Because the current code of stack_ method has different behavior when dealing with inconsistent keys:

In [2]: a = Batch(a=[1], b=[2], c=[3]) ...: b = Batch(a=[4], b=[5], d=[6]) ...: c = Batch(c=[3], b=[5], d=[9]) In [3]: Batch.cat([a,b,c]) Out[3]: Batch( b: array([2, 5, 5]), a: array([1, 4, 0]), c: array([3, 0, 3]), d: array([0, 6, 9]), ) In [4]: Batch.stack([a,b,c]) Out[4]: Batch( b: array([[2], [5], [5]]), )

We expect to get the stacked data with four keys.

It is not related to this part of the code, otherwise this would not work

In [1]: from tianshou.data import Batch In [2]: a = Batch(a=[1], b=[2], c=[3]) In [3]: b = Batch(a=[4], b=[5], d=[6]) In [4]: Batch.stack([a,b]) Out[4]: Batch( a: array([[1], [4]]), b: array([[2], [5]]), d: array([[0], [6]]), c: array([[3], [0]]), )

You can check by yourself that simply adding your modification line 604 set.union(*keys_map) - keys_shared does the trick:

In [1]: from tianshou.data import Batch In [2]: a = Batch(a=[1], b=[2], c=[3]) In [3]: b = Batch(a=[4], b=[5], d=[6]) In [4]: c = Batch(c=[3], b=[5], d=[9]) In [5]: Batch.stack([a,b,c]) Out[5]: Batch( b: array([[2], [5], [5]]), c: array([[3], [0], [3]]), a: array([[1], [4], [0]]), d: array([[0], [6], [9]]), )

Batch.stack([a,b]) works but Batch.stack([a,b,c]) does not work. Maybe there is a bug?

Emm, I'm sorry for this.

Emm, I'm sorry for this.

Don't be. You did nothing wrong but your best I'm pretty sure. Plus, replacing keys_partial = reduce(set.symmetric_difference, keys_map) by keys_partial = set.union(*keys_map) - keys_shared is VERY GOOD and fixes the bug with stack !

I'm not saying that you are doing anything wrong. Just try to keep in mind that open-source is great because it involves a community. So it means several things: many people are happy to spend times to improve the project, BUT, you also need to compromise and sometimes refrain your enthusiasm of changing many stuffs. Other people can only follow small steps. Trust me I know it is frustrating to spend time trying to fix code developed by others instead of doing something new, and that it is time consuming, but it is the only way to get and keep people involved.

Thank you. I have learned so much from you and from this project. ♡

I'll fix them tomorrow :)

Last but not last, you are doing a very good job so far. It just seems you need to learn take time to step back a little bit and get the full picture.

Thank you. I have learned so much from you and from this project. ♡
I'll fix them tomorrow :)

And it is just the beginning! There is a long way ahead of us

tianshou/data/batch.py

youkaichao · 2020-07-11T16:56:24Z

@duburcqa When I take a further look at the code, I find that axis arguments are really annoying. Current code actually does not deal with the axis argument.

try:
    self.__dict__[k][i] = val
except KeyError:
    self.__dict__[k] = \
        _create_value(val, len(batches))
    self.__dict__[k][i] = val

The question is, do we really need to support the axis argument?

Besides testcases, there are only one usage in buffer.py, which is actually not used by any script now.

duburcqa · 2020-07-11T17:04:11Z

The question is, do we really need to support the axis argument?
Besides testcases, there are only one usage in buffer.py, which is actually not used by any script now.

I guess if you can get it over with than you can remove it. I don't think I use it anywhere in my own code. So I encourage you doing it if it helps making things simpler.

youkaichao · 2020-07-11T17:16:22Z

The question is, do we really need to support the axis argument?
Besides testcases, there are only one usage in buffer.py, which is actually not used by any script now.

I guess if you can get it over with than you can remove it. I don't think I use it anywhere in my own code. So I encourage you doing it if it helps making things simpler.

It makes the world much simpler ...
If we support axis, we would need much more code than these 6 lines, but without any practical usage.
What's your opinion, @Trinkle23897

youkaichao · 2020-07-12T02:46:09Z

An example of failure case:

In [109]: a = Batch(a=np.zeros((3, 4))); b=Batch(b=np.zeros((5, 6)))
     ...: c = Batch.stack([a, b], axis=1)
     ...: c.a.shape # should be (3, 2, 4)
Out[109]: (2, 3, 4)

In [110]: c.b.shape # should be (5, 2, 6)
Out[110]: (2, 5, 6)

Trinkle23897 · 2020-07-12T03:06:51Z

The question is, do we really need to support the axis argument?
Besides testcases, there is only one usage in buffer.py, which is actually not used by any script now.

I guess if you can get it over with than you can remove it. I don't think I use it anywhere in my own code. So I encourage you to do it if it helps to make things simpler.

It makes the world much simpler ...
If we support axis, we would need much more code than these 6 lines, but without any practical usage.
What's your opinion, @Trinkle23897

In buffer, current implementation secures the consistency of shape when performing Batch.stack, because when we store the data into the buffer, it will automatically create those padding. So if you do not will to implement, you can raise an exception at the end of Batch.stack_.

codecov-commenter · 2020-07-12T03:20:00Z

Codecov Report

❗ No coverage uploaded for pull request base (dev@d1a2037). Click here to learn what that means.
The diff coverage is n/a.

@@          Coverage Diff           @@
##             dev     #130   +/-   ##
======================================
  Coverage       ?   87.38%           
======================================
  Files          ?       31           
  Lines          ?     2037           
  Branches       ?        0           
======================================
  Hits           ?     1780           
  Misses         ?      257           
  Partials       ?        0

Flag	Coverage Δ
#unittests	`87.38% <0.00%> (?)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d1a2037...a78d4ac. Read the comment docs.

tianshou/data/batch.py

duburcqa · 2020-07-12T15:41:14Z

@youkaichao I just wanted to stress the fact that you did a very nice job with this PR. As you can see, it was more work, but at the end the quality is much higher. With far less modifications, it fixes some bugs, and implement new features, without messing with code base and without any code duplication. That's what I call high-quality contribution. 👍

youkaichao · 2020-07-12T15:48:10Z

@youkaichao I just wanted to stress the fact that you did a very nice job with this PR. As you can see, it was more work, but at the end the quality is much higher. With far less modifications, it fixes some bugs, and implement new features, without messing with code base and without any code duplication. That's what I call high-quality contribution. 👍

Yes, I feel the code becomes much better. Thank you for the comment, otherwise these lines of complicated code will remain and make the maintenance much harder.

I too enjoy the process of collaborating with you in this project.

duburcqa · 2020-07-12T16:18:22Z

otherwise these lines of complicated code will remain and make the maintenance much harder.

Yes, that's my greatest fear when I see comment in any project saying "I will do it later in another PR". Often people forgot and it is never done, and things get messy without really realizing it. Then at some point it is too late and, it is necessary to start over from scratch over and over again.

* re-implement Batch.stack and add testcases * add doc for Batch.stack * reuse _create_values and refactor stack_ & cat_ * fix pep8 * fix docs * raise exception for stacking with partial keys and axis!=0 * minor fix * minor fix Co-authored-by: Trinkle23897 <463003665@qq.com>

…at (#130)" This reverts commit 5599a6d.

* make fileds with empty Batch rather than None after reset * dummy code * remove dummy * add reward_length argument for collector * Improve Batch (#126) * make sure the key type of Batch is string, and add unit tests * add is_empty() function and unit tests * enable cat of mixing dict and Batch, just like stack * bugfix for reward_length * add get_final_reward_fn argument to collector to deal with marl * minor polish * remove multibuf * minor polish * improve and implement Batch.cat_ * bugfix for buffer.sample with field impt_weight * restore the usage of a.cat_(b) * fix 2 bugs in batch and add corresponding unittest * code fix for update * update is_empty to recognize empty over empty; bugfix for len * bugfix for update and add testcase * add testcase of update * make fileds with empty Batch rather than None after reset * dummy code * remove dummy * add reward_length argument for collector * bugfix for reward_length * add get_final_reward_fn argument to collector to deal with marl * make sure the key type of Batch is string, and add unit tests * add is_empty() function and unit tests * enable cat of mixing dict and Batch, just like stack * dummy code * remove dummy * add multi-agent example: tic-tac-toe * move TicTacToeEnv to a separate file * remove dummy MANet * code refactor * move tic-tac-toe example to test * update doc with marl-example * fix docs * reduce the threshold * revert * update player id to start from 1 and change player to agent; keep coding * add reward_length argument for collector * Improve Batch (#128) * minor polish * improve and implement Batch.cat_ * bugfix for buffer.sample with field impt_weight * restore the usage of a.cat_(b) * fix 2 bugs in batch and add corresponding unittest * code fix for update * update is_empty to recognize empty over empty; bugfix for len * bugfix for update and add testcase * add testcase of update * fix docs * fix docs * fix docs [ci skip] * fix docs [ci skip] Co-authored-by: Trinkle23897 <463003665@qq.com> * refact * re-implement Batch.stack and add testcases * add doc for Batch.stack * reward_metric * modify flag * minor fix * reuse _create_values and refactor stack_ & cat_ * fix pep8 * fix reward stat in collector * fix stat of collector, simplify test/base/env.py * fix docs * minor fix * raise exception for stacking with partial keys and axis!=0 * minor fix * minor fix * minor fix * marl-examples * add condense; bugfix for torch.Tensor; code refactor * marl example can run now * enable tic tac toe with larger board size and win-size * add test dependency * Fix padding of inconsistent keys with Batch.stack and Batch.cat (#130) * re-implement Batch.stack and add testcases * add doc for Batch.stack * reuse _create_values and refactor stack_ & cat_ * fix pep8 * fix docs * raise exception for stacking with partial keys and axis!=0 * minor fix * minor fix Co-authored-by: Trinkle23897 <463003665@qq.com> * stash * let agent learn to play as agent 2 which is harder * code refactor * Improve collector (#125) * remove multibuf * reward_metric * make fileds with empty Batch rather than None after reset * many fixes and refactor Co-authored-by: Trinkle23897 <463003665@qq.com> * marl for tic-tac-toe and general gomoku * update default gamma to 0.1 for tic tac toe to win earlier * fix name typo; change default game config; add rew_norm option * fix pep8 * test commit * mv test dir name * add rew flag * fix torch.optim import error and madqn rew_norm * remove useless kwargs * Vector env enable select worker (#132) * Enable selecting worker for vector env step method. * Update collector to match new vecenv selective worker behavior. * Bug fix. * Fix rebase Co-authored-by: Alexis Duburcq <alexis.duburcq@wandercraft.eu> * show the last move of tictactoe by capital letters * add multi-agent tutorial * fix link * Standardized behavior of Batch.cat and misc code refactor (#137) * code refactor; remove unused kwargs; add reward_normalization for dqn * bugfix for __setitem__ with torch.Tensor; add Batch.condense * minor fix * support cat with empty Batch * remove the dependency of is_empty on len; specify the semantic of empty Batch by test cases * support stack with empty Batch * remove condense * refactor code to reflect the shared / partial / reserved categories of keys * add is_empty(recursive=False) * doc fix * docfix and bugfix for _is_batch_set * add doc for key reservation * bugfix for algebra operators * fix cat with lens hint * code refactor * bugfix for storing None * use ValueError instead of exception * hide lens away from users * add comment for __cat * move the computation of the initial value of lens in cat_ itself. * change the place of doc string * doc fix for Batch doc string * change recursive to recurse * doc string fix * minor fix for batch doc * write tutorials to specify the standard of Batch (#142) * add doc for len exceptions * doc move; unify is_scalar_value function * remove some issubclass check * bugfix for shape of Batch(a=1) * keep moving doc * keep writing batch tutorial * draft version of Batch tutorial done * improving doc * keep improving doc * batch tutorial done * rename _is_number * rename _is_scalar * shape property do not raise exception * restore some doc string * grammarly [ci skip] * grammarly + fix warning of building docs * polish docs * trim and re-arrange batch tutorial * go straight to the point * minor fix for batch doc * add shape / len in basic usage * keep improving tutorial * unify _to_array_with_correct_type to remove duplicate code * delegate type convertion to Batch.__init__ * further delegate type convertion to Batch.__init__ * bugfix for setattr * add a _parse_value function * remove dummy function call * polish docs Co-authored-by: Trinkle23897 <463003665@qq.com> * bugfix for mapolicy * pretty code * remove debug code; remove condense * doc fix * check before get_agents in tutorials/tictactoe * tutorial * fix * minor fix for batch doc * minor polish * faster test_ttt * improve tic-tac-toe environment * change default epoch and step-per-epoch for tic-tac-toe * fix mapolicy * minor polish for mapolicy * 90% to 80% (need to change the tutorial) * win rate * show step number at board * simplify mapolicy * minor polish for mapolicy * remove MADQN * fix pep8 * change legal_actions to mask (need to update docs) * simplify maenv * fix typo * move basevecenv to single file * separate RandomAgent * update docs * grammarly * fix pep8 * win rate typo * format in cheatsheet * use bool mask directly * update doc for boolean mask Co-authored-by: Trinkle23897 <463003665@qq.com> Co-authored-by: Alexis DUBURCQ <alexis.duburcq@gmail.com> Co-authored-by: Alexis Duburcq <alexis.duburcq@wandercraft.eu>

…ml#130) * re-implement Batch.stack and add testcases * add doc for Batch.stack * reuse _create_values and refactor stack_ & cat_ * fix pep8 * fix docs * raise exception for stacking with partial keys and axis!=0 * minor fix * minor fix Co-authored-by: Trinkle23897 <463003665@qq.com>

* make fileds with empty Batch rather than None after reset * dummy code * remove dummy * add reward_length argument for collector * Improve Batch (thu-ml#126) * make sure the key type of Batch is string, and add unit tests * add is_empty() function and unit tests * enable cat of mixing dict and Batch, just like stack * bugfix for reward_length * add get_final_reward_fn argument to collector to deal with marl * minor polish * remove multibuf * minor polish * improve and implement Batch.cat_ * bugfix for buffer.sample with field impt_weight * restore the usage of a.cat_(b) * fix 2 bugs in batch and add corresponding unittest * code fix for update * update is_empty to recognize empty over empty; bugfix for len * bugfix for update and add testcase * add testcase of update * make fileds with empty Batch rather than None after reset * dummy code * remove dummy * add reward_length argument for collector * bugfix for reward_length * add get_final_reward_fn argument to collector to deal with marl * make sure the key type of Batch is string, and add unit tests * add is_empty() function and unit tests * enable cat of mixing dict and Batch, just like stack * dummy code * remove dummy * add multi-agent example: tic-tac-toe * move TicTacToeEnv to a separate file * remove dummy MANet * code refactor * move tic-tac-toe example to test * update doc with marl-example * fix docs * reduce the threshold * revert * update player id to start from 1 and change player to agent; keep coding * add reward_length argument for collector * Improve Batch (thu-ml#128) * minor polish * improve and implement Batch.cat_ * bugfix for buffer.sample with field impt_weight * restore the usage of a.cat_(b) * fix 2 bugs in batch and add corresponding unittest * code fix for update * update is_empty to recognize empty over empty; bugfix for len * bugfix for update and add testcase * add testcase of update * fix docs * fix docs * fix docs [ci skip] * fix docs [ci skip] Co-authored-by: Trinkle23897 <463003665@qq.com> * refact * re-implement Batch.stack and add testcases * add doc for Batch.stack * reward_metric * modify flag * minor fix * reuse _create_values and refactor stack_ & cat_ * fix pep8 * fix reward stat in collector * fix stat of collector, simplify test/base/env.py * fix docs * minor fix * raise exception for stacking with partial keys and axis!=0 * minor fix * minor fix * minor fix * marl-examples * add condense; bugfix for torch.Tensor; code refactor * marl example can run now * enable tic tac toe with larger board size and win-size * add test dependency * Fix padding of inconsistent keys with Batch.stack and Batch.cat (thu-ml#130) * re-implement Batch.stack and add testcases * add doc for Batch.stack * reuse _create_values and refactor stack_ & cat_ * fix pep8 * fix docs * raise exception for stacking with partial keys and axis!=0 * minor fix * minor fix Co-authored-by: Trinkle23897 <463003665@qq.com> * stash * let agent learn to play as agent 2 which is harder * code refactor * Improve collector (thu-ml#125) * remove multibuf * reward_metric * make fileds with empty Batch rather than None after reset * many fixes and refactor Co-authored-by: Trinkle23897 <463003665@qq.com> * marl for tic-tac-toe and general gomoku * update default gamma to 0.1 for tic tac toe to win earlier * fix name typo; change default game config; add rew_norm option * fix pep8 * test commit * mv test dir name * add rew flag * fix torch.optim import error and madqn rew_norm * remove useless kwargs * Vector env enable select worker (thu-ml#132) * Enable selecting worker for vector env step method. * Update collector to match new vecenv selective worker behavior. * Bug fix. * Fix rebase Co-authored-by: Alexis Duburcq <alexis.duburcq@wandercraft.eu> * show the last move of tictactoe by capital letters * add multi-agent tutorial * fix link * Standardized behavior of Batch.cat and misc code refactor (thu-ml#137) * code refactor; remove unused kwargs; add reward_normalization for dqn * bugfix for __setitem__ with torch.Tensor; add Batch.condense * minor fix * support cat with empty Batch * remove the dependency of is_empty on len; specify the semantic of empty Batch by test cases * support stack with empty Batch * remove condense * refactor code to reflect the shared / partial / reserved categories of keys * add is_empty(recursive=False) * doc fix * docfix and bugfix for _is_batch_set * add doc for key reservation * bugfix for algebra operators * fix cat with lens hint * code refactor * bugfix for storing None * use ValueError instead of exception * hide lens away from users * add comment for __cat * move the computation of the initial value of lens in cat_ itself. * change the place of doc string * doc fix for Batch doc string * change recursive to recurse * doc string fix * minor fix for batch doc * write tutorials to specify the standard of Batch (thu-ml#142) * add doc for len exceptions * doc move; unify is_scalar_value function * remove some issubclass check * bugfix for shape of Batch(a=1) * keep moving doc * keep writing batch tutorial * draft version of Batch tutorial done * improving doc * keep improving doc * batch tutorial done * rename _is_number * rename _is_scalar * shape property do not raise exception * restore some doc string * grammarly [ci skip] * grammarly + fix warning of building docs * polish docs * trim and re-arrange batch tutorial * go straight to the point * minor fix for batch doc * add shape / len in basic usage * keep improving tutorial * unify _to_array_with_correct_type to remove duplicate code * delegate type convertion to Batch.__init__ * further delegate type convertion to Batch.__init__ * bugfix for setattr * add a _parse_value function * remove dummy function call * polish docs Co-authored-by: Trinkle23897 <463003665@qq.com> * bugfix for mapolicy * pretty code * remove debug code; remove condense * doc fix * check before get_agents in tutorials/tictactoe * tutorial * fix * minor fix for batch doc * minor polish * faster test_ttt * improve tic-tac-toe environment * change default epoch and step-per-epoch for tic-tac-toe * fix mapolicy * minor polish for mapolicy * 90% to 80% (need to change the tutorial) * win rate * show step number at board * simplify mapolicy * minor polish for mapolicy * remove MADQN * fix pep8 * change legal_actions to mask (need to update docs) * simplify maenv * fix typo * move basevecenv to single file * separate RandomAgent * update docs * grammarly * fix pep8 * win rate typo * format in cheatsheet * use bool mask directly * update doc for boolean mask Co-authored-by: Trinkle23897 <463003665@qq.com> Co-authored-by: Alexis DUBURCQ <alexis.duburcq@gmail.com> Co-authored-by: Alexis Duburcq <alexis.duburcq@wandercraft.eu>

youkaichao added 2 commits July 11, 2020 22:23

re-implement Batch.stack and add testcases

aefcdfc

add doc for Batch.stack

b3ab3fe

youkaichao requested a review from duburcqa July 11, 2020 14:39

duburcqa reviewed Jul 11, 2020

View reviewed changes

tianshou/data/batch.py Outdated Show resolved Hide resolved

duburcqa reviewed Jul 11, 2020

View reviewed changes

tianshou/data/batch.py Outdated Show resolved Hide resolved

duburcqa reviewed Jul 11, 2020

View reviewed changes

tianshou/data/batch.py Outdated Show resolved Hide resolved

Trinkle23897 assigned youkaichao Jul 11, 2020

youkaichao and others added 3 commits July 12, 2020 01:33

reuse _create_values and refactor stack_ & cat_

f7d7482

fix pep8

6bf23c7

fix docs

f434a26

Trinkle23897 changed the title ~~Batch.stack re-implementation~~ Fix padding of inconsistent keys with Batch.stack and Batch.cat Jul 12, 2020

raise exception for stacking with partial keys and axis!=0

a78d4ac

Trinkle23897 added 2 commits July 12, 2020 11:30

minor fix

1f08c4b

minor fix

c187390

youkaichao mentioned this pull request Jul 12, 2020

Add multi-agent example: tic-tac-toe #122

Merged

duburcqa approved these changes Jul 12, 2020

View reviewed changes

tianshou/data/batch.py Show resolved Hide resolved

Trinkle23897 merged commit a55ad33 into thu-ml:dev Jul 12, 2020

youkaichao deleted the stack branch July 13, 2020 13:22

youkaichao added a commit that referenced this pull request Jul 20, 2020

Revert "Fix padding of inconsistent keys with Batch.stack and Batch.c…

af4a608

…at (#130)" This reverts commit 5599a6d.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix padding of inconsistent keys with Batch.stack and Batch.cat #130

Fix padding of inconsistent keys with Batch.stack and Batch.cat #130

youkaichao commented Jul 11, 2020 •

edited by Trinkle23897

duburcqa Jul 11, 2020

Trinkle23897 Jul 11, 2020 •

edited

duburcqa Jul 11, 2020 •

edited by Trinkle23897

duburcqa Jul 11, 2020 •

edited by Trinkle23897

Trinkle23897 Jul 11, 2020 •

edited

youkaichao Jul 11, 2020

duburcqa Jul 11, 2020 •

edited

youkaichao Jul 11, 2020

duburcqa Jul 11, 2020 •

edited

duburcqa Jul 11, 2020 •

edited

youkaichao commented Jul 11, 2020 •

edited by Trinkle23897

duburcqa commented Jul 11, 2020

youkaichao commented Jul 11, 2020

youkaichao commented Jul 12, 2020 •

edited by Trinkle23897

Trinkle23897 commented Jul 12, 2020 •

edited

codecov-commenter commented Jul 12, 2020 •

edited

duburcqa commented Jul 12, 2020 •

edited

youkaichao commented Jul 12, 2020

duburcqa commented Jul 12, 2020

Fix padding of inconsistent keys with Batch.stack and Batch.cat #130

Fix padding of inconsistent keys with Batch.stack and Batch.cat #130

Conversation

youkaichao commented Jul 11, 2020 • edited by Trinkle23897

duburcqa Jul 11, 2020

Choose a reason for hiding this comment

Trinkle23897 Jul 11, 2020 • edited

Choose a reason for hiding this comment

duburcqa Jul 11, 2020 • edited by Trinkle23897

Choose a reason for hiding this comment

duburcqa Jul 11, 2020 • edited by Trinkle23897

Choose a reason for hiding this comment

Trinkle23897 Jul 11, 2020 • edited

Choose a reason for hiding this comment

youkaichao Jul 11, 2020

Choose a reason for hiding this comment

duburcqa Jul 11, 2020 • edited

Choose a reason for hiding this comment

youkaichao Jul 11, 2020

Choose a reason for hiding this comment

duburcqa Jul 11, 2020 • edited

Choose a reason for hiding this comment

duburcqa Jul 11, 2020 • edited

Choose a reason for hiding this comment

youkaichao commented Jul 11, 2020 • edited by Trinkle23897

duburcqa commented Jul 11, 2020

youkaichao commented Jul 11, 2020

youkaichao commented Jul 12, 2020 • edited by Trinkle23897

Trinkle23897 commented Jul 12, 2020 • edited

codecov-commenter commented Jul 12, 2020 • edited

Codecov Report

duburcqa commented Jul 12, 2020 • edited

youkaichao commented Jul 12, 2020

duburcqa commented Jul 12, 2020

youkaichao commented Jul 11, 2020 •

edited by Trinkle23897

Trinkle23897 Jul 11, 2020 •

edited

duburcqa Jul 11, 2020 •

edited by Trinkle23897

duburcqa Jul 11, 2020 •

edited by Trinkle23897

Trinkle23897 Jul 11, 2020 •

edited

duburcqa Jul 11, 2020 •

edited

duburcqa Jul 11, 2020 •

edited

duburcqa Jul 11, 2020 •

edited

youkaichao commented Jul 11, 2020 •

edited by Trinkle23897

youkaichao commented Jul 12, 2020 •

edited by Trinkle23897

Trinkle23897 commented Jul 12, 2020 •

edited

codecov-commenter commented Jul 12, 2020 •

edited

duburcqa commented Jul 12, 2020 •

edited