Advanced Batch slicing & minor fix of RNN support

[Trinkle23897]

Related to #104

[Trinkle23897]

@duburcqa I add as what you said, but it turns out that it is not compatible with tests in test/base.

Some issues:

• min(*data_shape) will return the smallest data_shape, not the smallest size in every dimension. E.g., min((1,2,3,4), (3,2,1)) will result in (1,2,3,4) instead of (1,2,1);
• if we want to force use np.array instead of list, we should further overwrite the __setattr__ method;
• some tests fail strangely, TypeError: zip argument #2 must support iteration, and in "test_collector_with_dict_state" it fails to deep-copy. Should we change all the list in test to np.ndarray or add extra except-handling method within Batch?

One more thing: ListReplayBuffer use list instead of np.ndarray. To entirely change list to np.ndarray, this class should be abandoned, which will related to #105 ?

[Trinkle23897]

tianshou/tianshou/data/collector.py

Lines 203 to 204 in c639446

	elif isinstance(self.state, (Batch, torch.Tensor, np.ndarray)):
	self.state[id] *= 0

I haven't figured it out the best practice to make it compatible with _create_value (None issue). Maybe we should create batch.zero_()?

[duburcqa]

Maybe we should create batch.zero_()

Yes it would be nice, or rather empty_, since it is doing something closer to empty than zero.

[Trinkle23897]

• some tests fail strangely, TypeError: zip argument #2 must support iteration, and in "test_collector_with_dict_state" it fails to deep-copy. Should we change all the list in test to np.ndarray or add extra except-handling method within Batch?

One more thing: ListReplayBuffer use list instead of np.ndarray. To entirely change list to np.ndarray, this class should be abandoned, which will related to #105 ?

How about these issue?

[duburcqa]

min(*data_shape) will return the smallest data_shape, not the smallest size in every dimension. E.g., min((1,2,3,4), (3,2,1)) will result in (1,2,3,4) instead of (1,2,1);

True. It should be replaced by list(map(min, zip(*data_shape)))

[duburcqa]

if we want to force use np.array instead of list, we should further overwrite the setattr method;

Yes, and I think it is a good thing to better control the data added to Batch instances or modified.

[duburcqa]

some tests fail strangely, TypeError: zip argument #2 must support iteration, and in "test_collector_with_dict_state" it fails to deep-copy. Should we change all the list in test to np.ndarray or add extra except-handling method within Batch?

I think it is related to __setattr__ issue, not converting list to array.

[duburcqa]

One more thing: ListReplayBuffer use list instead of np.ndarray. To entirely change list to np.ndarray, this class should be abandoned.

Good point. I don't know how to handle this, but it is only used by Collector, that you are refactoring, so maybe it would be better to refactor this part. I would only a raw list of Batch/dict, replacing:

self._cached_buf[i].add(**data)

by simply

self._cached_buf[i].append(Batch(data))

[Trinkle23897]

One more thing: ListReplayBuffer use list instead of np.ndarray. To entirely change list to np.ndarray, this class should be abandoned.

Good point. I don't know how to handle this, but it is only used by Collector, that you are refactoring, so maybe it would be better to refactor this part. I would only a raw list of Batch/dict, replacing:

self._cached_buf[i].add(**data)

by simply

self._cached_buf[i].append(Batch(data))

I found an interesting fact:

        if self._meta.__dict__.get(name, None) is None:
            self._meta.__dict__[name] = []
        self._meta.__dict__[name].append(inst)

this will bypass both of the __setitem__ and __setattr__ method. So the ListReplayBuffer would not be changed.

[Trinkle23897]

Why did you use @staticmethod in Batch.stack, @classmethod in Batch.cat?

[duburcqa]

this will bypass both of the setitem and setattr method. So the ListReplayBuffer would not be changed.

Yes, but in fact it would be better if we could use self._meta[name] instead of self._meta.__dict__[name], precisely to avoid such bypass behavior, but I was afraid of the impact on the performances (yet I haven't tried).

Why did you use @staticmethod in Batch.stack, @classmethod in Batch.cat?

No reason, @staticmethod is enough for both.

[Trinkle23897]

@duburcqa It should be ready now. Please have a code review at your convenience.
By the way, I have not removed the redundant code. You can comment and point them out (which will be added in this pr).

[duburcqa]

Very good job ! Still 2 comments to address 😁

[duburcqa]

I was thinking Batch.__init__ could have an extra argument copy similar to numpy (default to False), to enforce deepcopy of input argument. It could be convenient and it is only 2 lines of code.

[duburcqa]

By the way, did you do some profiling to check the performances ?

[Trinkle23897]

By the way, did you do some profiling to check the performances?

On my computer, it is the same as the last commit. However, GitHub shows that after the third commit of this PR, it shows down to some degree. I think maybe it is because of slicing: every time the __getitem__ will recursively call the _valid_bounds.

[duburcqa]

Apparently it is fine, at least for PPO.

[Trinkle23897]

I'd like to merge because I have other things to do (two projects and one deadline tomorrow). You can create a new PR to adapt your thought, and I can make a code review for you :)

Something to do:

• change Batch.empty to in-place fill
• Batch(copy=True)