How to restore from h5 file saved by PrioritizedVectorReplayBuffer?

[jaried]

• I have marked all applicable categories:
• I have visited the source website
• I have searched through the issue tracker for duplicates
• I have mentioned version numbers, operating system and environment, where applicable:

import tianshou, gym, torch, numpy, sys
print(tianshou.__version__, gym.__version__, torch.__version__, numpy.__version__, sys.version, sys.platform)
0.4.9 0.25.0 1.11.0+cu113 1.20.3 3.9.7 (default, Sep 16 2021, 16:59:28) [MSC v.1916 64 bit (AMD64)] win32

When I run the following statement, it prompts: KeyError: 'weight', how can I modify it?

data, idx = source.sample(0)

buff = PrioritizedVectorReplayBuffer.load_hdf5(buffer_path)

for buf_id, source in enumerate(buff.buffers):  # move data from buff to (old) buffer
    data, idx = source.sample(0)  # get all data
    train_collector.buffer.add(
        data, buffer_ids=np.full(shape=len(idx), fill_value=(buf_id % args.training_num)))
del buff

[Trinkle23897]

Sorry about that. It's a bug and we haven't added this kind of test before.

[jaried]

Is there a plan to fix this bug?

[Trinkle23897]

will do today

[jaried]

will do today

Thank you very much.

[Trinkle23897]

Could you please paste your full script to reproduce the bug? I just found I use PrioVecBuf to save and PrioBuf to load, which is absolutely wrong usage but can produce almost the same KeyError. After correcting this, I don't see any undesired behavior in my side.

[Trinkle23897]

The behavior in your script above is desirable. The PrioVecBuf needs to delete all of its sub-priobuf's weight and merge to a single segment tree, in order to enable global weight sampling.

[Trinkle23897]

            if args.prioritized_replay:
               train_collector.buffer.load_hdf5(buffer_path)

This should work I guess, if the buffer's type is the same.

[jaried]

            if args.prioritized_replay:
               train_collector.buffer.load_hdf5(buffer_path)

This should work I guess, if the buffer's type is the same.

I tried this, after reading, len(train_collector.buffer) is correct 650,000 steps. However, after continuing to collect data such as 8000 steps, len(train_collector.buffer)=8000.

[Trinkle23897]

Can you write a minimal example to show the bug?

[jaried]

Excuse me, do I need to add anything else?

[Trinkle23897]

After the above sentence is executed, len(train_collector.buffer) is still 0.

This is expected because it only returns a buffer instead of modifying one buffer in place.

Can you write a minimal example to show the bug?

I mean you can write something like the unit test code:

tianshou/test/base/test_buffer.py

Lines 935 to 964 in 278c91a

	def test_multibuf_hdf5():
	size = 100
	buffers = {
	"vector": VectorReplayBuffer(size * 4, 4),
	"cached": CachedReplayBuffer(ReplayBuffer(size), 4, size)
	}
	buffer_types = {k: b.__class__ for k, b in buffers.items()}
	device = 'cuda' if torch.cuda.is_available() else 'cpu'
	info_t = torch.tensor([1.]).to(device)
	for i in range(4):
	kwargs = {
	'obs': Batch(index=np.array([i])),
	'act': i,
	'rew': np.array([1, 2]),
	'done': i % 3 == 2,
	'info': {
	"number": {
	"n": i,
	"t": info_t
	},
	'extra': None
	},
	}
	buffers["vector"].add(
	Batch.stack([kwargs, kwargs, kwargs]), buffer_ids=[0, 1, 2]
	)
	buffers["cached"].add(
	Batch.stack([kwargs, kwargs, kwargs]), buffer_ids=[0, 1, 2]
	)

That is, manually add one or more fake data, save it, then immediately reload it, and check whether other functionality behaves correctly or not, instead of the whole training script...

[jaried]

My major is not computer, I haven't written unit tests yet, I need to learn to write.

[Trinkle23897]

Take your time, I believe you'll learn more after writing some tests to get a better understanding.

[jaried]

I just read the introduction to unit testing and it is worth learning.

[Trinkle23897]

On my side, this script can successfully run without error.

[Trinkle23897]

    # check shallow copy in PrioritizedVectorReplayBuffer
    new_buffers = {
        "vector": PrioritizedVectorReplayBuffer(size * 4, 4, alpha=0.6, beta=0.4),
    }

    for k in ["vector"]:
        new_buffers[k].info.number.n[0] = -100
        assert new_buffers[k].buffers[0].info.number.n[0] == -100

You access a buffer that contains nothing, this seems not correct.

[jaried]

At this point, index=='weight', there is no weight in the buffer, so an error is reported. I don't know how to write the unit test here.

tianshou/tianshou/data/batch.py

Lines 236 to 237 in 278c91a

	if isinstance(index, str):
	return self.__dict__[index]

Traceback (most recent call last):
   File "D:\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 3444, in run_code
     exec(code_obj, self.user_global_ns, self.user_ns)
   File "<ipython-input-11-5b960fabffe8>", line 1, in <module>
     self.__dict__[index]
KeyError: 'weight'
```

[jaried]

Use PrioritizedVectorReplayBuffer.load_hdf5(buffer_path) directly to solve the problem.
Thanks for your answer.

[Trinkle23897]

It's because you cannot directly sample data from PrioVectorReplayBuffer's sub_buffer. Just put buf = PrioVecBuf.load_hdf5(...) before train_collector's init, it should work.