Fix episodic training of DDPG

[muupan]

#317 broke DDPG's episodic training and made tests/agents_tests/test_ddpg.py fail. This PR fixes it.

[muupan]

Before this PR:

pytest -xsv --disable-pytest-warnings tests/agents_tests/test_ddpg.py
========================================================================================================================== test session starts ===========================================================================================================================
platform linux -- Python 3.5.2, pytest-3.8.2, py-1.6.0, pluggy-0.7.1 -- /home/fujita/.local/share/virtualenvs/chainerrl_20180827-9TPAa4mI/bin/python3
cachedir: .pytest_cache
rootdir: /home/fujita/chainerrl_20180827, inifile:
collected 8 items

tests/agents_tests/test_ddpg.py::TestDDPGOnContinuousPOABC::test_training_cpu <- tests/agents_tests/basetest_training.py FAILED

================================================================================================================================ FAILURES ================================================================================================================================
______________________________________________________________________________________________________________ TestDDPGOnContinuousPOABC.test_training_cpu _______________________________________________________________________________________________________________

self = <test_ddpg.TestDDPGOnContinuousPOABC testMethod=test_training_cpu>

    @testing.attr.slow
    def test_training_cpu(self):
>       self._test_training(-1, steps=100000)

tests/agents_tests/basetest_training.py:93:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/agents_tests/basetest_training.py:62: in _test_training
    eval_env=test_env)
chainerrl/experiments/train_agent.py:163: in train_agent_with_evaluation
    logger=logger)
chainerrl/experiments/train_agent.py:52: in train_agent
    action = agent.act_and_train(obs, r)
chainerrl/agents/ddpg.py:327: in act_and_train
    self.replay_updater.update_if_necessary(self.t)
chainerrl/replay_buffer.py:521: in update_if_necessary
    self.update_func(episodes)
chainerrl/agents/ddpg.py:276: in update_from_episodes
    transitions, xp=self.xp, phi=self.phi, gamma=self.gamma)
chainerrl/replay_buffer.py:453: in batch_experiences
    [elem[0]['state'] for elem in experiences], xp, phi),
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

.0 = <list_iterator object at 0x7f9676f58f98>

>   [elem[0]['state'] for elem in experiences], xp, phi),
            'action': xp.asarray([elem[0]['action'] for elem in experiences]),
            'reward': xp.asarray([sum((gamma ** i) * exp[i]['reward']
                          for i in range(len(exp)))
                      for exp in experiences],
                     dtype=np.float32),
            'next_state': batch_states(
    [elem[-1]['next_state']
     for elem in experiences], xp, phi),
            'is_state_terminal': xp.asarray(
    [any(transition['is_state_terminal']
         for transition in exp) for exp in experiences],
    dtype=np.float32),
            'discount': xp.asarray([(gamma ** len(elem))for elem in experiences],
                       dtype=np.float32)}
E                      KeyError: 0

chainerrl/replay_buffer.py:453: KeyError
======================================================================================================================== 1 failed in 8.84 seconds ========================================================================================================================

After this PR:

pytest -xsv --disable-pytest-warnings tests/agents_tests/test_ddpg.py
========================================================================================================================== test session starts ===========================================================================================================================
platform linux -- Python 3.5.2, pytest-3.8.2, py-1.6.0, pluggy-0.7.1 -- /home/fujita/.local/share/virtualenvs/chainerrl_20180827-9TPAa4mI/bin/python3
cachedir: .pytest_cache
rootdir: /home/fujita/chainerrl_20180827, inifile:
collected 8 items

tests/agents_tests/test_ddpg.py::TestDDPGOnContinuousPOABC::test_training_cpu <- tests/agents_tests/basetest_training.py Load agent from /tmp/tmp2inojjs1/agent_final
PASSED
tests/agents_tests/test_ddpg.py::TestDDPGOnContinuousPOABC::test_training_cpu_fast <- tests/agents_tests/basetest_training.py Load agent from /tmp/tmpu73pyv1c/agent_final
PASSED
tests/agents_tests/test_ddpg.py::TestDDPGOnContinuousPOABC::test_training_gpu <- tests/agents_tests/basetest_training.py Load agent from /tmp/tmp7b58w2iu/agent_final
PASSED
tests/agents_tests/test_ddpg.py::TestDDPGOnContinuousPOABC::test_training_gpu_fast <- tests/agents_tests/basetest_training.py Load agent from /tmp/tmpiwgxe9j8/agent_final
PASSED
tests/agents_tests/test_ddpg.py::TestDDPGOnContinuousABC::test_training_cpu <- tests/agents_tests/basetest_training.py Load agent from /tmp/tmp_n9ogrf_/agent_final
PASSED
tests/agents_tests/test_ddpg.py::TestDDPGOnContinuousABC::test_training_cpu_fast <- tests/agents_tests/basetest_training.py Load agent from /tmp/tmpb94phufh/agent_final
PASSED
tests/agents_tests/test_ddpg.py::TestDDPGOnContinuousABC::test_training_gpu <- tests/agents_tests/basetest_training.py Load agent from /tmp/tmpv5caxl_i/agent_final
PASSED
tests/agents_tests/test_ddpg.py::TestDDPGOnContinuousABC::test_training_gpu_fast <- tests/agents_tests/basetest_training.py Load agent from /tmp/tmpmiihmoj7/agent_final
PASSED

================================================================================================================ 8 passed, 16 warnings in 390.24 seconds =================================================================================================================

[muupan]

imosci test this please

[ninetan]

imosCI successfully created a job for commit 4a376ee:

• Dashboard for commit 4a376ee

[prabhatnagarajan]

LGTM