cartpole learning(continuous) : RoboschoolInvertedPendulum-v1

[CUN-bjy]

reinforcement learning
in RoboschoolInvertedPendulum-v1 environment

[CUN-bjy]

experiment specifications
hyperparameter:

• buffer_size = 20000, batch_size = 64
• prioritized buffer : False
• learning_rate: 1e-4,1e-3 for actor, critic
• tau(target update rate): 1e-3,1e-3 for actor, critic
• network
  • actor:

      # input layer(observations)
      input_ = Input(shape=self.obs_dim)
    
      # hidden layer 1
      h1_ = Dense(24,kernel_initializer=GlorotNormal())(input_)
      h1_b = BatchNormalization()(h1_)
      h1 = Activation('relu')(h1_b)
    
      # hidden_layer 2
      h2_ = Dense(16,kernel_initializer=GlorotNormal())(h1)
      h2_b = BatchNormalization()(h2_)
      h2 = Activation('relu')(h2_b)
    
      # output layer(actions)
      output_ = Dense(self.act_dim,kernel_initializer=GlorotNormal())(h2)
      output_b = BatchNormalization()(output_)
      output = Activation('tanh')(output_b)
      scalar = self.act_range * np.ones(self.act_dim)
      out = Lambda(lambda i: i * scalar)(output)

  • critic

      # input layer(observations and actions)
      input_obs = Input(shape=self.obs_dim)
      input_act = Input(shape=(self.act_dim,))
      inputs = [input_obs,input_act]
      concat = Concatenate(axis=-1)(inputs)
    
      # hidden layer 1
      h1_ = Dense(24, kernel_initializer=GlorotNormal(), kernel_regularizer=l2(0.01))(concat)
      h1_b = BatchNormalization()(h1_)
      h1 = Activation('relu')(h1_b)
    
      # hidden_layer 2
      h2_ = Dense(16, kernel_initializer=GlorotNormal(), kernel_regularizer=l2(0.01))(h1)
      h2_b = BatchNormalization()(h2_)
      h2 = Activation('relu')(h2_b)
    
      # output layer(actions)
      output_ = Dense(1, kernel_initializer=GlorotNormal(), kernel_regularizer=l2(0.01))(h2)
      output_b = BatchNormalization()(output_)
      output = Activation('linear')(output_b)

Performance

Critic Loss

Results

[CUN-bjy]

training time : about 5 hours(2500 epi)
on intel i7 cpu