Repository navigation
TD Agent
flowchart LR
subgraph Env
EnvironmentProc[Environment\nProcess]
Sensors
Actuators
end
subgraph Agent
Actor
Preprocessor
Trainer
subgraph NN[Neural Network]
NNLayers[States\nLayers]
CriticLayer[Critic\nLayer]
ActorLayer[Actor\nLayer]
end
end
Sensors -->|signals| EnvironmentProc
EnvironmentProc -->|state, reward| Trainer
EnvironmentProc -->|state| Preprocessor
EnvironmentProc -->|commands| Actuators
Preprocessor -->|state| NNLayers
Trainer -->|changes| NN
Actor -->|actions| EnvironmentProc
Actor -->|actions| Trainer
NNLayers --> CriticLayer
NNLayers --> ActorLayer
CriticLayer -->|residual\nadvantage\nprediction | Trainer
ActorLayer -->|action\npolicy| Trainer
ActorLayer -->|action\npolicy| Actor
The environment management process receives signals from sensors and translates them into status signals for the agent.
It is the task of this section to aggregate the sensor signals to produce a representation of the state of the system. In this section, for example, the signals from the proximity sensors are processed to create a two-dimensional spatial map of the obstacles.
It is also the task of this section to transform the actions into commands to be executed to control the robot.
The preprocessor has the task of transforming the state signals produced by the environment management process into input signals of the neural network.
The transformation can be for example the normalization of sensor signals or binary values coding (coarse or tiles coding).
The preprocessor available are
The implementing class is org.mmarini.rl.processors.NotProcessor.
The preprocessor transforms a single input signal into a feature vector, each feature represents the negation of input signal.
E.g.
Let the input signal be made up of a signal a of dimensions 3.
The output signal will therefore be made up of 3 features where the value is 1 if the input is not equal to 0
| a | out |
|---|---|
| a0 | !a0 |
| a1 | !a1 |
| a2 | !a1 |
The implementing class is org.mmarini.rl.processors.AndProcessor.
The preprocessor transforms a set of input signals into a feature vector, each feature represents the intersection of input signal not equals to 0.
The input signals must have the same dimensions
E.g.
Let the input signal be made up of signals a and b of dimensions 3.
The output signal will therefore be made up of 3 features where the value is 1 if all the respective dimension inputs is not equal 0
| a | b | out |
|---|---|---|
| a0 | b0 | a0 && b0 |
| a1 | b2 | a1 && b2 |
| a2 | b2 | a1 && b2 |
The implementing class is org.mmarini.rl.processors.DecodeProcessor.
The preprocessor transforms a list of integer input signals into a single decoded input signal The input signals must have the same shapes. Each output value dimension contains the unique value of the combination of its input dimensions.
E.g.
Let the input signals consist of two vectors a and b each of two dimensions with 3 and 2 integer possible values between 0, 2 and 0, 1 respectively.
Each dimension of output signal then has 3 x 2 = 6 possible values between 0 and 5.
| a0 | b0 | out0 | a1 | b1 | out1 |
|---|---|---|---|---|---|
| 0 | 0 | 0 + 3 * 0 = 0 | 1 | 0 | 1 + 3 * 0 = 1 |
| 1 | 1 | 1 + 3 * 1 = 4 | 2 | 1 | 2 + 3 * 1 = 5 |
The implementing class is org.mmarini.rl.processors.FeaturesProcessor.
The preprocessor transforms a integer input signals into features vector. Each input dimension value is transformed into features vectors.
E.g.
Let the input signals consist of vector a of two dimensions with 3 integer possible values between 0, 2.
Than the output signal is composed of a matrix of 2 x 3 binary values
| a | out | out | out |
|---|---|---|---|
| 0 | 1 | 0 | 0 |
| 2 | 0 | 0 | 1 |
The implementing class is org.mmarini.rl.processors.PartitionProcessor.
The preprocessor transforms the integer or float input signals into a class vector of integer, each value represents the classification for each dimension of the input signals.
E.g.
Let the input signal be made up of signal a of dimensions 2.
We want the input signals to be classified into 4 classes.
The output signal will therefore be made up of 2 values in the range 0, ..., 3
The main heart of the agent is the neural network that processes the input signals to produce the output signals of the actor/critic TD learning method.
The critic signal is the prediction of the residual advantage of the input state in relation to the reward received. The signal generated by the network is used to calculate the prediction error and then train the network accordingly.
The other set of values generated represent the policy of actions to be taken. This policy is nothing more than the distribution of probabilities of random choice of possible actions.
The network consists of a signal processing part and two output layers for two actor/critic signals.
Wheelly is actually using the following architecture
graph TB
L0["`canMoveFeatures
Input(6)`"]
L1["`hinderedDistanceFeatures
Input(552)`"]
L2["`labeledDistanceFeatures
Input(552)`"]
L3["`stateFeatures
Input(96)`"]
L4["`layer0[0]
Concat`"]
L5["`layer0[1]
Dense(144)
dropOut=0,800`"]
L6["`layer0
Relu`"]
L7["`hidden0[0]
Dense(144)
dropOut=0,500`"]
L8["`hidden0[1]
Relu`"]
L9["`hidden0
Dense(144)
dropOut=0,500`"]
L10["`resnet0[0]
Sum`"]
L11["`resnet0
Relu`"]
L12["`hidden1[0]
Dense(144)
dropOut=0,500`"]
L13["`hidden1[1]
Relu`"]
L14["`hidden1
Dense(144)
dropOut=0,500`"]
L15["`resnet1[0]
Sum`"]
L16["`resnet1
Relu`"]
L17["`hidden2[0]
Dense(144)
dropOut=0,500`"]
L18["`hidden2[1]
Relu`"]
L19["`hidden2
Dense(144)
dropOut=0,500`"]
L20["`resnet2[0]
Sum`"]
L21["`resnet2
Relu`"]
L22["`sensorAction[0]
Dense(7)`"]
L23["`sensorAction[1]
Tanh`"]
L24["`sensorAction
Softmax(0,434)`"]
L25["`move[0]
Dense(40)`"]
L26["`critic
Dense(1)`"]
L27["`move[1]
Tanh`"]
L28["`move
Softmax(0,290)`"]
L23 --> L24
L6 --> L7
L21 --> L25
L6 --> L10
L9 --> L10
L12 --> L13
L16 --> L17
L16 --> L20
L19 --> L20
L0 --> L4
L3 --> L4
L1 --> L4
L2 --> L4
L21 --> L26
L17 --> L18
L22 --> L23
L13 --> L14
L27 --> L28
L8 --> L9
L5 --> L6
L11 --> L15
L14 --> L15
L25 --> L27
L11 --> L12
L7 --> L8
L18 --> L19
L4 --> L5
L20 --> L21
L15 --> L16
L10 --> L11
L21 --> L22
The actor has the task of generating the actions randomly with a probability of choice given by the policy.
Finally we have the trainer who, with the reward signals, the critic's predictions, the action policy, the action chosen by the actor, calculates the error used to train the neural network.
Release 0.19.0