08. DLAgent

DLAgent implements an actor-critic reinforcement learning algorithm to generate the robot's actions, following the formulation presented in Reinforcement Learning: An Introduction (Sutton & Barto, 2018).

The network computes an action policy based on the state of the environment. The policy defines the probability distribution over actions, from which the agent samples the next action (Actor). The agent then observes the outcome of the selected action, and the policy is evaluated according to the average reward obtained per time step. The network also estimates the differential state-value function $\hat v(s_t)$ (Critic), which is used to update both the value estimate and the policy.

---

The differential TD error

The differential TD error is evaluated as

$$ \delta_t = R_{t+1} - \bar R_t + \hat v( s_{t+1}; \mathbf{w}) - \hat v( s_t; \mathbf{w}) $$

where $\bar R_t$ is the estimate of the average reward.

The estimate is updated as

$$\bar R_{t+1} = \gamma \bar R_{t} + \beta, \delta_t $$

with

• $\gamma= 0 \dots 1$ the decay factor that prevents the average reward estimate from diverging
• $\beta$ the average reward update factor.

The TD error acts as a common learning signal for both policy and value updates.

---

Network training

The network is then trained using the following update rules.

The critic network is trained by minimizing the squared TD error

$$ L_V(\mathbf{w}) = \frac{1}{2} \delta_t^2 $$

Verify the implementation

The policy is updated as

$$ \pi'(a|s_t) = f(\log \pi_t(a \mid s_t)+\alpha,\delta_t , m(a_t)) $$

where $f(\mathbf{x})$ denotes the softmax function

$$ f(\mathbf{x})_i = \frac{e^{x_i}}{\sum_j e^{x_j}} $$

where $m(a_t)$ is the action mask defined as

$$ m_i(a_t)= \begin{cases} 1 & \text{if } i=a_t,\\ 0 & \text{otherwise.} \end{cases} $$

and

• $\alpha$ the policy change factor

---

Online training

The following sequence diagrams summarize the interaction between the agent, the dataset iterators and the neural network during training.

sequenceDiagram
    actor Caller
    participant DLAgent
    participant Iterator as TrajectoryDatasetIterator
    participant Network as ComputationGraph
    participant KPIs

    Caller->>+DLAgent: train(trainingNetwork, trajectory)

    DLAgent->>DLAgent: createTrajectoryIterator(trajectory, batchSize, avgReward)
    DLAgent-->>+Iterator: iterator

    DLAgent->>Iterator: onKpis(kpis::onNext)

    DLAgent->>+Network: fit(iterator, numEpochs)

    loop Training batches
        Network->>Iterator: next()
        activate Iterator
        Iterator-->>Network: MultiDataSet
        Iterator-->>KPIs: onNext(kpi)
        deactivate Iterator
    end

    Network-->>-DLAgent: training completed

    DLAgent->>+Iterator: avgReward()
    Iterator-->>-DLAgent: avg

    Note over Iterator: close() automatico\n(try-with-resources)

    DLAgent->>DLAgent: status.updateAndGet(s -> s.trained(trainingNetwork, avg))

    DLAgent-->>-Caller: return

Loading

---

TrajectoryDatasetIterator

sequenceDiagram
    actor ComputationGraph
    participant Iterator as TrajectoryDatasetIterator
    participant Network as ComputationGraph
    participant Mediator as NNMediator
    participant Labels

    ComputationGraph->>+Iterator: next(numRecords)

    alt cursor == 0
        Iterator->>Iterator: disposeLabels()

        Iterator->>+Iterator: createLabels()

        Iterator->>+Network: predict(inputs)
        Network-->>-Iterator: predictions

        Iterator->>+Mediator: processRewards(rewards, critic, initialAvgReward, beta, gamma)
        Mediator-->>-Iterator: (deltas, avgReward)

        Iterator->>+Iterator: createLabels(predictions, deltas)

        Iterator->>+Labels: create critic label
        Labels-->>-Iterator: critic labels

        loop Policy outputs
            Iterator->>Iterator: compute delta policies
            Iterator->>+Mediator: computeNewPolicy(clipped, deltaMasks)
            Mediator-->>-Iterator: policy labels
        end

        Iterator-->>-Iterator: labels created
    end

    Iterator->>Iterator: n = min(numRecords, remaining)

    Iterator->>Iterator: extract minibatch inputs
    Iterator->>Iterator: extract minibatch labels

    Iterator->>+Labels: create MultiDataSet(minibatchInput, minibatchOutput)
    Labels-->>-Iterator: dataset

    Iterator->>Iterator: cursor += n

    Iterator-->>-ComputationGraph: dataset

Loading

---

Batch training

sequenceDiagram
    participant BatchApp
    participant DLAgent
    participant ComputationGraph
    participant KpisObserver
    
    create participant BinFilesDatasetIterator
    BatchApp-->BinFilesDatasetIterator: datasetIterator := create(...)

    BatchApp->>+DLAgent: trainedAgent := train(datasetIterator, numEpochs)
        DLAgent->>+ComputationGraph: fit(datasetIterator, numEpochs)
            ComputationGraph->>+BinFilesDatasetIterator: next()
                Note right of BinFilesDatasetIterator: Reads mini-batch size records from files
                BinFilesDatasetIterator->>+DLAgent: createDataset(states, action, rewards, avgRewards)
                    DLAgent-)KpisObserver: onNext(kpis)
                DLAgent-->>-BinFilesDatasetIterator: (multiDataset,reward)
            BinFilesDatasetIterator-->>-ComputationGraph: multiDataset
        ComputationGraph-->>-DLAgent:
    DLAgent-->>-BatchApp:

Loading

---

Release 0.27.0