Behavioural task MDP

#Notes State = location x texture (locations = {1234567}, textures = {ABC}) Some state transitions will be 0 (b/c cannot transition from 1 to 7, eg.) i.e. would have prob == 1 of moving from 1a to 2a if agent took action == south. have prob == 1 of staying in 1a if agent took action == stay or action == nosepoke.

21 states so large set of constraints; agent has to learn how to behave optimally. Reward ==1 if nosepoke in a rewarded chamber.

In Kirth's task, real state space depends on where the last rewarded nosepoke was. This component of the state is not fed directly to the agent. Our agent doesn't keep memory so doesn't store last rewarded nosepoke.

To represent in NN want to convert to series of distr representations. Have set of units that encodes location; set of units that encodes current texture all in one layer (i.e. input layer for NN). At any given time, have some activation vector in the input layer based on current state info.

Hidden layers = 2 for now. Can randomly assign some place cells to this environment i.e. each unit will have some activity fxn determined by loc of agent. Using sigmoid units for right now. Rate code a neuron would have. I.e. want each neuron to have activity pattern based on location (hard coded) Likewise for textures, would have distributed representation (i.e. 10 units activated to different degrees by each pattern). 20 loc units 10 tex units

30 dimensional input vector to NN. Forward pass feed up to some H.0 layer via some set of syn (W.0). H.0 = sig*W.0(in)

Then passing through to next H.1 layer via some set of syn W.1. H.1 = sig*W.1(H.0)

Output to V and PI through weights W.v and W.pi. V = (linear fxn)*W.v(H.1) and PI = (softmax)*W.pi(H.1) [Check https://en.wikipedia.org/wiki/Softmax_function]

:<math>\sigma(\mathbf{z})_j = \frac{e^{z_j}}{\sum_{k=1}^K e^{z_k}}</math> &nbsp;&nbsp; for ''j'' = 1, ..., ''K''

V is a single unit and basically just a scalar variable based on current state PI is some number of units = actions (i.e. 1 unit per action) == what is the prob of taking different Change weights by rule given in [http://arxiv.org/pdf/1602.01783v1.pdf -- p.9 algorithm 3, rule for updating \theta]

Want a few py files. Need object for representing environment -- stores Loc and Tex (and maybe last place reward was achieved)