Not meant as an issue, but a little perplexed by feature_dim...

[slerman12]

I noticed feature_dim is set to 50, which is quite a bottleneck from the encoding dim of 32 * 35 * 35 and the downstream hidden_dim of 1024. Very interesting. Do you think the bottleneck helps create some kind of better compression for learning?

[medric49]

Hi,
I am also interested in the answer !

[denisyarats]

Hi,

Yes, having such a bottleneck is a very crucial architectural choice. I did try a bunch of other variations (different size, having an mlp, etc.), but couldn't really come up with anything that works better than this. humanoid is the only environment where I use feature_dim=100, which makes sense because the actual dimensionality of humanoid's internal state is larger than 50.

[slerman12]

Very interesting! I suppose it helps generalization. I'm surprised by the massive de-compression of the 1024-size hidden_dim that follows — you'd think there wouldn't need to be such a massive layer of processing after such a small layer.

[denisyarats]

Initially I used 256 hidden size after the encoder's 50 dimensional output, which also works fine. Over time, however, I noticed that wider value/policy heads are useful for increasing stability of training. I suspect this is because with a wider network you have a better chance to end up in a good init position in the parameter space, so the seed won't collapse.

[slerman12]

Yeah, that makes sense. I hadn't thought of it that way.