Batch: don't just strip off empty entries when creating batches

[MischaPanch]

Batch can sometimes just remove entries without ever telling the user:

Batch(info=[{"a": 1}, {}])
>>>
Batch(
    info: Batch(
              a: array([1]),
          ),
)

This doesn't happen always though:

Batch(a=[1, {}])
>>>
Batch(
    a: array([1, {}], dtype=object),
)

What's even worse, caused by #1088 , bogus values are created:

Batch(info=[{"a": 1}, {}, {"a": 2, "b": 1}])
>>> 
Batch(
    info: Batch(
              a: array([1, 2]),
              b: array([0, 1]),
          ),
)

Note that the middle entry was removed, and for the first entry 0 was added as the value of b.

This issue is strongly related to #1088 and #1087

[maxhuettenrauch]

I'm quite sure these assignments shouldn't be valid for a batch, should they? If so, what should be the expected behavior in your given examples, especially if another batch item with the same key/structure should be concatenated or stacked? Maybe some kind of sanity check should run before constructing a batch, although this kind of data probably doesn't occur in the RL context.

[MischaPanch]

Yes, one way to solve this issue is to raise errors on invalid assignments. It's generally better to be strict. So many things can go wrong inside an RL training, we should help the users by eliminating as many possible sources of errors as we can

[MischaPanch]

We shouldn't completely eliminate object-type arrays though. For example our current vector envs return an array of dicts for the info entry. Custom envs are also free to return whatever they want as observations

[maxhuettenrauch]

Arrays of dicts should be fine (such as info entries), as long as the dict keys are consistent.

[MischaPanch]

I agree. It might not make sense to solve the batch related issues independently, maybe a single PR will have to address multiple of them