GUI doesn't update in the human mode

[Nanami18]

Hi!

I installed the benchmark following the instruction on the website, and was trying to play around with it in human mode using "python gui/run_gui.py --env "memory_maze:MemoryMaze-9x9-HD-v0". However, it looks like the observation never change no matter what action I do (only "exit" and "reset" works).

I have made sure that my keyboard works correctly, and I have also tried binding the actions to other keys instead of arrow keys, but it still doesn't work. When I printed out the action in line 174, the correct action is printed out after I pressed a key, but the observation still doesn't change. (didn't try to store the full observation, but obs.sum() is always the same no matter what I do, which probably indicates some errors).

Besides that, the only thing that is getting updated is the stuff in the bottom-left corner, I have no idea what that is but it seems to be changed according to my actions.

I have little to no experience in Pygame development so I hope to seek some help here. Let me know if you need more information from me, thanks!

[jurgisp]

That's very odd. When I put a print command on line 175

print(f'steps: {steps}, action: {action}, obs.mean: {obs.mean().round(2)}')

and turn around with left/right arrows (action: 2|3) - everything seems to be working fine, here is a screenshot:

Can you send a similar screenshot on your end?

[Nanami18]

I recorded a video with the random action mode turned on. I added some debugging statements that changed the line number, but they should not affect the behavior. In the middle, I manually clicked "reset" a few times. The console correctly outputs the reset action while the GUI is still not being updated.

2023-03-31.14-38-32.mp4

[jurgisp]

Yeah, it looks like a problem with GUI, that is, with pygame, on your system. I've tested it on Mac and Windows, but perhaps there is a problem in Linux.

Like you said, there is something drawn in the lower-bottom corner of the window, which looks like it's supposed to be the image, but it is rendered in an incorrect way and incorrect location. Unfortunately, I can't really help with that, I'm also not an expert in pygame. It may be worth checking if other people have similar problems with pygame?

[Nanami18]

Okay, thanks for your response. What's the pygame version you are using?

[jurgisp]

The version is not fixed, so you have whatever latest version there was when you did pip install pygame. I have the latest, pygame==2.3.0. Can you check what it is? Perhaps worth trying to upgrade pip install --upgrade pygame.

[Nanami18]

I have the same version, also installed through pip. I guess there are some system-dependent issues then, thanks for your help though.

[zoharri]

I'm having the exact same problem :(
Did you end up solving this?

[jurgisp]

@zoharri what is your OS? Personally I haven't been able to reproduce or solve it.

[dirkmcpherson]

Same problem on Ubuntu 20.04. The mujoco rendering setting was messing with the pygame rendering. Small PR here

[jurgisp]

@zoharri, @Nanami18 can you confirm if the patch by @dirkmcpherson solves the issue?