JSONDecode error while Training - Balancing Ball Example

[krishnanpc]

Hey guys,
had the same error @karthikveeramani had, but used 2017.1.0 , now the Basic works, but I get this error while training. The build is fine, (when i double click open the executable, it is working fine). The training throws this error leaving the executable running in a Not Responding condition.

I then changed the instantiation of env to:
env = UnityEnvironment(file_name=env_name, worker_id=2, base_port=8080)
as @Adamage suggested, but then get a different error

Any help is appreciated. :)

[vincentpierre]

I think when you set the baseport to 8080 you are listening to something other than your environment. The error The Unity environment took too long to respond usually happens when the brain in the scene is not set to external. You need to set the brain to external to perform training. Hope this helps.

[Adamage]

Hey - do not put the port. Just change the worker id, should be fine!

[krishnanpc]

Thanks so much for writing back!

The brain is set to external, the problem is when while I train I get a JSONDecoder error.
@Adamage , yup, changing the worker ID gets it "running", but I am not able to train it.

[krishnanpc]

Hi again, here is the exact problem I have. I am able to get it running, but while training it throws this JSON Error and crashes.

Traceback (most recent call last):
File "ppo.py", line 105, in
new_info = trainer.take_action(info, env, brain_name)
File "C:\Users\krishnan\Downloads\ml-agents-master (1)\ml-agents-master\python\ppo\trainer.py", line 55, in take_action
new_info = env.step(actions, value={brain_name: value})[brain_name]
File "C:\Users\krishnan\Downloads\ml-agents-master (1)\ml-agents-master\python\unityagents\environment.py", line 339, in step
return self._get_state()
File "C:\Users\krishnan\Downloads\ml-agents-master (1)\ml-agents-master\python\unityagents\environment.py", line 207, in _get_state
state_dict = self._get_state_dict()
File "C:\Users\krishnan\Downloads\ml-agents-master (1)\ml-agents-master\python\unityagents\environment.py", line 174, in get_state_dict
state_dict = json.loads(state)
File "F:\Program Files\Anaconda3\lib\json_init.py", line 319, in loads
return _default_decoder.decode(s)
File "F:\Program Files\Anaconda3\lib\json\decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "F:\Program Files\Anaconda3\lib\json\decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None

json.decoder.JSONDecodeError: Expecting value: line 92 column 17 (char 1460)
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "BestSplits" device_type: "CPU"') for unknown op: BestSplits
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "CountExtremelyRandomStats" device_type: "CPU"') for unknown op: CountExtremelyRandomStats
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "FinishedNodes" device_type: "CPU"') for unknown op: FinishedNodes
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "GrowTree" device_type: "CPU"') for unknown op: GrowTree
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "ReinterpretStringToFloat" device_type: "CPU"') for unknown op: ReinterpretStringToFloat
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "SampleInputs" device_type: "CPU"') for unknown op: SampleInputs
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "ScatterAddNdim" device_type: "CPU"') for unknown op: ScatterAddNdim
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "TopNInsert" device_type: "CPU"') for unknown op: TopNInsert
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "TopNRemove" device_type: "CPU"') for unknown op: TopNRemove
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "TreePredictions" device_type: "CPU"') for unknown op: TreePredictions
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "UpdateFertileSlots" device_type: "CPU"') for unknown op: UpdateFertileSlots

The JSON Decoder error Im getting is in the environment.py file. Any help is appreciated. :)

[Adamage]

Maybe the json state written previously was corrupted? Do you have access rights to all the files and folders ?

I looked at the code, and there are two json.loads. One is in environment init, and you have no trouble there. Second one is in line 339:
environment.py line 339, in step return self._get_state()
line 174, in get_state_dict : state_dict = json.loads(state)

Try to debug this line 174, see what is the content of "state" variable.

[vincentpierre]

I think @Adamage is on the right track. It might be possible that the json is not right. Can you replace line 174

state_dict = json.loads(state)

with

try: 
    state_dict = json.loads(state)
except:
    print(state)

and tell us what the state was ?

[krishnanpc]

Thanks for the reply guys.

INFO:unityagents.environment:
'Ball3DAcademy' started successfully!
{'--batch-size': '64',
'--beta': '1e-3',
'--buffer-size': '2048',
'--epsilon': '0.2',
'--gamma': '0.99',
'--help': False,
'--hidden-units': '64',
'--keep-checkpoints': '5',
'--lambd': '0.95',
'--learning-rate': '3e-4',
'--load': False,
'--max-steps': '1e6',
'--num-epoch': '5',
'--run-path': 'ppo',
'--save-freq': '50000',
'--summary-freq': '10000',
'--time-horizon': '2048',
'--train': True,
'--worker-id': '0',
'': 'thiswillwork'}
Unity Academy name: Ball3DAcademy
Number of brains: 1
Reset Parameters :

Unity brain name: Ball3DBrain
Number of observations (per agent): 0
State space type: continuous
State space size (per agent): 8
Action space type: continuous
Action space size (per agent): 2
Memory space size (per agent): 0
Action descriptions: ,
Loading Model...
{

"brain_name": "Ball3DBrain",

"agents": [

786,

790,

794,

798,

828,

838,

848,

850,

854,

860,

866,

878

],

"states": [

0.138556,

0.151974112,

0.212692112,

0.195137784,

0.00699138641,

-0.196405083,

-0.360402226,

0.249847561,

0.0534343533,

-0.177705958,

0.275514752,

0.7058239,

-0.202565953,

0.0,

-0.5885999,

0.0,

-0.206159458,

-0.0288430844,

0.2721056,

0.635192156,

0.07272377,

0.0,

-0.7847998,

0.0,

-0.205628216,

0.131468177,

-0.0459213741,

0.129028708,

0.09524937,

0.36126858,

-0.210067838,

0.160864562,

0.158972234,

0.144568011,

-0.3015967,

0.0203389637,

0.321134865,

-0.5403493,

-0.453727067,

0.06880206,

0.143490717,

-0.0785889,

-0.306238949,

-0.02469344,

-0.360634238,

-0.178046629,

-0.147952244,

-0.42478925,

-0.226226285,

0.178676635,

-0.210384756,

0.129888341,

0.31028223,

0.240352243,

-0.193306446,

0.166099608,

0.0334274247,

0.07945089,

0.2565689,

0.15646401,

-0.124899291,

0.0,

-1.56959939,

0.0,

0.0283072963,

-0.052078072,

0.206688121,

0.6351919,

0.244981959,

0.0,

-0.7847998,

0.0,

0.206082135,

0.0691380352,

Traceback (most recent call last):
File "ppo.py", line 108, in
new_info = trainer.take_action(info, env, brain_name)
File "C:\Users\krishnan\Documents\ml-agents-master\ml-agents-master\python\ppo\trainer.py", line 55, in take_action
new_info = env.step(actions, value={brain_name: value})[brain_name]
File "C:\Users\krishnan\Documents\ml-agents-master\ml-agents-master\python\unityagents\environment.py", line 348, in step
return self._get_state()
File "C:\Users\krishnan\Documents\ml-agents-master\ml-agents-master\python\unityagents\environment.py", line 216, in _get_state
state_dict = self._get_state_dict()
File "C:\Users\krishnan\Documents\ml-agents-master\ml-agents-master\python\unityagents\environment.py", line 184, in _get_state_dict
return state_dict
UnboundLocalError: local variable 'state_dict' referenced before assignment
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "BestSplits" device_type: "CPU"') for unknown op: BestSplits
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "CountExtremelyRandomStats" device_type: "CPU"') for unknown op: CountExtremelyRandomStats
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "FinishedNodes" device_type: "CPU"') for unknown op: FinishedNodes
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "GrowTree" device_type: "CPU"') for unknown op: GrowTree
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "ReinterpretStringToFloat" device_type: "CPU"') for unknown op: ReinterpretStringToFloat
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "SampleInputs" device_type: "CPU"') for unknown op: SampleInputs
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "ScatterAddNdim" device_type: "CPU"') for unknown op: ScatterAddNdim
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "TopNInsert" device_type: "CPU"') for unknown op: TopNInsert
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "TopNRemove" device_type: "CPU"') for unknown op: TopNRemove
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "TreePredictions" device_type: "CPU"') for unknown op: TreePredictions
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "UpdateFertileSlots" device_type: "CPU"') for unknown op: UpdateFertileSlots

and the Unity window is the folliwing,

[vincentpierre]

Ok, It looks to me that your message got cut off, which is weird since the message is not that long. We will be working on a way to split large messages into smaller messages. In the mean time, can you try removing all the agents but one in the scene? I would like to know if the communication for one agent can at least be handled.

[gerardsimons]

I am also facing this issue constantly now. Used to try with a bunch of agents, now I get the JSON Decode error even with just one agent. I changed the code to print the state, which shows CONFIG_REQUEST. Also the Unity app errors. Looking in the logs I found :

SocketException: The socket has been shut down
  at System.Net.Sockets.Socket.Send (System.Byte[] buffer, System.Int32 offset, System.Int32 size, System.Net.Sockets.SocketFlags socketFlags) [0x00016] in <344dc4d3f1ad41809df78607b6121a41>:0 
  at System.Net.Sockets.Socket.Send (System.Byte[] buffer) [0x00000] in <344dc4d3f1ad41809df78607b6121a41>:0 
  at ExternalCommunicator.GetResetParameters () [0x00016] in /Users/gerard/Git/boat-simulator/unity-env/Assets/ML-Agents/Scripts/ExternalCommunicator.cs:136 
  at Academy.RunMdp () [0x0010a] in /Users/gerard/Git/boat-simulator/unity-env/Assets/ML-Agents/Scripts/Academy.cs:307 
  at Academy.FixedUpdate () [0x0000e] in /Users/gerard/Git/boat-simulator/unity-env/Assets/ML-Agents/Scripts/Academy.cs:260 

Not sure if that causes the decode error or is rather caused by it. I only seem to have this error on Mac so far though, not on Linux / Ubuntu

[vincentpierre]

@gerardsimons If you see CONFIG_REQUEST my intuition is that the order of sending of the messages is not right. It might be because you compiled the environment with a newer or older version of ML-Agents compared to the python interface. I would recommend copying the files from you linux machine if they work and make sure you do pip install . and not pip install unityagents to be sure the library you have matches.

[gerardsimons]

@vincentpierre Thanks for the tip, but I don't think this is the issue. I use the two machines to work on the same Git project, sometimes working on one, then on the other. The unityagents code I took from this GitHub repo (I am not using a pip package), but as I said they share this same codebase. It was working for a very long time, and only recently started breaking. Any other ideas?

[vincentpierre]

The fact that this error happens on Mac and not Linux is puzzling me. The only thing I can think of is recompiling the environments for both Mac and Linux. Create a new virtualenv to make sure you import the right unityagents when calling from unityagents import *, and maybe update ML-Agents with the latest version.

[krishnanpc]

Hey @vincentpierre and @Adamage thanks for the replies. I tried with one agent, in a different computer and it seems to work perfectly. And now I use that .bytes file in my system with multiple agents, thanks for the support. :)

[gerardsimons]

@vincentpierre : I tried that, also checked and verified the code, but without much luck. I also re-cloned the repo and updated my Project with latest Python scripts. I don't always get the errors, and I am also very puzzled on what causes this. In fact, now I am getting them on Ubuntu too..

The strange thing is though that the actual Unity environment does launch, and I do see the agents move (while the brain is on External), so somehow I guess it's kind of working but I don't see how and I don't see any other processes running. It's also not with all the built environments, just some of them, but I am unsure what settings in Unity affect this behavior. Any ideas?

2017-11-06 09:53:29.549411: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2017-11-06 09:53:29.549435: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2017-11-06 09:53:29.549440: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2017-11-06 09:53:29.684041: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:893] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2017-11-06 09:53:29.684501: I tensorflow/core/common_runtime/gpu/gpu_device.cc:955] Found device 0 with properties:
name: GeForce GTX 1060 6GB
major: 6 minor: 1 memoryClockRate (GHz) 1.7085
pciBusID 0000:01:00.0
Total memory: 5.93GiB
Free memory: 4.54GiB
2017-11-06 09:53:29.684517: I tensorflow/core/common_runtime/gpu/gpu_device.cc:976] DMA: 0
2017-11-06 09:53:29.684522: I tensorflow/core/common_runtime/gpu/gpu_device.cc:986] 0:   Y
2017-11-06 09:53:29.684530: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1045] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1060 6GB, pci bus id: 0000:01:00.0)
Loading Model './models/ptr_sjso_h64'...
Model succesfully loaded from ./models/ptr_sjso_h64
INFO:tensorflow:Restoring parameters from ./models/ptr_sjso_h64/model-1850000.cptk
INFO:tensorflow:Restoring parameters from ./models/ptr_sjso_h64/model-1850000.cptk
/home/gerard/.virtualenvs/boatml/lib/python3.5/site-packages/numpy/core/fromnumeric.py:2909: RuntimeWarning: Mean of empty slice.
  out=out, **kwargs)
/home/gerard/.virtualenvs/boatml/lib/python3.5/site-packages/numpy/core/_methods.py:80: RuntimeWarning: invalid value encountered in double_scalars
  ret = ret.dtype.type(ret / rcount)
Mean Reward: nan
Traceback (most recent call last):
  File "ppo.py", line 156, in <module>
    info = env.reset(train_mode=train_model)[brain_name]
  File "/home/gerard/Git/boat-simulator/ml/unityagents/environment.py", line 197, in reset
    return self._get_state()
  File "/home/gerard/Git/boat-simulator/ml/unityagents/environment.py", line 208, in _get_state
    state_dict = self._get_state_dict()
  File "/home/gerard/Git/boat-simulator/ml/unityagents/environment.py", line 175, in _get_state_dict
    state_dict = json.loads(state)
  File "/usr/lib/python3.5/json/__init__.py", line 319, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.5/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.5/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

[vincentpierre]

This looks like a different error, this time, the json seems to be empty instead of too large. I think these issues are addressed in the development branch. Would it be possible for you to try the balance ball on this branch ?