TypeError: Object of type 'ndarray' is not JSON serializable #268
Description
WORKER 0 seed: 1566624902
Traceback (most recent call last):
File "./pytorch/train.py", line 306, in train
ret_dict = net_parallel(example_torch)
File "/home/chi_zhang/anaconda3/envs/second_alpha/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/home/chi_zhang/yz/second_alpha/second.pytorch/second/pytorch/models/voxelnet.py", line 363, in forward
preds_dict = self.network_forward(voxels, num_points, coors, batch_size_dev)
File "/home/chi_zhang/yz/second_alpha/second.pytorch/second/pytorch/models/voxelnet.py", line 335, in network_forward
preds_dict = self.rpn(spatial_features)
File "/home/chi_zhang/anaconda3/envs/second_alpha/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/home/chi_zhang/yz/second_alpha/second.pytorch/second/pytorch/models/rpn.py", line 394, in forward
res = super().forward(x)
File "/home/chi_zhang/yz/second_alpha/second.pytorch/second/pytorch/models/rpn.py", line 318, in forward
x = self.blocksi
File "/home/chi_zhang/anaconda3/envs/second_alpha/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/home/chi_zhang/yz/second_alpha/second.pytorch/torchplus/nn/modules/common.py", line 89, in forward
input = module(input)
File "/home/chi_zhang/anaconda3/envs/second_alpha/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/home/chi_zhang/anaconda3/envs/second_alpha/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 320, in forward
self.padding, self.dilation, self.groups)
RuntimeError: CUDA out of memory. Tried to allocate 34.38 MiB (GPU 0; 11.75 GiB total capacity; 780.10 MiB already allocated; 7.81 MiB free; 11.03 MiB cached)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "./pytorch/train.py", line 663, in
fire.Fire()
File "/home/chi_zhang/anaconda3/envs/second_alpha/lib/python3.6/site-packages/fire/core.py", line 138, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/home/chi_zhang/anaconda3/envs/second_alpha/lib/python3.6/site-packages/fire/core.py", line 471, in _Fire
target=component.name)
File "/home/chi_zhang/anaconda3/envs/second_alpha/lib/python3.6/site-packages/fire/core.py", line 675, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "./pytorch/train.py", line 421, in train
print(json.dumps(example["metadata"], indent=2))
File "/home/chi_zhang/anaconda3/envs/second_alpha/lib/python3.6/json/init.py", line 238, in dumps
**kw).encode(obj)
File "/home/chi_zhang/anaconda3/envs/second_alpha/lib/python3.6/json/encoder.py", line 201, in encode
chunks = list(chunks)
File "/home/chi_zhang/anaconda3/envs/second_alpha/lib/python3.6/json/encoder.py", line 428, in _iterencode
yield from _iterencode_list(o, _current_indent_level)
File "/home/chi_zhang/anaconda3/envs/second_alpha/lib/python3.6/json/encoder.py", line 325, in _iterencode_list
yield from chunks
File "/home/chi_zhang/anaconda3/envs/second_alpha/lib/python3.6/json/encoder.py", line 404, in _iterencode_dict
yield from chunks
File "/home/chi_zhang/anaconda3/envs/second_alpha/lib/python3.6/json/encoder.py", line 437, in _iterencode
o = _default(o)
File "/home/chi_zhang/anaconda3/envs/second_alpha/lib/python3.6/json/encoder.py", line 180, in default
o.class.name)
TypeError: Object of type 'ndarray' is not JSON serializable
i have reduce the batch size to 2, but the error is still there. i use the titan v which has 12G gpu memory.