What does this Error mean?

[Juicechen95]

Hi, I tried to load an LMDB data of ImageNet, it is very large, about 140G. And there are 1281167 instances in the data. But I meet this error. It seems to fail to load a too big LMDB file.

Traceback (most recent call last):
  File "main.py", line 388, in <module>
    main()
  File "main.py", line 154, in main
    num_workers=args.workers,)
  File "/home/jcz/github/pytorch_examples/imagenet/sequential_imagenet_dataloader/imagenet_seq/data.py", line 166, in __init__
    ds = td.LMDBData(lmdb_loc, shuffle=False)
  File "/home/jcz/github/tensorpack/tensorpack/dataflow/format.py", line 91, in __init__
    self._set_keys(keys)
  File "/home/jcz/github/tensorpack/tensorpack/dataflow/format.py", line 109, in _set_keys
    self.keys = loads(self.keys)
  File "/home/jcz/github/tensorpack/tensorpack/utils/serialize.py", line 29, in loads_msgpack
    return msgpack.loads(buf, raw=False, max_bin_len=1000000000)
  File "/home/jcz/Venv/pytorch/lib/python3.5/site-packages/msgpack_numpy.py", line 214, in unpackb
    return _unpackb(packed, **kwargs)
  File "msgpack/_unpacker.pyx", line 187, in msgpack._cmsgpack.unpackb
ValueError: 1281167 exceeds max_array_len(131072)

python env: 3.5
how can I fix this error?

[ppwwyyxx]

You can downgrade msgpack to 0.5.6

[Juicechen95]

You can downgrade msgpack to 0.5.6

how to do that?

[Juicechen95]

You can downgrade msgpack to 0.5.6

Great! It works. Thank you!