OverflowError: integer 2176051861 does not fit in 'int'

[tapadeep]

I'm trying to scatter data across 25 processors using:
attributes=comm.scatter(attributes,root=0)

But I always run into the following problem:

Traceback (most recent call last):
  File "PBGENA.py", line 99, in <module>
    attributes=comm.scatter(attributes,root=0)
  File "mpi4py/MPI/Comm.pyx", line 1267, in mpi4py.MPI.Comm.scatter
  File "mpi4py/MPI/msgpickle.pxi", line 730, in mpi4py.MPI.PyMPI_scatter
  File "mpi4py/MPI/msgpickle.pxi", line 125, in mpi4py.MPI.Pickle.dumpv
  File "mpi4py/MPI/msgbuffer.pxi", line 44, in mpi4py.MPI.downcast
OverflowError: integer 2176051861 does not fit in 'int'

I know for a fact that my data has no integer >=2176051861. The data I'm trying to scatter is a dictionary containing data of type <class 'scipy.sparse.csr.csr_matrix'>

Any help is greatly appreciated.

[dalcinl]

This happens when the total memory of the pickled object is larger than 2GB, the message count overflows a 32bit int. This is a well know limitation of MPI-1/2/3, hopefully fixed on MPI-4, but not yet available on a released Open MPI or MPICH (in mpi4py, there is already a branch with MPI-4 support).

My recommendation is to try replace your scatter call with a plain send/recv implementation. As long as each individual (pickled) item is less than 2GB, you should be fine.

# replacement for:  item = comm.scatter(attributes, root=0)

if comm.size == 0:
    (item,) = attributes
elif comm.rank == 0:
    for i, attr in enumerate(attributes):
        if i == 0:
            item = attr
        else:
            comm.send(attr, dest=i)
else:
    item = comm.recv(source=0)

[tapadeep]

Thanks, but this is not exactly the solution I was looking for. The data I'm working with is enormous, so if this is an issue, I need to manually keep track of the amount of information I'm sending across. Now I can do that, but the point of using scatter was to batch all the sends and save time.

[dalcinl]

If your data is enormous, you should still use send/recv, but use the auxiliary module mpi4py.util.pkl5. Of course, to take advantage of this, you really need the objects you communicate to support pickle protocol 5 and out of band buffers. NumPy does support pickle protocl version 5. Not sure about SciPy sparse matrices.

Using mpi4py.util.pkl5 will give you two benefits:

1. Overcome the 2GB limit of MPI-1/2/3.
2. Avoid copy of large buffer-like objects (e.g. numpy arrays).