Large memory requirements (for very large binning jobs)

[luispedro]

This is a bit of a meta-issue as this seems to be a oft-reported problem:

• numpy.core._exceptions._ArrayMemoryError: Unable to allocate 2.86 TiB for an array with shape (886662, 886662) and data type float32 #165
• SemiBin2 fails to generate bins for large assemblies #166 (@apcamargo rightfully comments that at least we should try to be more helpful with detecting and reporting this situation)
• Extract then merge #162 (a potential solution, but would change a lot of internals)

This has also been discussed on the mailing list: https://groups.google.com/g/semibin-users/c/lk1M3LYfsik

Fundamentally, SemiBin does not scale very well in terms of memory requirements for very large binning jobs (jobs larger than what we had originally envisioned)