std::bad_alloc in WriteSOCatalog

[rtobar]

VR is crashing in WriteSOCatalog with the following message:

[0000] [1445.514] [ info] io.cxx:1292 Saving SO particle lists to halos-2.catalog_SOlist.0
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
/var/slurm/slurmd/job2998665/slurm_script: Zeile 4: 16915 Abgebrochen             builds/53/stf -i /cosma/home/dc-borr1/c7dataspace/XL_wave_1/runs_correct_dt/Run_0/eagle_0000 -C /cosma/home/dc-borr1/c7dataspace/XL_wave_1/runs_correct_dt/vrconfig_3dfof_subhalos_SO_hydro.cfg -I 2 -o halos-2

This feels like a bug introduced with #57, but it will require a bit of investigation.

[rtobar]

I went back to this issue yesterday and today. This is caused by a copy operation of many PIDs from a array of vectors into a single vector, which is then used for writing the resulting data into disk. The same applies for an array of types:

VELOCIraptor-STF/src/io.cxx

Lines 1506 to 1525 in 64de17b

	if (nSOids>0) {
	idval.resize(nSOids,0);
	nSOids=0;
	for (Int_t i=1;i<=ngroups;i++) {
	for (Int_t j=0;j<SOpids[i].size();j++) {
	idval[nSOids++]=SOpids[i][j];
	}
	SOpids[i].clear();
	}
	#if defined(GASON) || defined(STARON) || defined(BHON)
	typeval.resize(nSOids,0);
	nSOids=0;
	for (Int_t i=1;i<=ngroups;i++) {
	for (Int_t j=0;j<SOtypes[i].size();j++) {
	typeval[nSOids++]=SOtypes[i][j];
	}
	SOtypes[i].clear();
	}
	#endif
	}

It just so happens that the original array of vectors holds quite a big number of IDs (in our case nSOids is 2718973906), and therefore these copies contribute greatly to bringing the memory requirements of the application beyond what's available on the machine:

#### At the beginning of GetSOMasses there's 65 GB of resident memory used:
[0000] [1148.534] [ info] substructureproperties.cxx:3234 Memory report at substructureproperties.cxx:3234 ... Resident: 64.978 [GiB] [...]

### Just before WriteSOCatalog is called, 100 GB are being used:
[0000] [1402.079] [ info] substructureproperties.cxx:3770 Memory report at substructureproperties.cxx:3770 [...] Resident: 99.044 [GiB] [...]

With nSOids == 2718973906 the two copies require ~20 GB of memory in a machine that usually offers ~120 of free memory to processes; hence we end up with a crash.

This has been a known problem for a while though:

VELOCIraptor-STF/src/io.cxx

Lines 1213 to 1214 in 64de17b

	///\todo optimisation memory wise can be implemented by not creating an array
	///to store all ids and then copying info from the array of vectors into it.

I'll relabel this as an improvement rather than a bug.

[rtobar]

In the original code, the original each of the vectors in the array-of-vectors was emptied after their contents were copied (but it wasn't shrunk via shrink_to_fit), and the full block of memory needed for the new 1D array was requested in full.

A non-invasive, possible improvement for this situation is then to resize the 1D array multiple times as we iterate over the original array-of-vectors, and as we effectively release the memory held by each of those individual vectors by calling shrink_to_fit. This will result in more memory allocations, but should keep the memory requirements in check, without having to touch more of the code.

[rtobar]

Two other ideas about this that can be quickly implemented:

• The flattening of ID and type values into their respective vectors doesn't need to happen at the same time. At the moment both are first flattened, and then sequentially written. Instead we could flatten one, write it, clear that memory, then do the same with the other
• The new vector of types could be declared with a one-byte integer type (i.e., char instead of int). This way the flat copy of the types array would be 1/4th of what's now

[rtobar]

The first bullet point mentioned in the last comment has now been merged into the master branch, as its implementation also fixed the problem described in #108. With this change The maximum RSS has now decreased, but extra copies are still required.

Although this is progress, a better solution is of course to avoid having to copy data in the first place, which should be possible if we collected PIDs and types into a single vector in the first place. This could be investigated in the future.