change the population indices to increase space efficiency

[mikejiang]

Right now the indices are stored as global bit vector for the purpose of fast indexing the raw data. It gives good performance of logical operations like unions and intersections compared to the integer indices,yet costing more space. For instance, the FCS file of 250k events is about 15M.
170 gates will end up having indices of N*170/10^6/8=5.2MB. which is 1/3 of raw data size.

We have following solutions based on the discussions so far:
1.compression of bit vector( run-length encoding is one of compressing techniques we can use)

2.using global integer indices, if most of gates are rare populations (like cytokine gates),it could be more efficient even integer vector is 32 times as big as bit vector

3.using local indices instead of global one(suggested by Raphael ages ago), it can be either bit or integer vector.

1st and 3rd can ease the space problem for sure,just at cost of some extra computation time (either sorting or converting local indices to global).

A modified version of 2nd could be promising too: basically we could mix two types of indices in one gating tree, each gate can store indices as either bit or integer vector, based on which ever is smaller.

[mikejiang]

it might be helpful to serialize the gatingset in xml format, to have a close look to find out why it takes so much space both in memory and disk.

[mikejiang]

After investigating the output of boost serialization in xml format, It is clear that boost does a good job of storing the nested data structures and pointers without repetitively storing the same data.
Here is the change of file size for single sample after removing indices:
Cytotrol B cell: (16 gates): 6.7M--->5.7M
HVTN ICS (150 gates): 54M ---> 202K
Most of 150 gates are boolean gates,which could be rare events and thus can stored as integer instead of bool vector.

I am gonna try the mixed-type indices to see how much gains we can achieve.

[mikejiang]

Here is the result after applying mixed-type indices:
Cytotrol B cell: (16 gates): 6.7M--->6.3M
HVTN ICS (150 gates): 54M ---> 10.4M
The second use case proves it to be 5-time smaller than before.

I am gonna try local indices to see if we can gain more.

[mikejiang]

Here is the disk usage after archiving the gating set contains 222 (samples ) * 33 (nodes/gates)
R object: 116K
CDF: 5 G
C object:451M (i.e 2M/sample)

I've calculated the cost of indices storage according to the gating results, it is about 297K/sample, and 66M in total for 222 samples,which is 15% of entire C structure.

When loading it back to R, the output from pmap shows the process used up to 454M memory in total,which is consistent with the disk usage (R+C).

So not sure where those 50G memory usage came from in the initial R session where gating was done.
It must be leaking somewhere, either by flowWorkspace or any other R packages used in flowAP.

More investigation is needed.

[mikejiang]

After rm all the R objects including gatingSet from R, 50G remained processed by R.
Then load 100 samples back to R as flowSet , and the memory footprint reported by pmap remained the same without increasing.
So conclusion would be:
R keeps the unused memory without returning to OS (regardless gc call) for future usage unless OS asks for them.
So there should be no obvious memory leaking.

[mikejiang]

Here are the file sizes of the GatingSet for 1518 samples:

total 62G
57G file32f41c13ab1b.nc  
5.1G file32f4224a318.dat
2.3M file32f4fb0dbd9.rds

Since we are dealing much bigger data sets than before, I am wondering if there is still room to improve the speed of initial loading C data structure by further reducing the dat file size.

Firstly,I cloned one sample from this dataset and saved the C data to disk

> .Call("R_saveGatingSet", test_gs@pointer, file.path(path,"output/test.dat"))

The dat file is about 3.8M.
Estimating the cost of pop indices by

> pop_event_count <- getPopStats(gh)[,2,drop=F]
> nTotalEvents <-  pop_event_count[1,]
> sum(
+     apply(pop_event_count,1,function(this_count){
+                           cost_int <- this_count * 4
+                           cost_bool <- nTotalEvents/8
+                           min(cost_int,cost_bool)
+             #              length(rle(this_ind)[[1]])
+                         }
+               )
+      )/2^20
[1] 0.5M

adding some overhead from STL containers (vector or vector ), the pop indices shouldn't be the major contribution to the dat file, It could be something other than indices, e.g. boost graph itself.

To verify this, I parsed this example without gating (execute=FALSE, which means pop indices are empty), the resulted dat file is 2.04M.

So the conclusion is that we may not gain significantly from further optimizing pop indices storage.
More memory profiling is need if we want to do something about the current size of dat file.

[ldash]

Sounds like you won't be able to decrease the overall size by more than 10% even if you managed to obliterate the dat file altogether, 5% at best: ~ half of ~5Gb out of the total over 50Gb...

[mikejiang]

no. Only dat file needs to be read into memory from disk.

[ldash]

Oh, that's right. Ok, then if the gains are on the order of 50%, then it is warranted...

[mikejiang]

Even after the optimizations done through changing data type from double to float and removing redundant bi-exponential transformations, the average size of C data structure for each sample (with pop indices, 3.7M) is still bigger than expected. After further troubleshooting this space issue, I found out it is the problem of boost serialization library. Here is a simple test: