Refactored GenomicAnnotation to reduce memory usage

[zhuchcn]

The GenomicAnnoation class was refactored slightly. When loading data from GTF, only useful annotation information is kept. For all the features of the same transcript, their IDs are now using the same object by reference. The memory usage for annotation itself is about 6.5 GB when reading directly from GTF and 10.5 GB when loading from the pickled object. Issue #394 is still open.

[lydiayliu]

The memory usage for annotation itself is about 6.5 GB when reading directly from GTF and 10.5 GB when loading from the pickled object.

Oh I missed this issue. Why would memory usage be double when reading from the pickled object? Isn't the pickled object just the loaded annotation saved?

[zhuchcn]

I was looking at this whole morning but still don't have a good conclusion. And this seems to only happen to the annotation file but not the genome fasta. Reading directly from FASTA uses just as much memory as loading from the pickled one.

[lydiayliu]

Wow interesting. Maybe a very specific data structure is not agreeing with pickling...

[zhuchcn]

Quoting from the PR that I accidentally opened for no reason..

I now save the genomic annotation in a compressed text format instead of pickle to solve #394 . The other objects are fine with pickle. This takes slightly longer time to load, but not a lot. The memory usage is now about 18 GB after loading everything, with 3.5 GB before launching. Still needs to be improved. Hopefully in the future.

I think we can close #394 for now. We should further optimize the memory usage but not the top priority for now.

[lydiayliu]

sorry i forgot about this. no thoughts whatsoever XD sounds like work