PDBReader reopens files reading every frame

[kain88-de]

The PDBReader opens and closes the trajectory file for reading every frame, see. Reopening the files all the time means we are doing direct I/O. This is an issue when MDAnalysis is run on an HPC cluster with 10 to 100 of different processes at the same time.

HPC cluster usually have some kind of heavily buffered distributed filesystem, like the GPFS. These type of filesystems hate direct I/O, they prefer buffered I/O. This would mean keeping a file open for longer period of time and reading from it. They can buffer that. Closing the file breaks their buffering mechanism. The end result is super slow file access for everybody on a cluster.

The solution is to keep the file pointer open. I did the same for the XTC/TRR readers.

[richardjgowers]

#239 was trying to move towards always accessing this way, but it sounds like this is a bad idea now

[kain88-de]

Yes if we want to allow MDAnalysis to run on HPC clusters for massively parallel analysis we have to keep the file descriptors open.