DOC: Example on how to parse comments

[summerela]

As a bioinformatician, I frequently use Pandas to parse hideous scientific file formats. Often standard biological file formats include a section of header comments that provide useful information about the file, such as reference build, species, versioning, etc.

For example, a typical VCF file:

#VCF v1.1.4
#RefBuild: hg19
#Assay:Oncopanel
#CHROM    POS    REF    ALT    QUAL    FILTER    INFO
22            123845    A      G          .             .           STRAND=+

Above, we'd need to know the RefBuild to determine which species and genome version we are working with, and Assay to let us know which assay of mutations that we are looking for. There are all strange manner of things found in these headers that is often important to our analysis.

Currently, I would need to read in the file once without Pandas to grab the header information, and then read it in again with pandas to skip the commented lines and turn the content into a dataframe. I primarily build pipelines for processing very large datasets and this little workaround is often the bane of my existence.

I would like to put in a feature request to add this ability. Similar questions/requests already exist on StackOverflow, as well Thank you.

[WillAyd]

I'm not sure this is a generalizable request. What in the context of pandas would you expect to be done with the first few lines?

[summerela]

Great point, but it's a very common need in the science field. I currently have to read in the file once to grab the header, pull out and store the information as variables, or use the information as a check.

For example, pull out the first commented line from two different files (such as VCF and FASTA), and make sure the reference build matches, as there is no sense in continuing a pipeline if one genome is human and the other is canine.

I suppose you could easily just store the commented lines as a string and let the user parse the info from there. It would save I/O on reopening a file more than once.. and most of the files I deal with are upwards of 30 gigs each. For the love of science?

[WillAyd]

I've seen this before but I'm still -1 as I don't believe it is in the scope of pandas to resolve, nor do I think there would be a large enough consensus as to how to actually handle that type of header information (ex: should it be a string or dict return?)

I've been wrong before so let's leave open to see what others think.

I currently have to read in the file once to grab the header

IIUC you can just open a file handle, loop until you don't see comments to start and then simply pass that opened handle to read_csv (assuming your checks on comments have passed) to prevent having to close / open the file multiple times

For the love of science?

Don't get me started

[summerela]

All valid points. Thanks for taking a look.

I didn't realize you could pass the file handle to read_csv; I'll check that out, as that would be a vast improvement.

For the love of science.. yeah, that can definitely be a bit of a love/hate relationship.
Cheers!

[jorisvandenbossche]

I didn't realize you could pass the file handle to read_csv; I'll check that out, as that would be a vast improvement.

It would be really nice to have an example on how to do this in the documentation (open a file, parse the first few manually, pass the rest to read_csv), as it is indeed something not that rare.
Would like to do a PR for that?

[WillAyd]

As pointed out by @datapythonista here: #26141 (comment) you should be able to use comment="#" for your purposes

[datapythonista]

If I understand correctly, the problem is not skipping the comments, but ignoring all them except the last one, which contains the header.

I don't think there is any way to generalize this, in the StackOverflow question the column names are available one per row, one in the description of the issue they are all in the last comment.

From pandas point of view, I think we're providing comment='#' and skiprows=4, and header=None which can be used for parsing the file correctly, without the header names. Parsing the header names should be trivial with Python, but totally dependent on a custom format.

In the case of the example with skiprows=3 would get the column names, with the # prefixed to the first column name. But I don't think we want to add something tricky (and potentially confusing) like this in the docs:

df = pandas.read_csv(fd, skiprows=3)
df.columns = [df.columns[0][1:], *df.columns[1:]]

-1 on this issue. Personally I think parsing a csv comments is a general Python problem, and very specific to the csv format, and I don't think we should have an example in the docs.

[WillAyd]

Thanks for the input @datapythonista ! After reading again I agree with you. Closing out