calls file not recognised

[Coracollar]

Hi, I've been using methplotlib for representing my methylation frequencies. However I am having problems for representing the calls. The file is in the nanopolish call format and does not seem to have any problems when I check it (I tried with more than one calls file)

$ methplotlib -m /data/cephfs/punim1048/allbarcoded/CORT/CORT13/methylation_chromosomes/chr10_methylation_calls.tsv -n CORT13 -w ch10:63273264-63314576 -g /data/cephfs/punim1048/GRCm38_genome/annotation/mm10.ensGene.gtf --simplify

Input file /data/cephfs/punim1048/allbarcoded/CORT/CORT13/methylation_chromosomes/chr10_methylation_calls.tsv not recognized!

Detailed error:
Traceback (most recent call last):
File "/home/coracollar/anaconda3/envs/methplot/bin/methplotlib", line 10, in
sys.exit(main())
File "/home/coracollar/anaconda3/envs/methplot/lib/python3.7/site-packages/methplotlib/methplotlib.py", line 18, in main
meth_data = get_data(args.methylation, args.names, window, args.smooth)
File "/home/coracollar/anaconda3/envs/methplot/lib/python3.7/site-packages/methplotlib/import_methylation.py", line 165, in get_data
return [read_meth(f, n, window, smoothen) for f, n in zip(methylation_files, names)]
File "/home/coracollar/anaconda3/envs/methplot/lib/python3.7/site-packages/methplotlib/import_methylation.py", line 165, in
return [read_meth(f, n, window, smoothen) for f, n in zip(methylation_files, names)]
File "/home/coracollar/anaconda3/envs/methplot/lib/python3.7/site-packages/methplotlib/import_methylation.py", line 32, in read_meth
return parse_nanopolish(filename, file_type, name, window, smoothen=smoothen)
File "/home/coracollar/anaconda3/envs/methplot/lib/python3.7/site-packages/methplotlib/import_methylation.py", line 51, in parse_nanopolish
gr.pos = np.floor(gr.drop().df[["Start", "End"]].mean(axis=1))
File "/home/coracollar/.local/lib/python3.7/site-packages/pandas/core/frame.py", line 2806, in getitem
indexer = self.loc._get_listlike_indexer(key, axis=1, raise_missing=True)[1]
File "/home/coracollar/.local/lib/python3.7/site-packages/pandas/core/indexing.py", line 1552, in _get_listlike_indexer
keyarr, indexer, o._get_axis_number(axis), raise_missing=raise_missing
File "/home/coracollar/.local/lib/python3.7/site-packages/pandas/core/indexing.py", line 1639, in _validate_read_indexer
raise KeyError(f"None of [{key}] are in the [{axis_name}]")
KeyError: "None of [Index(['Start', 'End'], dtype='object')] are in the [columns]"

[wdecoster]

Do you think it would be possible to share that file with me to debug?

[Coracollar]

Sure.

…

On Mon, May 25, 2020 at 5:17 PM Wouter De Coster ***@***.***> wrote: Do you think it would be possible to share that file with me to debug? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#23 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANZ7PMECHWTIW3AHQYKIPLLRTILRZANCNFSM4NJEOWBA> .

[wdecoster]

There is a column of index numbers in the file which shouldn't be there. You can fix it with:

zcat chr10_methylation_calls.tsv.gz | cut -f2- | gzip > fixed_calls.tsv.gz

The output of that command worked for me with methplotlib.

Did you do any manual manipulations to that file or is it straight from nanopolish? If so, which version did you use?

[Coracollar]

Hi, Sorry I sent you the indexed one, but that’s not the problem, I tried with no indexed files too. Anyway I have tried your command in this particular file but I keep having the same problem. [coracollar@spartan-login2 methylation_chromosomes]$ zcat chr10_methylation_calls.tsv | cut -f2- | gzip > fixed_calls.tsv.gz (methplot) [coracollar@spartan-login1 calls]$ methplotlib -m /data/cephfs/punim1048/allbarcoded/CORT/CORT13/methylation_chromosomes/fixed_calls.tsv.gz -n CORT13 -w ch10:6,000-7,000 -g /data/cephfs/punim1048/GRCm38_genome/annotation/mm10.ensGene.gtf --simplify Input file /data/cephfs/punim1048/allbarcoded/CORT/CORT13/methylation_chromosomes/fixed_calls.tsv.gz not recognized! Detailed error: Traceback (most recent call last): File "/home/coracollar/anaconda3/envs/methplot/bin/methplotlib", line 10, in <module> sys.exit(main()) File "/home/coracollar/anaconda3/envs/methplot/lib/python3.7/site-packages/methplotlib/methplotlib.py", line 18, in main meth_data = get_data(args.methylation, args.names, window, args.smooth) File "/home/coracollar/anaconda3/envs/methplot/lib/python3.7/site-packages/methplotlib/import_methylation.py", line 165, in get_data return [read_meth(f, n, window, smoothen) for f, n in zip(methylation_files, names)] File "/home/coracollar/anaconda3/envs/methplot/lib/python3.7/site-packages/methplotlib/import_methylation.py", line 165, in <listcomp> return [read_meth(f, n, window, smoothen) for f, n in zip(methylation_files, names)] File "/home/coracollar/anaconda3/envs/methplot/lib/python3.7/site-packages/methplotlib/import_methylation.py", line 32, in read_meth return parse_nanopolish(filename, file_type, name, window, smoothen=smoothen) File "/home/coracollar/anaconda3/envs/methplot/lib/python3.7/site-packages/methplotlib/import_methylation.py", line 51, in parse_nanopolish gr.pos = np.floor(gr.drop().df[["Start", "End"]].mean(axis=1)) File "/home/coracollar/.local/lib/python3.7/site-packages/pandas/core/frame.py", line 2806, in __getitem__ indexer = self.loc._get_listlike_indexer(key, axis=1, raise_missing=True)[1] File "/home/coracollar/.local/lib/python3.7/site-packages/pandas/core/indexing.py", line 1552, in _get_listlike_indexer keyarr, indexer, o._get_axis_number(axis), raise_missing=raise_missing File "/home/coracollar/.local/lib/python3.7/site-packages/pandas/core/indexing.py", line 1639, in _validate_read_indexer raise KeyError(f"None of [{key}] are in the [{axis_name}]") KeyError: "None of [Index(['Start', 'End'], dtype='object')] are in the [columns]" (methplot) [coracollar@spartan-login1 calls]$ methplotlib --version methplotlib 0.14.0 In response to your second question I did manipulate the files from the nanopolish output (nanopolish version 0.11.3) because from nanopolish I obtain all the chromosomes and together when calling methylationin a format where instead of chr1 I have only 1, chr2 only 2 etc., thus I add chr and separate in individual files for each chromosome this output. The weed thing is that I used this files before with nano polish to show calls and frequencies for each sample and it use to work. Thanks, Coralina

…

On 26 May 2020, at 5:36 pm, Wouter De Coster ***@***.***> wrote: There is a column of index numbers in the file which shouldn't be there. You can fix it with: zcat chr10_methylation_calls.tsv.gz | cut -f2- | gzip > fixed_calls.tsv.gz The output of that command worked for me with methplotlib. Did you do any manual manipulations to that file or is it straight from nanopolish? If so, which version did you use? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#23 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANZ7PMED7EMHV6R24HJLMQTRTNWQ7ANCNFSM4NJEOWBA>.

[wdecoster]

-w ch10:6,000-7,000 is that a typo here on GitHub or did you use chr10?

[Coracollar]

I did use chr10, is the one in sent you, and then modified to the the fiche_calls file

…

On 27 May 2020, at 6:02 pm, Wouter De Coster ***@***.***> wrote: -w ch10:6,000-7,000 is that a typo here on GitHub or did you use chr10? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#23 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANZ7PMH3JWUZEUERHZ2PQ6DRTTCIJANCNFSM4NJEOWBA>.

[wdecoster]

But did you do -w ch10 or -w chr10? Which command exactly caused the problem?

[Coracollar]

Ah, it’s regardless. It never got to the typing error since it doe not recognise the file in the first place. Here’s with -w chr10 (methplot) [coracollar@spartan-login2 methplots]$ methplotlib -m /data/cephfs/punim1048/allbarcoded/CORT/CORT13/methylation_chromosomes/fixed_calls.tsv.gz -n CORT13 -w chr10:6,000-7,000 -g /data/cephfs/punim1048/GRCm38_genome/annotation/mm10.ensGene.gtf --simplify Input file /data/cephfs/punim1048/allbarcoded/CORT/CORT13/methylation_chromosomes/fixed_calls.tsv.gz not recognized! Detailed error: Traceback (most recent call last): File "/home/coracollar/anaconda3/envs/methplot/bin/methplotlib", line 10, in <module> sys.exit(main()) File "/home/coracollar/anaconda3/envs/methplot/lib/python3.7/site-packages/methplotlib/methplotlib.py", line 18, in main meth_data = get_data(args.methylation, args.names, window, args.smooth) File "/home/coracollar/anaconda3/envs/methplot/lib/python3.7/site-packages/methplotlib/import_methylation.py", line 165, in get_data return [read_meth(f, n, window, smoothen) for f, n in zip(methylation_files, names)] File "/home/coracollar/anaconda3/envs/methplot/lib/python3.7/site-packages/methplotlib/import_methylation.py", line 165, in <listcomp> return [read_meth(f, n, window, smoothen) for f, n in zip(methylation_files, names)] File "/home/coracollar/anaconda3/envs/methplot/lib/python3.7/site-packages/methplotlib/import_methylation.py", line 32, in read_meth return parse_nanopolish(filename, file_type, name, window, smoothen=smoothen) File "/home/coracollar/anaconda3/envs/methplot/lib/python3.7/site-packages/methplotlib/import_methylation.py", line 51, in parse_nanopolish gr.pos = np.floor(gr.drop().df[["Start", "End"]].mean(axis=1)) File "/home/coracollar/.local/lib/python3.7/site-packages/pandas/core/frame.py", line 2806, in __getitem__ indexer = self.loc._get_listlike_indexer(key, axis=1, raise_missing=True)[1] File "/home/coracollar/.local/lib/python3.7/site-packages/pandas/core/indexing.py", line 1552, in _get_listlike_indexer keyarr, indexer, o._get_axis_number(axis), raise_missing=raise_missing File "/home/coracollar/.local/lib/python3.7/site-packages/pandas/core/indexing.py", line 1639, in _validate_read_indexer raise KeyError(f"None of [{key}] are in the [{axis_name}]") KeyError: "None of [Index(['Start', 'End'], dtype='object')] are in the [columns]"

…

On 28 May 2020, at 8:31 pm, Wouter De Coster ***@***.***> wrote: But did you do -w ch10 or -w chr10? Which command exactly caused the problem? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#23 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANZ7PMDXL3IGBUUCYKHDP4TRTY4RDANCNFSM4NJEOWBA>.

[Coracollar]

Follow up. I tried with a different chromosome and region and worked in one region and then in another it didn’t. The same “problematic region” failed in the frequencies too. But worked when I made the region bigger. The region I was checking (and failed) , I was suspecting there might be no calls in there and actually when I checked in the bigger one that worked I can see that this small first region has nothing in it (so my theory that there were no calls there is true). So maybe that’s the reason is not working, when the region you are calling in you file has no calls/frecuencies it says the error file not recognised. (methplot) [coracollar@spartan-login2 pruebaRegionsNotRoundinDMLTest]$ methplotlib -m /data/cephfs/punim1048/allbarcoded/Control/C3/methylation_chromosomes/C3_chr1_calls.tsv /data/cephfs/punim1048/allbarcoded/CORT/CORT13/methylation_chromosomes/CORT13_chr1_calls.tsv -n C3 CORT13 -w chr1:4878621-4913856 -g /data/cephfs/punim1048/GRCm38_genome/annotation/mm10.ensGene.gtf --simplify Found 19 gene(s) in the region. (methplot) [coracollar@spartan-login2 pruebaRegionsNotRoundinDMLTest]$ methplotlib -m /data/cephfs/punim1048/allbarcoded/Control/C3/methylation_chromosomes/C3_chr1_calls.tsv /data/cephfs/punim1048/allbarcoded/CORT/CORT13/methylation_chromosomes/CORT13_chr1_calls.tsv -n C3 CORT13 -w chr1:4913784-4913856 -g /data/cephfs/punim1048/GRCm38_genome/annotation/mm10.ensGene.gtf --simplify Input file /data/cephfs/punim1048/allbarcoded/Control/C3/methylation_chromosomes/C3_chr1_calls.tsv not recognized! Detailed error: Traceback (most recent call last): File "/home/coracollar/anaconda3/envs/methplot/bin/methplotlib", line 10, in <module> sys.exit(main()) File "/home/coracollar/anaconda3/envs/methplot/lib/python3.7/site-packages/methplotlib/methplotlib.py", line 18, in main meth_data = get_data(args.methylation, args.names, window, args.smooth) File "/home/coracollar/anaconda3/envs/methplot/lib/python3.7/site-packages/methplotlib/import_methylation.py", line 165, in get_data return [read_meth(f, n, window, smoothen) for f, n in zip(methylation_files, names)] File "/home/coracollar/anaconda3/envs/methplot/lib/python3.7/site-packages/methplotlib/import_methylation.py", line 165, in <listcomp> return [read_meth(f, n, window, smoothen) for f, n in zip(methylation_files, names)] File "/home/coracollar/anaconda3/envs/methplot/lib/python3.7/site-packages/methplotlib/import_methylation.py", line 32, in read_meth return parse_nanopolish(filename, file_type, name, window, smoothen=smoothen) File "/home/coracollar/anaconda3/envs/methplot/lib/python3.7/site-packages/methplotlib/import_methylation.py", line 51, in parse_nanopolish gr.pos = np.floor(gr.drop().df[["Start", "End"]].mean(axis=1)) File "/home/coracollar/.local/lib/python3.7/site-packages/pandas/core/frame.py", line 2806, in __getitem__ indexer = self.loc._get_listlike_indexer(key, axis=1, raise_missing=True)[1] File "/home/coracollar/.local/lib/python3.7/site-packages/pandas/core/indexing.py", line 1552, in _get_listlike_indexer keyarr, indexer, o._get_axis_number(axis), raise_missing=raise_missing File "/home/coracollar/.local/lib/python3.7/site-packages/pandas/core/indexing.py", line 1639, in _validate_read_indexer raise KeyError(f"None of [{key}] are in the [{axis_name}]") KeyError: "None of [Index(['Start', 'End'], dtype='object')] are in the [columns]" (methplot) [coracollar@spartan-login2 pruebaRegionsNotRoundinDMLTest]$ (methplot) [coracollar@spartan-login2 pruebaRegionsNotRoundinDMLTest]$ methplotlib -m /data/cephfs/punim1048/allbarcoded/Control/C3/methylation_chromosomes/C3_chr1_frec.tsv /data/cephfs/punim1048/allbarcoded/CORT/CORT13/methylation_chromosomes/CORT13_chr1_frec.tsv -n C3 CORT13 -w chr1:4913784-4913856 -g /data/cephfs/punim1048/GRCm38_genome/annotation/mm10.ensGene.gtf --simplify Input file /data/cephfs/punim1048/allbarcoded/Control/C3/methylation_chromosomes/C3_chr1_frec.tsv not recognized! Detailed error: Traceback (most recent call last): File "/home/coracollar/anaconda3/envs/methplot/bin/methplotlib", line 10, in <module> sys.exit(main()) File "/home/coracollar/anaconda3/envs/methplot/lib/python3.7/site-packages/methplotlib/methplotlib.py", line 18, in main meth_data = get_data(args.methylation, args.names, window, args.smooth) File "/home/coracollar/anaconda3/envs/methplot/lib/python3.7/site-packages/methplotlib/import_methylation.py", line 165, in get_data return [read_meth(f, n, window, smoothen) for f, n in zip(methylation_files, names)] File "/home/coracollar/anaconda3/envs/methplot/lib/python3.7/site-packages/methplotlib/import_methylation.py", line 165, in <listcomp> return [read_meth(f, n, window, smoothen) for f, n in zip(methylation_files, names)] File "/home/coracollar/anaconda3/envs/methplot/lib/python3.7/site-packages/methplotlib/import_methylation.py", line 32, in read_meth return parse_nanopolish(filename, file_type, name, window, smoothen=smoothen) File "/home/coracollar/anaconda3/envs/methplot/lib/python3.7/site-packages/methplotlib/import_methylation.py", line 51, in parse_nanopolish gr.pos = np.floor(gr.drop().df[["Start", "End"]].mean(axis=1)) File "/home/coracollar/.local/lib/python3.7/site-packages/pandas/core/frame.py", line 2806, in __getitem__ indexer = self.loc._get_listlike_indexer(key, axis=1, raise_missing=True)[1] File "/home/coracollar/.local/lib/python3.7/site-packages/pandas/core/indexing.py", line 1552, in _get_listlike_indexer keyarr, indexer, o._get_axis_number(axis), raise_missing=raise_missing File "/home/coracollar/.local/lib/python3.7/site-packages/pandas/core/indexing.py", line 1639, in _validate_read_indexer raise KeyError(f"None of [{key}] are in the [{axis_name}]") KeyError: "None of [Index(['Start', 'End'], dtype='object')] are in the [columns]" (methplot) [coracollar@spartan-login2 pruebaRegionsNotRoundinDMLTest]$ methplotlib -m /data/cephfs/punim1048/allbarcoded/Control/C3/methylation_chromosomes/C3_chr1_frec.tsv /data/cephfs/punim1048/allbarcoded/CORT/CORT13/methylation_chromosomes/CORT13_chr1_frec.tsv -n C3 CORT13 -w chr1:4913784-4923856 -g /data/cephfs/punim1048/GRCm38_genome/annotation/mm10.ensGene.gtf --simplify Found 11 gene(s) in the region. (methplot) [coracollar@spartan-login2 pruebaRegionsNotRoundinDMLTest]$

…

On 29 May 2020, at 8:30 am, Coralina Collar Fernandez ***@***.***> wrote: Ah, it’s regardless. It never got to the typing error since it doe not recognise the file in the first place. Here’s with -w chr10 (methplot) ***@***.*** methplots]$ methplotlib -m /data/cephfs/punim1048/allbarcoded/CORT/CORT13/methylation_chromosomes/fixed_calls.tsv.gz -n CORT13 -w chr10:6,000-7,000 -g /data/cephfs/punim1048/GRCm38_genome/annotation/mm10.ensGene.gtf --simplify Input file /data/cephfs/punim1048/allbarcoded/CORT/CORT13/methylation_chromosomes/fixed_calls.tsv.gz not recognized! Detailed error: Traceback (most recent call last): File "/home/coracollar/anaconda3/envs/methplot/bin/methplotlib", line 10, in <module> sys.exit(main()) File "/home/coracollar/anaconda3/envs/methplot/lib/python3.7/site-packages/methplotlib/methplotlib.py", line 18, in main meth_data = get_data(args.methylation, args.names, window, args.smooth) File "/home/coracollar/anaconda3/envs/methplot/lib/python3.7/site-packages/methplotlib/import_methylation.py", line 165, in get_data return [read_meth(f, n, window, smoothen) for f, n in zip(methylation_files, names)] File "/home/coracollar/anaconda3/envs/methplot/lib/python3.7/site-packages/methplotlib/import_methylation.py", line 165, in <listcomp> return [read_meth(f, n, window, smoothen) for f, n in zip(methylation_files, names)] File "/home/coracollar/anaconda3/envs/methplot/lib/python3.7/site-packages/methplotlib/import_methylation.py", line 32, in read_meth return parse_nanopolish(filename, file_type, name, window, smoothen=smoothen) File "/home/coracollar/anaconda3/envs/methplot/lib/python3.7/site-packages/methplotlib/import_methylation.py", line 51, in parse_nanopolish gr.pos = np.floor(gr.drop().df[["Start", "End"]].mean(axis=1)) File "/home/coracollar/.local/lib/python3.7/site-packages/pandas/core/frame.py", line 2806, in __getitem__ indexer = self.loc._get_listlike_indexer(key, axis=1, raise_missing=True)[1] File "/home/coracollar/.local/lib/python3.7/site-packages/pandas/core/indexing.py", line 1552, in _get_listlike_indexer keyarr, indexer, o._get_axis_number(axis), raise_missing=raise_missing File "/home/coracollar/.local/lib/python3.7/site-packages/pandas/core/indexing.py", line 1639, in _validate_read_indexer raise KeyError(f"None of [{key}] are in the [{axis_name}]") KeyError: "None of [Index(['Start', 'End'], dtype='object')] are in the [columns]" > On 28 May 2020, at 8:31 pm, Wouter De Coster ***@***.*** ***@***.***>> wrote: > > > But did you do -w ch10 or -w chr10? Which command exactly caused the problem? > > — > You are receiving this because you authored the thread. > Reply to this email directly, view it on GitHub <#23 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANZ7PMDXL3IGBUUCYKHDP4TRTY4RDANCNFSM4NJEOWBA>. >

[wdecoster]

Thanks for following up! That sounds likely to be the issue, but methplotlib should give you a better error message there. I'll take care of it.

[wdecoster]

Thanks for your patience, methplotlib should now give you a better error message (in version 0.14.1).
Please let me know if you encounter further issues.