bioawk vs awk give different results (locale-specific)

[janxkoci]

Hi,

I'm trying to check my VCF filtering and I wanted to play around with bioawk but I noticed it gives different results than regular awk present on my system (it's a server that should have OpenSUSE according to admin).

See this example:

grep -v '#' r123_newrbmcl100_fb.vcf | awk '$6>20 {print $1,$2,$6}' | head
E57432_L179 11 7368.02
E57432_L179 19 8461.42
E57432_L179 35 12953.8
E57432_L179 43 14922.7
E57432_L179 51 19613.4
E57432_L179 62 20413.3
E57432_L179 70 20282.5
E57432_L179 108 25482.3
E57432_L179 118 27017.1
E57432_L179 149 24639.4

compared to:

grep -v '#' r123_newrbmcl100_fb.vcf | bioawk '$6>20 {print $1,$2,$6}' | head
E57432_L179 11 7368.02
E57432_L179 19 8461.42
E57432_L179 62 20413.3
E57432_L179 70 20282.5
E57432_L179 108 25482.3
E57432_L179 118 27017.1
E57432_L179 124 9.09527e-05
E57432_L179 128 4.50905e-05
E57432_L179 129 3.73425e-05
E57432_L179 138 4.61921e-05

Note the only difference between the commands is replacing awk with bioawk - no special features of bioawk are called. However I noticed this originally when I did use the features (namely -tc hdr and calling variables with colnames) and I got the same wrong result. I then tested it as above.

For some reason bioawk doesn't print some lines that pass the $6>20 filter (like positions 35, 43, and 51) while it prints possitions that should not pass the filter (positions 124-138). System awk prints it correctly.

I installed bioawk using conda manager and bioconda channel. The version is 1.0 and build h84994c4_4.

Thanks for any input on what could go wrong.

[neptuneyt]

It should be your bioawk version.It is without any error checking in my bioawk version 20110810 by your data.

…

------------------ 原始邮件 ------------------ 发件人: "janxkoci"<notifications@github.com>; 发送时间: 2019年7月20日(星期六) 下午4:36 收件人: "lh3/bioawk"<bioawk@noreply.github.com>; 抄送: "Subscribed"<subscribed@noreply.github.com>; 主题: [lh3/bioawk] bioawk vs awk give different results (#34) Hi, I'm trying to check my VCF filtering and I wanted to play around with bioawk but I noticed it gives different results than regular awk present on my system (it's a server that should have OpenSUSE according to admin). See this example: grep -v '#' r123_newrbmcl100_fb.vcf | awk '$6>20 {print $1,$2,$6}' | head E57432_L179 11 7368.02 E57432_L179 19 8461.42 E57432_L179 35 12953.8 E57432_L179 43 14922.7 E57432_L179 51 19613.4 E57432_L179 62 20413.3 E57432_L179 70 20282.5 E57432_L179 108 25482.3 E57432_L179 118 27017.1 E57432_L179 149 24639.4 compared to: grep -v '#' r123_newrbmcl100_fb.vcf | bioawk '$6>20 {print $1,$2,$6}' | head E57432_L179 11 7368.02 E57432_L179 19 8461.42 E57432_L179 62 20413.3 E57432_L179 70 20282.5 E57432_L179 108 25482.3 E57432_L179 118 27017.1 E57432_L179 124 9.09527e-05 E57432_L179 128 4.50905e-05 E57432_L179 129 3.73425e-05 E57432_L179 138 4.61921e-05 Note the only difference between the commands is replacing awk with bioawk - no special features of bioawk are called. However I noticed this originally when I did use the features (namely -tc hdr and calling variables with colnames) and I got the same wrong result. I then tested it as above. For some reason bioawk doesn't print some lines that pass the $6>20 filter (like positions 35, 43, and 51) while it prints possitions that should not pass the filter (positions 124-138). System awk prints it correctly. I installed bioawk using conda manager and bioconda channel. The version is 1.0 and build h84994c4_4. Thanks for any input on what could go wrong. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[janxkoci]

Ok I git cloned bioawk to another server today and it works correctly. Closing.

[janxkoci]

No actually.

So cloning worked on two servers (CentOS and gentoo), however I had an error when doing make on the OpenSUSE server. Installing bison with conda install bison made it go away and I compiled succesfully, but when I then use bioawk the parsing is still wrong and produces the output as in the examples above. I made sure I'm calling it from the git folder and not from conda, but it still doesn't work.

Maybe there is an issue with the yacc?

[janxkoci]

I will close this as I think it's a problem with the parser generator bison.

[janxkoci]

Hi again,

I'm still having this issue with pretty much every installation I attempt (I cannot even reproduce correct behaviour with older installs on the servers mentioned in this thread).

Today I cloned this git repo to my new laptop with elementary OS 5.1.7 (based on Ubuntu 18.04 LTS). I successfully compiled it (I just run sudo apt install bison before I started) and tried to use it on some example data. And the parsing was wrong.

I'm attaching an example csv file bioawk-test.csv, created in a spreadsheet. I tried several commands and got really strange behaviours:

$ cat bioawk-test.csv 
1	1	2	3	4	4
2	10	20	30	0.40000000000000002	0.40000000000000002
3	19	38	57	-3.2000000000000011	3.2000000000000011
4	28	56	84	-6.8000000000000007	6.8000000000000007
5	37	74	111	-10.400000000000002	10.400000000000002
6	46	92	138	-14.000000000000004	14.000000000000004
7	55	110	165	-17.600000000000001	17.600000000000001
8	64	128	192	-21.200000000000003	21.200000000000003
9	73	146	219	-24.800000000000004	24.800000000000004
10	82	164	246	-28.400000000000006	28.400000000000006

## first failed test
$ bioawk '$6<10' bioawk-test.csv 
1	1	2	3	4	4
2	10	20	30	0.40000000000000002	0.40000000000000002

## this is also wrong
$ bioawk '$6>10' bioawk-test.csv 
3	19	38	57	-3.2000000000000011	3.2000000000000011
4	28	56	84	-6.8000000000000007	6.8000000000000007
5	37	74	111	-10.400000000000002	10.400000000000002
6	46	92	138	-14.000000000000004	14.000000000000004
7	55	110	165	-17.600000000000001	17.600000000000001
8	64	128	192	-21.200000000000003	21.200000000000003
9	73	146	219	-24.800000000000004	24.800000000000004
10	82	164	246	-28.400000000000006	28.400000000000006

## one more test
$ bioawk '$5<-20' bioawk-test.csv 
5	37	74	111	-10.400000000000002	10.400000000000002
6	46	92	138	-14.000000000000004	14.000000000000004
7	55	110	165	-17.600000000000001	17.600000000000001

## this is the weirdest one
$ bioawk '$5>-20' bioawk-test.csv 
1	1	2	3	4	4
2	10	20	30	0.40000000000000002	0.40000000000000002
3	19	38	57	-3.2000000000000011	3.2000000000000011
4	28	56	84	-6.8000000000000007	6.8000000000000007
8	64	128	192	-21.200000000000003	21.200000000000003
9	73	146	219	-24.800000000000004	24.800000000000004
10	82	164	246	-28.400000000000006	28.400000000000006

It looks to me that bioawk has some trouble with interpreting numbers with decimal points. Parsing other columns of this file is otherwise fine.

[janxkoci]

Real cause: locale

Hi again,

recently, I encountered a weird issue with system sort (GNU coreutils), where I was numerically sorting a text file on a column with US-style decimal numbers (with dot as decimal point) and my sort was producing wrong results. After a few minutes of frustration, I've found the issue was caused by my Czech locale - running the command export LC_ALL=C and rerunning the sort command fixed my issue. I still don't understand why the parsing of such decimals would be wrong, but nevertheless - it is wrong.

Anyway - yesterday, I got the idea if perhaps this issue with bioawk could be caused by locale too. And it seems it is!

Specifically, I can get wrong parsing under Czech locale with bioawk, and mawk 1.3.4, but not with gawk 4.0.2 or miller. Setting my locale to "C" as above fixes the issue and bioawk (as well as mawk) starts parsing as expected. Note that my tests in previous comments were testing against gawk, which is not affected by locale in this way.

My original locale:

$ locale                                                                                                                                                                                                            [94/94]
LANG=cs_CZ.UTF-8                                                                                                                                                                                                                               
LC_CTYPE="cs_CZ.UTF-8"                                                                                                                                                                                                                         
LC_NUMERIC=cs_CZ.UTF-8                                                                                                                                                                                                                         
LC_TIME=cs_CZ.UTF-8                                                                                                                                                                                                                            
LC_COLLATE="cs_CZ.UTF-8"                                                                                                                                                                                                                       
LC_MONETARY=cs_CZ.UTF-8                                                                                                                                                                                                                        
LC_MESSAGES="cs_CZ.UTF-8"                                                                                                                                                                                                                      
LC_PAPER=cs_CZ.UTF-8                                                                                                                                                                                                                           
LC_NAME=cs_CZ.UTF-8                                                                                                                                                                                                                            
LC_ADDRESS=cs_CZ.UTF-8                                                                                                                                                                                                                         
LC_TELEPHONE=cs_CZ.UTF-8                                                                                                                                                                                                                       
LC_MEASUREMENT=cs_CZ.UTF-8                                                                                                                                                                                                                     
LC_IDENTIFICATION=cs_CZ.UTF-8                                                                                                                                                                                                                  
LC_ALL=