Insert peaks (again but in 2.0.0-rc4)

[mscook]

Hi Seb,

I'm running - Ray version: 2.0.0-rc4

We still get artificially detected "large peaks".

I've included the metrics:

General

['kmer', '17', '19', '21', '23', '25', '27', '29', '31', '33', '35']
['c > 100 bp', '143', '147', '145', '142', '137', '124', '128', '132', '134', '126']
['c > 100 bp total len', '5241893', '5465571', '5354470', '5255806', '5242270', '5257263', '5385963', '5405457', '5327053', '5406021']
['c > 100 bp av len', '36656', '37180', '36927', '37012', '38264', '42397', '42077', '40950', '39754', '42904']
['c > 100 bp N50', '124514', '139408', '161895', '161895', '162956', '159457', '164333', '159457', '159457', '159457']
['c > 100 bp largest', '345228', '345245', '348268', '348294', '349160', '345344', '345370', '345526', '345526', '345439']
['c > 500 bp', '121', '119', '113', '109', '112', '112', '118', '118', '115', '114'], ['c > 500 bp total len', '5238060', '5460711', '5348486', '5249896', '5237169', '5255489', '5384460', '5402805', '5323604', '5404203']
['c > 500 bp average len', '43289', '45888', '47331', '48164', '46760', '46924', '45631', '45786', '46292', '47405']
['c > 500 bp N50', '124514', '139408', '161895', '161895', '162956', '159457', '164333', '159457', '159457', '159457']
['c > 500 bp largest', '345228', '345245', '348268', '348294', '349160', '345344', '345370', '345526', '345526', '345439']
['s > 100', '125', '132', '129', '121', '125', '110', '115', '116', '117', '108'], ['s > 100 bp total len', '5245763', '5467786', '5356812', '5260052', '5244192', '5307461', '5432292', '5470661', '5389713', '5478137']
['s > 100 bp av len', '41966', '41422', '41525', '43471', '41953', '48249', '47237', '47160', '46065', '50723']
['s > 100 bp N50', '159462', '161919', '171861', '164333', '190540', '186846', '171660', '172749', '171272', '187079']
['s > 100 bp largest', '345228', '345245', '535229', '535191', '535193', '345724', '345370', '345526', '345526', '345439']
['s > 500 bp', '103', '104', '97', '88', '100', '98', '105', '102', '98', '96']
['s > 500 bp total len', '5241930', '5462926', '5350828', '5254142', '5239091', '5305687', '5430789', '5468009', '5386264', '5476319']
['s > 500 bp av', '50892', '52528', '55163', '59706', '52390', '54139', '51721', '53607', '54961', '57044']
['s > 500 bp N50', '159462', '161919', '171861', '164333', '190540', '186846', '171660', '172749', '171272', '187079']
['s > 500 bp largest', '345228', '345245', '535229', '535191', '535193', '345724', '345370', '345526', '345526', '345439']

Scaffolding Stats

['NumberOfNChars', 3870, 2215, 2342, 4246, 1922, 50198, 46329, 65204, 62660, 72116]
['NumberOfNCharRuns', 18, 15, 16, 21, 12, 14, 13, 16, 17, 18]
['MinNChars', 29, 23, 37, 56, 17, 881, 506, 3594, 83, 3474]
['MaxNChars', 390, 312, 252, 363, 357, 4357, 4334, 4458, 4379, 4384]
['AvNChars', 215.0, 147.66666666666666, 146.375, 202.1904761904762, 160.16666666666666, 3585.5714285714284, 3563.769230769231, 4075.25, 3685.8823529411766, 4006.4444444444443]]

Insert Stats (: sepearates multiple)

['meanIns', '344', '350', '351', '351', '351', '351:4422', '351:4426', '351:4426', '351:4424', '351:4423'],
['sdIns', '56', '50', '49', '49', '49', '49:24', '49:20', '49:20', '49:22', '49:24']

I've included the LibraryStatistics and Library0 files for kmer 35:

LibraryNumber: 0
InputFormat: Interleaved,Paired
DetectionType: Automatic
File: XXXX_proc.fastq
NumberOfSequences: 8205596
Distribution: 35/Library0.txt
Peak 0
AverageOuterDistance: 351
StandardDeviation: 49
Peak 1
AverageOuterDistance: 4423
StandardDeviation: 24

47 1
54 1
57 1
59 1
65 1
66 1
68 1
69 2
71 3
72 2
74 7
75 5
76 5
77 9
78 3
79 4
80 2
81 5
82 1
83 1
84 4
85 4
86 2
87 6
88 1
89 3
90 3
91 2
92 4
93 3
94 3
95 6
96 3
97 7
98 6
99 4
100 7
101 121
102 113
103 93
104 95
105 112
106 128
107 133
108 104
109 127
110 145
111 178
112 180
113 187
114 193
115 201
116 215
117 208
118 265
119 257
120 273
121 263
122 296
123 344
124 345
125 362
126 375
127 342
128 397
129 446
130 482
131 494
132 513
133 515
134 581
135 609
136 661
137 683
138 663
139 733
140 762
141 806
142 821
143 891
144 961
145 959
146 1046
147 1027
148 1114
149 1075
150 1189
151 1188
152 1225
153 1269
154 1340
155 1423
156 1403
157 1454
158 1505
159 1521
160 1562
161 1598
162 1620
163 1719
164 1710
165 1754
166 1757
167 2012
168 1931
169 1912
170 1982
171 1961
172 2029
173 2075
174 2119
175 2110
176 2147
177 2184
178 2158
179 2248
180 2222
181 2314
182 2346
183 2351
184 2361
185 2318
186 2362
187 2352
188 2379
189 2508
190 2416
191 2508
192 2536
193 2425
194 2375
195 2483
196 2544
197 2413
198 2402
199 2469
200 2509
201 2435
202 2573
203 2428
204 2433
205 2496
206 2510
207 2493
208 2578
209 2531
210 2652
211 2472
212 2421
213 2488
214 2457
215 2382
216 2553
217 2475
218 2456
219 2440
220 2431
221 2343
222 2500
223 2450
224 2377
225 2307
226 2384
227 2463
228 2330
229 2389
230 2362
231 2399
232 2349
233 2388
234 2353
235 2386
236 2284
237 2325
238 2374
239 2287
240 2347
241 2366
242 2269
243 2262
244 2261
245 2242
246 2220
247 2348
248 2197
249 2284
250 2332
251 2214
252 2268
253 2243
254 2143
255 2214
256 2226
257 2234
258 2142
259 2118
260 2192
261 2168
262 2152
263 2126
264 2229
265 2128
266 2162
267 2158
268 2151
269 2136
270 2156
271 2201
272 2200
273 2264
274 2250
275 2223
276 2263
277 2135
278 2231
279 2183
280 2336
281 2233
282 2268
283 2252
284 2308
285 2267
286 2329
287 2340
288 2293
289 2404
290 2272
291 2288
292 2469
293 2391
294 2441
295 2534
296 2448
297 2516
298 2529
299 2425
300 2612
301 2662
302 2732
303 2782
304 2876
305 2952
306 3094
307 3176
308 3301
309 3518
310 3664
311 3986
312 4174
313 4420
314 4805
315 5167
316 5657
317 6207
318 6835
319 7489
320 8414
321 9325
322 10344
323 11486
324 12684
325 14228
326 15928
327 17390
328 19347
329 20962
330 22709
331 24658
332 26985
333 28822
334 31205
335 32772
336 34470
337 36416
338 37924
339 39824
340 40501
341 41865
342 43719
343 44970
344 45913
345 46423
346 47696
347 48126
348 48847
349 49953
350 50303
351 50775
352 51165
353 51567
354 51607
355 52142
356 52288
357 52254
358 52345
359 52346
360 52078
361 51349
362 51441
363 51071
364 50682
365 50937
366 50411
367 50102
368 49922
369 49540
370 48842
371 48466
372 48273
373 47883
374 47479
375 46592
376 46501
377 45421
378 44737
379 44276
380 43646
381 43146
382 41802
383 41218
384 40094
385 39390
386 38392
387 37312
388 36124
389 34988
390 33370
391 32068
392 30878
393 29090
394 27928
395 26226
396 25191
397 23977
398 22608
399 21323
400 20070
401 18912
402 17747
403 16524
404 15490
405 14380
406 13500
407 12759
408 11992
409 11160
410 10352
411 9636
412 9086
413 8313
414 7739
415 7460
416 6908
417 6255
418 5861
419 5610
420 5171
421 4593
422 4294
423 4030
424 3728
425 3496
426 3164
427 2873
428 2653
429 2404
430 2215
431 2005
432 1861
433 1640
434 1485
435 1305
436 1250
437 1062
438 943
439 819
440 745
441 674
442 567
443 515
444 460
445 380
446 336
447 319
448 276
449 207
450 197
451 161
452 138
453 122
454 104
455 100
456 79
457 68
458 52
459 46
460 33
461 31
462 29
463 10
464 19
465 23
466 11
467 14
468 10
469 9
470 12
471 10
472 2
473 2
474 4
475 3
476 4
477 4
478 2
479 1
480 2
481 2
482 3
483 1
484 1
485 2
486 2
488 1
489 1
490 4
493 2
494 1
495 1
497 1
499 3
505 2
506 2
507 2
508 1
511 3
513 1
517 1
520 2
523 1
525 1
528 1
529 1
535 1
537 4
538 1
540 1
544 2
545 3
546 1
549 1
551 1
552 1
554 1
555 1
559 1
561 1
567 1
571 1
572 1
586 1
587 3
592 1
594 1
598 1
604 1
607 1
612 1
621 1
630 1
632 1
636 1
639 1
649 1
655 1
656 1
657 1
670 1
679 1
682 1
683 1
684 1
690 1
693 1
695 2
708 1
713 1
716 1
721 1
732 2
733 1
755 1
779 1
780 1
808 1
895 1
973 1
1302 1
1497 1
2056 1
2237 1
3744 1
3937 1
3981 1
4279 2
4345 1
4350 1
4352 1
4353 1
4355 2
4358 1
4359 1
4361 1
4362 1
4363 2
4365 2
4368 1
4369 2
4370 3
4371 4
4372 2
4373 1
4375 2
4377 1
4379 1
4380 2
4382 2
4384 2
4385 1
4386 1
4387 4
4388 3
4389 2
4390 2
4391 3
4392 4
4393 5
4395 1
4396 1
4397 8
4398 6
4399 2
4401 3
4402 3
4403 5
4404 3
4405 6
4406 6
4407 4
4408 7
4409 4
4410 5
4411 5
4412 5
4413 6
4414 3
4415 4
4416 4
4417 12
4418 3
4419 5
4420 6
4421 6
4422 9
4423 7
4424 6
4425 5
4426 8
4427 8
4428 4
4429 2
4430 2
4431 5
4432 4
4433 5
4434 7
4435 4
4436 7
4437 6
4438 3
4439 3
4440 6
4441 6
4442 5
4443 4
4444 2
4445 6
4446 4
4447 4
4448 4
4449 6
4450 5
4451 4
4452 3
4453 3
4454 7
4455 6
4456 1
4457 2
4458 3
4459 3
4460 3
4461 1
4462 2
4463 2
4464 2
4466 1
4467 1
4468 2
4470 1
4471 1
4474 2
4479 1
4480 1
4484 1
4492 1
4494 1
4496 1
4501 1
4510 1
4511 1
4521 1
4532 1
4536 2
4543 1
4556 1
4561 1
4568 1
4581 1
4583 1
4585 1
4587 1
4597 1
4602 1
4616 1
4637 1
4721 1
4871 1
5126 1
5276 1
5848 1
6751 1
6868 1
7544 1
7996 1
8213 1
8528 1
13467 1
14233 1
16717 1
19963 1
20008 1
32159 1
34585 1
39611 1
40416 1
45612 1

Hope this helps.

Cheers

Mitch

[sebhtml]

You can clone Ray-TestSuite, Ray and RayPlatform to fix this in
code/plugin_Library/LibraryPeakFinder.cpp

The script to test peak finding is located in
unit-tests/test_peakFinder.sh (Ray-TestSuite).

Otherwise, I will look into that in a few weeks when I have some time.

On Thu, 2012-03-22 at 19:40 -0700, Mitchell J Stanton-Cook wrote:

Hi Seb,

I'm running - Ray version: 2.0.0-rc4

We still get artificially detected "large peaks".

I've included the metrics:

#General
['kmer', '17', '19', '21', '23', '25', '27', '29', '31', '33', '35']
['c > 100 bp', '143', '147', '145', '142', '137', '124', '128', '132', '134', '126']
['c > 100 bp total len', '5241893', '5465571', '5354470', '5255806', '5242270', '5257263', '5385963', '5405457', '5327053', '5406021']
['c > 100 bp av len', '36656', '37180', '36927', '37012', '38264', '42397', '42077', '40950', '39754', '42904']
['c > 100 bp N50', '124514', '139408', '161895', '161895', '162956', '159457', '164333', '159457', '159457', '159457']
['c > 100 bp largest', '345228', '345245', '348268', '348294', '349160', '345344', '345370', '345526', '345526', '345439']
['c > 500 bp', '121', '119', '113', '109', '112', '112', '118', '118', '115', '114'], ['c > 500 bp total len', '5238060', '5460711', '5348486', '5249896', '5237169', '5255489', '5384460', '5402805', '5323604', '5404203']
['c > 500 bp average len', '43289', '45888', '47331', '48164', '46760', '46924', '45631', '45786', '46292', '47405']
['c > 500 bp N50', '124514', '139408', '161895', '161895', '162956', '159457', '164333', '159457', '159457', '159457']
['c > 500 bp largest', '345228', '345245', '348268', '348294', '349160', '345344', '345370', '345526', '345526', '345439']
['s > 100', '125', '132', '129', '121', '125', '110', '115', '116', '117', '108'], ['s > 100 bp total len', '5245763', '5467786', '5356812', '5260052', '5244192', '5307461', '5432292', '5470661', '5389713', '5478137']
['s > 100 bp av len', '41966', '41422', '41525', '43471', '41953', '48249', '47237', '47160', '46065', '50723']
['s > 100 bp N50', '159462', '161919', '171861', '164333', '190540', '186846', '171660', '172749', '171272', '187079']
['s > 100 bp largest', '345228', '345245', '535229', '535191', '535193', '345724', '345370', '345526', '345526', '345439']
['s > 500 bp', '103', '104', '97', '88', '100', '98', '105', '102', '98', '96']
['s > 500 bp total len', '5241930', '5462926', '5350828', '5254142', '5239091', '5305687', '5430789', '5468009', '5386264', '5476319']
['s > 500 bp av', '50892', '52528', '55163', '59706', '52390', '54139', '51721', '53607', '54961', '57044']
['s > 500 bp N50', '159462', '161919', '171861', '164333', '190540', '186846', '171660', '172749', '171272', '187079']
['s > 500 bp largest', '345228', '345245', '535229', '535191', '535193', '345724', '345370', '345526', '345526', '345439']
#Scaffolding Stats
['NumberOfNChars', 3870, 2215, 2342, 4246, 1922, 50198, 46329, 65204, 62660, 72116]
['NumberOfNCharRuns', 18, 15, 16, 21, 12, 14, 13, 16, 17, 18]
['MinNChars', 29, 23, 37, 56, 17, 881, 506, 3594, 83, 3474]
['MaxNChars', 390, 312, 252, 363, 357, 4357, 4334, 4458, 4379, 4384]
['AvNChars', 215.0, 147.66666666666666, 146.375, 202.1904761904762, 160.16666666666666, 3585.5714285714284, 3563.769230769231, 4075.25, 3685.8823529411766, 4006.4444444444443]]
#Insert Stats (: sepearates multiple)
['meanIns', '344', '350', '351', '351', '351', '351:4422', '351:4426', '351:4426', '351:4424', '351:4423'],
['sdIns', '56', '50', '49', '49', '49', '49:24', '49:20', '49:20', '49:22', '49:24']

I've included the LibraryStatistics and Library0 files for kmer 35:

LibraryNumber: 0
InputFormat: Interleaved,Paired
DetectionType: Automatic
File: XXXX_proc.fastq
NumberOfSequences: 8205596
Distribution: 35/Library0.txt
Peak 0
AverageOuterDistance: 351
StandardDeviation: 49
Peak 1
AverageOuterDistance: 4423
StandardDeviation: 24

47 1
54 1
57 1
59 1
65 1
66 1
68 1
69 2
71 3
72 2
74 7
75 5
76 5
77 9
78 3
79 4
80 2
81 5
82 1
83 1
84 4
85 4
86 2
87 6
88 1
89 3
90 3
91 2
92 4
93 3
94 3
95 6
96 3
97 7
98 6
99 4
100 7
101 121
102 113
103 93
104 95
105 112
106 128
107 133
108 104
109 127
110 145
111 178
112 180
113 187
114 193
115 201
116 215
117 208
118 265
119 257
120 273
121 263
122 296
123 344
124 345
125 362
126 375
127 342
128 397
129 446
130 482
131 494
132 513
133 515
134 581
135 609
136 661
137 683
138 663
139 733
140 762
141 806
142 821
143 891
144 961
145 959
146 1046
147 1027
148 1114
149 1075
150 1189
151 1188
152 1225
153 1269
154 1340
155 1423
156 1403
157 1454
158 1505
159 1521
160 1562
161 1598
162 1620
163 1719
164 1710
165 1754
166 1757
167 2012
168 1931
169 1912
170 1982
171 1961
172 2029
173 2075
174 2119
175 2110
176 2147
177 2184
178 2158
179 2248
180 2222
181 2314
182 2346
183 2351
184 2361
185 2318
186 2362
187 2352
188 2379
189 2508
190 2416
191 2508
192 2536
193 2425
194 2375
195 2483
196 2544
197 2413
198 2402
199 2469
200 2509
201 2435
202 2573
203 2428
204 2433
205 2496
206 2510
207 2493
208 2578
209 2531
210 2652
211 2472
212 2421
213 2488
214 2457
215 2382
216 2553
217 2475
218 2456
219 2440
220 2431
221 2343
222 2500
223 2450
224 2377
225 2307
226 2384
227 2463
228 2330
229 2389
230 2362
231 2399
232 2349
233 2388
234 2353
235 2386
236 2284
237 2325
238 2374
239 2287
240 2347
241 2366
242 2269
243 2262
244 2261
245 2242
246 2220
247 2348
248 2197
249 2284
250 2332
251 2214
252 2268
253 2243
254 2143
255 2214
256 2226
257 2234
258 2142
259 2118
260 2192
261 2168
262 2152
263 2126
264 2229
265 2128
266 2162
267 2158
268 2151
269 2136
270 2156
271 2201
272 2200
273 2264
274 2250
275 2223
276 2263
277 2135
278 2231
279 2183
280 2336
281 2233
282 2268
283 2252
284 2308
285 2267
286 2329
287 2340
288 2293
289 2404
290 2272
291 2288
292 2469
293 2391
294 2441
295 2534
296 2448
297 2516
298 2529
299 2425
300 2612
301 2662
302 2732
303 2782
304 2876
305 2952
306 3094
307 3176
308 3301
309 3518
310 3664
311 3986
312 4174
313 4420
314 4805
315 5167
316 5657
317 6207
318 6835
319 7489
320 8414
321 9325
322 10344
323 11486
324 12684
325 14228
326 15928
327 17390
328 19347
329 20962
330 22709
331 24658
332 26985
333 28822
334 31205
335 32772
336 34470
337 36416
338 37924
339 39824
340 40501
341 41865
342 43719
343 44970
344 45913
345 46423
346 47696
347 48126
348 48847
349 49953
350 50303
351 50775
352 51165
353 51567
354 51607
355 52142
356 52288
357 52254
358 52345
359 52346
360 52078
361 51349
362 51441
363 51071
364 50682
365 50937
366 50411
367 50102
368 49922
369 49540
370 48842
371 48466
372 48273
373 47883
374 47479
375 46592
376 46501
377 45421
378 44737
379 44276
380 43646
381 43146
382 41802
383 41218
384 40094
385 39390
386 38392
387 37312
388 36124
389 34988
390 33370
391 32068
392 30878
393 29090
394 27928
395 26226
396 25191
397 23977
398 22608
399 21323
400 20070
401 18912
402 17747
403 16524
404 15490
405 14380
406 13500
407 12759
408 11992
409 11160
410 10352
411 9636
412 9086
413 8313
414 7739
415 7460
416 6908
417 6255
418 5861
419 5610
420 5171
421 4593
422 4294
423 4030
424 3728
425 3496
426 3164
427 2873
428 2653
429 2404
430 2215
431 2005
432 1861
433 1640
434 1485
435 1305
436 1250
437 1062
438 943
439 819
440 745
441 674
442 567
443 515
444 460
445 380
446 336
447 319
448 276
449 207
450 197
451 161
452 138
453 122
454 104
455 100
456 79
457 68
458 52
459 46
460 33
461 31
462 29
463 10
464 19
465 23
466 11
467 14
468 10
469 9
470 12
471 10
472 2
473 2
474 4
475 3
476 4
477 4
478 2
479 1
480 2
481 2
482 3
483 1
484 1
485 2
486 2
488 1
489 1
490 4
493 2
494 1
495 1
497 1
499 3
505 2
506 2
507 2
508 1
511 3
513 1
517 1
520 2
523 1
525 1
528 1
529 1
535 1
537 4
538 1
540 1
544 2
545 3
546 1
549 1
551 1
552 1
554 1
555 1
559 1
561 1
567 1
571 1
572 1
586 1
587 3
592 1
594 1
598 1
604 1
607 1
612 1
621 1
630 1
632 1
636 1
639 1
649 1
655 1
656 1
657 1
670 1
679 1
682 1
683 1
684 1
690 1
693 1
695 2
708 1
713 1
716 1
721 1
732 2
733 1
755 1
779 1
780 1
808 1
895 1
973 1
1302 1
1497 1
2056 1
2237 1
3744 1
3937 1
3981 1
4279 2
4345 1
4350 1
4352 1
4353 1
4355 2
4358 1
4359 1
4361 1
4362 1
4363 2
4365 2
4368 1
4369 2
4370 3
4371 4
4372 2
4373 1
4375 2
4377 1
4379 1
4380 2
4382 2
4384 2
4385 1
4386 1
4387 4
4388 3
4389 2
4390 2
4391 3
4392 4
4393 5
4395 1
4396 1
4397 8
4398 6
4399 2
4401 3
4402 3
4403 5
4404 3
4405 6
4406 6
4407 4
4408 7
4409 4
4410 5
4411 5
4412 5
4413 6
4414 3
4415 4
4416 4
4417 12
4418 3
4419 5
4420 6
4421 6
4422 9
4423 7
4424 6
4425 5
4426 8
4427 8
4428 4
4429 2
4430 2
4431 5
4432 4
4433 5
4434 7
4435 4
4436 7
4437 6
4438 3
4439 3
4440 6
4441 6
4442 5
4443 4
4444 2
4445 6
4446 4
4447 4
4448 4
4449 6
4450 5
4451 4
4452 3
4453 3
4454 7
4455 6
4456 1
4457 2
4458 3
4459 3
4460 3
4461 1
4462 2
4463 2
4464 2
4466 1
4467 1
4468 2
4470 1
4471 1
4474 2
4479 1
4480 1
4484 1
4492 1
4494 1
4496 1
4501 1
4510 1
4511 1
4521 1
4532 1
4536 2
4543 1
4556 1
4561 1
4568 1
4581 1
4583 1
4585 1
4587 1
4597 1
4602 1
4616 1
4637 1
4721 1
4871 1
5126 1
5276 1
5848 1
6751 1
6868 1
7544 1
7996 1
8213 1
8528 1
13467 1
14233 1
16717 1
19963 1
20008 1
32159 1
34585 1
39611 1
40416 1
45612 1

Hope this helps.

Cheers

Mitch

---

Reply to this email directly or view it on GitHub:
#49

[sebhtml]

Hello,

I added a unit test for your case in Ray-TestSuite.

The problem was that the signal was being poluted by points with a frequency of only 1.

Therefore, the new code discards frequencies of 1.

All the tests pass for this suite:

[sboisver12@colosse1 unit-tests]$ bash test_peakFinder.sh > log
[sboisver12@colosse1 unit-tests]$ grep error log|wc -l
0
[sboisver12@colosse1 unit-tests]$ grep FAIL log|wc -l
0
[sboisver12@colosse1 unit-tests]$ grep PASS log|wc -l
61

Thank you for helping improving the software !

[sboisver12@colosse1 unit-tests]$ ./peaks Libraries/mitch-issue-49.txt
Mode= 2 signalAverage= 4
CURRENT IS NOISE, 485
GOT HIT 359
Peak 0 351 49

See this commit 538550f