better alternative to 'click and drag your uploaded files to the correct file type'

[colinbrislawn]

I feel this may be covered in another issue, so feel free to point me in the right direction!

When setting up the Prep Info with paired reads, is there a good way to get the reads in the right column? My forward reads are *.f.fastq.gz and my reverse reads are *.r.fastq.gz files.

Thank you for your time!

[josenavas]

Hi @colinbrislawn
you can click on the file name, drag and drop it in the correct column. In this case, drag your 'Hans_1_8_5.r.fastq.gz' file into the raw reverse seqs column.

[colinbrislawn]

I tried out the 'click and drag' and it worked pretty well. Unfortunately, I have 268 files. Is there another way to organize them other than clicking and dragging 268 times?

I also considered uploading the fastq files after joining them. Would that be a good option?

[antgonza]

Sadly there is no way to automatically define what's forward/reverse (it's a complex task cause you can imagine people naming the forward as 01, 1, a, f, fwd, forward, etc) but you could upload your joined, not demuxed files ...

[colinbrislawn]

Yeah, I imagine automation would be pretty hard. Maybe if you have two different columns for the two file names in the Prep Info file...

I'll upload the paired files. This is a good solution because they were sequenced with strange overlap, so joining would be a pain even if I could easily upload both.

Thanks guys!

(You can close this if you want.)

[josenavas]

That actually gave me a good idea:

Check in the Prep info file if there are columns with the filepath type (in this case they should match the name, e.g. raw_reverse_reads and raw_forward_seqs). If those columns exist, then populate the files based on those names.

@biocore/qiita-admin what do you think?

[colinbrislawn]

👍 This would elegantly support Illumina MiSeq, perhaps the most common amplicon sequencing platform.

[squirrelo]

This would mean having code and columns specifically for the target gene plugin built into qiita, which I thought we were avoiding.

[colinbrislawn]

So maybe a javascript tool to supplement drag and drop could avoid this; some tool that would 'move items that match this pattern?'

[josenavas]

@squirrelo that's not correct

In my description, I pointed out the they should have the same name. Each artifact type have the filepath types that it supports, so we just need to search that list in the column names, that is completely agnostic to the actual data.

[ElDeveloper]

I think the solution would need to be on both sides. On the user interface, we should allow to bulk-select files using simple wildcard matching, no regular expressions. This will probably capture most of the cases where forward reads are [somestring].R1.fastq.gz or in @colinbrislawn's case [somestring].f.fastq.gz. And on the data side of things, it would still be nice to implement the solution that @josenavas suggests, this would allow for more complicated filenames where simple wildcards won't cut it.

[josenavas]

I like that idea @ElDeveloper
Are you aware of any JS library that already does that (or can be easily adapted) to accomplish this?

[squirrelo]

You can use underscore.js to do this filtering of filenames in a single line:

var matches = _.filter(fileNamesArray, function(val) { return val == searchString; });

[squirrelo]

Alternately you can do the same thing with jquery and not rely on a new dependency using grep or filter

[colinbrislawn]

Yeah, jquery could do it, but we would still have to wrap it up in a usable interface. I'm not sure what a good UI for this would be (I've seen this in the unix cli, but never in a gui).

[antgonza]

This issue is super annoying and annoyance grows with number of samples. We should fix ASAP, changing deadline.

[antgonza]

I have been thinking about this issue and the problem comes when you have per sample FASTQ and to be able to process them correctly you need to have a run_prefix - either for target gene or shotgun. Currently, we auto-populate the forward based on these values and it shouldn't be that difficult to check if the same prefix has 2 files, if so, after ordering we can place the 1st in the forward column and the reverse in the 2nd column. This will solve this issue and minimize the GUI changes. I'll fix following this idea. I plan to add a note saying that if the user doesn't see the auto-population or the files are not correctly distributed it's highly probable that something is wrong with their run_prefix.

[colinbrislawn]

I just tested this auto-population system and it works great!

However, it's now telling me the paired list is wrong. How can I fix this?

This is on qiita study 11114, 16S prep

Thanks Antonio!

[antgonza]

You have 91 samples in your prep template, with 91 run_prefixes but only 9 pairs of files. Solution, either upload the rest of the samples or delete the other samples from your prep.

[colinbrislawn]

That's all correct! I can upload the rest of my files.

Did I miss the error on the page, or is this an error we could add?

[antgonza]

Mind opening a new issue to add those error messages?

[colinbrislawn]

Sure. Sorry to double post.

[antgonza]

Thanks, not a problem, just cleaner.