-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
FASTQ datatype enhancements #2
Comments
great idea - could be a big speed improvement for slow IO environments |
So are you talking about uploading and storeing/manageing the new uploaded data as compressed data? Or just uploading a compressed fastq file and then extacting it once the upload is complete server-side? Or do you want the data to remain compressed in storage on the server? |
This is a good idea. @timothom I think Galaxy already decompresses fastq files that are uploaded, so I assume this would involve storing as compressed data? Would this involve disabling the decompression of uploaded fastq files? It would be nice for us in our lab to be able to operate directly on compressed fastq files though. |
I'd love if even linking in compressed files worked seemlessly. That wouldn't require mucking with the upload tool to not autodecompress stuff. |
This would make very happy several french galaxy admins! |
There has also been some work by @yhoogstrate in galaxyproject#2535 with a different approach. |
@mvdbeek: @ashvark and I reviewed @yhoogstrate's work and it requires this squashfs thing to be installed all over the cluster, no? |
@ashvark has started some work on a compressed Fastq type, see #38. @bgruening: how would this work with tools that do not support compressed fastq? And how would compressing existing datasets work - would set_meta() compress / decompress if that key changed? Finally, see the issue @frederikcoppens mentioned on #38 - something to look out for. |
@pvanheus With a new compressed fastq datatype, this would require updating the wrappers to also allow this datatype I assume? Then tools that do not support it require a conversion to use it as input. |
Yes. I am planning to trying to add converters but i am afraid that would not be good idea for larger fastq files |
Why a new format, just annotate the old format and convert tools that do not support compressed fastq to react on the metadata. This should be compatible and doable without much effort. I'm assuming here that most of the tools already have native support for gzipped fastq. |
@bgruening because metadata is per-user not per-dataset. However, how about we make a new type: uncompressed fastq. So Fastq is compressed fastq. I'm just thinking of a way to convert existing datasets... @natefoo also pointed out to me that the correct way to handle tools that depend on .gz extension is that at job run time the dataset is linked in with the extension as per datatypes_conf.xml. |
@bgruening and @pvanheus . I have created a separate branch (https://github.com/ashvark/galaxy/tree/fastq_enhancements) in my repository for the enhancement of fastq datatype to handle gzipped fastq files as such. I have tested this only with simpe testcases. Below is the explanation of the changes
TO DO
I would like to know your suggestions and improvements. |
Merge pull request #2 from galaxyproject/dev
@yhoogstrate that pull request remains open and seems no further development has been done against it. Another discussing is here: #38 Perhaps we should a combined efforts around this. @ashvark I briefly tested your changes locally and worked ok. The other issue is file/dataset extension that sometimes tools use to determine the format of the file, is there any reasons why Galaxy forces the .dat extension. I know it will be a big change, but can files be stored and tracked in their original extension in Galaxy? |
As discussed with @frederikcoppens it would be nice to upload compressed FASTQ-files and to handle them properly in tools.
One way to do this is to enhance the already existing FASTQ datatype with a new metadata element that indicates if it compressed or not. Tools should then be able to recognize this metadata element. GSNAP is one of the tools which would benefit from it.
You can learn about:
Skills:
The text was updated successfully, but these errors were encountered: